Documentation
Initialization

Prior to running any additional commands, it is assumed you have run the following:

from ai2thor.controller import Controller
controller = Controller(scene='FloorPlan28', gridSize=0.25, rotateStepDegrees=90.0, agentMode='default', agentControllerType='physics')

Initialization Parameters

Parameter Type Description Default
scene string The name of which scene to initialize. Kitchens: FloorPlan1 - FloorPlan30, Living rooms: FloorPlan201 - FloorPlan230, Bedrooms: FloorPlan301 - FloorPlan330, Bathrooms: FloorPlan401 - FloorPlan430  
gridSize float Determines the size of the grid blocks in meters. Value for gridSize must be nonzero and no greater than 5.0 meters. This grid determines the step size that the agent takes with the actions MoveAhead,MoveBack, MoveRight, and MoveLeft. The grid is centered around the default position of the agent in a scene, and is expanded based on this value. Note that the gridSize will only be respected if snapToGrid = True 0.25
snapToGrid bool Determines wether the agent’s position is clamped after any movement action (eg. MoveAhead,TeleportFull….), to end in a discrete position on a grid. Grid blocks are defined by gridSize. Set to “False” to allow diagonal movement or other movement/teleportation that might put the agent “off-grid.” True
rotateStepDegrees float The default value for all rotation actions. Change this to allow all rotate actions to default to the given rotation amount. 90.0
renderDepthImage bool When enabled a depth image is sent and made available on the returned Event as the attribute depth_frame. False
renderClassImage bool When enabled a class segmentation image is sent and made available on the returned Event as the attribute class_segmentation_frame. False
renderObjectImage bool When enabled an object segmentation image is sent and made available on the returned Event as the attribute instance_segmentation_frame. False
visibilityDistance float Distance in meters from the agent’s vertical y axis that an object should be considered visible. This value defines the radius of a cylinder centered about the y axis of the agent. There are three steps to determine if an object is visible. First, an object must bet within the camera frustrum. Second, the object must be within the area defined by a cylinder centered around the agent’s vertical axis with radius visibilityDistance. Last, if a line can be drawn unobstrected from the agent’s camera position to a point on the object without obstruction, it is considered visible. Note that the agent’s camera is also centered on the vertical axis of the agent. 1.5
fieldOfView float Field of view for the agent’s camera. Corresponds to the value Camera.fieldOfView. Valid values for fieldOfView are (0-180), non-inclusive. 90.0
width int Sets the width of the returned images in pixels. Must be >= 300. The image quality is improved with larger images, as opposed to simply resizing the default 300px by 300px image. 300
height int Sets the height of the returned images in pixels. Must be >= 300. The image quality is improved with larger images, as opposed to simply resizing the default 300px by 300px image. 300
agentMode string This determines which mode preset the agent initializes to. The mode determines the agent’s size and look. Valid mode strings are default bot or drone. The default mode is a tall, custom model with a face. The bot mode uses collision and has a 3D model that is accurate to the LoCoBot. The drone mode changes the agent to a small, flying drone with a basket on top of it. default
agentControllerType string This determines which controller type the agent uses. Valid strings are: physics stochastic drone. Each agent mode is compatible with a subset of controller types. The default agentMode can use either the physics or stochastic controller. The bot agentMode can use the stochastic controller. The drone agentMode can use the drone controller. For more information on the capabilities of each controller type, see the Agent Controller Type section below. physics
applyActionNoise bool If the agentControllerType is set to stochastic, this bool will apply movement and rotation Gaussian noise to Move and Rotate actions automatically. True
movementGaussianMu float This value is only applied if agentControllerType is set to stochastic and applyActionNoise is set to True. If nonzero, the agent’s position after a move action will be offset by a random value taken from a Gaussian with mean movementGaussianMu. 0.001
movementGaussianSigma float This value is only applied if agentControllerType is set to stochastic and applyActionNoise is set to True. If nonzero, the agent’s position after a move action will be offset by a random value taken from a Gaussian with standard deviation movementGaussianSigma. 0.005
rotateGaussianMu float This value is only applied if agentControllerType is set to stochastic and applyActionNoise is set to True. If nonzero, the agent’s rotation after a rotate action will be offset by a random value taken from a Gaussian with mean rotateGaussianMu. 0.0
rotateGaussianSigma float This value is only applied if agentControllerType is set to stochastic and applyActionNoise is set to True. If nonzero, the agent’s rotation after a rotate action will be offset by a random value taken from a Gaussian with standard deviation rotateGaussianSigma. 0.5

Agent Controller Type

There are three controller types you can choose to initialize the agent into. Depending on which agentMode the agent is initialized into, different controller types will be compatible. The mode determines the agent’s look and size, the controller type determines the agent’s functionality.

agentMode compatible agentControllerType(s)
default physics, stochastic
bot stochastic
drone drone


Each controller has a different set of capabilities:

  • physics - allows the agent to interact with object and manipulate the environment using physics simulation. This controller must be used to perform physics based tasks (things like picking up objects, throwing objects, shoving objects around, etc).
  • stochastic - this controller has built in parameters to allow for randomized stochastic movement and rotation. This controller can only be used for navigation tasks, as it does not have the functionality to interact with objects in the same way the physics controller can. Set applyActionNoise = True as well as the movement and rotation gaussians (movementGaussianMu movementGaussianSigma rotateGaussianMu rotateGaussianSigma) to automatically have “move” and “rotate” actions apply noise stochastically.
  • drone - this controller forces the environment to pause real-time physics simulation to be in sync with metadata returns, to simulate thrust/velocity and other drone-like movement over multiple actions. This way, every movement action with the drone agent will have the influence of previous movement actions to determine its movement over time. This controller also allows drone-specific metadata and drone-specific actions to be recorded and used.

Reset

Use this method to change to a new scene after a Controller has been initialized.

controller.reset(scene='FloorPlan30')
Parameter Type Description Default
scene string The name of which scene to load. Kitchens: FloorPlan1 - FloorPlan30, Living rooms: FloorPlan201 - FloorPlan230, Bedrooms: FloorPlan301 - FloorPlan330, Bathrooms: FloorPlan401 - FloorPlan430  

Object Position Randomization

After initializing the scene, pickupable objects can have their default positions randomized to any valid receptacle they could be placed in within the scene. Objects are not guaranteed to move from their default position, as sometimes there is not enough free space in the scene for an object to move.

controller.step(action='InitialRandomSpawn', randomSeed=0, forceVisible=False, numPlacementAttempts=5, placeStationary=True
numDuplicatesOfType = [{'objectTypeCount': 'Statue', 'count': 20}, {'objectTypeCount': 'Bowl', 'count': 20}],
              excludedReceptacles={'CounterTop', 'DiningTable'})

Remember to reset and initialize the scene before using the Position Randomizer, otherwise seeded values will be inaccurate.

Parameter Type Description Default
randomSeed int Used to seed the randomization for duplicatable scene states. Because this seed depends on the current state of the scene, remember to reset the scene with controller.reset() before running InitialRandomSpawn(), otherwise the seeded randomization will not be accurate. 0
forceVisible bool When enabled, the scene will attempt to randomize all moveable objects outside of receptacles in plain view. Use this if you want to avoid objects spawning out of view inside closed drawers, cabinets, etc. False
numPlacementAttempts int How many times each object in the scene attempts to randomly spawn. Setting this value higher will lead to fewer spawn failures at the cost of performance. 5
placeStationary bool Determines if spawned objects will be settled completely static and unmoving, or if non-determenistic physics resolve their final position. Setting this to False will allow physics to resolve final positions, which can be used to spawn an object on a sloped receptacle but have it end up rolling off. True
numDuplicatesOfType list of ObjectTypeCount dictionary Used to specify how many objects of a certain type will attempt to be duplicated somewhere in the scene. It does not guarantee this number of duplicated objects, only the number of attempted spawned objects, so this is the max it will be. This will only create copies of objects already in the scene, so if you request an object which is not in the scene upon reset, it will not work. None
excludedReceptacles list of strings A list of receptacle object types to exclude from valid receptacles that can be randomly chosen as a spawn location. An example use case is excluding all CounterTop receptacles to allow for a scene configuration that has more free space on CounterTops in case you need free space for interaction. Note that this will not guarantee all listed receptacles as being completely clear of objects, as any objects that failed to reposition will remain in their default position, which might have been on the excluded receptacle type. Check the Actionable Properties section of the Objects documenation for a full list of Receptacle objects. None

Restore Object Positions

This is useful for setting a scene to the exact state of a previous initialization without relying on the Object Position Randomization’s random seed. Use a previous metadata dump of SimObjects to get position/rotation data for each SimObject. Sets up a scene according to the provided objects and poses. Objects which are specified twice will be copied. Objects which are not specified but are in the original scene will be removed.

object_poses = [
	{
		"objectName": "Alarm_Clock_19",
		"rotation": {"y": 270, "x": 0, "z": 0},
		"position": {"y": 0.8197357, "x": 2.45610785, "z": 0.054755792}
	},
	{
		"objectName": "Book_19",
		"rotation": {"y": 0, "x": 0, "z": 0},
		"position": {"y": 0.7308113, "x": 0.3694262, "z" :1.497829}
	},
	{
		"objectName": "pillow_19",
		"rotation": {"y": 0, "x": 0, "z": 0},
		"position": {"y": 0.809355736, "x": -0.5954236, "z": 0.17262423}
	},
	{
		"objectName": "Alarm_Clock_19",
		"rotation": {"y": 270, "x": 0, "z": 0},
		"position": {"y": 0.8197357, "x": 2.64169645, "z": -0.134690791}
	}
]
controller.step(action='SetObjectPoses', objectPoses=object_poses)
Parameter Type Description Default
objectPoses list List of object names and their poses. This information can be dumped from the metadata of a prior episode. None

objectPoses attributes

The objectPoses list should have the following attributes for each member. Each attribute can be found in the object’s metadata return (see the Object Attributes section for event.metadata['objects'].

Attribute Type Description Example
objectName string The name of the Sim Object reported by the object metadata’s name Alarm_Clock_19
position xyz dictionary x, y, z coordinates of the objects position. This is reported by the object metadata’s position (0.54, 0.3, 12.3)
rotation xyz dictionary x, y, z coordinates of the objects rotation. This is reported by the object metadata’s rotation (0, 90, 0)

Set Objects States by Object Type

Enables setting the state of various states of an entire object type.

This example toggles off all DeskLamp objects in a scene, changing their metadata to isToggled=False.

event = controller.step(action='SetObjectStates', SetObjectStates={'objectType': 'DeskLamp', 'stateChange': 'toggleable', 'isToggled': False})

This example opens all Microwaves in the scene, setting their metadata to isOpen = True.

event = controller.step(action='SetObjectStates', SetObjectStates={'objectType': 'Microwave', 'stateChange': 'openable', 'isOpen': True})

This example cooks all Potatoes in the scene, setting their metadata to isCooked = True.

event = controller.step(action='SetObjectStates', SetObjectStates={'objectType': 'Potato', 'stateChange': 'cookable', 'isCooked': True})
Parameter Type Description Default
SetObjectStates dictionary Dictionary of attributes to determine what object type, state change, and boolean state will be set by this action. None

SetObjectStates object attributes

The SetObjectStates object has the following members. Set the objectType to whatever object type that is being changed. Set stateChange to the state that the object has that will change (Some objects can have multiple manipulateable states- for example, Microwaves can be opened/closed because it is openable = True and can be toggled on/off because it is also toggleable = True).

Attribute Type Description Example
objectType string The type of object in the scene to be toggled. DeskLamp
stateChange string A string describing which state change the objects specified by objectType will be changed. These states are explicitly defined as some objects may have multiple states (ie: Microwave is both toggleable and openable). Valid strings are: openable, toggleable, breakable, canFillWithLiquid, dirtyable, cookable, sliceable, canBeUsedUp toggleable
isOpen boolean Only used if stateChange="openable", sets value of isOpened object metadata and visually changes all objects in scene of type objectType to open or closed. True
isToggled boolean Only used if stateChange="toggleable", sets value of isToggled object metadata and visually changes all objects in scene of type objectType to on or off. True
isBroken boolean Only used if stateChange="breakable", sets value of isBroken object metadata and visually changes all objects in scene of type objectType to broken. Note that this is a one-way modification to objects, so once an object is broken, it cannot be repaired unless the scene is reset to default True
isFilledWithLiquid boolean Only used if stateChange="canFillWithLiquid", sets value of isFilledWithLiquid object metadata and visually changes all objects in scene of type objectType to filled or not. True
isDirty boolean Only used if stateChange="dirtyable", sets value of isDirty object metadata and visually changes all objects in scene of type objectType to dirty or clean. True
isCooked boolean Only used if stateChange="cookable", sets value of isCooked object metadata and visually changes all objects in scene of type objectType to cooked. Note that this is a one-way modification to objects, so once an object is cooked, it cannot be uncooked unless the scene is reset to default True
isSliced boolean Only used if stateChange="sliceable", sets value of isSliced object metadata and visually changes all objects in scene of type objectType to sliced. Note that this is a one-way modification to objects, so once an object is sliced, it cannot be put back together unless the scene is reset to default True
isUsedUp boolean Only used if stateChange="canBeUsedUp", sets value of isUsedUp object metadata and visually changes all objects in scene of type objectType to used up (ie: empty toilet paper roll). Note that this is a on-way modification to objects, so once an object is used up, it can’t be refilled unless the scene is reset to default True

Unbreakable Objects

Objects with the Breakable property break by default if enough force is applied to them. This means dropping these objects from a high enough distance will cause automatic state changes. You can use this helper initialization function to set all objects of a given Type to unbreakable, preving this automatic interaction. This example sets all Eggs in the scene to be unbreakable unless explicitly using the BreakObject action.

controller.step(action='MakeObjectsOfTypeUnbreakable', objectType="Egg")
Parameter Type Description Default
objectType string Specify an object type to be set as unbreakable except by explicit use of the BreakObject action None

Make Objects Moveable

Use this if running an episode with the intention of realistically throwing/pushing/colliding objects around. This action resets kinematics of all non-static objects in the scene so that they are affected by gravity. This is to ensure immediate physics reactions in cases where objects are thrown or pushed into each other and they need to bounce and react immediately. Without running this, some objects might default to stationary placement, where they will not immediately respond to physics.

event = controller.step(action='MakeAllObjectsMoveable')