Documentation
Object Interaction

These actions allow the agent to interact with Sim Objects in the scene in various ways.

Pickup Object

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

Pick up an Interactable object specified by objectId and move it to the Agent’s Hand. Compatible objects have pickupable return true in their Object Metadata and return their current state with isPickedUp. Note that the agent’s hand must be clear of obstruction or this action will fail. If the target object being in the Agent’s Hand would cause it to clip into the environment, this will fail.

Picked up objects can also obstruct the Agent’s view of the environment since the Agent’s hand is always in camera view, so know that picking up larger objects will obstruct the field of vision.

Moveable Receptacles: Note that certain objects are Receptacles that can themselves be picked up. If a moveable receptacle is picked up while other Sim Objects are inside of it, the contained objects will be picked up with the moveable receptacle. This allows for sequences like “Place Egg on Plate -> Pick Up Plate” to move both the Plate and Egg.

#pickup a specific mug
event = controller.step(action='PickupObject', objectId="Mug|0.25|-0.27")
#attempt to pickup an object positioned at (0.5, 0.7)
event = controller.step(action='PickupObject', x=0.5, y = 0.7)
Parameter Type Description Default
objectId string the string unique id of the target object. Leave this null in order to use (x, y) screen coordinates to target a potential object for Pickup null
forceAction bool Set to true to allow an object to be picked up regardless of if it is within interaction range of the agent. False
manualInteract bool By default, objects picked up by the agent teleport into the agent’s hand at a default position in front of the agent camera. Set this to True in order to instead pick up an object at the object’s location. This allows the agent to manipulate the object via object manipulation actions without the abstraction of the picked up object teleporting to the agent’s hand. False
x float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized x coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the left most side of the screen. The x axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for x will be the exact middle of the screen in the x direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0
y float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized y coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the upper most side of the screen. The y axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for y will be the exact middle of the screen in the y direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0

Put Object

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

Attempt to Put an object the Agent is holding onto/in the target Receptacle. Valid target receptacle objects will return receptacle as true in their Object Metadata.

# place a held object onto a specific tabletop
event = controller.step(action='PutObject', objectId="TableTop|0.25|-0.27|0.95")
#place a held object onto a receptacle at (0.2, 0.5)
event = controller.step(action='PutObject', x=0.2, y=0.5)
Parameter Type Description Default
objectId string string unique id of target receptacle to attempt putting the held object in/on. Leave this null in order to use (x, y) screen coordinates to target a potential receptacle object null
forceAction bool Enable to ignore any Receptacle Restrictions when attempting to place objects. Normally objects will fail to be put on a receptacle if that receptacle is not valid for the object. This will also ignore interaction range restrictions of the agent. Note this does not gaurantee an object will be placed in a receptacle, as some objects will not fit inside a receptacle regardless of the default object restrictions. False
placeStationary bool If placeStationary = False is passed in, a placed object will use the physics engine to resolve the final position. This means placing an object on an uneven surface may cause inconsistent results due to the object rolling around or even falling off of the target receptacle. Note that because of variances in physics resolution, this placement mode is non-determanistic! If placeStationary = True, the object will be placed in/on the valid receptacle without using physics to resolve the final position. This means that the object will be placed so that it will not roll around. For determanistic placement make sure to set to true! True
x float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized x coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the left most side of the screen. The x axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for x will be the exact middle of the screen in the x direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0
y float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized y coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the upper most side of the screen. The y axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for y will be the exact middle of the screen in the y direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0

Drop Object

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

Drop an object currently in the Agent’s hand and let Physics resolve where it lands. Note that this is different from the Put Object function, as this does not guarantee the held object will be put into a specified receptacle. This is meant to be used in tandem with the Move/Rotate Hand functions to maneuver a held object to a target area, and the let it drop.

Additionally, this Drop action will fail if the held object is not clear from all collisions. Most importantly, the Agent’s collision will prevent Drop, as dropping an object if it is “inside” the agent will lead to unintended behavior.

event = controller.step(action='DropHandObject')
Parameter Type Description Default
forceAction bool Set to true to forcibly drop an object even if this would cause clipping with the environment, other objects, or the agent False

Throw Object

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

An extension of the Drop function-throw an object currently in the Agent’s hand in the current forward direction of the Agent at a force specified by moveMagnitude. Because objects can have different Mass properties, certain objects will require more or less force to throw the same distance.

event = controller.step(action='ThrowObject', moveMagnitude=150.0)
Parameter Type Description Default
moveMagnitude float The amount of force used to throw the object in newtons. Note that objects of different masses will have different throw distances if this magnitude is not changed. 0.0
forceAction bool Set to true to forcibly throw an object even if this would cause clipping with the environment or other objects False

Push Object

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

Push an object specified by objectId away from the Agent at a force specified by moveMagnitude. Because objects can have different Mass properties, certain objects will require more or less force to push the same distance. Only certain objects can be pushed (i.e., Chair, Ottoman, Apple, CellPhone etc.). Some objects are structural and cannot be pushed (i.e., Refrigerator, Countertop, Microwave). The target object cannot be in the Agent’s hand. All Pickupable and Moveable objects can be pushed.

#push a mug
event = controller.step(action='PushObject', objectId="Mug|0.25|-0.27", moveMagnitude=150.0)
#push an object at (0.1, 0.4)
event = controller.step(action='PushObject', x=0.1, y=0.4, moveMagnitude=150.0)
Parameter Type Description Default
moveMagnitude float The amount of force used to push the object in newtons. Note that objects of different masses will move different distances if this magnitude is not changed 0.0
objectId string the string unique id of the target object. Leave this null to use (x,y) screen coordinates to target objects. null
forceAction bool Force the object, if the objectId is valid, to be pushed regardless of visibility to agent. False
x float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized x coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the left most side of the screen. The x axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for x will be the exact middle of the screen in the x direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0
y float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized y coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the upper most side of the screen. The y axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for y will be the exact middle of the screen in the y direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0

Pull Object

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

Pull an object specified by objectId towards the Agent at a force specified by moveMagnitude. Because objects can have different Mass properties, certain objects will require more or less force to push the same distance. All Pickupable and Moveable objects can be pulled.

#pull a mug
event = controller.step(action='PullObject', objectId="Mug|0.25|-0.27", moveMagnitude=150.0)
#pull an object at (0.1, 0.4)
event = controller.step(action='PullObject', x=0.1, y=0.4, moveMagnitude=150.0)
Parameter Type Description Default
moveMagnitude float The amount of force used to pull the object in newtons. Note that objects of different masses will move different distances if this magnitude is not changed. 0.0
objectId string the string unique id of the target object. Leave this null to use (x, y) screen coordinates to target objects null
forceAction bool Force the object, if the objectId is valid, to be pulled regardless of visibility to agent. False
x float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized x coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the left most side of the screen. The x axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for x will be the exact middle of the screen in the x direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0
y float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized y coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the upper most side of the screen. The y axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for y will be the exact middle of the screen in the y direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0

Directional Push

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

Push an object specified by objectId in a direction vector determined by pushAngle at a magnitude of moveMagnitude. Because objects can have different Mass properties, certain objects will require more or less force to push the same distance. All Pickupable and Moveable objects can be pulled.

#push a sofa 90 degrees with magnitude 100
event = controller.step(action='DirectionalPush', objectId=Sofa|3|2|1, moveMagnitude=100.0, pushAngle=90.0)
#push an object at (0.3, 0.5) 90 degrees with magnitude 100
event = controller.step(action='DirectionalPush', x=0.3, y=0.5, moveMagnitude=100.0, pushAngle=90.0)
Parameter Type Description Default
moveMagnitude float The amount of force used to pull the object in newtons. Note that objects of different masses will move different distances if this magnitude is not changed. 0.0
objectId string the string unique id of the target object. Leave this null to use (x, y) screen coordinates to target objects. null
pushAngle float The direction vector to push the object. Values from 0 to 360 are valid, 0 being the current forward direction of the agent. This value will change the push direction clockwise from the agent’s forward. (ie: 90.0 will push the object directly right, a value of 180.0 will push the object backwards) 0.0
forceAction bool Force the object, if the objectId is valid, to be pushed regardless of visibility to agent. False
x float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized x coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the left most side of the screen. The x axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for x will be the exact middle of the screen in the x direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0
y float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized y coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the upper most side of the screen. The y axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for y will be the exact middle of the screen in the y direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0

Place Object At Point

This will attempt to place an object flush with the surface of a Receptacle. This can only be used on objects that are Pickupable or Moveable. The point to place an object at can be generated by the GetSpawnCoordinatesAboveReceptacle action below. Combine these two actions in order to teleport objects onto different receptacle surfaces.

event = controller.step(action = 'PlaceObjectAtPoint', objectId=Toaster|1|1|1, position=dict(x=-1.25, y=1.0, z=-1.0))
Parameter Type Description Default
objectId string the string unique id of the target object null
position xyz dictionary x, y, z coordinates for the point to try and place the object null

Get Spawn Coordinate Above Receptacle

This action is to explicitly be used in tandem with the PlaceObjectAtPoint action. Returns an array of xyz dictionary representing the x, y, z coordinates of valid spawn positions above a Receptacle object. The array of xyz dictionary is returned in the actionReturn metadata.

event = controller.step(action = 'GetSpawnCoordinatesAboveReceptacle', objectId=CounterTop|1|1|1, anywhere=True)
Parameter Type Description Default
objectId string the string unique id of the target object null
anywhere bool If True, spawn coordinates will be returned even if the exact position of the coordinate is outside of the agent’s field of view. Keep False to return spawn coordinates even if they are not in view of the agent. False

Touch Then Apply Force

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

Will cast a ray from a point on the screen into the simulation world. The ray’s maximum length is determined by handDistance. If a Pickupable or Moveable sim object is hit along the path of this ray, it will have a force determined by moveMagnitude applied to it instantaneously. This action will return feedback in the actionReturn attribute of the event metadata return.

event = controller.step(action= 'TouchThenApplyForce', x=0.5, y=0.5, handDistance = 1.0, direction=dict(x=0.0, y=1.0, z=0.0), moveMagnitude = 80.0)
event.actionReturn
Parameter Type Description Default
x float The normalized x coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the left most side of the screen. The x axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for x will be the exact middle of the screen in the x direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0
y float The normalized y coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the upper most side of the screen. The y axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for y will be the exact middle of the screen in the y direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0
handDistance float The maximum length of the ray cast from screen point (x, y) out into the scene’s world space in meters. The starting point of this ray is on the near-plane of the agent’s camera. 0.0
direction xyz dictionary The direction vector relative to the agent’s current forward to push any object touched.  
moveMagnitude float The amount of force to apply to a touched object in newtons. 0.0

Metadata actionReturn Feedback

Attribute Type Description Example
didHandTouchSomething bool returns True if a sim object was touched within the ray’s length of handDistance meters True
objectId string The unique string id of the object touched by the ray. If the object touched in the scene has collision but is not a sim object, this objectId attribute will be “not a sim object, a structure was touched.” Apple|+1.0|+1.0|+1.0
armsLength float The distance from the touched point on the object to the agent’s camera. This distance will never exceed the handDistance value passed in originally unless the action finishes as a failed action (see below) 1.20


This action will return a failure (metadata’s lastActionSuccess=False) only in two cases:

  • 1) The raycast hit an object, but the object was outside of the maximum visibility range of the agent (ie: a handDistance of 10 is passed in, and hits an object 9 meters away, but the agent’s max visibility distance is 1.5 meters, causing a failure). The feedback object generated in this case will be didHandTouchSomething=False, objectId="", armsLength=handDistance.
  • 2) The handDistance of the action is larger than the agent’s max visibility distance and no object was hit. The feedback object generated in this case will be didHandTouchSomething=False, objectId="", armsLength=handDistance with a metadata errormessage of “the position the hand would have moved to is outside the agent’s max interaction range”.

Note that this action interacts with the visibility of an object in order to determine what can be poked. The visibility of an object is defined by:

  • 1) An object must be within the agent camera’s frustrum (field of view).
  • 2) An object must be within the area of a cylinder defined by a radius of length visibilityDistance around the agent’s vertical y axis.
  • 2) If that object is within the cylinder, a line must be able to be cast from the agent’s camera position to a point on that object unobstructed.

Because visiblity is defined by the cylinder with radius visibilityDistance, the total area of objects that are touchable by this action is the intersection of the sphere of radius handDistance centered around the agent’s camera, and the cylinder of radius visibilityDistance about the agent’s vertical axis. Note that the agent camera is also centered around the agent’s vertical axis.

Open Object

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

Open an object specified by objectId. Compatible objects will return openable as true in their Object Metadata, and return their current open state with isOpen.

The target object must be within range of the Agent and Interactable in order for this action to succeed. An object can fail to open if it hits another object as it is opening. In this case the action will fail and the target object will reset to the position it was last in.

#open a fridge
event = controller.step(action='OpenObject', objectId="Fridge|0.25|0.75")
#open a fridge halfway
event = controller.step(action='OpenObject', objectId="Fridge|0.25|0.75", moveMagnitude=0.5)
#try and open an object at (0.2, 0.5)
event = controller.step(action='OpenObject', x=0.2, y=0.4)
Parameter Type Description Default
objectId string the string unique id of the target object. Leave this null to use (x, y) screen coordinates to target objects null
moveMagnitude float Pass a magnitude between 0.0 and 1.0 to open by the corresponding percentage. For example, a magnitude of 0.5 will cause the object to open halfway, a value of 0.25 will open the object a quarter of its full open position 0.0
forceAction bool Force the object, if the objectId is valid, to open regardless of visibility to agent. False
x float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized x coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the left most side of the screen. The x axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for x will be the exact middle of the screen in the x direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0
y float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized y coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the upper most side of the screen. The y axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for y will be the exact middle of the screen in the y direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0

Close Object

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

Close an object specified by objectId. Compatible objects will return openable as true in their Object Metadata, and return their current open state with isOpen.

The target object must be within range of the Agent and Interactable in order for this action to succeed. An object can fail to open if it hits another object as it is closing. In this case the action will fail and the target object will reset to the position it was last in.

#close a fridge
event = controller.step(action='CloseObject', objectId="Fridge|0.25|0.75")
#try and close an object at (1.0, 0.4)
event = controller.step(action='CloseObject', x=1.0, y=0.4)
Parameter Type Description Default
objectId string the string unique id of the target object. Leave this null to use (x, y) screen coordinates to target objects. null
forceAction bool Force the object, if the objectId is valid, to close regardless of visibility to agent. False
x float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized x coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the left most side of the screen. The x axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for x will be the exact middle of the screen in the x direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0
y float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized y coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the upper most side of the screen. The y axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for y will be the exact middle of the screen in the y direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0

Toggle On

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

Toggles an object specified by objectId into the On state. Compatible objects will return toggleable as true in their Object Metadata, and will return their current on/off state with isToggled. Noteable examples are Lamps, Light Switches, Stove Knobs, and Laptops.

#toggle on lightswitch
event = controller.step(action='ToggleObjectOn', objectId="LightSwitch|0.25|-0.27|0.95")
#try and toggle on an object at (0.2, 0.5)
event = controller.step(action='ToggleObjectOn', x=0.2, y=0.5)
Parameter Type Description Default
objectId string the string unique id of the target object. Leave this null to use (x, y) screen coordinates to target objects null
forceAction bool Force the object, if the objectId is valid, to toggle on regardless of visibility to agent. False
x float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized x coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the left most side of the screen. The x axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for x will be the exact middle of the screen in the x direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0
y float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized y coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the upper most side of the screen. The y axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for y will be the exact middle of the screen in the y direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0

Toggle Off

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

Toggles an object specified by objectId into the Off state if applicable. Compatible objects will return toggleable as true in their Object Metadata, and will return their current on/off state with isToggled.

#toggle off a lightswitch
event = controller.step(action='ToggleObjectOff', objectId="LightSwitch|0.25|-0.27|0.95")
#try and toggle off an object at (0.9, 1.0)
event = controller.step(action='ToggleObjectOff', x=0.9, y=1.0)
Parameter Type Description Default
objectId string the string unique id of the target object. Leave this null to use (x, y) screen coordinates to target objects null
forceAction bool Force the object, if the objectId is valid, to toggle off regardless of visibility to agent. False
x float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized x coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the left most side of the screen. The x axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for x will be the exact middle of the screen in the x direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0
y float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized y coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the upper most side of the screen. The y axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for y will be the exact middle of the screen in the y direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0

Cook Object

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

Switches the object specified by objectId to its cooked state if compatible. Compatible objects will return cookable as true in their object metadata, and will return thier current cooked state with isCooked. Note that this is a one-way state change, so only a scene rest will revert a cooked object back to its uncooked state.

#cook a potato
event = controller.step(action='CookObject', objectId="Potato|0.25|-0.27|0.95")
#try and cook an object at (0.6, 0.5)
event = controller.step(action='CookObject', x=0.6, y=0.5)
Parameter Type Description Default
objectId string the string unique id of the target object. Leave this null to use (x, y) screen coordinates to target objects null
forceAction bool Force the object, if the objectId is valid, to be cooked on regardless of visibility to agent. False
x float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized x coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the left most side of the screen. The x axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for x will be the exact middle of the screen in the x direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0
y float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized y coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the upper most side of the screen. The y axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for y will be the exact middle of the screen in the y direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0

Slice Object

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

Slices an object specified by objectId. Compatible objects that have sliceable return true in their Object Metadata. This action destroys the source object and spawns in multiple “sliced” objects based on the source object in the exact same location. This is a one-way state change, so only a scene reset will revert sliced objects to their whole versions. Sliced objects will still report metadata information (isSliced) even after being destroyed. This allows you to check the last position the source object was before the Slice action finished.

#slice a potato
event = controller.step(action='SliceObject', objectId="Potato|0.25|-0.27|0.95")
#try and slice an object at (0.0, 0.2)
event = controller.step(action='SliceObject', x=0.0, y=0.2)
Parameter Type Description Default
objectId string the string unique id of the target object. Leave this null to use (x, y) screen coordinates to target objects null
forceAction bool Force the object, if the objectId is valid, to be sliced on regardless of visibility to agent. False
x float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized x coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the left most side of the screen. The x axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for x will be the exact middle of the screen in the x direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0
y float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized y coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the upper most side of the screen. The y axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for y will be the exact middle of the screen in the y direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0

Break Object

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

Breaks an object specified by objectId. Compatible objects have breakable return true in their Object Metadata. This action will cause a visible change in the target object to show it is broken. This is a one-way state change, so only a scene reset will revert sliced objects to their whole versions. Broken objects will still report metadata information (isBroken) even after being destroyed. Some breakable objects shatter completely into pieces, while others will have a visual change such as a cracked screen.

#break a window
event = controller.step(action='BreakObject', objectId="Window|0.25|-0.27|0.95")
#try and break an object at (0.5, 0.5)
event = controller.step(action='BreakObject', x=0.5, y=0.5)
Parameter Type Description Default
objectId string the string unique id of the target object. Leave this null to use (x, y) screen coordinates to target objects. null
forceAction bool Force the object, if the objectId is valid, to break regardless of visibility to agent. False
x float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized x coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the left most side of the screen. The x axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for x will be the exact middle of the screen in the x direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0
y float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized y coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the upper most side of the screen. The y axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for y will be the exact middle of the screen in the y direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0

Dirty Object

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

Makes an object specified by objectId look dirty or messy. Compatible objects have dirtyable return true in their Object Metadata and return their current state with isDirty. This only works on objects that return isDirty=False.

#make a bed messy
event = controller.step(action='DirtyObject', objectId="Bed|0.25|-0.27|0.95")
#try and make an object at (0.5, 0.5) dirty
event = controller.step(action='DirtyObject', x=0.5, y=0.5)
Parameter Type Description Default
objectId string the string unique id of the target object. Leave this null to use (x, y) screen coordinates to target objects null
forceAction bool Force the object, if the objectId is valid, to become dirty on regardless of visibility to agent. False
x float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized x coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the left most side of the screen. The x axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for x will be the exact middle of the screen in the x direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0
y float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized y coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the upper most side of the screen. The y axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for y will be the exact middle of the screen in the y direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0

Clean Object

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

Makes an object specified by objectId look cleaner. Compatible objects have dirtyable return true in their Object Metadata and return their current state with isDirty. This only works on objects that return isDirty=True.

#clean a mug
event = controller.step(action='CleanObject', objectId="Mug|0.25|-0.27|0.95")
#try and clean an object at (0.1, 0.9)
event = controller.step(action='CleanObject', x=0.1, y=0.9
Parameter Type Description Default
objectId string the string unique id of the target object. Leave this null to use (x, y) screen coordinates to target object. null
forceAction bool Force the object, if the objectId is valid, to become clean on regardless of visibility to agent. False
x float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized x coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the left most side of the screen. The x axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for x will be the exact middle of the screen in the x direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0
y float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized y coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the upper most side of the screen. The y axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for y will be the exact middle of the screen in the y direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0

Fill Object With Liquid

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

Makes an object specified by objectId filled with liquid specified by fillLiquid. Compatible objects have canFillWithLiquid return true in their Object Metadata and return their current state with isFilledWithLiquid. Only objects that are empty and return isFilledWithLiquid=False can be filled.

#fill a mug with coffee
event = controller.step(action='FillObjectWithLiquid', objectId="Mug|0.25|-0.27|0.95", fillLiquid ="coffee")
#try and fill an object at (0.5, 0.5) with coffee
event = controller.step(action='FillObjectWithLiquid', x=0.5, y=0.5, fillLiquid ="coffee")
Parameter Type Description Default
objectId string the string unique id of the target object. Leave this null to use (x, y) screen coordinates to target object. null
fillLiquid string a string specifying the type of liquid the object should be filled with. Valid strings: coffee, wine, water null
forceAction bool Force the object, if the objectId is valid, to fill on regardless of visibility to agent. False
x float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized x coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the left most side of the screen. The x axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for x will be the exact middle of the screen in the x direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0
y float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized y coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the upper most side of the screen. The y axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for y will be the exact middle of the screen in the y direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0

Empty Liquid from Object

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

Empties an object specified by objectId if it is currently filled with liquid (isFilledWithLiquid=True). Note that objects filled with liquid will automatically empty themselves if rotated to an angle extreme enough that the liquid would fall out.

#empty a mug
event = controller.step(action='EmptyLiquidFromObject', objectId="Mug|0.25|-0.27|0.95")
#empty an object at (0.5, 0.5)
event = controller.step(action='EmptyLiquidFromObject', x=0.5, y=0.5)
Parameter Type Description Default
objectId string the string unique id of the target object. Leave this null to use (x, y) screen coordinates to target object. null
forceAction bool Force the object, if the objectId is valid, to empty on regardless of visibility to agent. False
x float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized x coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the left most side of the screen. The x axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for x will be the exact middle of the screen in the x direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0
y float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized y coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the upper most side of the screen. The y axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for y will be the exact middle of the screen in the y direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0

Use Up Object

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

Uses up parts of an object specified by objectId. Compatible objects have canBeUsedUp return true in their Object Metadata and return their current state with isUsedUp. This is an action specifically for objects like ToiletPaper, TissueBox, and PaperTowel that all can have bits of themselves used up to cause a change in state.

#use up a toilet paper roll
event = controller.step(action='UseUpObject', objectId="ToiletPaper|0.25|-0.27|0.95")
#try and use up an object at (0.5, 0.5)
event = controller.step(action='UseUpObject', x=0.5, y=0.5)
Parameter Type Description Default
objectId string the string unique id of the target object. Leave this null to use (x, y) screen coordinates to target object. null
forceAction bool Force the object, if the objectId is valid, to be used up regardless of visibility to agent. False
x float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized x coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the left most side of the screen. The x axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for x will be the exact middle of the screen in the x direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0
y float Used to target objects via screen coordinates instead of by objectId. The objectId parameter must be left null to use this. The normalized y coordinate on the screen to cast a ray from. Value must be between 0 and 1 with 0 representing the upper most side of the screen. The y axis of the screen is normalized relative to the screen’s resolution, so a value of 0.5 for y will be the exact middle of the screen in the y direction no matter your resolution. A value of x=0, y=0 will represent the top left of the screen. 0.0

Change Room Temperature Decay Time for Specific Object Type.

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

This changes all objects specified by objectType in the current scene to have a new Room Temp Decay Time specified by TimeUntilRoomTemp. This can be done to adjust the amount of time it takes for specific object types to return to room temperature. By default, all objects begin at Room Temperature. Objects placed on/in Hot or Cold sources (stove burner, microwave, fridge etc.) will have their ObjectTemperature value changed to Hot or Cold. If the object is removed from a Hot or Cold source, they will return to room temperature over time. The default time it takes for an object to decay to Room Temp is 10 seconds.

event = controller.step(action='SetRoomTempDecayTimeForType', objectType="Bread", TimeUntilRoomTemp=20.0)
Parameter Type Description Default
objectType string The object type to change the decay timer of. See a full list of Object Types on the Object Types page. null
TimeUntilRoomTemp float The amount of time it will take for an object to decay from Hot/Cold to Room Temp. 0.0

Set Global Temperature Decay Time

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

This changes all objects in the current scene to have a new Room Temp Decay Time specified by TimeUntilRoomTemp. By default, all objects begin at Room Temperature. Objects placed on/in Hot or Cold sources (stove burner, microwave, fridge etc.) will have their ObjectTemperature value changed to Hot or Cold. If the object is removed from a Hot or Cold source, they will return to room temperature over time. The default time it takes for an object to decay to Room Temp is 10 seconds.

event = controller.step(action='SetGlobalRoomTempDecayTime', TimeUntilRoomTemp=20.0)
Parameter Type Description Default
TimeUntilRoomTemp float The amount of time it will take for an object to decay from Hot/Cold to Room Temp. 0.0

Disable Temperature Decay

Only to be used if the agent is initialized with agentMode = 'default' and agentControllerType = 'physics'.

This disables the decay over time of the ObjectTemperature of objects. If set to False, objects will not decay to room temperature and will remain Hot/Cold even if removed from the Hot/Cold source.

event = controller.step(action='SetDecayTemperatureBool', allowDecayTemperature=False)
Parameter Type Description Default
allowDecayTemperature bool Set to allow object Temperatures to decay over time True