Object Interaction

These actions allow the agent to interact with Sim Objects in the scene in various ways.


Pickup Object

Pick up an Interactable object specified by objectId and move it to the Agent’s Hand. Compatible objects have pickupable return true in their Object Metadata and return their current state with isPickedUp. Note that the agent’s hand must be clear of obstruction or this action will fail. If the target object being in the Agent’s Hand would cause it to clip into the environment, this will fail.

Picked up objects can also obstruct the Agent’s view of the environment since the Agent’s hand is always in camera view, so know that picking up larger objects will obstruct the field of vision.

Moveable Receptacles: Note that certain objects are Receptacles that can themselves be picked up. If a moveable receptacle is picked up while other Sim Objects are inside of it, the contained objects will be picked up with the moveable receptacle. This allows for sequences like “Place Egg on Plate -> Pick Up Plate” to move both the Plate and Egg.

event = controller.step(dict(action='PickupObject', objectId="Mug|0.25|-0.27"))
Parameter Type Description Default
objectId string the string unique id of the target object null

Put Object

Attempt to Put an object the Agent is holding onto/in the target Receptacle. Valid target receptacle objects will return receptacle as true in their Object Metadata.

event = controller.step(dict(action='PutObject', objectId = "Tomato|0.1|3.2|0.43", receptacleObjectId = "TableTop|0.25|-0.27|0.95"))
Parameter Type Description Default
objectId string The string unique id of the object in the Agent’s hand that it is trying to put down. Note that an error will be thrown if trying to put an object the Agent is not holding null
receptacleObjectId string string unique id of target receptacle to attempt putting the object in/on. null
forceAction bool Enable to ignore any Receptacle Restrictions when attempting to place objects. Normally objects will fail to be put on a receptacle if that receptacle is not valid for the object. See the Receptacle Object Types page for more details. False
placeStationary bool If placeStationary = False is passed in, a placed object will use the physics engine to resolve the final position. This means placing an object on an uneven surface may cause inconsistent results due to the object rolling around or even falling off of the target receptacle. Note that because of variances in physics resolution, this placement mode is non-determanistic! If placeStationary = True, the object will be placed in/on the valid receptacle without using physics to resolve the final position. This means that the object will be placed so that it will not roll around. For determanistic placement make sure to set to true! True

Drop Object

Drop an object currently in the Agent’s hand and let Physics resolve where it lands. Note that this is different from the Put Object function, as this does not guarantee the held object will be put into a specified receptacle. This is meant to be used in tandem with the Move/Rotate Hand functions to maneuver a held object to a target area, and the let it drop.

Additionally, this Drop action will fail if the held object is not clear from all collisions. Most importantly, the Agent’s collision will prevent Drop, as dropping an object if it is “inside” the agent will lead to unintended behavior.

event = controller.step(dict(action='DropHandObject')))

Throw Object

An extention of the Drop function-throw an object currently in the Agent’s hand in the current forward direction of the Agent at a force specified by moveMagnitude. Because objects can have different Mass properties, certain objects will require more or less force to throw the same distance.

event = controller.step(dict(action='ThrowObject', moveMagnitude= 150.0 )))
Parameter Type Description Default
moveMagnitude float The amount of force used to throw the object in Newtons. Note that objects of different masses will have different throw distances if this magnitude is not changed 0.0

Push Object

Push an object specified by objectId away from the Agent at a force specified by moveMagnitude. Because objects can have different Mass properties, certain objects will require more or less force to push the same distance. Only certain objects can be pushed (ie: Chair, Ottoman, Apple, CellPhone etc.). Some objects are structural and cannot be pushed (ie:Refrigerator, Countertop, Microwave). The target object cannot be in the Agent’s hand.

event = controller.step(dict(action='PushObject', moveMagnitude= 150.0 )))
Parameter Type Description Default
moveMagnitude float The amount of force used to push the object in Newtons. Note that objects of different masses will move different distances if this magnitude is not changed 0.0
objectId string the string unique id of the target object null

Pull Object

Pull an object specified by objectId towards the Agent at a force specified by moveMagnitude. Because objects can have different Mass properties, certain objects will require more or less force to push the same distance. Only certain objects can be pushed (ie: Chair, Ottoman, Apple, CellPhone etc.). Some objects are structural and cannot be pushed (ie:Refrigerator, Countertop, Microwave). The target object cannot be in the Agent’s hand.

event = controller.step(dict(action='PullObject', moveMagnitude= 150.0 )))
Parameter Type Description Default
moveMagnitude float The amount of force used to pull the object in Newtons. Note that objects of different masses will move different distances if this magnitude is not changed 0.0
objectId string the string unique id of the target object null

OpenObject

Open an object specified by objectId. Compatible objects will return openable as true in their Object Metadata, and return their current open state with isOpen.

The target object must be within range of the Agent and Interactable in order for this action to succeed. An object can fail to open if it hits another object as it is opening. In this case the action will fail and the target object will reset to the position it was last in.

event = controller.step(dict(action='OpenObject', objectId="Fridge|0.25|0.75"))

Here is an example of opening the Fridge halfway:

event = controller.step(dict(action='OpenObject', objectId="Fridge|0.25|0.75", moveMagnitude = 0.5))
Parameter Type Description Default
objectId string the string unique id of the target object null
moveMagnitude float Pass a magnitude between 0.0 and 1.0 to open by the corresponding percentage. For example, a magnitude of 0.5 will cause the object to open halfway, a value of 0.25 will open the object a quarter of it’s full open position 0.0

CloseObject

Close an object specified by objectId. Compatible objects will return openable as true in their Object Metadata, and return their current open state with isOpen.

The target object must be within range of the Agent and Interactable in order for this action to succeed. An object can fail to open if it hits another object as it is closing. In this case the action will fail and the target object will reset to the position it was last in.

event = controller.step(dict(action='CloseObject', objectId="Fridge|0.25|0.75"))
Parameter Type Description Default
objectId string the string unique id of the target object null

Toggle On

Toggles an object specified by objectId into the On state. Compatible objects will return toggleable as true in their Object Metadata, and will return their current on/off state with isToggled. Noteable examples are Lamps, Light Switches, Stove Knobs, and Laptops.

event = controller.step(dict(action='ToggleObjectOn', objectId= "LightSwitch|0.25|-0.27|0.95")))
Parameter Type Description Default
objectId string the string unique id of the target object null

Toggle Off

Toggles an object specified by objectId into the Off state if applicable. Compatible objects will return toggleable as true in their Object Metadata, and will return their current on/off state with isToggled.

event = controller.step(dict(action='ToggleObjectOff', objectId= "LightSwitch|0.25|-0.27|0.95")))
Parameter Type Description Default
objectId string the string unique id of the target object null

Slice Object

Slices an object specified by objectId. Compatible objects that have sliceable return true in their Object Metadata. This action destroys the source object and spawns in multiple “sliced” objects based on the source object in the exact same location. This is a one-way state change, so only a scene reset will revert sliced objects to their whole versions. Sliced objects will still report metadata information (isSliced) even after being destroyed. This allows you to check the last position the source object was before the Slice action finished.

event = controller.step(dict(action='SliceObject', objectId= "Potato|0.25|-0.27|0.95")))
Parameter Type Description Default
objectId string the string unique id of the target object null

Break Object

Breaks an object specified by objectId. Compatible objects have breakable return true in their Object Metadata. This action will cause a visible change in the target object to show it is broken. This is a one-way state change, so only a scene reset will revert sliced objects to their whole versions. Broken objects will still report metadata information (isBroken) even after being destroyed. Some breakable objects shatter completely into pieces, while others will have a visual change such as a cracked screen.

event = controller.step(dict(action='BreakObject', objectId= "Potato|0.25|-0.27|0.95")))
Parameter Type Description Default
objectId string the string unique id of the target object null

Dirty Object

Makes an object specified by objectId look dirty or messy. Compatible objects have dirtyable return true in their Object Metadata and return their current state with isDirty. This only works on objects that return isDirty = false.

event = controller.step(dict(action='DirtyObject', objectId= "Bed|0.25|-0.27|0.95")))
Parameter Type Description Default
objectId string the string unique id of the target object null

Clean Object

Makes an object specified by objectId look cleanor tidy. Compatible objects have dirtyable return true in their Object Metadata and return their current state with isDirty. This only works on objects that return isDirty = true.

event = controller.step(dict(action='CleanObject', objectId= "Bed|0.25|-0.27|0.95")))
Parameter Type Description Default
objectId string the string unique id of the target object null

Fill Object With Liquid

Makes an object specified by objectId filled with liquid specified by fillLiquid. Compatible objects have canFillWithLiquid return true in their Object Metadata and return their current state with isFilledWithLiquid. Only objects that are empty and return isFilledWithLiquid = false can be filled.

event = controller.step(dict(action='FillObjectWithLiquid', objectId= "Mug|0.25|-0.27|0.95", fillLiquid = "coffee")))
Parameter Type Description Default
objectId string the string unique id of the target object null
fillLiquid string a string specifying the type of liquid the object should be filled with. Valid strings: coffee, wine, water null

Empty Liquid from Object

Empties an object specified by objectId if it is currently filled with liquid (isFilledWithLiquid = true). Note that objects filled with liquid will automatically empty themselves if rotated to an angle extreme enough that the liquid would fall out.

event = controller.step(dict(action='EmptyLiquidFromObject', objectId= "Mug|0.25|-0.27|0.95")))
Parameter Type Description Default
objectId string the string unique id of the target object null

Use Up Object

Uses up parts of an object specified by objectId. Compatible objects have canBeUsedUp return true in their Object Metadata and return their current state with isUsedUp. This is an action specifically for objects like ToiletPaper, TissueBox, and PaperTowel that all can have bits of themselves used up to cause a change in state.

event = controller.step(dict(action='UseUpObject', objectId= "ToiletPaper|0.25|-0.27|0.95")))
Parameter Type Description Default
objectId string the string unique id of the target object null

Change Room Temperature Decay Time for Specific Object Type.

This changes all objects specified by objectType in the current scene to have a new Room Temp Decay Time specified by TimeUntilRoomTemp. This can be done to adjust the amount of time it takes for specific object types to return to room temperature. By default, all objects begin at Room Temperature. Objects placed on/in Hot or Cold sources (stove burner, microwave, fridge etc.) will have their ObjectTemperature value changed to Hot or Cold. If the object is removed from a Hot or Cold source, they will return to room temperature over time. The default time it takes for an object to decay to Room Temp is 10 seconds.

event = controller.step(dict(action='SetRoomTempDecayTimeForType', objectType= "Bread", TimeUntilRoomTemp = 20.0)))
Parameter Type Description Default
objectType string The object type to change the decay timer of. See a full list of Object Types on the Object Types page. null
TimeUntilRoomTemp float The amount of time it will take for an object to decay from Hot/Cold to Room Temp. 0.0

Set Global Temperature Decay Time

This changes all objects in the current scene to have a new Room Temp Decay Time specified by TimeUntilRoomTemp. By default, all objects begin at Room Temperature. Objects placed on/in Hot or Cold sources (stove burner, microwave, fridge etc.) will have their ObjectTemperature value changed to Hot or Cold. If the object is removed from a Hot or Cold source, they will return to room temperature over time. The default time it takes for an object to decay to Room Temp is 10 seconds.

event = controller.step(dict(action='SetGlobalRoomTempDecayTime', TimeUntilRoomTemp = 20.0)))
Parameter Type Description Default
TimeUntilRoomTemp float The amount of time it will take for an object to decay from Hot/Cold to Room Temp. 0.0

Disable Temperature Decay

This disables the decay over time of the ObjectTemperature of objects. If set to false, objects will not decay to room temperature and will remain Hot/Cold even if removed from the Hot/Cold source.

event = controller.step(dict(action='SetDecayTemperatureBool', allowDecayTemperature = false)))
Parameter Type Description Default
allowDecayTemperature bool Set to allow object Temperatures to decay over time True

Next Steps

Continue on to the Held Object Manipulation documentation for information on how to manipulate picked up objects.