Held Objects

Set to

True

to allow an object to be picked up regardless of if it is within the

visibilityDistance

of the agent.

manualInteract

By default, objects picked up by the agent teleport into the agent’s hand at a default position in front of the agent camera. Set this to

True

in order to instead pick up an object at the object’s location. This allows the agent to manipulate the object via object manipulation actions without the abstraction of the picked up object teleporting to the agent’s hand.

Put Object

PutObject

attempts to put an object the agent is holding onto or into the target receptacle. Valid target receptacle objects have

receptacle

True

in their object metadata.

controller.step(
    action="PutObject",
    objectId="Apple|1|1|1",
    forceAction=False,
    placeStationary=True
)

Put Object Parameters

objectId

: string

required

The target object's

objectId

, found in the object's metadata.

forceAction

Enable to ignore any Receptacle Restrictions when attempting to place objects. Normally objects will fail to be put on a receptacle if that receptacle is not valid for the object. This will also ignore interaction range restrictions of the agent. Note this does not guarantee an object will be placed in a receptacle, as some objects will not fit inside a receptacle regardless of the default object restrictions.

placeStationary

: bool = True

False

, a placed object will use the physics engine to resolve the final position. This means placing an object on an uneven surface may cause inconsistent results due to the object rolling around or even falling off of the target receptacle. Note that because of variances in physics resolution, this placement mode is non-deterministic! If

True

, the object will be placed in/on the valid receptacle without using physics to resolve the final position. This means that the object will be placed so that it will not roll around. For deterministic placement make sure to set to

True

Drop Object

DropHandObject

attempts to drop an object currently in the agent’s hand and let physics resolve where it lands. The action is different from

PutObject

, as this does not guarantee the held object will be put into a specified receptacle. This is meant to be used in tandem with the Move/Rotate Hand functions to maneuver a held object to a target area, and the let it drop.

Additionally, this drop action will fail if the held object is not clear from all collisions. Most importantly, the agent’s collision will prevent drop, as dropping an object if it is “inside” the agent will lead to unintended behavior.

controller.step(
    action="DropHandObject",
    forceAction=False
)

Drop Object Parameters

forceAction

Set to

True

to forcibly drop an object even if this would cause clipping with the environment, other objects, or the agent.

Throw Object

ThrowObject

is an extension of

DropHandObject

. It throws the object currently in the agent’s hand in the forward facing direction of the agent with a force of

moveMagnitude

newtons. Since objects have different mass properties, objects require different forces to throw them same distance.

controller.step(
    action="ThrowObject",
    moveMagnitude=150.0,
    forceAction=False
)

Throw Object Parameters

moveMagnitude

required

The amount of force used to throw the object in newtons. Note that objects of different masses will have different throw distances if this magnitude is not changed.

forceAction

Set to

True

to forcibly throw an object even if this would cause clipping with the environment or other objects.

Move Held Object

While the agent is holding an object, it has several available actions to manipulate it. One such action is the ability to move the held object closer or further away from it. Held object movement is useful if we want to drop an object on a surface that is relatively far away.

There are several directions that the agent can move the held object. They include:

MoveHeldObjectAhead

MoveHeldObjectBack

MoveHeldObjectLeft

MoveHeldObjectRight

MoveHeldObjectUp

, and

MoveHeldObjectDown

. Each direction supports the same additional parameters.

Warning

After calling a move action (e.g.,

MoveAhead

) or a rotate action (e.g.,

RotateRight

) the position of the held object will reset to its default position in-front of the agent.

controller.step(
    action="MoveHeldObjectAhead",
    moveMagnitude=0.1,
    forceVisible=False
)

# Other supported directions
controller.step("MoveHeldObjectBack")
controller.step("MoveHeldObjectLeft")
controller.step("MoveHeldObjectRight")
controller.step("MoveHeldObjectUp")
controller.step("MoveHeldObjectDown")

Move Held Object Parameters

moveMagnitude

required

The distance, in meters, to move the held object in the specified direction.

forceVisible

Setting

forceVisible=True

results in action failing if the object is not visible to the agent. This prevents the agent from hiding the held object inside or behind another object, or moving it too far, such that it's out of the agent's field of view.

We also provide a separate helper action,

MoveHeldObject

, which allows the held object to move in several directions with only a single action:

controller.step(
    action="MoveHeldObject",
    ahead=0.1,
    right=0.05,
    up=0.12,
    forceVisible=False
)

Move Held Object Parameters

ahead

required

The distance, in meters, to move the held object forward, from the agent's current facing direction.

right

required

The distance, in meters, to move the held object rightwards, from the agent's current facing direction.

up

required

The distance, in meters, to move the held object upwards, from the agent's current facing direction.

forceVisible

Setting

forceVisible=True

Rotate Held Object

RotateHeldObject

attempts to rotate the held object relative to its current rotation.

Warning

After calling a move action (e.g.,

MoveAhead

) or a rotate action (e.g.,

RotateRight

) the rotation of the held object will reset to its default rotation.

controller.step(
    action="RotateHeldObject",
    pitch=90,
    yaw=25,
    roll=45
)

Rotate Held Object Parameters

pitch

: float = 0

required

Increments the pitch of an object from the agent's current facing direction. Specified in degrees. The video below shows the held object's

pitch

incrementing in

15

degree increments.

yaw

: float = 0

required

Increments the yaw of an object from the agent's current facing direction. Specified in degrees. The video below shows the held object's

yaw

incrementing in

15

degree increments.

roll

: float = 0

required

Increments the roll of an object from the agent's current facing direction. Specified in degrees. The video below shows the held object's

roll

incrementing in

15

degree increments.

An overload of

RotateHeldObject

can be used to rotate the held object to a fixed rotation.

controller.step(
    action="RotateHeldObject",
    rotation=dict(x=90, y=15, z=25)
)

Rotate Held Object Parameters

rotation

: dict[str, float]

required

Sets the rotation of the object to the provided rotation, in degrees. The rotation is relative to the object's axes. So, the set rotation is independent of the object's current rotation. Rotations are specified in degrees.

Pushing Objects

Directional Push

DirectionalPush

attempts to push an object in a given direction. Since objects can have different mass properties, pushing different objects often requires differing amounts of force to move them the same distance. Only

pickupable

and

moveable

objects can be pushed.

controller.step
    action="DirectionalPush",
    objectId="Sofa|3|2|1",
    moveMagnitude="100",
    pushAngle="90"
)

Directional Push Parameters

moveMagnitude

required

The amount of force used to pull the object in newtons. Following natural physics, objects of different masses may move different distances with the same amount of force.

pushAngle

required

The direction vector to push the object. Values in $[0:360]$ are valid, with $0$ being the current forward direction of the agent. This value will change the push direction clockwise from the agent’s forward. (i.e., $90$ will push the object directly right, a value of $180$ will push the object backwards).

objectId

required

The target object's

objectId

, found in the object's metadata.

forceAction

Set to

True

to allow an object to be picked up regardless of if it is within the

visibilityDistance

of the agent.

Push Object

PushObject

is syntactic sugar for

DirectionalPush

with a

pushAngle

0

controller.step(
    action="PushObject",
    objectId="Mug|0.25|-0.27",
    forceAction=False
)

Push Object Parameters

moveMagnitude

required

The amount of force used to push the object in newtons. Note that objects of different masses will move different distances if this magnitude is not changed.

objectId

required

The target object's

objectId

, found in the object's metadata.

forceAction

Set to

True

to allow an object to be picked up regardless of if it is within the

visibilityDistance

of the agent.

Pull Object

PullObject

is syntactic sugar for

DirectionalPush

with a

pushAngle

180

controller.step(
    action="PullObject",
    objectId="Mug|0.25|-0.27",
    forceAction=False
)

Push Object Parameters

moveMagnitude

required

The amount of force used to pull the object in newtons. Note that objects of different masses will move different distances if this magnitude is not changed.

objectId

required

The target object's

objectId

, found in the object's metadata.

forceAction

Set to

True

to allow an object to be picked up regardless of if it is within the

visibilityDistance

of the agent.

Touch Then Apply Force

TouchThenApplyForce

allows the agent to push an object at a certain point on the object, in a given direction. If a

pickupable

moveable

sim object is hit along the path of this ray, it will have a force determined by

moveMagnitude

applied to it instantaneously. This action will return feedback in the

actionReturn

attribute of the event metadata return.

event = controller.step(
    action="TouchThenApplyForce",
    x=0.5,
    y=0.5,
    direction={
        "x": 0.0,
        "y": 1.0,
        "z": 0.0
    },
    moveMagnitude=80,
    handDistance=1.5
)

Parameters

x

required

Used instead of

objectId

as an alternate way to target an object based on the last image frame. Valid values are in

[0:1]

, corresponding to how far left the image is from the previous frame.

y

required

Used in tandem with

to target an object based on the last image frame, rather than an

objectId

. Valid values are in

[0:1]

, corresponding to how from the top the object is from the previous frame.

direction

: dict[str, float]

required

The direction vector relative to the agent’s current forward to push any object touched.

moveMagnitude

required

The amount of force to apply to a touched object in newtons.

handDistance

required

The maximum Euclidean distance from the agent's camera that the (x, y) point that targets the object can be, in meters. If the point on the object is further away than

handDistance

, the action will fail.

Response

{
    "didHandTouchSomething": True
    "objectId": "Apple|+1|+1|+1"
    "armsLength": 1.20
}

event.metadata["actionReturn"]

didHandTouchSomething

: bool

Returns

True

if a sim object was touched within the ray’s length of

handDistance

meters.

objectId

The unique string id of the object touched by the ray. If the object touched in the scene has collision but is not a sim object, this

objectId

attribute will be "not a sim object, a structure was touched".

armsLength

The distance from the touched point on the object to the agent’s camera. This distance will never exceed the handDistance value passed in originally unless the action finishes as a failed action (see below).

This action will return a failure (event.metadata["lastActionSuccess"] == False) only in two cases:

The raycast hit an object, but the object was outside of the maximum visibility range of the agent (ie: a handDistance of 10 is passed in, and hits an object 9 meters away, but the agent’s max visibility distance is 1.5 meters, causing a failure). The feedback object generated in this case will be didHandTouchSomething=False, objectId="", armsLength=handDistance.
The handDistance of the action is larger than the agent’s max visibility distance and no object was hit. The feedback object generated in this case will be didHandTouchSomething=False, objectId="", armsLength=handDistance with a metadata errormessage of “the position the hand would have moved to is outside the agent’s max interaction range”.

Note that this action interacts with the visibility of an object in order to determine what can be poked. The visibility of an object is defined by:

An object must be within the agent camera’s field of view.
An object must be within the area of a cylinder defined by a radius of length visibilityDistance around the agent’s vertical y axis.
If that object is within the cylinder, a line must be able to be cast from the agent’s camera position to a point on that object unobstructed.

Because visibility is defined by the cylinder with radius visibilityDistance, the total area of objects that are touchable by this action is the intersection of the sphere of radius handDistance centered around the agent’s camera, and the cylinder of radius visibilityDistance about the agent’s vertical axis. Note that the agent camera is also centered around the agent’s vertical axis.

Teleporting Objects

Place Object At Point

PlaceObjectAtPoint

attempts to place an object flush with the surface of a receptacle. This can only be used on objects that are

pickupable

moveable

. The point to place an object at can be generated by the

GetSpawnCoordinatesAboveReceptacle

action below. Combine these two actions in order to teleport objects onto different receptacle surfaces.

controller.step(
    action="PlaceObjectAtPoint",
    objectId="Toaster|1|1|1",
    position={
        "x": -1.25,
        "y": 1.0,
        "z": -1.0
    }
)

Place Object At Point Parameters

objectId

required

The target object's

objectId

, found in the object's metadata.

position

: dict[str, float]

required

Global $(x, y, z)$ coordinates for the point to try and place the object.

Get Spawn Coordinate Above Receptacle

GetSpawnCoordinatesAboveReceptacle

is to explicitly be used in tandem with the

PlaceObjectAtPoint

action. It returns an array of xyz dictionary representing the

(x, y, z)

coordinates of valid spawn positions above a

receptacle

object. The array of xyz dictionary is returned in the actionReturn metadata.

controller.step(
    action="GetSpawnCoordinatesAboveReceptacle",
    objectId="CounterTop|1|1|1",
    anywhere=False
)

Parameters

objectId

required

The target object's

objectId

, found in the object's metadata.

anywhere