NEW
Big News! We've announced ProcTHOR

ManipulaTHOR API Reference

ManipulaTHOR comes packaged with AI2-THOR, which can be installed in Python using pip:

pip install ai2thor

It is currently designed to work with iTHOR scenes. But, compared to iTHOR, the actions are significantly lower level by leveraging a robotic arm.

After installing AI2-THOR, we can initialize a controller, which will allow us to execute actions in the environment.

from ai2thor.controller import Controller

controller = Controller(
    agentMode="arm",
    massThreshold=None,
    scene="FloorPlan203",
    visibilityDistance=1.5,
    gridSize=0.25,
    renderDepthImage=False,
    renderInstanceSegmentation=False,
    width=300,
    height=300,
    fieldOfView=60
)

Controller Parameters

Always set this to

arm
in order to use the agent designed for ManipulaTHOR.

Warning
If unset, the agent will be the default iTHOR agent, which is not the agent designed for ManipulaTHOR. Thus, the agent will have different properties than these docs.

A RuntimeError is raised if

agentMode
is not set to "default", "locobot", "drone", or "arm".

Blocks the arm from being able to directly interact and move objects with a mass above

massThreshold
. Empirically, it may help make the environment more deterministic.

Remark
This only stops the arm from knocking over things above the threshold that it directly touches. For example, if the arm touches an object with a mass below
massThreshold
that is adjacent to an object with a mass above
massThreshold
, pushing the smaller object can still cause the larger object to move since no direct contact was made.

The name of which scene to initialize. ManipulaTHOR is currently set up to work with iTHOR scenes. All iTHOR scene names and descriptions can be found here.

Used to set the maximum goal distance in meters between the agent and an object. This affects each object's visible property, where when True, the agent is within

visibilityDistance
of that object and the object appears in the agent's current egocentric frame.

With the ManipulaTHOR agent, this only affects how far positions are spaced with

GetReachablePositions
.

When

True
, a depth frame is rendered and made available as an event.depth_frame attribute.

We require this to be explicitly passed in because rendering depth takes longer than rendering RGB alone.

When

True
, a instance segmentation frame is rendered and made available as an
Event.instance_segmentation_frame
attribute. Instance segmentation provides segmentation by object instance, where all objects in view are distinguishable.

We require this to be explicitly passed in because rendering instance segmentation takes longer than rendering RGB alone.

The number of horizontal sampled pixels for each frame. This affects every rendered image frame (e.g., RGB, depth, BGR).

The number of vertical sampled pixels for each frame. This affects every rendered image frame (e.g., RGB, depth, BGR).

Changes the camera's optical field of view, in degrees. Valid values are in the domain (0:180)(0:180). The default field of view for the LoCoBot is 60 degrees.

Remark

Not all agent modes have the same default field of view. For instance, the drone's field of view is 150 degrees, by default.

Any

Initialization parameter can later be changed by calling the reset method. For instance, we can call:

controller.reset(scene="FloorPlan15", fieldOfView=80)

The values will default to what they were upon the most recent reset or initialization. For instance, if you initialized with

fieldOfView=45
, then called
reset
with
fieldOfView=60
, calling
reset
again, without passing in
fieldOfView
will keep
fieldOfView=60
.

The arm agent has three components that may move: the arm, which has several degrees of freedom, the arm base, and the agent's body.

MoveAgent
moves the agent's body from it's current facing direction. Compared to iTHOR move actions (e.g.,
MoveAhead
),
MoveAgent
can be used to move the agent diagonally (e.g., move ahead and left at the same time).

The video belows shows the result of consecutively calling the action with (ahead, right) =(0.25,0.0)(0.5,0.0)(0.0,0.25)(0.0,0.5)(0.5,0.5)= (0.25, 0.0)\to (-0.5, 0.0)\to (0.0, 0.25)\to (0.0, -0.5)\to (0.5, 0.5).

Move Agent Parameters

The distance, in meters, that the agent attempts to move in its forward facing direction.

Remark
Use negative values to move the agent backwards.

The distance, in meters, that the agent attempts to move rightward from its forward facing direction.

Remark
Use negative values to move the agent leftwards.

Sets the agent back to its starting pose if it collides with an object while attempting to move.

How fast the agent moves, in meters per second.

The interval, in seconds, with which that physics time steps are simulated.

For instance, if the interval is large, then the agent may move to the location in a manner that looks like teleportation. But, then there may not be many checks along the path to see if the agent collides with an object as it moves.

RotateAgent
rotates the agent's body from it's current facing direction.

Rotate Agent Parameters

Amount to rotate the agent rightward of its current facing direction.

Remark
Use negative values to rotate the agent leftwards.

Sets the agent back to its starting pose if it collides with an object while attempting to move.

How fast the agent moves, in degrees per second.

The interval, in seconds, with which physics time steps are simulated.

If the interval is large, for instance, then the agent may move to the location in a manner that looks like teleportation. But, then there may not be many checks along the path to see if the agent were to collide with an object as it moved.

Move the wrist joint of the Mid-Level-Arm to position coordinate. Position coordinates are relative to the arm’s forward direction. This means using (0, 0, 1) will extend the arm straight forward in the arm’s positive z direction. Note that the coordinates can also be passed in relative to the camera on the agent’s hand. See the handCameraSpace parameter below for more info.

controller.step(
    action="MoveArm",
    position=dict(x=0, y=0.5, z=0),
    coordinateSpace="armBase",
    restrictMovement=False,
    speed=1,
    returnToStart=True,
    fixedDeltaTime=0.02
)

Pickup Object Parameters

The position of where to move the arm. See the

coordinateSpace
parameter for how the position is interpreted. All positions are given in terms of meters.

Defines how the

position
parameter should be interpreted. Valid values are:

  • armBase
    moves the arm in the coordinate system relative to the spot where the spot where the arm's "shoulder" is being held by the base.
    Analogous to
    Teleport
    , using the same coordinate global space, but with the arm. For instance, one may use object metadata to move the arm near an object's global position.
    After initializing a scene, the arm starts at (x=0,y=0,z=0.5)(x=0, y=0, z=0.5), relative to the
    armBase
    .
    The video below shows many examples of
    moveArm
    with
    armBase
    as the
    coordinateSpace
    . Notice:
    • xx corresponds to the left and right direction (with x=0x=0 in the middle).
    • yy corresponds to the vertical direction (with y=0y=0 being the middle).
    • zz corresponds to the distance from the agent, with z=0z=0 being next to the agent, and z=0.75z=0.75 being 0.750.75 meters in front of the agent.
  • wrist
    moves the arm relative to its current position and rotation.
    Analogous to an action like
    MoveAgent
    , where calling it consecutively attempts to keep moving the agent to a new position, whereas calling
    Teleport
    consecutively will never change the goal position, after the first call.
    The video below shows the arm starting at
    armBase
    coordinate (x=0.5,y=0.5,z=0.5)(x=0.5, y=-0.5, z=0.5) and consecutively calling
    MoveArm
    with (x=0.2,y=0.2,z=0)(x=-0.2, y=0.2, z=0). Here, notice that since the position is relative to the agent's current arm position, the arm continues to move after passing in the same position consecutively.
  • world
    moves the arm to a position in global, world space.
    Analogous to
    Teleport
    , using the same coordinate global space, but with the arm. For instance, one may use object metadata to move the arm near an object's global position.

Restricts the arm to only being able to reach certain positions that are realistic. If

False
, the agent will be able to reach behind itself and clipping into itself is ignored.

The video below shows an example where

restrictMovement
is
False
, which allows the agent to reach behind itself.

How fast the arm moves, in meters per second.

Reverts the arm's position to its pose before the action was called, if the arm collides with an object while attempting to move it into position.

The interval, in seconds, with which that physics time steps are simulated.

For instance, if the interval is large, then the agent's arm may move to the location in a manner that looks like teleportation. But, then there may not be many checks along the path to see if the agent's arm collides with an object as it moves.

Raise and lower the height of the entire arm. This height is relative to the minimum and maximum extents, relative to the agent's size.

Move Arm Base Parameters

y

: float
required

The height of the arm. Value values are in [0:1][0:1], where 00 is the lowest possible height of the arm and 11 is the largest hight of the arm.

The video below displays the arm base

y
moving between 0.51.00.00.50.750.250.5 \to 1.0\to 0.0\to 0.5\to 0.75\to 0.25 (in that order).

How fast the arm moves, in meters per second.

Sets the agent's arm base position back to its starting

y
coordinate if it collides with an object while attempting to move.

The interval, in seconds, with which physics time steps are simulated.

If the interval is large, for instance, then the agent's arm may move into position in a manner that looks like teleportation. But, then there may not be many checks along the path to see if the arm were to collide with an object as it moved.

We can adjust the camera's pitch by utilizing the

LookUp
and
LookDown
commands. Both increment in 3030^\circ intervals, with angles being clamped between 6060^\circ in the downward direction and 3030^\circ in the upward direction.

controller.step("LookUp")
controller.step("LookDown")

Teleport
allows the agent to set its pose to anywhere in the scene in a single step. Valid poses do not place the agent outside the outer boundaries of the scene or in an area that it collides with an object.

Teleport Parameters

The position of the agent's body in global 3D coordinate space. If unspecified, the position of the agent remains the same.

The rotation of the agent's body in global 3D space. Here, the rotation changes the agent's yaw rotation.

Remark

The default agent's body cannot change in pitch or roll; hence, why the x,z=0x, z=0.

If unspecified, the rotation of the agent remains the same.

The

horizon
change's the camera's rotation. Values are clamped between [-30:60].

Since the agent looks up and down in 3030^\circ increments, it most common for the horizon to be in {30,0,30,60}\lbrace -30, 0, 30, 60\rbrace.

Warning

Negative camera

horizon
values correspond to agent looking up, whereas positive
horizon
values correspond to the agent looking down.

If unspecified, the horizon of the agent remains the same.

Denotes whether the agent is in a standing or crouched position. If unspecified, the standing state of the agent remains the same.

In addition to being able to

Teleport
, we also have an equivalent action
TeleportFull
that strictly requires every degree of freedom to be specified. The strictness of this action is often useful in large projects, where we don't want to any part of the agent implied implicitly.

Warning

TeleportFull
does not guarantee backwards compatibility in future releases. If a new degree of freedom is added to the agent, it will be added as a required parameter. If this is an issue, we recommend using
Teleport
.

controller.step(
    action="TeleportFull",
    position=dict(x=1, y=0.9, z=-1.5),
    rotation=dict(x=0, y=270, z=0),
    horizon=30,
    standing=True
)

TeleportFull Parameters

The position of the agent's body in global 3D coordinate space.

The rotation of the agent's body in global 3D space. Here, the rotation changes the agent's yaw rotation.

Remark

The default agent's body cannot change in pitch or roll; hence, why the x,z=0x, z=0.

The

horizon
change's the camera's rotation. Values are clamped between [-30:60].

Since the agent looks up and down in 3030^\circ increments, it most common for the horizon to be in {30,0,30,60}\lbrace -30, 0, 30, 60\rbrace.

Warning

Negative camera

horizon
values correspond to agent looking up, whereas positive
horizon
values correspond to the agent looking down.

Denotes whether the agent is in a standing or crouched position.

It is often useful to randomize the position of the agent in the scene, before starting an episode. Here, we can use:

  1. GetReachablePositions, which does an optimized BFS over a grid spaced out by the initialized
    gridSize
    . The valid positions are then added and returned in a list.
  2. Teleport
    , which can take a given position, and transform the agent to that position.

The process is illustrated below:

Step 1: Get the Positions
positions = controller.step(
    action="GetReachablePositions"
).metadata["actionReturn"]
Response
[
    dict(x=(...), y=(...), z=(...)),
    dict(x=(...), y=(...), z=(...)),
    dict(x=(...), y=(...), z=(...)),
    {...}
    dict(x=(...), y=(...), z=(...)),
]
Step 2: Teleport to a Position
import random
position = random.choice(positions)
controller.step(
    action="Teleport",
    position=position
)

Get Reachable Positions Response

A list of each position that the agent can be at, without colliding into any other objects or going beyond the walls.

The

Done
action nothing to the state of the environment. But, it returns a cleaned up event with respect to the metadata.

controller.step(action="Done")

It is often used in the definition of a successful navigation task (see Anderson et al.), where the agent must call the done action to signal that it knows that it's done, rather than arbitrarily or biasedly guessing.

Warning
The
Done
action does literally nothing to the state of the environment. For instance, if the agent calls
Done
and then
MoveAgent
, the
Done
action will have no affect on preventing a
MoveAgent
action from executing.

It is often used to return a cleaned up version of the metadata.

This section provides several actions that facilitate interaction with the ManipulaTHOR agent. If you have any additional actions that you would like to see, please create an issue and we'll see if we can support it!

Picks up all pickupable sim objects that are within the agent's hand sphere radius.

Pickup Object Parameters

Restricts the arm to only being able to pickup these specific objects. These objects will only be picked up iff they are within the agent's hand sphere radius.

controller.step(action="ReleaseObject")

Release Object Description

Drops any objects that the agent is currently holding.

Changes the radius of the agent's "magnet sphere" hand. Here, if a pickupable object is within

radius
of the agent's hand and
PickupObject
is called, the object will be picked up.

controller.step(
    action="SetHandSphereRadius",
    radius=0.1
)

Parameters

The radius on the agent's "magnet sphere" hand, in meters. Valid values are in (0.04:0.5)(0.04:0.5) meters.

The figure above shows AI2-THOR's agent-simulator interaction loop. Our backend scenes, actions, and observations are stored within Unity, a powerful real-time game engine.

From the Python side, we sequentially interact with Unity by taking actions using AI2-THOR's Python

Controller
. We have already seen several actions, such as
MoveAgent
,
RotateAgent
, and
LookUp
.

The returned data after executing any action is an

Event
. Each
Event
contains a ton of information to encode the state of the environment after the action has been executed:

event = controller.step(...)
References the last executed Event
controller.last_event
Response
<ai2thor.server.Event
    .metadata: {...}

    .frame: {...}
    .cv2img: {...}

    .depth_frame: {...}

    .instance_segmentation_frame: {...}
    .instance_masks: {...}
    .instance_detections2D: {...}

    .color_to_object_id: {...}
    .object_id_to_color: {...}
/>

Event Response

Contains information about the action and how it affected the environment. See the

Metadata
section for more.

RGB frame of the scene from the agent egocentric perspective. The size of the frame is the initialized (

height
,
width
, 3) stored in the numpy.uint8 format.

Same as the

frame
, except image channels are in BGR ordering. This is often useful with Python's OpenCV module (i.e., cv2), which expects images with BGR orderings.

Measures the distance, in meters, from the agent to each pixel in the frame. The size of the frame is the initialized (

height
,
width
) stored in the numpy.float32 format.

Remark

Upon

initialization
or
reset
,
renderDepthImage
must be set to
True
in order for the depth frame to appear.

Segments every object instance in the scene. The size of the frame is (

height
,
width
, 3) stored in the numpy.uint8 format. Each unique pixel color corresponds to a different object, which are indexable with
color_to_object_id
or
object_id_to_color
.

Remark

Upon

initialization
or
reset
,
renderInstanceSegmentation
must be set to
True
in order for the instance segmentation frame to appear.

Segments every object instance in the scene, storing each segmentation as a boolean image mask.

The keys are object IDs visible in the frame and each value is the boolean numpy.ndarray of size

height
,
width
) that segments the image.

Remark

Upon

initialization
or
reset
,
renderInstanceSegmentation
must be set to
True
in order for the instance masks to appear.

Provides the 2D bounding box around every object instance in the scene.

The keys are object IDs and the values are [Upper Left xx, Upper Left yy, Lower Right xx, Lower Right yy], where each element is the number of pixels it is from the top left corner of the image.

Remark

Upon

initialization
or
reset
,
renderInstanceSegmentation
must be set to
True
in order for the 2D instance detections to appear.

Provides a mapping from each pixel's color, represented as a hashable tuple, in the

instance_segmentation_frame
to what object it represents.

Same as

color_to_object_id
, except for the keys and values being swapped.

The metadata dictionary contains summary information about the state of the environment.

event.metadata
Response
{
    "errorMessage": {...}
    "lastActionSuccess": {...}
    "actionReturn": {...}
    "lastAction": {...}

    "sceneName": {...}
    "sceneBounds": {...}

    "arm": {...}
    "agent": {...}
    "objects": {...}

    "fov": {...}
    "screenWidth": {...}
    "screenHeight": {...}
    {...}
}

Metadata Response

If an action fails, feedback is provided as to what caused the failure.

States whether the last action was able to successfully execute.

An exception will not be raised upon invalid states. For instance, if an agent is standing right in-front of a wall and tries to

MoveAhead
, it will fail, but an exception will not be raised.

If an action is unsuccessful, the state of the environment will not have changed.

Holds the resulting data for actions that query the environment, such as

Get Reachable Positions
.

The name of the action passed into the Controller.

The name of the scene that the agent is currently located in.

This object provides all coordinates that are within bounds of the scene. This can be used in tandem with actions like PlaceObjectAtPoint to make sure the coordinate used is not out of bounds. This returns a sceneBounds object that includes an 8x3 matrix of xyz coordinates that represent the 8 corners of the box encompassing the entire scene, an xyz dictionary for the coordinates of the center of that box, and an xyz dictionary for the size (extents) of that box. Example:

{
  "center": {
    'x': -1.45,
    'y': 1.407,
    'z': 0.2
  },
 "cornerPoints": [
   [ 1.5,   2.92,  3.2],
   [ 1.5,   2.92, -2.8],
   [ 1.5, -0.106,  3.2],
   [ 1.5, -0.106, -2.8],
   [-4.4,   2.92,  3.2],
   [-4.4,   2.92, -2.8],
   [-4.4, -0.106,  3.2],
   [-4.4, -0.106, -2.8]
  ],
 "size": {
   "x": 5.9,
   "y": 3.02,
   "z": 6.0
  }
}

ManipulaTHOR specific information about the state of the agent's arm. See Arm Metadata for more.

Information about the pose of the agent. See Agent Metadata for more.

Information about the state of each object in the scene. See Object Metadata for more.

The field of view of the agent's camera, which can be changed upon initialization or reset.

The number of pixels that make up the width of each rendered image frame from the agent.

The number of pixels that make up the height of each rendered image frame from the agent.

Metadata specific to the arm ManipulaTHOR agent.

event.metadata["arm"]
Response
{
    "heldObjects": {...},
    "pickupableObjects": {...},

    "handSphereCenter": {...},
    "handSphereRadius": {...},

    "joints": {...},
}

Arm Metadata Response

A list of objectId's currently held by the agent.

All pickupable objects located within the agent's hand sphere radius, which could be picked up.

The global, world coordinate of the center (x,y,z)(x, y, z) position of the agent's hand sphere.

The radius of the hand sphere, which determines how close objects must be to allow them to be picked up. The value can be changed with the Set Hand Sphere Radiusaction.

Metadata for each of the 44 joints on the arm. See the Joint Metadata section for more.

Metadata for each of the 44 joints of the ManipulaTHOR arm.

import random
i = random.choice(range(4))

event.metadata["arm"]["joints"][i]
Response
{
    "name": {...},

    "position": {...},
    "rootRelativePosition": {...},

    "rotation": {...},
    "rootRelativeRotation": {...},
    "localRotation": {...},
}

Joint Metadata Response

The name of the joint. Names are in the form

robot_arm_i_jnt
for i{1,2,3,4}i\in\{1, 2, 3, 4\}.

The global, world (x,y,z)(x, y, z) position of the joint.

The (x,y,z)(x, y, z) position of the joint, relative to the

armBase
.

The global, world (x,y,z,w)(x, y, z, w) rotation quaternion of the joint.

The (x,y,z,w)(x, y, z, w) rotation quaternion of this joint, relative to the first joint of the arm.

The (x,y,z,w)(x, y, z, w) rotation quaternion of this joint, relative to the previous joint of the arm, which is the joint's parent in the hierarchy.

Within the metadata dictionary, the agent's key contains the pose of the agent after the action has executed.

event.metadata["agent"]
Response
{
    "cameraHorizon": {...},
    "isStanding": {...},
    "position": {...},
    "rotation": {...},
    {...}
}

Agent Metadata Response

The angle in degrees that the camera's pitch is rotated.

Warning

As illustrated in the figure below, negative camera

horizon
values correspond to agent looking up, whereas positive
horizon
values correspond to the agent looking down.

True if the agent is currently in a standing position, otherwise false. This bool can be changed if the agent uses the Stand or Crouch actions, which will change the camera height.

Warning

The default agent is currently the only agent with the ability to stand.

The global position of the agent, with keys for xx, yy, and zz.

Note that yy corresponds to the upward coordinate in 3D space.

The local rotation of the agent's body, with keys for xx (pitch), yy (yaw), and zz (roll). Since the LoCoBot's body can only change its yaw rotation, both xx and zz will always be approximately 0.

Each object has a plethora of information exposed about it during each step.

event.metadata["objects"]
Response
{
    "objectId": {...},
    "objectType": {...},
    "name": {...},

    "distance": {...},
    "visible": {...},

    "position": {...},
    "rotation": {...},

    "axisAlignedBoundingBox": {...},
    "objectOrientedBoundingBox": {...},

    "mass": {...},
    "salientMaterials": {...},

    "parentReceptacles": {...},
    "receptacle": {...},
    "receptacleObjectIds": {...},

    "ObjectTemperature": {...},
    "canChangeTempToHot": {...},
    "canChangeTempToCold": {...},

    "breakable": {...},
    "isBroken": {...},

    "canBeUsedUp": {...},
    "isUsedUp": {...},

    "openable": {...},
    "isOpen": {...},
    "openness": {...},

    "moveable": {...},
    "isMoving": {...},

    "pickupable": {...},
    "isPickedUp": {...},

    "sliceable": {...},
    "isSliced": {...},

    "toggleable": {...},
    "isToggled": {...},

    "canFillWithLiquid": {...},
    "isFilledWithLiquid": {...},

    "cookable": {...},
    "isCooked": {...},

    "dirtyable": {...},
    "isDirty": {...},
}

Object Metadata Response

The unique ID of each object in the scene. It is generated at runtime and composed of an object's

objectType
and
position
.

Example:

AlarmClock|-02.08|+00.94|-03.62
.

The annotated type of the object. Each time is specified in the

Object Types
section.

Name of the object in Unity Scene. These names are unique within any individual scene.

The Euclidean distance from near the center-point of the object to the agent's camera.

Indicates whether the object is visible, and within the initialized visibility distance of the agent.

Warning

The visible property does not mean the object is literally visible in the frame. Rather, the object both has to be visible and at a distance of less than the initialized

visibilityDistance
away.

The global position of the object, with keys for xx, yy, and zz.

Note that yy corresponds to the upward coordinate in 3D space.

The local rotation of the object, with keys for xx (pitch), yy (yaw), and zz (roll).

Returns an axisAlignedBoundingBox object that includes an 8x3 matrix of xyz coordinates that represent the 8 corners of the box, an xyz dictionary for the coordinates of the center of the box, and an xyz dictionary for the size (extents) of the box. This axis aligned bounding box is the smallest box that can completely encloses the sim object that is aligned to the world axis. This means that if the object is rotated or moved, the overall size and volume of this axis aligned box will change in order to remain aligned relative to the static, world-axis. This is best used for rough approximations of the area/volume an object takes up. Do note that large, oddly shaped objects like countertops that wrap around kitchens may have awkwardly defined axis-aligned bounds. Example:

{
  "center": {
    "x": -1.336,
    "y":  1.098,
    "z":  0.221
  },
 "cornerPoints": [
   [-1.232, 1.277, 0.319],
   [-1.232, 1.277, 0.124],
   [-1.232, 0.919, 0.319],
   [-1.232, 0.919, 0.124],
   [ -1.44, 1.277, 0.319],
   [ -1.44, 1.277, 0.124],
   [ -1.44, 0.919, 0.319],
   [ -1.44, 0.919, 0.124],
 ],
 "size": {
   "x": 0.208,
   "y": 0.358,
   "z": 0.195
  }
}

Returns an objectOrientedBoundingBox object that includes an 8x3 matrix of xyz coordinates that represent the 8 corners of the object oriented box. This object oriented bounding box is a box that completely encloses a sim object. The difference between this object oriented box and the axis aligned box above is this box’s dimensions are static relative to the object’s rotation and position. This means this object oriented box will always have the same total volume regardless of how the object is manipulated/rotated/moved. Note that only Pickupable objects have an objectOrientedboundingBox at this time. Example:

{
  'cornerPoints': [
    [-1.445, 0.910, 0.115],
    [-1.228, 0.910, 0.115],
    [-1.228, 0.910, 0.328],
    [-1.445, 0.910, 0.328],
    [-1.445, 1.284, 0.115],
    [-1.228, 1.284, 0.115],
    [-1.228, 1.284, 0.328],
    [-1.445, 1.284, 0.328]
  ]
}

The mass of a Pickupable or Moveable sim object in Kilograms

Array of strings listing the salient materials a pickupable object is composed of. Valid strings are: Metal, Wood, Plastic, Glass, Ceramic, Stone, Fabric, Rubber, Food, Paper, Wax, Soap, Sponge, and Organic.

A list of objectId strings of all receptacles that contain this object.

If True, this object has the Receptacle property and can contain other objects.

If the object is a receptacle, this is an array of objectIds that the receptacle contains.

String that indicates what this object’s current abstracted temperature is. Valid strings are: Hot, Cold, RoomTemp.

If True, this object is a source of Heat and can contextually change other object’s Temperature to Hot.

If True, this object is a source of Cold and can contextually change other object’s Temperature to Cold.

If True, this object has the Breakable property and can be broken with the BreakObject action or contextually if enough force is applied to it.

Only used if this object is breakable = True. If True, this object is currently in its broken state. If False, this object is not broken.

If True, this object has the UsedUp property and can change its state to a depleted form with the UsedUpObject action.

Only used if this object is canBeUsedUp = True. If True, this object is currently in its used up state. If False, the object is full and not used up yet.

If True, this object has the openable property and can be switched to an open or closed state with the OpenObject and CloseObject actions.

Only used if this object is openable = True. If True, this object is currently in its opened state. If False, this object is in its closed state.

The proportion that the object is open, linearly scaled to the range [0:1][0:1].

If True, this object has the Moveable property and can be moved around the environment with actions like PushObject.

A bool tracking if this object is actively in motion. This can be useful when tracking object behaviors while using PausePhysicsAutoSim and AdvancePhysicsStep actions.

If True, this object has the Pickupable property and can be picked up by the agent with the PickupObject action.

Only used if this object is pickupable = True. If True, this object is currently being picked up by the agent. If False, the object is not being held by the agent.

If True, this object has the sliceable property and can be sliced into multiple pieces with the SliceObject action.

Only used if this object is sliceable = True. If True, this object has been sliced. If False, this object is full and not sliced.

If True, this object has the Toggleable property and can be manipulated with the ToggleObjectOn and ToggleObjectOff actions.

Only used if this object is Toggleable = True. If True, this object is toggled on. If False, the object is toggled off.

If True, this object has the Fillable property and can be filled with liquid using the FillObjectWithLiquid action or via contextual interactions with some parts of the environment.

Only used if this object is canFillWithLiquid = True. If True, this object is currently filled. If False, this object is empty.

If True, this object has the cookable property and can change its state to a cooked form with the CookObject action.

Only used if this object is cookable = True. If True, this object is currently in its cooked state. If False, the object is not cooked.

If True, this object has the Dirty property, allowing it to be switched between clean and dirty states using the DirtyObject and CleanObject actions.

Only used if this object is dirtyable = True. If True, this object is currently in its dirty state. If False, this object is in its clean state.