ManipulaTHOR API Reference

required

Always set this to

arm

in order to use the agent designed for ManipulaTHOR.

Warning

If unset, the agent will be the default iTHOR agent, which is not the agent designed for ManipulaTHOR. Thus, the agent will have different properties than these docs.

A RuntimeError is raised if

agentMode

is not set to "default", "locobot", "drone", or "arm".

massThreshold

: Optional[float] = None

Blocks the arm from being able to directly interact and move objects with a mass above

massThreshold

. Empirically, it may help make the environment more deterministic.

Remark

This only stops the arm from knocking over things above the threshold that it directly touches. For example, if the arm touches an object with a mass below

massThreshold

that is adjacent to an object with a mass above

massThreshold

, pushing the smaller object can still cause the larger object to move since no direct contact was made.

scene

: Optional[str] = None

The name of which scene to initialize. ManipulaTHOR is currently set up to work with iTHOR scenes. All iTHOR scene names and descriptions can be found here.

visibilityDistance

: float = 1.5

Used to set the maximum goal distance in meters between the agent and an object. This affects each object's visible property, where when True, the agent is within

visibilityDistance

of that object and the object appears in the agent's current egocentric frame.

gridSize

: float = 0.25

With the ManipulaTHOR agent, this only affects how far positions are spaced with

GetReachablePositions

renderDepthImage

: bool = False

When

True

, a depth frame is rendered and made available as an event.depth_frame attribute.

We require this to be explicitly passed in because rendering depth takes longer than rendering RGB alone.

renderInstanceSegmentation

: bool = False

When

True

, a instance segmentation frame is rendered and made available as an

Event.instance_segmentation_frame

attribute. Instance segmentation provides segmentation by object instance, where all objects in view are distinguishable.

We require this to be explicitly passed in because rendering instance segmentation takes longer than rendering RGB alone.

width

: int = 300

The number of horizontal sampled pixels for each frame. This affects every rendered image frame (e.g., RGB, depth, BGR).

height

: int = 300

The number of vertical sampled pixels for each frame. This affects every rendered image frame (e.g., RGB, depth, BGR).

fieldOfView

: int = 60

Changes the camera's optical field of view, in degrees. Valid values are in the domain $(0:180)$ . The default field of view for the LoCoBot is 60 degrees.

Remark

Not all agent modes have the same default field of view. For instance, the drone's field of view is 150 degrees, by default.

Reset

Any

Controller

Initialization parameter can later be changed by calling the reset method. For instance, we can call:

controller.reset(scene="FloorPlan15", fieldOfView=80)

The values will default to what they were upon the most recent reset or initialization. For instance, if you initialized with

fieldOfView=45

, then called

reset

with

fieldOfView=60

, calling

reset

again, without passing in

fieldOfView

will keep

fieldOfView=60

Movement

The arm agent has three components that may move: the arm, which has several degrees of freedom, the arm base, and the agent's body.

Move Agent

MoveAgent

moves the agent's body from it's current facing direction. Compared to iTHOR move actions (e.g.,

MoveAhead

MoveAgent

can be used to move the agent diagonally (e.g., move ahead and left at the same time).

controller.step(
    action="MoveAgent",
    ahead=0.25,
    right=0.25,
    returnToStart=True,
    speed=1,
    fixedDeltaTime=0.02
)

The video belows shows the result of consecutively calling the action with (ahead, right) $= (0.25, 0.0)\to (-0.5, 0.0)\to (0.0, 0.25)\to (0.0, -0.5)\to (0.5, 0.5)$ .

Move Agent Parameters

ahead

The distance, in meters, that the agent attempts to move in its forward facing direction.

Remark

Use negative values to move the agent backwards.

right

The distance, in meters, that the agent attempts to move rightward from its forward facing direction.

Remark

Use negative values to move the agent leftwards.

returnToStart

Sets the agent back to its starting pose if it collides with an object while attempting to move.

speed

How fast the agent moves, in meters per second.

fixedDeltaTime

The interval, in seconds, with which that physics time steps are simulated.

For instance, if the interval is large, then the agent may move to the location in a manner that looks like teleportation. But, then there may not be many checks along the path to see if the agent collides with an object as it moves.

Rotate Agent

RotateAgent

rotates the agent's body from it's current facing direction.

controller.step(
    action="RotateAgent",
    degrees=30,
    returnToStart=True,
    speed=1,
    fixedDeltaTime=0.02
)

The video belows shows the result of consecutively the action with degrees $= 60\to -120$ .

Rotate Agent Parameters

degrees

required

Amount to rotate the agent rightward of its current facing direction.

Remark

Use negative values to rotate the agent leftwards.

returnToStart

Sets the agent back to its starting pose if it collides with an object while attempting to move.

speed

How fast the agent moves, in degrees per second.

fixedDeltaTime

The interval, in seconds, with which physics time steps are simulated.

If the interval is large, for instance, then the agent may move to the location in a manner that looks like teleportation. But, then there may not be many checks along the path to see if the agent were to collide with an object as it moved.

Move Arm

Move the wrist joint of the Mid-Level-Arm to position coordinate. Position coordinates are relative to the arm’s forward direction. This means using (0, 0, 1) will extend the arm straight forward in the arm’s positive z direction. Note that the coordinates can also be passed in relative to the camera on the agent’s hand. See the handCameraSpace parameter below for more info.

controller.step(
    action="MoveArm",
    position=dict(x=0, y=0.5, z=0),
    coordinateSpace="armBase",
    restrictMovement=False,
    speed=1,
    returnToStart=True,
    fixedDeltaTime=0.02
)

Pickup Object Parameters

position

: Dict[str, float]

required

The position of where to move the arm. See the

coordinateSpace

parameter for how the position is interpreted. All positions are given in terms of meters.

coordinateSpace

: str = "armBase"

Defines how the

position

parameter should be interpreted. Valid values are:

armBase
moves the arm in the coordinate system relative to the spot where the spot where the arm's "shoulder" is being held by the base.
Analogous to
Teleport
, using the same coordinate global space, but with the arm. For instance, one may use object metadata to move the arm near an object's global position.
After initializing a scene, the arm starts at $(x=0, y=0, z=0.5)$ , relative to the
armBase
.
The video below shows many examples of
moveArm
with
armBase
as the
coordinateSpace
. Notice:
- $x$ corresponds to the left and right direction (with $x=0$ in the middle).
- $y$ corresponds to the vertical direction (with $y=0$ being the middle).
- $z$ corresponds to the distance from the agent, with $z=0$ being next to the agent, and $z=0.75$ being $0.75$ meters in front of the agent.
wrist
moves the arm relative to its current position and rotation.
Analogous to an action like
MoveAgent
, where calling it consecutively attempts to keep moving the agent to a new position, whereas calling
Teleport
consecutively will never change the goal position, after the first call.
The video below shows the arm starting at
armBase
coordinate $(x=0.5, y=-0.5, z=0.5)$ and consecutively calling
MoveArm
with $(x=-0.2, y=0.2, z=0)$ . Here, notice that since the position is relative to the agent's current arm position, the arm continues to move after passing in the same position consecutively.
world
moves the arm to a position in global, world space.
Analogous to
Teleport
, using the same coordinate global space, but with the arm. For instance, one may use object metadata to move the arm near an object's global position.

restrictMovement

: bool = False

Restricts the arm to only being able to reach certain positions that are realistic. If

False

, the agent will be able to reach behind itself and clipping into itself is ignored.

The video below shows an example where

restrictMovement

False

, which allows the agent to reach behind itself.

speed

How fast the arm moves, in meters per second.

returnToStart

Reverts the arm's position to its pose before the action was called, if the arm collides with an object while attempting to move it into position.

fixedDeltaTime

The interval, in seconds, with which that physics time steps are simulated.

For instance, if the interval is large, then the agent's arm may move to the location in a manner that looks like teleportation. But, then there may not be many checks along the path to see if the agent's arm collides with an object as it moves.

Move Arm Base

Raise and lower the height of the entire arm. This height is relative to the minimum and maximum extents, relative to the agent's size.

controller.step(
    action="MoveArmBase",
    y=0.5,
    speed=1,
    returnToStart=True,
    fixedDeltaTime=0.02
)

Move Arm Base Parameters

y

required

The height of the arm. Value values are in $[0:1]$ , where $0$ is the lowest possible height of the arm and $1$ is the largest hight of the arm.

The video below displays the arm base

moving between

0.5 \to 1.0\to 0.0\to 0.5\to 0.75\to 0.25

(in that order).

speed

How fast the arm moves, in meters per second.

returnToStart

Sets the agent's arm base position back to its starting

coordinate if it collides with an object while attempting to move.

fixedDeltaTime

: Optional[dict[str, float]] = None

The interval, in seconds, with which physics time steps are simulated.

If the interval is large, for instance, then the agent's arm may move into position in a manner that looks like teleportation. But, then there may not be many checks along the path to see if the arm were to collide with an object as it moved.

Camera Rotation

We can adjust the camera's pitch by utilizing the

LookUp

and

LookDown

commands. Both increment in

30^\circ

intervals, with angles being clamped between

60^\circ

in the downward direction and

30^\circ

in the upward direction.

controller.step("LookUp")
controller.step("LookDown")

Teleport

allows the agent to set its pose to anywhere in the scene in a single step. Valid poses do not place the agent outside the outer boundaries of the scene or in an area that it collides with an object.

controller.step(
    action="Teleport",
    position=dict(x=1, y=0.9, z=-1.5),
    rotation=dict(x=0, y=270, z=0),
    horizon=30,
    standing=True
)

Teleport Parameters

position

The position of the agent's body in global 3D coordinate space. If unspecified, the position of the agent remains the same.

rotation

: Optional[dict[str, float]] = None

The rotation of the agent's body in global 3D space. Here, the rotation changes the agent's yaw rotation.

Remark

The default agent's body cannot change in pitch or roll; hence, why the $x, z=0$ .

If unspecified, the rotation of the agent remains the same.

horizon

: Optional[float] = None

The

horizon

change's the camera's rotation. Values are clamped between [-30:60].

Since the agent looks up and down in $30^\circ$ increments, it most common for the horizon to be in $\lbrace -30, 0, 30, 60\rbrace$ .

Warning

Negative camera

horizon

values correspond to agent looking up, whereas positive

horizon

values correspond to the agent looking down.

If unspecified, the horizon of the agent remains the same.

standing

: Optional[bool] = None

Denotes whether the agent is in a standing or crouched position. If unspecified, the standing state of the agent remains the same.

In addition to being able to

Teleport

, we also have an equivalent action

TeleportFull

that strictly requires every degree of freedom to be specified. The strictness of this action is often useful in large projects, where we don't want to any part of the agent implied implicitly.

Warning

TeleportFull

does not guarantee backwards compatibility in future releases. If a new degree of freedom is added to the agent, it will be added as a required parameter. If this is an issue, we recommend using

Teleport

controller.step(
    action="TeleportFull",
    position=dict(x=1, y=0.9, z=-1.5),
    rotation=dict(x=0, y=270, z=0),
    horizon=30,
    standing=True
)

TeleportFull Parameters

position

required

The position of the agent's body in global 3D coordinate space.

rotation

required

The rotation of the agent's body in global 3D space. Here, the rotation changes the agent's yaw rotation.

Remark

The default agent's body cannot change in pitch or roll; hence, why the $x, z=0$ .

horizon

required

The

horizon

change's the camera's rotation. Values are clamped between [-30:60].

Since the agent looks up and down in $30^\circ$ increments, it most common for the horizon to be in $\lbrace -30, 0, 30, 60\rbrace$ .

Warning

Negative camera

horizon

values correspond to agent looking up, whereas positive

horizon

values correspond to the agent looking down.

standing

required

Denotes whether the agent is in a standing or crouched position.

It is often useful to randomize the position of the agent in the scene, before starting an episode. Here, we can use:

GetReachablePositions, which does an optimized BFS over a grid spaced out by the initialized
gridSize
. The valid positions are then added and returned in a list.
Teleport
, which can take a given position, and transform the agent to that position.

The process is illustrated below:

Step 1: Get the Positions

positions = controller.step(
    action="GetReachablePositions"
).metadata["actionReturn"]

Response

[
    dict(x=(...), y=(...), z=(...)),
    dict(x=(...), y=(...), z=(...)),
    dict(x=(...), y=(...), z=(...)),
    {...}
    dict(x=(...), y=(...), z=(...)),
]

Step 2: Teleport to a Position

import random
position = random.choice(positions)
controller.step(
    action="Teleport",
    position=position
)

Get Reachable Positions Response

actionReturn

: list[dict[str, float]]

A list of each position that the agent can be at, without colliding into any other objects or going beyond the walls.

Done

The

Done

action nothing to the state of the environment. But, it returns a cleaned up event with respect to the metadata.

controller.step(action="Done")

It is often used in the definition of a successful navigation task (see Anderson et al.), where the agent must call the done action to signal that it knows that it's done, rather than arbitrarily or biasedly guessing.

Warning

The

Done

action does literally nothing to the state of the environment. For instance, if the agent calls

Done

and then

MoveAgent

, the

Done

action will have no affect on preventing a

MoveAgent

action from executing.

It is often used to return a cleaned up version of the metadata.

Interaction

This section provides several actions that facilitate interaction with the ManipulaTHOR agent. If you have any additional actions that you would like to see, please create an issue and we'll see if we can support it!

Pickup Object

Picks up all pickupable sim objects that are within the agent's hand sphere radius.

controller.step(
    action="PickupObject",
    objectIdCandidates=["Apple|1|1|0", "Fork|2|1|1"]
)

Pickup Object Parameters

objectIdCandidates

: list[str] = []

Restricts the arm to only being able to pickup these specific objects. These objects will only be picked up iff they are within the agent's hand sphere radius.

Release Object

controller.step(action="ReleaseObject")

Release Object Description

Drops any objects that the agent is currently holding.

Set Hand Sphere Radius

Changes the radius of the agent's "magnet sphere" hand. Here, if a pickupable object is within

radius

of the agent's hand and

PickupObject

is called, the object will be picked up.

controller.step(
    action="SetHandSphereRadius",
    radius=0.1
)

Parameters

radius

required

The radius on the agent's "magnet sphere" hand, in meters. Valid values are in $(0.04:0.5)$ meters.

Environment State

The figure above shows AI2-THOR's agent-simulator interaction loop. Our backend scenes, actions, and observations are stored within Unity, a powerful real-time game engine.

From the Python side, we sequentially interact with Unity by taking actions using AI2-THOR's Python

Controller

. We have already seen several actions, such as

MoveAgent

RotateAgent

, and

LookUp

Events

The returned data after executing any action is an

Event

. Each

Event

contains a ton of information to encode the state of the environment after the action has been executed:

event = controller.step(...)

References the last executed Event

controller.last_event

Response

<ai2thor.server.Event
    .metadata: {...}

    .frame: {...}
    .cv2img: {...}

    .depth_frame: {...}

    .instance_segmentation_frame: {...}
    .instance_masks: {...}
    .instance_detections2D: {...}

    .color_to_object_id: {...}
    .object_id_to_color: {...}
/>

Event Response

metadata

: Optional[numpy.ndarray]

Contains information about the action and how it affected the environment. See the

Metadata

section for more.

frame

: numpy.ndarray

RGB frame of the scene from the agent egocentric perspective. The size of the frame is the initialized (

height

width

, 3) stored in the numpy.uint8 format.

cv2img

: numpy.ndarray

Same as the

frame

, except image channels are in BGR ordering. This is often useful with Python's OpenCV module (i.e., cv2), which expects images with BGR orderings.

depth_frame

Measures the distance, in meters, from the agent to each pixel in the frame. The size of the frame is the initialized (

height

width

) stored in the numpy.float32 format.

Remark

Upon

initialization

reset

renderDepthImage

must be set to

True

in order for the depth frame to appear.

instance_segmentation_frame

: Optional[numpy.ndarray]

Segments every object instance in the scene. The size of the frame is (

height

width

, 3) stored in the numpy.uint8 format. Each unique pixel color corresponds to a different object, which are indexable with

color_to_object_id

object_id_to_color

Remark

Upon

initialization

reset

renderInstanceSegmentation

must be set to

True

in order for the instance segmentation frame to appear.

instance_masks

: Optional[dict[str, numpy.ndarray]]

Segments every object instance in the scene, storing each segmentation as a boolean image mask.

The keys are object IDs visible in the frame and each value is the boolean numpy.ndarray of size

height

width

) that segments the image.

Remark

Upon

initialization

reset

renderInstanceSegmentation

must be set to

True

in order for the instance masks to appear.

instance_detections2D

: Optional[dict[str, numpy.array]]

Provides the 2D bounding box around every object instance in the scene.

The keys are object IDs and the values are [Upper Left $x$ , Upper Left $y$ , Lower Right $x$ , Lower Right $y$ ], where each element is the number of pixels it is from the top left corner of the image.

Remark

Upon

initialization

reset

renderInstanceSegmentation

must be set to

True

in order for the 2D instance detections to appear.

color_to_object_id

: dict[tuple[int, int, int], str]

Provides a mapping from each pixel's color, represented as a hashable tuple, in the

instance_segmentation_frame

to what object it represents.

object_id_to_color

: dict[str, tuple[int, int, int]]

Same as

color_to_object_id

, except for the keys and values being swapped.

Metadata

The metadata dictionary contains summary information about the state of the environment.

event.metadata

Response

{
    "errorMessage": {...}
    "lastActionSuccess": {...}
    "actionReturn": {...}
    "lastAction": {...}

    "sceneName": {...}
    "sceneBounds": {...}

    "arm": {...}
    "agent": {...}
    "objects": {...}

    "fov": {...}
    "screenWidth": {...}
    "screenHeight": {...}
    {...}
}

Metadata Response

errorMessage

If an action fails, feedback is provided as to what caused the failure.

lastActionSuccess

States whether the last action was able to successfully execute.

An exception will not be raised upon invalid states. For instance, if an agent is standing right in-front of a wall and tries to

MoveAhead

, it will fail, but an exception will not be raised.

If an action is unsuccessful, the state of the environment will not have changed.

actionReturn

: any

Holds the resulting data for actions that query the environment, such as

Get Reachable Positions

lastAction

The name of the action passed into the Controller.

sceneName

The name of the scene that the agent is currently located in.

sceneBounds

This object provides all coordinates that are within bounds of the scene. This can be used in tandem with actions like PlaceObjectAtPoint to make sure the coordinate used is not out of bounds. This returns a sceneBounds object that includes an 8x3 matrix of xyz coordinates that represent the 8 corners of the box encompassing the entire scene, an xyz dictionary for the coordinates of the center of that box, and an xyz dictionary for the size (extents) of that box. Example:

{
  "center": {
    'x': -1.45,
    'y': 1.407,
    'z': 0.2
  },
 "cornerPoints": [
   [ 1.5,   2.92,  3.2],
   [ 1.5,   2.92, -2.8],
   [ 1.5, -0.106,  3.2],
   [ 1.5, -0.106, -2.8],
   [-4.4,   2.92,  3.2],
   [-4.4,   2.92, -2.8],
   [-4.4, -0.106,  3.2],
   [-4.4, -0.106, -2.8]
  ],
 "size": {
   "x": 5.9,
   "y": 3.02,
   "z": 6.0
  }
}

arm

ManipulaTHOR specific information about the state of the agent's arm. See Arm Metadata for more.

agent

Information about the pose of the agent. See Agent Metadata for more.

objects

: list[dict[str, any]]

Information about the state of each object in the scene. See Object Metadata for more.

fov

The field of view of the agent's camera, which can be changed upon initialization or reset.

screenWidth

: int

The number of pixels that make up the width of each rendered image frame from the agent.

screenHeight

: int

The number of pixels that make up the height of each rendered image frame from the agent.

Arm Metadata

Metadata specific to the arm ManipulaTHOR agent.

event.metadata["arm"]

Response

{
    "heldObjects": {...},
    "pickupableObjects": {...},

    "handSphereCenter": {...},
    "handSphereRadius": {...},

    "joints": {...},
}

Arm Metadata Response

heldObjects

A list of objectId's currently held by the agent.

pickupableObjects

All pickupable objects located within the agent's hand sphere radius, which could be picked up.

handSphereCenter

The global, world coordinate of the center $(x, y, z)$ position of the agent's hand sphere.

handSphereRadius

The radius of the hand sphere, which determines how close objects must be to allow them to be picked up. The value can be changed with the Set Hand Sphere Radiusaction.

joints

: list[dict[str, any]]

Metadata for each of the $4$ joints on the arm. See the Joint Metadata section for more.

Joint Metadata

Metadata for each of the $4$ joints of the ManipulaTHOR arm.

import random
i = random.choice(range(4))

event.metadata["arm"]["joints"][i]

Response

{
    "name": {...},

    "position": {...},
    "rootRelativePosition": {...},

    "rotation": {...},
    "rootRelativeRotation": {...},
    "localRotation": {...},
}

Joint Metadata Response

name

The name of the joint. Names are in the form

robot_arm_i_jnt

for

i\in\{1, 2, 3, 4\}

position

The global, world $(x, y, z)$ position of the joint.

rootRelativePosition

The $(x, y, z)$ position of the joint, relative to the

armBase

rotation

The global, world $(x, y, z, w)$ rotation quaternion of the joint.

rootRelativeRotation

The $(x, y, z, w)$ rotation quaternion of this joint, relative to the first joint of the arm.

localRotation

The $(x, y, z, w)$ rotation quaternion of this joint, relative to the previous joint of the arm, which is the joint's parent in the hierarchy.

Agent Metadata

Within the metadata dictionary, the agent's key contains the pose of the agent after the action has executed.

event.metadata["agent"]

Response

{
    "cameraHorizon": {...},
    "isStanding": {...},
    "position": {...},
    "rotation": {...},
    {...}
}

Agent Metadata Response

cameraHorizon

The angle in degrees that the camera's pitch is rotated.

Warning

As illustrated in the figure below, negative camera

horizon

values correspond to agent looking up, whereas positive

horizon

values correspond to the agent looking down.

isStanding

True if the agent is currently in a standing position, otherwise false. This bool can be changed if the agent uses the Stand or Crouch actions, which will change the camera height.

Warning

The default agent is currently the only agent with the ability to stand.

position

The global position of the agent, with keys for $x$ , $y$ , and $z$ .

Note that $y$ corresponds to the upward coordinate in 3D space.

rotation

The local rotation of the agent's body, with keys for $x$ (pitch), $y$ (yaw), and $z$ (roll). Since the LoCoBot's body can only change its yaw rotation, both $x$ and $z$ will always be approximately 0.

Object Metadata

Each object has a plethora of information exposed about it during each step.

event.metadata["objects"]

Response

{
    "objectId": {...},
    "objectType": {...},
    "name": {...},

    "distance": {...},
    "visible": {...},

    "position": {...},
    "rotation": {...},

    "axisAlignedBoundingBox": {...},
    "objectOrientedBoundingBox": {...},

    "mass": {...},
    "salientMaterials": {...},

    "parentReceptacles": {...},
    "receptacle": {...},
    "receptacleObjectIds": {...},

    "ObjectTemperature": {...},
    "canChangeTempToHot": {...},
    "canChangeTempToCold": {...},

    "breakable": {...},
    "isBroken": {...},

    "canBeUsedUp": {...},
    "isUsedUp": {...},

    "openable": {...},
    "isOpen": {...},
    "openness": {...},

    "moveable": {...},
    "isMoving": {...},

    "pickupable": {...},
    "isPickedUp": {...},

    "sliceable": {...},
    "isSliced": {...},

    "toggleable": {...},
    "isToggled": {...},

    "canFillWithLiquid": {...},
    "isFilledWithLiquid": {...},

    "cookable": {...},
    "isCooked": {...},

    "dirtyable": {...},
    "isDirty": {...},
}

Object Metadata Response

objectId

The unique ID of each object in the scene. It is generated at runtime and composed of an object's

objectType

and

position

Example:

AlarmClock|-02.08|+00.94|-03.62

objectType

The annotated type of the object. Each time is specified in the

Object Types

section.

name

Name of the object in Unity Scene. These names are unique within any individual scene.

distance

The Euclidean distance from near the center-point of the object to the agent's camera.

visible

Indicates whether the object is visible, and within the initialized visibility distance of the agent.

Warning

The visible property does not mean the object is literally visible in the frame. Rather, the object both has to be visible and at a distance of less than the initialized

visibilityDistance

away.

position

The global position of the object, with keys for $x$ , $y$ , and $z$ .

Note that $y$ corresponds to the upward coordinate in 3D space.

rotation

: dict[str, union[str, any]]

The local rotation of the object, with keys for $x$ (pitch), $y$ (yaw), and $z$ (roll).

axisAlignedBoundingBox

Returns an axisAlignedBoundingBox object that includes an 8x3 matrix of xyz coordinates that represent the 8 corners of the box, an xyz dictionary for the coordinates of the center of the box, and an xyz dictionary for the size (extents) of the box. This axis aligned bounding box is the smallest box that can completely encloses the sim object that is aligned to the world axis. This means that if the object is rotated or moved, the overall size and volume of this axis aligned box will change in order to remain aligned relative to the static, world-axis. This is best used for rough approximations of the area/volume an object takes up. Do note that large, oddly shaped objects like countertops that wrap around kitchens may have awkwardly defined axis-aligned bounds. Example:

{
  "center": {
    "x": -1.336,
    "y":  1.098,
    "z":  0.221
  },
 "cornerPoints": [
   [-1.232, 1.277, 0.319],
   [-1.232, 1.277, 0.124],
   [-1.232, 0.919, 0.319],
   [-1.232, 0.919, 0.124],
   [ -1.44, 1.277, 0.319],
   [ -1.44, 1.277, 0.124],
   [ -1.44, 0.919, 0.319],
   [ -1.44, 0.919, 0.124],
 ],
 "size": {
   "x": 0.208,
   "y": 0.358,
   "z": 0.195
  }
}

objectOrientedBoundingBox

: optional[dict[str, list[list[float]]]]

Returns an objectOrientedBoundingBox object that includes an 8x3 matrix of xyz coordinates that represent the 8 corners of the object oriented box. This object oriented bounding box is a box that completely encloses a sim object. The difference between this object oriented box and the axis aligned box above is this box’s dimensions are static relative to the object’s rotation and position. This means this object oriented box will always have the same total volume regardless of how the object is manipulated/rotated/moved. Note that only Pickupable objects have an objectOrientedboundingBox at this time. Example:

{
  'cornerPoints': [
    [-1.445, 0.910, 0.115],
    [-1.228, 0.910, 0.115],
    [-1.228, 0.910, 0.328],
    [-1.445, 0.910, 0.328],
    [-1.445, 1.284, 0.115],
    [-1.228, 1.284, 0.115],
    [-1.228, 1.284, 0.328],
    [-1.445, 1.284, 0.328]
  ]
}

mass

The mass of a Pickupable or Moveable sim object in Kilograms

salientMaterials

Array of strings listing the salient materials a pickupable object is composed of. Valid strings are: Metal, Wood, Plastic, Glass, Ceramic, Stone, Fabric, Rubber, Food, Paper, Wax, Soap, Sponge, and Organic.

parentReceptacles

A list of objectId strings of all receptacles that contain this object.

receptacle

If True, this object has the Receptacle property and can contain other objects.

receptacleObjectIds

If the object is a receptacle, this is an array of objectIds that the receptacle contains.

ObjectTemperature

String that indicates what this object’s current abstracted temperature is. Valid strings are: Hot, Cold, RoomTemp.

canChangeTempToHot

If True, this object is a source of Heat and can contextually change other object’s Temperature to Hot.

canChangeTempToCold

If True, this object is a source of Cold and can contextually change other object’s Temperature to Cold.

breakable

If True, this object has the Breakable property and can be broken with the BreakObject action or contextually if enough force is applied to it.

isBroken

Only used if this object is breakable = True. If True, this object is currently in its broken state. If False, this object is not broken.

canBeUsedUp

If True, this object has the UsedUp property and can change its state to a depleted form with the UsedUpObject action.

isUsedUp

Only used if this object is canBeUsedUp = True. If True, this object is currently in its used up state. If False, the object is full and not used up yet.

openable

If True, this object has the openable property and can be switched to an open or closed state with the OpenObject and CloseObject actions.

isOpen

Only used if this object is openable = True. If True, this object is currently in its opened state. If False, this object is in its closed state.

openness

The proportion that the object is open, linearly scaled to the range $[0:1]$ .

moveable

If True, this object has the Moveable property and can be moved around the environment with actions like PushObject.

isMoving

A bool tracking if this object is actively in motion. This can be useful when tracking object behaviors while using PausePhysicsAutoSim and AdvancePhysicsStep actions.

pickupable

If True, this object has the Pickupable property and can be picked up by the agent with the PickupObject action.

isPickedUp

Only used if this object is pickupable = True. If True, this object is currently being picked up by the agent. If False, the object is not being held by the agent.

sliceable

If True, this object has the sliceable property and can be sliced into multiple pieces with the SliceObject action.

isSliced

Only used if this object is sliceable = True. If True, this object has been sliced. If False, this object is full and not sliced.

toggleable

If True, this object has the Toggleable property and can be manipulated with the ToggleObjectOn and ToggleObjectOff actions.

isToggled

Only used if this object is Toggleable = True. If True, this object is toggled on. If False, the object is toggled off.

canFillWithLiquid

If True, this object has the Fillable property and can be filled with liquid using the FillObjectWithLiquid action or via contextual interactions with some parts of the environment.

isFilledWithLiquid

Only used if this object is canFillWithLiquid = True. If True, this object is currently filled. If False, this object is empty.

cookable

If True, this object has the cookable property and can change its state to a cooked form with the CookObject action.

isCooked

Only used if this object is cookable = True. If True, this object is currently in its cooked state. If False, the object is not cooked.

dirtyable

If True, this object has the Dirty property, allowing it to be switched between clean and dirty states using the DirtyObject and CleanObject actions.

isDirty