Given only their egocentric visual observations, agents jointly hold a lifted piece of furniture in a living room scene and must collaborate to move it to a visually distinct goal location. As a piece of furniture cannot be moved without both agents agreeing on the direction, agents must explicitly coordinate at every timestep.

In FurnMove, each agent at every timestep receives an egocentric observation (a $3\times 84 \times 84$ RGB image) from AI2-THOR. In addition, agents are allowed to communicate with other agents at each timestep via a low bandwidth communication channel. Based on their local observation and communication, each agent must take an action from the set $A$.

At test time each agent must take only a egocentric observation (a $3\times 84 \times 84$ RGB image) from AI2-THOR.

Do not exploit the metadata in test-scenes: You cannot use additional depth, mask, metadata info etc. from the simulator on test scenes. However, during training you are free to use additional info for things like auxiliary losses.

If you use additional sensory information from AI2-THOR as input (e.g., depth, segmentation masks, class masks, panoramic images) during test-time, your entry will not be considered. For official consideration to the CVPR 2022 challenge, agents should just use RGB input.

Each agent can take has an action space defined by $A = A^{NAV} ∪ A^{MWO} ∪ A^{MO} ∪ A^{RO}$, where:

• $A^{NAV} = \lbrace \text{MoveAhead}, \text{RotateLeft}, \text{RotateRight}, \text{Pass}\rbrace$ used to independently move each agent.
• $A^{MWO} = \lbrace \text{MoveWithObject}X \mid X \in \lbrace \text{Ahead}, \text{Right}, \text{Left}, \text{Back} \rbrace\rbrace$ to move the lifted object and the agents simultaneously in the same direction.
• $A^{MO} = \lbrace \text{MoveObject}X \mid X \in \lbrace \text{Ahead}, \text{Right}, \text{Left}, \text{Back} \rbrace\rbrace$ used to move the lifted object while the agents stay in place.
• $A^{RO} = \lbrace \text{RotateObjectRight} \rbrace$ to rotate the lifted object clockwise.

So the two agents, together, have joint action space $13 \times 13 = 169$ actions. The coordination of this action space is defined by the following coordination matrix:

We assume that all movement actions for agents and the lifted object result in a displacement of $0.25$ meters and all rotation actions result in a rotation of $90$ degrees counter-clockwise when viewing the agents from above.

Winners of the challenge will have the opportunity to present their work at the CVPR 2022 Embodied AI Workshop.

More details will be updated on March 1st, 2022.

To cite this work, please cite our papers on multi-agent furniture moving:

BibTeX

Copy

@InProceedings{CordialSync,
  author = {Jain, Unnat and Weihs, Luca and Kolve, Eric and Farhadi, Ali and Lazebnik, Svetlana and Kembhavi, Aniruddha and Schwing, Alexander G.},
  title = {A Cordial Sync: Going Beyond Marginal Policies For Multi-Agent Embodied Tasks},
  booktitle = {ECCV},
  year = {2020},
  note = {first two authors contributed equally},
}

@InProceedings{TwoBody,
  author = {Jain, Unnat and Weihs, Luca and Kolve, Eric and Rastegari, Mohammad and Lazebnik, Svetlana and Farhadi, Ali and Schwing, Alexander G. and Kembhavi, Aniruddha},
  title = {Two Body Problem: Collaborative Visual Task Completion},
  booktitle = {CVPR},
  year = {2019},
  note = {first two authors contributed equally},
}