FurnMove Challenge 2022

Held in conjunction with the CVPR 2022 Embodied AI Workshop

Example of agents solving the FurnMove Task.

Welcome to the 2022 AI2-THOR Furniture Moving (FurnMove) Challenge hosted at the CVPR 2022 Embodied AI Workshop. The goal of this challenge is to develop collaborative embodied agents. Particularly, two agents need to work together to move a piece of furniture through a living room to a goal. We are interested in the more realistic decentralized setting enabled via low-bandwidth communication.

We'll be updating more details on March 1st, 2022. So, stay tuned for updates!

Given only their egocentric visual observations, agents jointly hold a lifted piece of furniture in a living room scene and must collaborate to move it to a visually distinct goal location. As a piece of furniture cannot be moved without both agents agreeing on the direction, agents must explicitly coordinate at every timestep.

In FurnMove, each agent at every timestep receives an egocentric observation (a 3×84×843\times 84 \times 84 RGB image) from AI2-THOR. In addition, agents are allowed to communicate with other agents at each timestep via a low bandwidth communication channel. Based on their local observation and communication, each agent must take an action from the set AA.

At test time each agent must take only a egocentric observation (a 3×84×843\times 84 \times 84 RGB image) from AI2-THOR.

Do not exploit the metadata in test-scenes: You cannot use additional depth, mask, metadata info etc. from the simulator on test scenes. However, during training you are free to use additional info for things like auxiliary losses.

If you use additional sensory information from AI2-THOR as input (e.g., depth, segmentation masks, class masks, panoramic images) during test-time, your entry will not be considered. For official consideration to the CVPR 2022 challenge, agents should just use RGB input.

Each agent can take has an action space defined by A=ANAVAMWOAMOAROA = A^{NAV} ∪ A^{MWO} ∪ A^{MO} ∪ A^{RO}, where:

  • ANAV={MoveAhead,RotateLeft,RotateRight,Pass}A^{NAV} = \lbrace \text{MoveAhead}, \text{RotateLeft}, \text{RotateRight}, \text{Pass}\rbrace used to independently move each agent.
  • AMWO={MoveWithObjectXX{Ahead,Right,Left,Back}}A^{MWO} = \lbrace \text{MoveWithObject}X \mid X \in \lbrace \text{Ahead}, \text{Right}, \text{Left}, \text{Back} \rbrace\rbrace to move the lifted object and the agents simultaneously in the same direction.
  • AMO={MoveObjectXX{Ahead,Right,Left,Back}}A^{MO} = \lbrace \text{MoveObject}X \mid X \in \lbrace \text{Ahead}, \text{Right}, \text{Left}, \text{Back} \rbrace\rbrace used to move the lifted object while the agents stay in place.
  • ARO={RotateObjectRight}A^{RO} = \lbrace \text{RotateObjectRight} \rbrace to rotate the lifted object clockwise.

So the two agents, together, have joint action space 13×13=16913 \times 13 = 169 actions. The coordination of this action space is defined by the following coordination matrix:

We assume that all movement actions for agents and the lifted object result in a displacement of 0.250.25 meters and all rotation actions result in a rotation of 9090 degrees counter-clockwise when viewing the agents from above.

FurnMove Challenge Announced
Feb 14, 2022
Challenge Code and Data Release
March 1, 2022
Submissions Close
June 3, 2022
Winner Announcement
Jun 19, 2022

More details will be updated on March 1st, 2022.

To cite this work, please cite our papers on multi-agent furniture moving:

  author = {Jain, Unnat and Weihs, Luca and Kolve, Eric and Farhadi, Ali and Lazebnik, Svetlana and Kembhavi, Aniruddha and Schwing, Alexander G.},
  title = {A Cordial Sync: Going Beyond Marginal Policies For Multi-Agent Embodied Tasks},
  booktitle = {ECCV},
  year = {2020},
  note = {first two authors contributed equally},

  author = {Jain, Unnat and Weihs, Luca and Kolve, Eric and Rastegari, Mohammad and Lazebnik, Svetlana and Farhadi, Ali and Schwing, Alexander G. and Kembhavi, Aniruddha},
  title = {Two Body Problem: Collaborative Visual Task Completion},
  booktitle = {CVPR},
  year = {2019},
  note = {first two authors contributed equally},
The FurnMove challenge organizers are listed below:
Unnat Jain
Ani Kembhavi
Alexander Schwing
Luca Weihs