Building path-instruction pairs from BnB dataset

Though navigation instructions are rarely found on the Internet, image-caption pairs from home environments are abundant in online marketplaces (e.g. Airbnb), which include images and descriptions of rental listings.

We collect BnB, a new large-scale dataset with 1.4M indoor images and 0.7M captions. First, we show that in-domain image-caption pairs bring additional benefits for downstream VLN tasks when applied with generic web data [17]. In order to further reduce the domain gap between the BnB pretraining and the VLN task, we present an approach to transform static image-caption pairs into visual paths and navigation-like instructions, leading to large additional performance gains.

Airbert: A Pretrained VLN Model

Our pretrained model, Airbert, is a generic transformer backbone that can be readily integrated in both discriminative VLN tasks such as path-instruction compatibility prediction [17] and generative VLN tasks [18] in R2R navigation [2] and REVERIE remote referring expression [19]. We achieve state-of-the-art performance on these VLN tasks with our pretrained model. Beyond the standard evaluation, our in-domain pretraining opens an exciting new direction of one/few-shot VLN where the agent is trained on examples only from one/few environment(s) and expected to generalize to other unseen environments.

We also propose a shuffling loss that improves the model’s temporal reasoning abilities by learning a temporal alignment between a path and the corresponding instruction.

Results

We evaluate Airbert on the test set by submitting our best method to the R2R leaderboard. As seen on the following Table, our method achieves the highest success rate at 77% and is ranked first as of the submission deadline.