1. Introduction

In vision-and-language navigation (VLN), an agent is asked to navigate in home environments following natural language instructions [1], [2]. This task is attractive to many real-world applications such as domestic robotics and personal assistants. However, given the high diversity of VLN data across environments and the difficulty of the manual collection and annotation of VLN training data at scale, the performance of current methods remains limited, especially for previously unseen environments [3].

Our work is motivated by significant improvements in vision and language pretraining [4]–[9], where deep transformer models [10] are trained via self-supervised proxy tasks [11] using large-scale, automatically harvested image-text datasets [12], [13]. Such pretraining enables learning transferable multi-modal representations achieving state-of-the-art performance in various vision and language tasks. Similarly, with the goal of learning an embodied agent that generalizes, recent works [14]–[17] have explored different pretraining approaches for VLN tasks.

In [14], [15], annotated path-instruction pairs are augmented with a speaker model that generates instructions for random unseen paths. However, as these paths originate from a small set of 61 houses used during training, they are limited in visual diversity. The limited pretraining environments do not equip agents with visual understanding abilities that enable generalization to unseen houses, see Fig. 1. To address this problem, VLN-BERT [17] proposes to pretrain the agent on generic image-caption datasets that are abundant and cover diverse visio-linguistic knowledge. However, these image-caption pairs are quite different from the dynamic visual stream (path) and navigable instructions observed by a VLN agent, and such out-of-domain pretraining, although promising, only brings limited gains to the navigation performance. Besides the above limitations, existing pretraining methods do not place much emphasis on temporal reasoning abilities in their self-supervised proxy tasks such as one-step action prediction [14] and path-instruction pairing [17], while such reasoning is important to a sequential decision making task like VLN. As a result, even if performance in downstream tasks is improved, the pretrained models may still be brittle. For example, a simple corruption of instructions by swapping noun phrases within the instruction, or replacing them with other nouns, leads to significant confusion as models are unable to pick the correct original pair.

In this paper, we explore a different data source and proxy tasks to address the above limitations in pretraining a generic VLN agent. Though navigation instructions are rarely found on the Internet, image-caption pairs from home environments are abundant in online marketplaces (e.g.. Airbnb), which include images and descriptions of rental listings. We collect BnB, a new large-scale dataset with 1.4M indoor images and 0.7M captions. First, we show that in-domain image-caption pairs bring additional benefits for downstream VLN tasks when applied with generic web data [17]. In order to further reduce the domain gap between the BnB pretraining and the VLN task, we present an approach to transform static image-caption pairs into visual paths and navigation-like instructions (Fig. 1 bottom), leading to large additional performance gains. We also propose a shuffling loss that improves the model’s temporal reasoning abilities by learning a temporal alignment between a path and the corresponding instruction.

Our pretrained model, Airbert, is a generic transformer backbone that can be readily integrated in both discriminative VLN tasks such as path-instruction compatibility prediction [17] and generative VLN tasks [18] in R2R navigation [2] and REVERIE remote referring expression [19]. We achieve state-of-the-art performance on these VLN tasks with our pretrained model. Beyond the standard evaluation, our in-domain pretraining opens an exciting new direction of one/few-shot VLN where the agent is trained on examples only from one/few environment(s) and expected to generalize to other unseen environments.

In summary, the contributions of this work are three-fold. (1) We collect a new large-scale in-domain dataset, BnB, to promote pretraining for vision-and-language navigation tasks. (2) We curate the dataset in different ways to reduce the distribution shift between pretraining and VLN and also propose the shuffling loss to improve temporal reasoning abilities. (3) Our pretrained Airbert can be plugged into generative or discriminative architectures and achieves state-of-the-art performance on R2R and REVERIE datasets. Moreover, our model generalizes well under a challenging one/few-shot VLN evaluation, truly highlighting the capabilities of our learning paradigm. We will release the code, model, and data.

2. Related Work

Vision-and-language navigation. VLN [2] has received significant attention with a large number of followup tasks introduced in recent years [1], [19]–[26]. Early days of VLN saw the use of sequence-to-sequence LSTMs to predict low-level actions [2] or high-level directions in a panoramic action space [27]. For better cross-modal alignment, a visio-linguistic co-grounding attention mechanism is proposed in [28], and instructions are further disentangled into objects and directions in [29]. To alleviate exposure bias in supervised training of the agent, reinforcement learning has been adopted through planning [30], REINFORCE [31], A2C [32] and reward learning [33]. A few works also explore different search algorithms such as backtracking by monitoring progress [28], [34] or beam search [27], [32], [35] in environment exploration.

To improve an agent’s generalization to unseen environments, data augmentation is performed by using a speaker model [27] that generates instructions for random paths in seen environments, and environment dropout [32] is used to mimic new environments. While pretraining LSTMs to learn vision and language representations is adopted by [15], recently, there has been a shift towards transformer models [14] to learn generic multimodal representations. This is further extended to a recurrent model that significantly improves sequential action prediction [18]. However, the limited environments in pretraining [14], [15] constrain the generalization ability to unseen scenarios. Most related to this work, VLN-BERT [17] transfers knowledge from abundant, but out-of-domain image-text data to improve path-instruction matching. In this work, we not only create a large-scale, in-domain BnB dataset to improve visual diversity, but also propose effective pretraining strategies to mitigate the domain-shift between webly crawled image-text pairs and VLN data.

Large-scale visio-linguistic pretraining. Thanks to large-scale vision-language pairs automatically collected from the web [12], [13], [36], [37], visio-linguistic pretraining (VLP) has made great breakthroughs in recent years towards learning transferable multimodal representations. Several VLP models [5]–[7], [38] have been proposed based on the transformer architecture [10]. These models are often pretrained with self-supervised objectives akin to those in BERT [11]: masked language modeling, masked region modeling and vision-text pairing. Fine-tuning them on downstream datasets achieves state-of-the-art performance on various VL tasks [39]–[42]. While such pretraining focuses on learning correlations between vision and text, it is not designed for sequential decision making as required in embodied VLN. The goal of this work is not to improve VLP architectures but to present in-domain training strategies that lead to performance improvements for VLN tasks.

3. BnB Dataset

Hosts that rent places on online marketplaces often upload attractive and unique photos along with descriptions. One such marketplace, Airbnb, has 5.6M listings from over 100K cities all around the world [43]. We propose to use this abundant and curated data for large-scale in-domain VLN pretraining. In this section, we first describe how we collect image-caption pairs from Airbnb. Then, we propose methods to transform images and captions into VLN-like path-instruction pairs to reduce the domain gap between webly crawled image-text pairs and VLN tasks (see Fig. 2).

3.1. Collecting BnB Image-Caption Pairs

Collection process. We restrict our dataset to listings from the US (about 10% of Airbnb) to ensure high quality English captions and visual similarity with Matterport environments [44]. The data collection proceeds as follows: (1) obtain a list of locations from Wikipedia; (2) find listings in these locations by querying the Airbnb search engine; (3) download listings and their metadata; (4) remove outdoors images³ as classified by a ResNet model pretrained on Places365 [45]; and (5) remove invalid image captions such as emails, URLs and duplicates.

Statistics. We downloaded almost 150k listings and their metadata (1/4 of the listings in the US) in step 3, leading to over 3M images and 1M captions. After data cleaning with steps 4 and 5, we obtain 713K image-caption pairs and 676K images without captions. Table 2 compares our BnB dataset to other datasets used in previous works for VLN (pre-)training. It is larger than R2R [2], REVERIE [19] and includes a large diversity of rooms and objects, which is not the case for Conceptual Captions [13]. We posit that such in-domain data is crucial to deal with the data scarcity challenge in VLN environments as illustrated above. We use 95% of our BnB dataset for training and the remaining 5% for validation.

Apart from images and captions, our collected listings contain structured data including a list of amenities, a general description, reviews, location, and rental price, which may offer additional applications in the future. More details about the dataset and examples are presented in the supplementary material.

3.2. Creating BnB Path-Instruction Pairs

BnB image-caption (IC) pairs are complementary to Conceptual Captions (ConCaps) as they capture diverse VLN environments. However, they still have large differences from path-instruction (PI) pairs in VLN tasks. For example, during navigation, an agent observes multiple panoramic views of a sequence of locations rather than a single image, and the instruction may contain multiple sentences describing different locations along the way. To mitigate this domain gap, we propose strategies to automatically craft path-instruction pairs starting from BnB-IC pairs.

Building path-instruction pairs

Images in a BnB listing usually depict different locations in a house, mimicking the sequential visual observations an agent makes while navigating in the house. To create a VLN-like path-instruction pair, we randomly select and concatenate K⁴ image-caption pairs from the listing . In between each caption, we randomly add a word from “and”, “then”, “.” or nothing to make the concatenated instruction more fluent and diverse.

Augmenting Paths with Visual Contexts

In the above concatenated path, each location only contains one BnB image, and perhaps with a limited view angle as hosts may focus on objects or amenities they wish to highlight. Therefore, it lacks the panoramic visual context at each location that the agent receives in real navigation paths. Moreover, each location in the concatenated instruction is described by a unique sentence, while adjacent locations are often expressed together in one sentence in VLN instructions [46]. To address the above issues with concatenation, we propose two approaches to compose paths that have more visual context and can also leverage the abundant images without captions (denoted as captionless images).

Image merging extends the panoramic context of a location by grouping images from similar room categories (see Fig. 2). For example, if the image depicts a kitchen sink, it is natural to expect images of other objects such as forks and knives nearby. Specifically, we first cluster images of similar categories (e.g.. kitchen) using room labels predicted by a pretrained Places365 model [45]. Then, we extract multiple regions from this merged set of images, and use them as an approximation to the panoramic visual representation.

Captionless image insertion. The Table 1 shows that half of the BnB images are captionless. Using them allows to increase the size of the dataset. When creating a path-instruction pair from the concatenation approach, a captionless image is inserted as if its caption was an empty string. The BnB PI pairs hence generated better approximate the distribution of the R2R path-instructions: (1) some images in the path are not described and (2) instructions have similar number of noun phrases.

Crafting Instructions with Fluent Transitions

The concatenated captions mainly describe rooms or objects at different locations, but do not contain any of the actionable verbs as in navigation instructions, e.g.. “turn left at the door” or “walk straight down the corridor”. We suggest two strategies to create fake instructions that have fluent transitions between sentences.

Instruction rephrasing. We use a fill-in-the-blanks approach to replace noun-phrases in human annotated navigation instructions [2] by those in BnB captions (see Fig. 2). Concretely, we create more than 10K instruction templates containing 2-7 blanks, and fill the blanks with noun-phrases extracted from BnB captions. The noun-phrases matched to object categories from the Visual Genome [47] dataset are preferred during selection. This allows us to create VLN-like instructions with actionable verbs interspersed with room and object references for visual cues that are part of the BnB path (see Fig. 2).

Instruction rephrasing. It is a video captioning like model that takes in a sequence of images and generates an instruction corresponding to an agent’s path through an environment. To train this model, we adopt ViLBERT and train it to generate captions for single BnB image-caption pairs. Further, this model is fine-tuned on trajectories of the R2R dataset to generate corresponding instructions. Finally, we use this model to generate BnB PI pairs by producing an instruction for a concatenated image sequence from BnB (the path).

4. Airbert: A Pretrained VLN Model

In this section, we present Airbert, our multi-modal transformer pretrained on the BnB dataset with masking and shuffling losses. We first introduce the architecture of Airbert, and then describe datasets and pretext tasks in pretraining. Finally, we show how Airbert can be adapted to downstream VLN tasks.

4.1. ViLBERT-like Architecture

ViLBERT [7] is a multi-modal transformer extended from BERT [11] to learn joint visio-linguistic representations from image-text pairs, as illustrated in Fig. 3.

Given an image-text pair (V, C), the model encodes the image as region features [v₁, …, v_𝒱] via a pretrained Faster R-CNN [48], and embeds the text as a series of tokens: [[CLS], w₁, …, w_T, [SEP]], where [CLS]and [SEP]are special tokens added to the text. ViLBERT contains two separate transformers that encode V and C and it learns cross-modal interactions via co-attention [7].

We follow a similar strategy to encode path-instruction pairs (created in Sec. 3.1) that contain multiple images and captions {(V_k, C_k)}_k = 1^K. Here, each V_k is represented as visual regions v_i^k and C_k as word tokens w_t^k. Respectively, the visual and text inputs to Airbert are:
$$\begin{aligned} X_V &= [\texttt{[IMG]}, v^1_1, \ldots, v^1_{\mathcal{V}_1}, \ldots, \texttt{[IMG]}, v^K_1, \ldots, v^K_{\mathcal{V}_K}], \\ X_C &= [\texttt{[CLS]}, w^1_1, \ldots, w^1_{T_1}, \ldots, w^K_1, \ldots, w^K_{T_K}, \texttt{[SEP]}] ,\end{aligned}$$
where the [IMG] token is used to separate image region features taken at different locations.

Note that while our approach is not limited to a ViLBERT-like architecture, we choose ViLBERT for a fair comparison with previous work [15].

4.2. Datasets and Pretext Tasks for Pretraining

We use Conceptual Captions (ConCaps) [37] and BnB-PI in subsequent pretraining steps (see Fig. [fig:model]) to reduce the domain gap for downstream VLN tasks.

Previous multi-modal pretraining efforts [7], [15], [17] commonly use two self-supervised losses given image-caption (IC) pairs or path-instruction (PI) pairs: (1) Masking loss: An input image region or word is randomly replaced by a [MASK] token. The output feature of this masked token is trained to predict the region label or the word given its multi-modal context. (2) Pairing loss: Given the output features of [IMG]and [CLS] tokens, a binary classifier is trained to predict whether the image (path) and caption (instruction) are paired.

The above two pretext tasks mainly focus on learning object-word associations instead of reasoning about the temporal order of paths and instructions. For example, if an image V_i appears before V_j, then words from its caption C_i should appear before C_j. In order to promote such a temporal reasoning ability, we propose an additional shuffling loss to enforce alignment between PI pairs.

Given an aligned PI pair X⁺ = {(V_k, C_k)}_k = 1^K, we generate 𝒩 negative pairs X_n⁻ = {(V_k, C_l)}, k ≠ l, by shuffling the composed images or the captions. We train our model to choose the aligned PI pair as compared to the shuffled negatives by minimizing the cross-entropy loss:
$$L = -\log \frac{\exp(f(X^+))}{\exp(f(X^+)) + \sum_n \exp(f(X^-_n))} \, ,$$
where f(X) denotes the similarity score (logit) computed via Airbert for some PI pair X.

4.3. Adaptations for Downstream VLN tasks

We consider two VLN tasks: goal-oriented navigation (R2R [2]) and object-oriented navigation (REVERIE [19]). Airbert can be readily integrated in discriminative and generative models for the above VLN tasks.

The navigation problem on the R2R dataset is formulated as a path selection task in [15]. Several candidate paths are generated via beam search from a navigation agent such as [32], and a discriminative model is trained to choose the best path among them. We fine-tune Airbert on the R2R dataset for path selection. A two-stage fine-tuning process is adopted: in the first phase, we use masking and shuffling losses on the PI pairs of the target VLN dataset in a manner similar to BnB PI pairs; in the second phase, we choose a positive candidate path as one that arrives within 3m of the goal, and contrast it against 3 negative candidate paths. We also compare multiple strategies to mine additional negative pairs (other than the 3 negative candidates), and in fact, empirically show that negatives created using shuffling outperform other options.

Generative Model: Recurrent VLN-BERT [18]. The Recurrent VLN-BERT model adds recurrence to a state in the transformer to sequentially predict actions, achieving state-of-the-art performance on R2R and REVERIE tasks. We use our Airbert architecture as its backbone and apply it to the two tasks as follows. First, the language transformer encodes the instruction via self-attention. Then, the embedded [CLS] token in the instruction is used to track history and concatenated with visual tokens (observable navigable views or objects) in each action step. Self-attention and cross-attention on embedded instructions are employed to update the state and visual tokens and the attention score from the state token to visual tokens is used to decide the action at each step. We fine-tune the Recurrent VLN-BERT model with Airbert as the backbone in the same way as [18].

Please refer to the supplementary material for additional details about the models and their implementation.

Experimental Results

We first perform ablation studies evaluating alternative ways to pretrain Airbert in Sec. 5.1. Then, we compare Airbert with state-of-the-art methods on R2R and REVERIE tasks in Sec. 5.2. Finally, in Sec. 5.3, we evaluate models in a more challenging setup: VLN few-shot learning where an agent is trained on examples taken from one/few houses.

R2R Setup. We briefly describe the two evaluation datasets used in our work: R2R [2] and REVERIE [19]. Most of our experiments are conducted on the R2R dataset [2], where we adopt standard splits and metrics defined by the task. We focus on success rate (SR), which is the ratio of predicted paths that stop within 3m of the goal. Please refer to [2], [17] for a more detailed explanation of the metrics. In particular, as the discriminative model uses path selection for R2R, we follow the pre-explored environment setting adopted by VLN-BERT [17], and compute metrics on the selected path.

REVERIE Setup. We also adopt standard splits and metrics on the REVERIE task [19]. Here, the success rate (SR) is the ratio of paths for which the agent stops at a viewpoint where the target object is visible. Remote Grounding Success Rate (RGS) measures accuracy of localizing the target object in the stopped viewpoint, and RGS per path length (RGSPL) is a path length weighted version.

5.1 Pretraining with BnB

We perform ablation studies on the impact of various methods for creating path-instruction pairs. We also present ablation studies that highlight the impact of using the shuffling loss during Airbert’s pretraining as well as fine-tuning stages. Throughout this section, our primary focus is on the SR on the unseen validation set and we compare our results against VLN-BERT [17], which achieves a SR of 59.26%.

1. Impact of creating path-instruction pairs. Table 2 presents the performance of multiple ways of using the BnB dataset after ConCaps pretraining as illustrated in Fig. 3. In row 1, we show that directly using BnB IC pairs without any strategies to reduce domain gap improves performance over VLN-BERT by 3.2%. Even if we skip ConCaps pretraining, we achieve 60.54% outperforming 59.26% of VLN-BERT. It proves that our BnB dataset is more beneficial to VLN than the generic ConCaps dataset.

Naive concatenation (row 2) does only slightly better than using the IC pairs (row 1) as there are still domain shifts with respect to fluency of transitions and lack of visual context. Rows 3-6 show that each method mitigates domain-shift to some extent. Instruction rephrasing (row 3) performs better at improving instructions than instruction generation (row 4), possibly since the generator is unable to use the diverse vocabulary of the BnB captions. Inserting captionless images at random locations (row 6) reduces the domain-shift significantly and achieves the highest individual performance. Finally, a combination of instruction rephrasing, image merging and captionless insertion provides an overall 3.8% improvement over concatenation, and a large 7.2% improvement over VLN-BERT.

2. Shuffling loss applied during pretraining. Table 3 demonstrates that shuffling is an effective strategy to train the model to reason about temporal order, and enforce alignment between PI pairs. Rows 2-4 show that shuffling is beneficial both during pretraining with BnB-PI data, or during fine-tuning with R2R data, and results in 2.3% and 0.4% improvements respectively. In combination with the Speaker dataset (paths from seen houses with generated instruction yielding 178K additional PI pairs [32]), we see that shuffling has a major role to play and provides 3.1% overall improvement (row 5 vs. 6). 68.67% is also our highest single-model performance on the R2R dataset.

3. Shuffling loss applied during fine-tuning. The final stage of model training on R2R involves fine-tuning to rank multiple candidate paths that form the path selection task. We compare various approaches to improve this fine-tuning procedure (results in Table 4). (1) In row 2, we explore the impact of using additional negative paths. Unsurprisingly, this does not improve performance. (2) Inspired by [49], we highlight keywords in the instruction using a part-of-speech tagger [50], and include an extra loss term that encourages the model to pay attention to their similarity scores (row 3). (3) Another alternative suggested by [49] involves masking keywords in the instruction and using VLP models to suggest replacements, resulting in hard negatives (row 4).

Hard negatives and highlighting keywords show good performance improvements, about 2.1-2.3%, but at the cost of extra parsers or VLP models. On the other hand, shuffling visual paths to create two additional negatives results in highest performance improvement (row 5, +2.7% on val unseen) and appears to be a strong strategy to enforce temporal order reasoning, that neither requires an external parser nor additional VLP models.

4. Error analysis. We study the areas in which Airbert brings major improvements by analyzing scores for aligned PI pairs and simple corruptions that involve replacing noun phrases (e.g.. bedroom by sofa), swapping noun phrases appearing within the instruction, or switching left and right directions (e.g.. turn left/right or leftmost/rightmost chair). In particular, for every ground-truth aligned PI pair, we create 10 additional negatives by corrupting the instruction, and measure how well the model is able to assign the highest score to the correct pair as accuracy. Table 3 shows that Airbert with in-domain training and the shuffling loss achieves large improvements (> 8%) for corruptions involving replacement or swapping of noun phrases. On the other hand, distinguishing directions continues to be a challenging problem; but here as well we see Airbert outperform VLN-BERT by 4.5%.

5.2. Comparison against state-of-the-art

R2R. We first evaluate the discriminative model for the R2R task. Similar to VLN-BERT, we evaluate Airbert as an ensemble model created by a linear combination (chosen through grid search) of multiple model outputs (see Table 6). First, we see that Airbert alone (row 2) outperforms VLN-BERT (row 1) by 9.4% on the unseen environments and a strong ensemble of speaker and follower models [32] (row 3) by 0.7%. Ensembling Airbert results in a gain of 1.4% over the VLN-BERT ensemble (row 4 vs. 5).

We also obtain results on the test set by submitting our best method to the R2R leaderboard.⁵ As seen from Table 9, our method of ensembling Airbert, speaker, and follower (similar to VLN-BERT with speaker and follower [11]) achieves the highest success rate at 77% and is ranked first as of the submission deadline. Airbert also benefits generative models for the R2R task. The results are presented in the supplementary material. In both VLN-BERT and Airbert, 30 candidate trajectories are sampled using beam search with the EnvDrop [32] approach, inducing the same path length (PL) for all three methods (687 in 8. The SPL metric on the leaderboard takes into account the total path length over the 30 trajectories. This explains why SPL is very low and similar across multiple approaches.

REVERIE. Table 7 presents results for the REVERIE dataset. The last four rows in the table use Recurrent VLN-BERT [18] with different backbones or parameter initialization. The OSCAR and ViLBERT backbones are pretrained on out-of-domain image-caption pairs. As compared to OSCAR, we observe slight improvements using the ViLBERT backbone for the REVERIE task. VLN-BERT shares the same architecture as ViLBERT, but is pretrained on the R2R dataset, resulting in performance improvement on the unseen environments. Our pretrained Airbert achieves significantly better performance than VLN-BERT, with over 2.4% gain on navigation SR and 1.8% gain on RGS in unseen environments (val unseen). Without any special adaptation, we see that Airbert brings benefits from pretraining on the BnB dataset. We also achieve the state-of-the-art performance on the REVERIE test set, surpassing previous works by a large margin.

5.3. Training a navigation agent on few houses

We hypothesize that in-domain pretraining, especially one that leverages proposed PI pair generation methods, can achieve superior performance while requiring less training data. To evaluate this, we propose a novel few shot evaluation paradigm for VLN: models are allowed to fine-tune on samples (PI pairs) from one (or few) environments. Few-shot learning for VLN is particularly interesting as visual appearance of houses may differ vastly across geographies, and while training data is hard to obtain, pretraining data like BnB may be readily available.

One/few shot tasks. We considered two types of setups: (1) learning from a single environment, which we refer as one-shot learning; and (2) learning from 6 environments (representing 10% of the total training size). For both cases, we randomly sample 5 sets of environments, and report average results (standard deviations in the supplementary material). As the number of paths in an environment may have a large impact on performance, we exclude 17 of 61 environments with less than 80 paths.

Results. We adopt VLN-BERT, pretrained on ConCaps, as a baseline for few-shot tasks. Recall that fine-tuning VLN-BERT and Airbert on R2R relies on candidate paths drawn from an existing model (EnvDrop [32]). However, as this would lead to unfair comparisons (EnvDrop is trained on the full dataset), we sample candidate paths by random walks from the starting position in the environment.

Table 9 shows that Airbert outperforms VLN-BERT by very large margins on the unseen validation set: 22.4% with 1 house and 21% with 6 houses. In fact, Airbert fine-tuned on 6 houses is almost as good as VLN-BERT on the entire training set. Interestingly, as seen in the last two rows of the table, using random paths for fine-tuning does not lead to a large performance drop for both models and is a testament to the power of pretrained networks.

6. Conclusion

We introduced BnB, a large-scale, in-domain, image-text dataset from houses listed on online rental marketplaces and showed how domain gaps between BnB image-caption pairs and VLN tasks can be mitigated through the creation of path-instruction pairs. We also proposed shuffling, as a means to improve an agent’s reasoning about temporal order. Our pretrained model Airbert, achieved state-of-the-art on R2R through the discriminative path-selection setting, and REVERIE through a generative setting. We also demonstrated large performance improvements when applying our model to a challenging one/few-shot VLN setup, highlighting the impact of good pretraining in VLN tasks.