Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

Fusion-in-Decoder and Atlas: Reading Many Passages and Learning Retrieval

4 minute read

Published:

TL;DR: FiD encodes question–passage pairs independently and lets a decoder attend across their representations. Atlas combines a retrieval-augmented reader with retrieval-aware training for few-shot learning. Multi-passage access creates an opportunity to combine evidence, not a guarantee that the model will do so correctly.

FiD changes where passages meet

Fusion-in-Decoder uses an encoder–decoder architecture. Each retrieved passage is paired with the question and encoded separately. The resulting token representations are concatenated for decoder cross-attention, allowing generation to draw on several passages.

This avoids full encoder self-attention across all passages. It also differs from the original RAG mixture, where separate document-conditioned answer distributions are marginalised. FiD gives one decoder direct access to multiple encoded sources within its computation.

Compare the attention costs

If there are \(K\) passages of length \(L\), separately encoding them has an encoder self-attention term proportional to \(KL^2\). Jointly encoding the entire concatenated input has a corresponding term proportional to \((KL)^2\). These expressions isolate the attention component, not all runtime or memory costs.

For ten passages of 100 tokens, the pair-count scale is 100,000 for separate encoding versus 1,000,000 for full concatenation. FiD’s decoder must still attend to about 1,000 encoder positions per output step. Independent encoding does not make the evidence free.

The repeated question and passage formatting also consume tokens. Report actual sequence lengths and memory use when comparing architectures, rather than treating this simplified ratio as a measured speedup.

Two passages can supply complementary facts

Suppose one passage identifies a product’s manufacturer, while another gives that manufacturer’s headquarters. A decoder with access to both can answer the composed question. But it can also conflate similarly named manufacturers or use an unrelated location.

Evaluate the two pieces of evidence separately and together. If the answer remains unchanged when either supporting passage is removed, inspect whether the model uses parametric knowledge or a shortcut. Attention weights alone do not establish which passage causally supported a claim.

Atlas adds retrieval-aware learning

Atlas builds on a retrieval-augmented encoder–decoder setup and studies pretraining and few-shot adaptation with a learned retriever. It investigates ways to transfer the reader’s assessment of passage usefulness into retriever training, rather than relying only on a fixed external relevance model.

One general distillation pattern compares a reader-derived distribution over passages with retriever probabilities:

\[ \mathcal L_{\mathrm{ret}}=D_{\mathrm{KL}}(p_{\mathrm{teacher}}(d\mid q,y)\Vert p_\eta(d\mid q)). \]

This equation illustrates the learning relationship; Atlas evaluates several objectives and update choices, so a reproduction must identify the exact one. Reader likelihood and attention-based scores are not interchangeable definitions of a teacher distribution.

A toy teacher signal

For three passages, suppose the retriever assigns (0.6, 0.3, 0.1) while a normalised reader-derived target is (0.1, 0.2, 0.7). The distillation loss encourages more mass on the third passage. Whether that change improves factual support depends on why the reader prefers it.

A passage that repeats an answer string can improve likelihood without providing a valid explanation. Pair retriever-training diagnostics with supporting-evidence labels and examples containing misleading answer mentions.

What to measure in a multi-passage system

Vary the passage count under a fixed or explicitly reported context budget. Measure sufficient-evidence coverage, redundant context, latency, reader loss and claim support. Include source-order and distractor tests, especially when several documents discuss different versions of the same subject.

Compare a frozen retriever with an adapted retriever while holding the reader fixed where possible. Then compare reader adaptation on the same retrieved sets. This separates gains from better evidence access and gains from better evidence use.

For the long-input alternative, see retrieval versus long context. A larger context window and FiD’s multi-passage interface solve different computational problems even when they expose the same source material.

Key Insight: FiD provides a multi-source reading architecture; Atlas studies learning to make retrieval useful for that kind of reader. Access, fusion and training are separate design choices.
Common pitfall: More passages can add contradictory or redundant evidence. Report the passage budget and test whether newly retrieved material actually improves supported answering.

Recap

  • FiD fuses separately encoded passages in decoder cross-attention.
  • Independent encoding reduces one attention cost while leaving decoder reading costs.
  • Atlas connects retrieval-aware training with few-shot adaptation.
  • Reader-derived retriever targets still require evidence-quality checks.

References

  1. Izacard and Grave. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. EACL, 2021.
  2. Izacard et al. Atlas: Few-shot Learning with Retrieval Augmented Language Models. Preprint, 2022; JMLR, 2023.
  3. Lewis et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS, 2020.

You May Also Enjoy

The Standard Library Modules Worth Knowing Before You pip install

8 minute read

Published:

Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed, most of what people reach for a package to solve is one import away.

Files and Context Managers: Why with Is Not Optional

9 minute read

Published:

A file object that falls out of scope without with does get closed eventually, but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.

Errors and Exceptions: EAFP, the Hierarchy, and Reading a Traceback

9 minute read

Published:

Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first, which is faster, shorter, and free of race conditions.