REPLUG: Retrieval for a Frozen Black-Box Language Model

Retrieval without changing model weights

Many applications can query a language model but cannot modify its architecture or inspect its parameters. REPLUG studies how retrieval can improve such a model by formatting retrieved documents as inputs and ensembling their conditional predictions.

The paper also introduces a language-model-supervised retrieval variant. A document that helps the frozen model predict training text can provide a learning signal for the retriever. This differs from REALM, where retrieval is integrated into pretraining, and from merely reranking documents by a generic relevance classifier.

Mix predictions, not just strings

For retrieved documents \(d_i\) with normalised weights \(w_i\), a document-conditioned next-token ensemble can be written

The same generated prefix is supplied to each branch. This is not equivalent to asking each branch for a full answer and majority-voting the strings, nor to concatenating all documents into one prompt.

Suppose two branches have weights 0.7 and 0.3. For a candidate token, their probabilities are 0.1 and 0.9. The mixture gives \(0.7(0.1)+0.3(0.9)=0.34\). Selecting only the top-retrieved document would give 0.1. Ensembling preserves uncertainty about which source context is useful.

What “black box” must expose

A text-only interface that returns a completed string does not necessarily expose the probabilities needed for exact token-level ensembling. Retriever training based on likelihood also needs access to a suitable scoring signal.

Before claiming a reproduction, identify whether the interface supports token probabilities, scoring supplied target text and coordinated decoding across branches. A simpler retrieve-and-prompt approximation can still be useful, but it should be named as an approximation rather than silently equated with the paper’s computation.

Reader feedback can train retrieval

A general reader-guided approach computes a target distribution from document-conditioned likelihoods and trains retrieval probabilities toward it. For example,

This schematic highlights the role of the reader; an implementation must follow the paper’s precise normalisation, temperatures and sampling choices. The retriever can be updated without differentiating through the frozen language model.

A document that improves answer likelihood might still exploit a shortcut. Compare with evidence-based relevance judgements and inspect whether the gain survives misleading answer-string passages.

Cost grows with document branches

With \(K\) separately evaluated document contexts, generation can require more model work than one combined prompt. Shared-prefix caching or batched execution may reduce costs, but the number and length of branches still matter.

Report model calls, input and output tokens, latency and retrieval cost. A comparison against a single-call baseline at an unspecified budget can confuse additional computation with a better retrieval strategy.

When the method is a useful reference

REPLUG is particularly useful conceptually when the generator is fixed but retrieval can be learned or changed. It demonstrates that adaptation need not always mean changing the generator’s weights.

For a practical study, keep the model version and corpus fixed, compare no retrieval, prompt concatenation and document ensembling, then evaluate a generic retriever against the reader-guided one. Use the same questions and report both answer quality and source support. This separates the architectural benefit of ensembling from the training benefit of a better retriever.

References

1. Shi et al. REPLUG: Retrieval-Augmented Black-Box Language Models. NAACL, 2024.
2. Lewis et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS, 2020.
3. Izacard et al. Atlas: Few-shot Learning with Retrieval Augmented Language Models. Preprint, 2022; JMLR, 2023.