FLARE: Retrieve for the Sentence the Model Is About to Write

Why one initial search can be insufficient

A long answer may begin with a definition, continue with an example and finish with a historical comparison. The original query may not name the entities introduced later. Retrieving once can therefore provide adequate context for the opening while leaving later claims unsupported.

FLARE, introduced in Active Retrieval Augmented Generation, explores forward-looking retrieval based on generated upcoming text. Its confidence-based variant anticipates the next sentence, identifies low-confidence content, retrieves using the anticipated information need and regenerates with evidence.

This differs from searching only after a completed answer has been written. The temporary continuation is part of deciding what information to obtain next.

An anticipated sentence is a query proposal

Suppose a draft continuation says “The system was introduced at the 2018 robotics conference.” If the event name and year are uncertain, that continuation can suggest a focused search about the system’s introduction.

The retrieved documents may establish 2019 instead, so the sentence should be regenerated. The original draft must not be retained as independent evidence or treated as a fact to preserve at all costs.

A practical trace stores the provisional sentence, retrieval trigger, actual query, returned passages and final sentence separately. This makes it possible to inspect whether the new evidence corrected or merely reinforced the provisional guess.

Confidence is a trigger, not a truth score

One illustrative uncertainty rule triggers retrieval if any content token’s probability falls below a threshold \(\tau\). Other aggregation choices use a proportion of uncertain tokens or a sequence score. These rules can behave differently on long sentences and rare names.

For token probabilities (0.9, 0.8, 0.15, 0.7), a minimum-probability threshold of 0.2 triggers retrieval because of the third token. But a familiar false date can receive high probability, while a correct uncommon name can receive low probability. Language-model likelihood measures predictive familiarity, not factual correctness.

The paper’s masking and query-construction details matter because including an uncertain invented entity can steer search toward that invention. State which FLARE variant and thresholds are being reproduced.

Regeneration needs a stopping rule

After retrieval, the model may still be uncertain. Unlimited retrieve-and-regenerate loops can repeat queries or accumulate irrelevant context. Define a retrieval budget, a maximum number of attempts and an outcome when sufficient evidence remains unavailable.

A sentence that repeatedly changes its date after each search is a useful diagnostic. Inspect source versions and contradictions rather than assuming one more generation will resolve them. More calls can increase exposure to noise as well as to evidence.

Evaluate the value of the trigger

Compare initial-query-only retrieval, periodic retrieval and confidence-triggered retrieval under a matched call or token budget. Measure how often a trigger obtains useful evidence and how often regeneration improves support.

Separate four cases: uncertain and wrong, uncertain and correct, confident and wrong, confident and correct. A good trigger should direct effort toward errors without wasting all of its budget on harmless rare wording. Calibration and domain shifts can change that balance.

Connect to the other adaptive methods

Self-RAG learns explicit retrieval and critique signals. IRCoT uses intermediate reasoning to guide subsequent evidence acquisition. FLARE emphasises the upcoming sentence and uncertainty in its generation.

These methods can be compared through when they search, what query they issue and what they do after retrieval. Avoid treating their different control signals as interchangeable implementations of the same algorithm. A clean experiment holds the search corpus and base generator fixed while varying that control loop.

References

1. Jiang et al. Active Retrieval Augmented Generation. EMNLP, 2023.
2. Asai et al. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. ICLR, 2024.
3. Trivedi et al. Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions. ACL, 2023.