Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

Question Answering: Retrieval, Span Extraction and Answerability

4 minute read

Published:

TL;DR: Question answering combines evidence retrieval with an answer decision. Extractive systems choose a span; generative systems produce text; both need an answerability policy. Evaluate retrieval, reading and complete evidence-supported answering separately.

Specify the information boundary

A reading-comprehension task supplies a passage and asks for an answer from it. Open-domain QA must retrieve evidence from a collection. Closed-book QA answers from model parameters without a supplied source. These settings test different capabilities and have different failure modes.

For a corpus-grounded assistant, a correct answer should be supported by the accessible evidence under a stated time and domain scope. A fact remembered from elsewhere may be true yet still fail a task requiring an answer from a particular document.

A QA pipeline separates evidence access from the decision to answer.Question|and scope → Retrieve|passages → Read and|check support → Answer|or abstain Question and scope → Retrieve passages → Read and check support → Answer or abstain
A QA pipeline separates evidence access from the decision to answer.

Extract a span

An extractive reader can produce start logits \(s_i\) and end logits \(e_j\) over passage tokens. A simple decoder chooses

\[ (i^*,j^*)=\arg\max_{i\leq j,\ j-i+1\leq L}\left(s_i+e_j\right). \]

The ordering and maximum-length constraints prevent invalid spans. Choosing the highest start and end independently can produce an end before the start. A joint span model can represent further dependencies; the factorised score above is a modelling simplification.

For passage “Rossi joined the lab in 2021”, a start distribution peaking at 2021 is helpful for “When did Rossi join?” but irrelevant to “Why did Rossi join?” The latter has no answer in the passage. The existence of a high-scoring span does not establish answerability.

A constrained example

Suppose start scores for three positions are 3, 1 and 0, while end scores are 0, 2 and 4. With maximum span length two, span [1,3] in one-based indexing is illegal even though its unconstrained score would be seven. The best legal spans [1,2] and [2,3] both score five. Define tie handling or retain uncertainty.

This example shows why decoder constraints must be applied before interpreting the output. A model that was trained on legal spans but decoded without the same restrictions can emit impossible answers.

No-answer decisions

A reader can include a null option or compare the best answer score with a no-answer score. Tune the decision threshold on validation data containing unanswerable questions. A threshold learned only on answerable examples cannot estimate the intended false-answer tradeoff.

Unanswerability has several causes: no relevant passage was retrieved, the source does not contain the fact, the question contains a false premise, or the answer requires combining evidence the reader cannot handle. Keep these categories separate in error analysis.

Generating an answer adds another failure mode

A generative reader can paraphrase and combine evidence, but it can also introduce unsupported detail. Returning a source link does not by itself show that the source supports the generated sentence. Check claims against specific spans, including conditions, dates and negation.

If two passages disagree, the task may require reporting the disagreement rather than selecting the more fluent answer. Preserve provenance and source dates so the system can reason about scope rather than silently merging incompatible statements.

Measure each stage

Retrieval recall asks whether a supporting passage appears among candidates. Reader accuracy conditioned on gold evidence isolates comprehension. End-to-end evaluation measures the pipeline actually used. If retrieval misses the evidence, a stronger reader may have no opportunity to help.

Exact match requires an answer string to match a reference after specified normalisation. Token F1 measures lexical overlap; for predicted North Harbour and gold North Harbour Labs, precision is one, recall is \(2/3\) and F1 is 0.8. That partial score does not necessarily mean the named organisation was identified correctly.

Multiple valid answer forms and aliases require an evaluation policy. Also report answerability, unsupported-answer rates and performance on questions requiring more than one sentence. A high extractive F1 should not be advertised as evidence of broad conversational reasoning.

Key Insight: Answerability is a separate prediction from span selection. Every passage has a highest-scoring span, including passages that contain no answer.
Common pitfall: Evaluating only with gold evidence hides retrieval failures. Conversely, evaluating only the whole pipeline can hide whether the bottleneck is retrieval, comprehension or answer rejection.

Recap

  • Define whether evidence is supplied, retrieved or absent.
  • Span decoding must enforce start/end and length constraints.
  • No-answer thresholds require appropriate validation data.
  • Exact match, token F1 and evidence support measure different properties.

References

  1. Rajpurkar et al. SQuAD: 100,000+ Questions for Machine Comprehension of Text. EMNLP, 2016.
  2. Rajpurkar, Jia and Liang. Know What You Don’t Know: Unanswerable Questions for SQuAD. ACL, 2018.

You May Also Enjoy

The Standard Library Modules Worth Knowing Before You pip install

8 minute read

Published:

Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed, most of what people reach for a package to solve is one import away.

Files and Context Managers: Why with Is Not Optional

9 minute read

Published:

A file object that falls out of scope without with does get closed eventually, but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.

Errors and Exceptions: EAFP, the Hierarchy, and Reading a Traceback

9 minute read

Published:

Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first, which is faster, shorter, and free of race conditions.