Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

ColBERT and SPLADE: Beyond One Dense Vector per Passage

4 minute read

Published:

TL;DR: ColBERT preserves multiple contextual token vectors and matches them through late interaction. SPLADE learns sparse vocabulary-space weights that can include expansion terms. These methods retain different kinds of evidence from a single-vector retriever and have different indexing costs.

What a single vector compresses

A passage can mention several entities and conditions. Compressing all of them into one vector may blur the exact combination requested by a query. A cross-encoder can model detailed query–passage interactions, but running it over an entire corpus is expensive.

ColBERT offers a middle position: encode the query and document separately, retain token representations, then compute a lightweight interaction score. SPLADE takes another route by learning sparse term-weight vectors compatible with inverted-index retrieval.

ColBERT’s MaxSim score

For contextual query vectors \(q_i\) and document vectors \(d_j\), a simplified late-interaction score is

\[ s(Q,D)=\sum_i\max_j q_i^\top d_j. \]

Each query token finds its strongest document-token match, and the matches are summed. The original system includes additional representation and indexing details; this equation exposes the central interaction.

For a two-token query, suppose token similarities to a three-token document are (0.9, 0.2, 0.1) and (0.1, 0.4, 0.8). MaxSim gives \(0.9+0.8=1.7\). A competing document with maxima 0.95 and 0.3 scores 1.25. Strong coverage of both query components beats an excellent match to only one.

The same document token may maximise more than one query term. MaxSim is not a one-to-one alignment, and a high score does not enforce factual entailment or relational consistency.

Storage and search consequences

A multi-vector document representation contains more stored information than one vector per passage. Compression, pruning and candidate generation therefore matter to practical performance. Compare systems under a stated memory and latency budget rather than attributing all improvements to the scoring formula.

For an identifier-heavy query, retaining token-level distinctions can be useful. Yet contradictory passages can still match all query terms. “R7 does not support outdoor charging” is highly relevant to the query even though it supports a negative answer.

SPLADE learns lexical expansion

SPLADE produces nonnegative weights over a vocabulary using a language model and sparsity regularisation. The sparse vector can assign weight to terms not literally present, allowing learned expansion while preserving term-based indexing.

The original SPLADE and later variants differ in pooling choices. A general schematic is

\[ w_v(d)=\operatorname{pool}_{i}\log(1+\operatorname{ReLU}(a_{iv})), \qquad s(q,d)=\sum_v w_v(q)w_v(d). \]

The pooling operation must be specified: sum and max do not define the same model. Sparsity regularisation controls how many vocabulary dimensions remain active and therefore affects both retrieval quality and index cost.

If a query activates battery with weight two and charging with weight one, while a document activates those terms with weights one and three, their dot product contribution is five. Expanded terms can help bridge wording differences, but an expansion into the wrong sense can also introduce false matches.

Pick a representation through failure analysis

RepresentationRetained evidenceMain tradeoff
Single dense vectorGlobal passage geometryCompact but compressed interactions
ColBERT-style vectorsContextual token matchesMore storage and interaction work
SPLADE-style sparse vectorLearned vocabulary weightsExpansion quality and posting-list cost
Cross-encoderJoint query–passage featuresUsually limited to a candidate set

Use a fixed corpus and supporting-evidence labels. Measure rare-term behaviour, compositional queries, index size and end-to-end answer support. A retriever’s improved matching can be lost if the context packer removes the useful detail before generation.

Key Insight: Late interaction and sparse expansion preserve different information. Compare the matching failure each representation fixes, not just the headline label “neural retrieval”.
Common pitfall: MaxSim similarity and expanded lexical overlap do not prove that a passage supports the intended answer. Negation and entity relationships still need to be read.

Recap

  • ColBERT aggregates strongest token-level matches after separate encoding.
  • SPLADE learns sparse vocabulary weights and expansion terms.
  • Pooling and sparsity settings are part of the model definition.
  • Compare retrieval quality alongside memory, latency and evidence preservation.

References

  1. Khattab and Zaharia. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. SIGIR, 2020.
  2. Formal, Piwowarski and Clinchant. SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking. SIGIR, 2021.
  3. Karpukhin et al. Dense Passage Retrieval for Open-Domain Question Answering. EMNLP, 2020.

You May Also Enjoy

The Standard Library Modules Worth Knowing Before You pip install

8 minute read

Published:

Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed, most of what people reach for a package to solve is one import away.

Files and Context Managers: Why with Is Not Optional

9 minute read

Published:

A file object that falls out of scope without with does get closed eventually, but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.

Errors and Exceptions: EAFP, the Hierarchy, and Reading a Traceback

9 minute read

Published:

Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first, which is faster, shorter, and free of race conditions.