Cross-Attention: How Models Attend to Another Sequence

Self-Attention vs Cross-Attention

In self-attention, every token looks at every other token within the same sequence:

Q = X · W_Q    (from the same sequence X)
K = X · W_K    (from the same sequence X)
V = X · W_V    (from the same sequence X)

In cross-attention, queries come from one sequence, but keys and values come from another:

Q = X_dec · W_Q    (from the decoder sequence)
K = X_enc · W_K    (from the encoder output)
V = X_enc · W_V    (from the encoder output)

The output has the same length as the query sequence (decoder), but each position has gathered information from the full key-value sequence (encoder).

The Translation Analogy

Think of translating English → French.

The encoder reads the full English sentence and computes rich contextual representations: every English word has seen every other English word.

The decoder generates French words one at a time. At each step, it needs to ask: which English words are most relevant to the French word I am about to produce?

That is cross-attention:

• Q = the current French position being generated (decoder)
• K, V = all English token representations (encoder)
• Output = a blend of English information, weighted by relevance to the current French word

Where Cross-Attention Appears

1. Encoder-Decoder Transformers (T5, BART, original Transformer)

Each decoder layer has three sub-layers:

1. Masked self-attention (decoder attends to its own past tokens)
2. Cross-attention (decoder attends to encoder output)
3. Feed-forward network

The cross-attention layer is what connects the two towers. Remove it, and the decoder has no way to condition on the input.

2. Image Captioning

• Encoder: a vision model (CNN or ViT) processes the image → spatial feature map
• Decoder: a language model generates the caption
• Cross-attention: each generated word queries which image regions are most relevant

3. Diffusion Models (Stable Diffusion, DALL-E 2)

• Encoder: CLIP or T5 encodes the text prompt → contextual embeddings
• Decoder: the UNet denoising network
• Cross-attention: each spatial location in the noisy image queries the text tokens to determine what to generate there

This is why changing a single word in a prompt changes the relevant regions of the generated image — cross-attention routes each spatial location to the relevant text signal.

4. Multimodal Models (Flamingo, BLIP-2)

Cross-attention allows visual tokens to query language tokens and vice versa — the fundamental mechanism for grounding language in images.

Worked Example: 3-Token Translation

Translating “The cat sat” (3 English tokens) → “Le chat s’est assis” (4 French tokens).

Encoder processes [The, cat, sat] → produces key-value pairs \(K_{\mathrm{enc}}, V_{\mathrm{enc}}\) (shape \(3 \times d_{\mathrm{model}}\)).

Decoder generates each French token one at a time. When generating “chat” (token 2):

Q_dec  = W_Q · h_decoder["chat position"]   → shape 1×d_k
K_enc  = W_K · [The, cat, sat]              → shape 3×d_k
V_enc  = W_V · [The, cat, sat]              → shape 3×d_v

scores = Q_dec · K_enc^T = [s_The, s_cat, s_sat]
       ≈ [0.10,  0.85,  0.05]   (after softmax)

output = 0.10·v_The + 0.85·v_cat + 0.05·v_sat

“chat” attends mostly to “cat” — the cross-attention map recovers the word alignment without any explicit supervision.

The Attention Map Has a New Shape

In self-attention on a sequence of length \(N\), the attention matrix is \(N \times N\).

In cross-attention, if the query sequence has length \(M\) (decoder) and the key-value sequence has length \(N\) (encoder), the attention matrix is \(M \times N\).

Each of the \(M\) output positions independently attends over all \(N\) input positions. The output tensor is \(M \times d_v\) — same length as the query sequence, same value dimension.

Cross-Attention Visualised

For the translation pair “The cat sat” → “Le chat s’est assis”:

            The   cat   sat
Le          0.8   0.1   0.1   →  "Le" attends mostly to "The"
chat        0.1   0.85  0.05  →  "chat" attends mostly to "cat"
s'est       0.1   0.05  0.85  →  "s'est" attends mostly to "sat"
assis       0.05  0.1   0.85  →  "assis" attends mostly to "sat"

The attention pattern learned by a well-trained translation model tends to align source and target words — a property that emerged from training, not from any explicit alignment supervision.

Summary

Cross-attention is the fundamental building block for any model that needs to condition generation on a separate encoded representation — translation, captioning, diffusion, and multimodal understanding all rely on it.

References

• Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. NeurIPS 2017 (introduces cross-attention in the decoder: queries from the target sequence attend over encoder keys and values).
• Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR 2020 (T5: encoder-decoder model built entirely around cross-attention for text-to-text transfer learning).
• Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022 (Stable Diffusion: cross-attention between text and image latents for text-conditioned image generation).