Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

VL-JEPA: Predicting Answer Embeddings Before Decoding Words

3 minute read

Published:

TL;DR: VL-JEPA predicts the embedding of a textual answer from visual features and a query. It trains with a bidirectional contrastive loss and can decode the predicted embedding into words when needed. The embedding also supports retrieval and classification, so producing a new sentence is not required for every visual update.
First page of VL-JEPA: Joint Embedding Predictive Architecture for Vision-language
Paper preview: VL-JEPA: Joint Embedding Predictive Architecture for Vision-language.
Delong Chen, Mustafa Shukor, Théo Moutakanni et al.
Read the paper (PDF)

Several sentences can describe the same event

Intuition first. “A person lifts the cup” and “The cup is picked up” may describe the same visible event. A representation can place those descriptions close together while preserving the distinction from “The cup is put down.” That creates a prediction task whose target is less tied to one particular wording.

Chen et al. build VL-JEPA with a visual encoder, a query-conditioned predictor, a target text encoder, and a text decoder used when an output sentence is needed. The predictor and target encoder learn jointly with bidirectional InfoNCE. This is a contrastive JEPA variant, so the name does not imply the absence of negative pairs.

Predict a point in text-representation space

Let \(v\) be the visual input, \(q\) the query, and \(y\) its target answer. The prediction path is

\[ \hat z_y=g(f_{\mathrm{vision}}(v),q), \qquad z_y=f_{\mathrm{text}}(y). \]

For normalised embeddings in a batch of size \(B\), one direction of the contrastive loss is

\[ \mathcal L_{\rightarrow}=-\frac{1}{B}\sum_i \log\frac{\exp(\hat z_i^\top z_i/\tau)} {\sum_j\exp(\hat z_i^\top z_j/\tau)}. \]

The reverse direction exchanges predictions and targets. The temperature \(\tau\) controls score scaling. The method section explains why alignment and separation both matter, and distinguishes embedding training from the text-decoding component.

A predicted embedding can support several outputsVisual features and a query feed a predictor. Its answer embedding can be compared with candidate descriptions or sent to an optional text decoder. Visual featuresand queryPredicted answerembeddingCompare withcandidate textsDecode wordswhen needed
The prediction has uses before it becomes a sentence. Text generation remains a separate operation.

Worked example: a contrastive decision

Suppose the predicted embedding has cosine similarities \(0.8\) and \(0.2\) with the correct answer and one competing answer. With an illustrative temperature \(\tau=0.2\), the scores become 4 and 1. The probability assigned to the correct pair is

\[ p=\frac{e^4}{e^4+e^1}=\frac{1}{1+e^{-3}} \approx0.9526,\qquad -\log p\approx0.0486. \]

Equal similarities would give \(p=0.5\) and loss \(\log 2\approx0.6931\). The loss rewards being closer to the paired answer than to the competitor, rather than simply shrinking both distances.

This also exposes a data issue: a competing caption may actually describe the same event. Batch construction and duplicate or equivalent answers can affect the learning signal. A small loss is not evidence that every “negative” is semantically wrong.

Selective decoding has a useful trade-off

For a video stream, nearby windows may yield similar predicted meanings. VL-JEPA uses that stability to support selective text decoding; its reported efficiency result concerns fewer decoding operations under the paper’s evaluation protocol. It should not be read as a universal end-to-end speedup. Selective decoding experiments.

As an illustrative accounting example, 30 visual updates with 10 decoder calls use one-third as many decodes as decoding every update. The visual encoder and predictor still run, so the overall latency saving depends on how expensive decoding was in the first place.

Evaluate the events skipped between decoded outputs. A small embedding change may conceal a critical transition, such as a hand releasing an object. Decoding less often is useful when the retained updates preserve the distinctions the application needs.

Key Insight: Predicting an answer representation separates semantic estimation from sentence generation. Both the representation and the decision to decode need task-specific evaluation.

Compare LeJEPA for a distribution-regularised objective, and return to the JEPA overview for the complete reading route.

References

  1. Chen, D., et al. VL-JEPA: Joint Embedding Predictive Architecture for Vision-language. 2025; revised 2026.

You May Also Enjoy

The Standard Library Modules Worth Knowing Before You pip install

8 minute read

Published:

Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed, most of what people reach for a package to solve is one import away.

Files and Context Managers: Why with Is Not Optional

9 minute read

Published:

A file object that falls out of scope without with does get closed eventually, but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.

Errors and Exceptions: EAFP, the Hierarchy, and Reading a Traceback

9 minute read

Published:

Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first, which is faster, shorter, and free of race conditions.