Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

V-JEPA 2.1: Why Dense Features Need Local Supervision

3 minute read

Published:

TL;DR: Recognising an action does not require every patch feature to describe its own location well. V-JEPA 2.1 adds a weighted prediction loss on visible context tokens and supervises several encoder depths. Together with image/video tokenisers and scaling, these changes improve features used for dense spatial tasks.
First page of V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning
Paper preview: V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning.
Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar et al.
Read the paper (PDF)

Global understanding can coexist with poor local features

Intuition first. A model may recognise “someone is cooking” while representing the handle of a pan poorly. Classification can succeed with aggregated evidence. Grasping that handle, estimating its depth, or assigning each pixel a class requires more local information.

Mur-Labadia et al. investigate this gap in V-JEPA features. Their hypothesis is that visible context tokens can become global information aggregators when the loss directly supervises only masked positions. V-JEPA 2.1 adds supervision at the context positions and at multiple representation levels.

Make the visible positions accountable

Let \(M\) and \(C\) denote masked and visible token indices, and let \(e_i=\|\hat z_i-\operatorname{sg}(z_i)\|_1\). At one supervised level, the objective has the form

\[ \mathcal L=\underbrace{\frac{1}{|M|}\sum_{i\in M}e_i}_{\text{masked targets}} +\underbrace{\frac{1}{|C|}\sum_{i\in C}\lambda_i e_i}_{\text{visible context}}. \]

The context weights emphasise positions near a masked region. The paper uses an inverse-square-root distance weighting, together with its training schedule. The full method applies losses at multiple encoder levels and uses separate image and video patch tokenisers feeding a shared encoder. Method, official code and models.

Both masked and context positions contribute to the lossAn earlier masked-only objective scores two hidden positions. A dense objective also scores four visible context positions with weights. Masked onlyDense loss Blue: masked-target loss · Green: weighted context loss
The change concerns where the prediction loss is applied. Visible tokens still interact with the rest of the scene through attention.

Worked example: distance changes a token’s weight

Use the paper’s weighting shape with an illustrative base coefficient \(\lambda=0.2\). A context token one block from its closest masked token receives weight \(0.2\); a token four blocks away receives weight \(0.1\):

\[ \lambda_i=\frac{\lambda}{\sqrt{d_{\min}(i,M)}}, \qquad \frac{0.2}{\sqrt{1}}=0.2, \quad \frac{0.2}{\sqrt{4}}=0.1. \]

If both have feature error \(0.5\), their weighted contributions before averaging are \(0.10\) and \(0.05\). For a context set containing just these two tokens, the context loss is \(0.075\).

The nearer position receives stronger local consistency pressure. These numbers illustrate weighting, not the coefficients of every released checkpoint. Masked positions use their separate loss term; substituting distance zero into the context expression would misuse the formula.

Why multiple depths are useful

A transformer can change what its tokens represent from layer to layer. Supervising several depths supplies direct training signals before the final output. The paper evaluates this together with context supervision rather than assuming that a final-layer objective gives every intermediate representation the same properties.

For an independent diagnostic, freeze the backbone and train the same small segmentation head on features from different depths. Hold resolution, labels, optimisation, and evaluation split fixed. This asks where local information is accessible, rather than whether a large task-specific decoder can compensate for weak features.

Read visualisations beside quantitative evidence

Colouring the leading principal components of a feature map can expose coherent regions. But a smooth colour map does not establish correct boundaries or calibrated depth. Compare task metrics, probe capacity, and failure cases as well as pictures.

Key Insight: A token can help predict another location without describing its own location accurately. Dense supervision makes local feature quality an explicit part of the training problem.

Read V-JEPA 2 for the preceding world-model pipeline and V-JEPA for the original masked-video objective.

References

  1. Mur-Labadia, L., et al. V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning. 2026.
  2. Meta FAIR. V-JEPA 2.1 code and models in the V-JEPA 2 repository.

You May Also Enjoy

The Standard Library Modules Worth Knowing Before You pip install

8 minute read

Published:

Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed, most of what people reach for a package to solve is one import away.

Files and Context Managers: Why with Is Not Optional

9 minute read

Published:

A file object that falls out of scope without with does get closed eventually, but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.

Errors and Exceptions: EAFP, the Hierarchy, and Reading a Traceback

9 minute read

Published:

Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first, which is faster, shorter, and free of race conditions.