Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

V-JEPA: Learning Motion by Predicting Video Features

3 minute read

Published:

TL;DR: V-JEPA extends feature prediction to video: an encoder sees a masked clip and a predictor estimates features at missing space-time positions. The original model learns from video without text or action labels. Masked clip prediction is not automatically causal forecasting, and action recognition is not the same capability as controlling a robot.
First page of Revisiting Feature Prediction for Learning Visual Representations from Video
Paper preview: Revisiting Feature Prediction for Learning Visual Representations from Video.
Adrien Bardes, Quentin Garrido, Jean Ponce et al.
Read the paper (PDF)

Time changes what a feature must explain

Intuition first. A still frame of a hand beside a drawer does not tell you whether the drawer is being opened or closed. The surrounding frames can disambiguate the motion. A video representation should preserve this temporal information when it matters for the task.

Bardes et al. train V-JEPA by hiding parts of a clip and predicting their representations. The context encoder processes visible video tokens; a moving-average target encoder supplies features from the complete clip. The predictor receives positional information identifying the missing tokens. The training objective uses feature prediction without pixel reconstruction or negative examples.

A loss indexed by space and time

Let \(M\) index masked video tokens, \(\hat z_i\) be a prediction, and \(z_i\) the target feature. The feature loss is

\[ \mathcal L=\frac{1}{|M|}\sum_{i\in M} \|\hat z_i-\operatorname{sg}(z_i)\|_1. \]

Each index now identifies a spatial location and temporal position. The paper describes the space-time masks, while the official code contains the tokenisation and training configurations.

A masked region can extend through a video clipThree frame grids show the same central spatial region masked at successive times. Remaining regions provide space-time context for predicting missing features. EarlierMiddleLaterGrey regions are hidden; green regions remain visible.
A schematic temporal mask. Visible information can span the clip, so the training problem need not use only the past.

Worked example: counting tokens, not frames

Consider an illustrative clip containing 16 frames at \(224\times224\) resolution. Suppose a token covers two frames and a \(16\times16\) spatial patch. The full clip contains

\[ N=\frac{16}{2}\frac{224}{16}\frac{224}{16} =8\cdot14\cdot14=1568 \]

tokens. At an illustrative 75% mask rate, the context encoder processes 392 visible tokens, while 1176 positions are prediction targets. These numbers explain token accounting; they are not a claim about one particular released configuration.

This also separates encoder cost from whole-system cost. A smaller visible sequence reduces context-encoder attention work, but the target encoder and predictor still consume computation. Multiplying a mask percentage by total training cost would therefore give a misleading saving estimate.

Masked prediction and forecasting ask different questions

Suppose the model must predict the middle of a clip while seeing both its beginning and end. It can use evidence unavailable to a system operating online at the middle timestamp. A low error on that task does not measure past-only forecasting.

To evaluate online use, explicitly constrain the observation window, sample frames at the intended rate, and account for sensing and inference delay. For motion recognition, compare clips with similar objects but different temporal order. Otherwise, a model may succeed using scene appearance while missing the action distinction.

What a frozen evaluation measures

The original work evaluates learned features with a frozen backbone and trained downstream components on appearance and motion tasks. The backbone weights remain unchanged, but the task head still learns from labels. This distinction matters when comparing the results to zero-shot inference or full fine-tuning. Evaluation protocol.

Key Insight: Temporal information enters through the observations, masks, and prediction task. A video input alone does not establish causal forecasting or action-conditioned dynamics.

Next, V-JEPA 2 adds the bridge from video representations to action-conditioned robot planning. I-JEPA explains the image version of the prediction problem.

References

  1. Bardes, A., et al. Revisiting Feature Prediction for Learning Visual Representations from Video. 2024.
  2. Meta FAIR. Official V-JEPA code and models.

You May Also Enjoy

The Standard Library Modules Worth Knowing Before You pip install

8 minute read

Published:

Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed, most of what people reach for a package to solve is one import away.

Files and Context Managers: Why with Is Not Optional

9 minute read

Published:

A file object that falls out of scope without with does get closed eventually, but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.

Errors and Exceptions: EAFP, the Hierarchy, and Reading a Traceback

9 minute read

Published:

Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first, which is faster, shorter, and free of race conditions.