Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

V-JEPA 2: From Watching Video to Planning Robot Actions

3 minute read

Published:

TL;DR: V-JEPA 2 learns video representations through action-free pretraining. V-JEPA 2-AC adds action-conditioned dynamics using robot interaction data, enabling planning towards an image goal. The distinction is essential: watching motion supplies visual experience, while interaction data connects robot commands to their consequences.
First page of V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
Paper preview: V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning.
Mahmoud Assran, Adrien Bardes, David Fan et al.
Read the paper (PDF)

Observation and intervention carry different information

Intuition first. Watching a person move a cup reveals how the scene changes. To make a robot reproduce the change, the system also needs to relate its own commands to future observations. A visual feature extractor and an action-conditioned dynamics model solve different parts of that problem.

Assran et al. first train V-JEPA 2 on large-scale video and image data. They then train the action-conditioned V-JEPA 2-AC using fewer than 62 hours of robot interaction data from DROID. The paper reports image-goal planning on Franka arms in two labs without collecting training data in those deployment environments.

“Zero-shot” here describes that deployment setting. Robot interaction data still enters the earlier post-training stage.

From representations to candidate futures

To expose the planning idea, write a simplified latent rollout as

\[ \begin{aligned} z_t&=f(o_t), & z_g&=f(o_{\mathrm{goal}}),\\ \hat z_{t+k+1}&=F(\hat z_{t+k},a_{t+k}),\\ A^*&=\underset{A}{\arg\min}\; D(\hat z_{t+H}(A),z_g). \end{aligned} \]

Here \(A\) is a proposed action sequence. This notation suppresses history, robot state, and patch structure to highlight the optimisation problem; it is not the complete model interface. The paper uses sampling-based planning with the cross-entropy method. Consult the planning details and official implementation for the actual conditioning and cost.

Plan in feature space and return to observationsThe current observation is encoded. Candidate actions are rolled out and compared with a goal embedding. An action is executed, followed by a fresh observation. Encode currentobservationRoll out candidateaction sequencesScore againstgoal featuresExecute an action, observe, and plan againThe model predicts consequences; the planner chooses actions.
A conceptual feedback loop. Fresh observations limit how long the controller relies on its predicted future.

Worked example: a planner chooses between two sequences

For an illustrative two-dimensional goal \(z_g=(1,0)\), candidate A predicts a final state \((0.8,0.1)\) and candidate B predicts \((0.6,0)\). Using a squared Euclidean cost solely for this example,

\[ \begin{aligned} J(A)&=(0.8-1)^2+0.1^2=0.05,\\ J(B)&=(0.6-1)^2+0^2=0.16. \end{aligned} \]

The planner prefers A under this model and cost. That is a conditional decision: a prediction error could reverse the ranking. Suppose execution produces \((0.5,0.2)\) instead. The observed error is then \(0.29\), and another observation gives the controller a chance to revise its plan.

The numbers are latent coordinates, not metres. Their scale and meaning depend on the representation. A lower feature distance is useful only insofar as it tracks the task’s actual progress.

When the representation hides a control-critical detail

Imagine two states that look nearly identical: a cup is firmly grasped in one and merely touched in the other. If their embeddings are indistinguishable, a goal-distance planner may confidently lift in both. Improving the optimiser cannot recover information absent from its state representation.

Evaluate the complete loop with independent task success criteria. Include different initial positions, lighting, and contact conditions; record whether failures come from perception, prediction, candidate search, or execution. The paper’s robot demonstrations establish results in its tested settings, not unrestricted physical reliability.

Key Insight: Action-conditioned prediction supplies a way to compare possible interventions. Planning becomes useful when both the learned dynamics and the goal distance preserve the distinctions the robot needs.

For the control mechanics, read learned MPC. For a policy that directly predicts action chunks from demonstrations, compare ACT.

References

  1. Assran, M., et al. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning. 2025.
  2. Meta FAIR. Official V-JEPA 2 code and models.

You May Also Enjoy

The Standard Library Modules Worth Knowing Before You pip install

8 minute read

Published:

Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed, most of what people reach for a package to solve is one import away.

Files and Context Managers: Why with Is Not Optional

9 minute read

Published:

A file object that falls out of scope without with does get closed eventually, but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.

Errors and Exceptions: EAFP, the Hierarchy, and Reading a Traceback

9 minute read

Published:

Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first, which is faster, shorter, and free of race conditions.