JEPA: Learning What to Predict in Representation Space
Published:
Start with the target
Intuition first. Watch a cup move behind a book. You may predict that the cup will reappear on the other side without knowing the exact reflection on its surface. A useful predictive model should preserve information needed for that judgement. Whether it actually learns to do so is an experimental question.
JEPA makes the prediction target a learned representation. In I-JEPA, the target describes hidden image regions; in V-JEPA, it describes hidden parts of a video. The word predictive does not require the target to be in the future: a missing patch in the same image is also a prediction problem.
The common pattern
Let \(x\) be visible context, \(y\) the target observation, and \(c\) information such as target position, a question, or an action. A schematic objective is
Here \(D\) measures prediction error, while \(\mathcal R\) stands for an explicit regulariser when the method uses one. This is a reading aid, not a universal implementation: the papers use different losses, encoders, and mechanisms for maintaining informative features.
Worked example: a perfect loss can be useless
Take two unrelated images. Suppose both target embeddings equal \((0,0)\), and the predictor always returns \((0,0)\). Their squared prediction errors are both zero. Yet the representation cannot tell the images apart.
This is representation collapse. Measuring prediction error alone cannot establish usefulness when the model also learns the targets. LeJEPA tackles this through distribution regularisation; I-JEPA and V-JEPA use a moving-average target encoder within their training recipes. These mechanisms should be examined together with masking, architecture, and evaluation.
A second failure is subtler. Suppose two scenes differ only in whether a cup is inside a gripper. A representation that erases this distinction may be adequate for identifying the object and inadequate for deciding whether to lift. The downstream task decides which invariances are helpful.
A reading route through the book
| Chapter | Question to carry into the paper |
|---|---|
| I-JEPA | How do masking and target features shape image learning? |
| LeJEPA | Why must agreement be paired with a distribution constraint? |
| V-JEPA | What changes when the missing information spans time? |
| V-JEPA 2 | What additional information makes action-conditioned planning possible? |
| V-JEPA 2.1 | Why can good global features still be poor local features? |
| LeWorldModel | How can visual representations and action-conditioned dynamics be learned together? |
| VL-JEPA | When should a predicted meaning be decoded into words? |
| Audio-JEPA | Which sound information survives masked spectrogram prediction? |
| LLM-JEPA | How can a representation objective complement token generation? |
| D-JEPA | How can predicted features condition a generative denoising model? |
Reading the evidence
Check whether a result freezes the encoder and trains a small task head, fine-tunes the whole model, or evaluates a complete control system. These measure different capabilities. Also check which labels, captions, actions, and pretrained components enter each stage.
V-JEPA 2 distinguishes action-free video learning from robot interaction post-training. VL-JEPA uses vision-language supervision and a contrastive objective. The JEPA name therefore does not imply that every model is action-aware, label-free, or non-contrastive.
For the control background, continue into learned model predictive control. The paper-specific chapters above each include the original first page, source links, and a worked explanation.