Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

LeJEPA: Prediction Needs a Non-Collapsing Representation

3 minute read

Published:

TL;DR: Making related views agree is insufficient: constant embeddings agree perfectly. LeJEPA combines a prediction objective with Sketched Isotropic Gaussian Regularization (SIGReg), which shapes the distribution of embeddings. Its theoretical claims depend on the paper's assumptions about downstream prediction; a Gaussian-shaped feature cloud alone does not establish semantic understanding.
First page of LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
Paper preview: LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics.
Randall Balestriero and Yann LeCun
Read the paper (PDF)

Agreement has a trivial solution

Intuition first. Ask a model to give two photographs of the same object similar representations. Returning the same vector for every photograph satisfies the instruction. The loss needs a second requirement that distinguishes useful agreement from this shortcut.

Balestriero and LeCun propose LeJEPA, pairing prediction with a regulariser that pushes embeddings towards an isotropic Gaussian distribution. Their formulation trains without the moving-average teacher or stop-gradient used in several earlier JEPA recipes.

Two terms with different jobs

In the paper’s image-view formulation, let \(z_{n,v}\) be the embedding of view \(v\) of sample \(n\), and \(\bar z_n\) the mean embedding of its global views. A compact expression is

\[ \begin{aligned} \mathcal L_{\mathrm{pred}}&=\frac{1}{NVd} \sum_{n,v}\|z_{n,v}-\bar z_n\|_2^2,\\ \mathcal L&=(1-\lambda)\mathcal L_{\mathrm{pred}} +\lambda\mathcal L_{\mathrm{SIGReg}}. \end{aligned} \]

Here \(d\) is feature dimension; SIGReg is evaluated across samples for each view and then averaged. The centre remains part of the differentiable computation. See the method and implementation and official repository.

Prediction ties together related observations. Regularisation constrains the population of representations. Neither role can be inferred from the other term’s scalar value.

How random projections reveal a collapsed cloud

SIGReg compares projected embedding distributions to a standard Gaussian using characteristic functions. For a unit direction \(u\) and projected samples \(s_n=u^\top z_n\), the empirical characteristic function is

\[ \hat\varphi_u(t)=\frac{1}{N}\sum_{n=1}^{N}e^{it s_n}, \qquad \varphi_{\mathcal N(0,1)}(t)=e^{-t^2/2}. \]

The method aggregates discrepancies over frequencies and sampled directions. This checks more than means and variances; the paper supplies the statistical construction and its assumptions. SIGReg derivation.

Agreement and distribution regularisation address different failuresA constant representation is one point, a dimensionally collapsed representation is a line of points, and a more spread representation occupies both dimensions. ConstantOne dimensionSpread across space
Illustrative point clouds. A spread cloud is a useful diagnostic; this sketch does not itself demonstrate Gaussianity or useful semantics.

Worked example: the zero embedding

If every embedding is zero, every projection is zero. Hence \(\hat\varphi_u(t)=1\) for every direction and frequency. At \(t=1\), the target value is \(e^{-1/2}\approx0.6065\). The squared discrepancy at that single frequency is

\[ |1-e^{-1/2}|^2\approx0.1548. \]

The prediction term may be zero while this distribution discrepancy is positive. This calculation illustrates the mechanism; it is not the complete SIGReg loss, which aggregates and weights its comparisons.

Now consider embeddings drawn independently of their images from a Gaussian. They could have an excellent marginal distribution but would not consistently agree across related views. A useful representation must satisfy both requirements while preserving information relevant to downstream tasks.

What the theory does and does not establish

Read the optimality result together with its task family, probe, distributional assumptions, and constraints. A theorem about favourable feature geometry does not show that an arbitrary optimiser will discover the right visual concepts on an arbitrary dataset.

For practical evaluation, keep the encoder fixed and compare task heads under the same data split. Also inspect feature variance and covariance, and test whether changes that matter to the task remain distinguishable. These diagnostics answer different questions and should not be compressed into one “healthy representation” score.

Key Insight: Predictive agreement couples related examples. Distribution regularisation prevents that agreement from being satisfied by an uninformative representation. The downstream evaluation must still test meaning.

Compare the target-encoder approach in I-JEPA and the contrastive training used by VL-JEPA.

References

  1. Balestriero, R., & LeCun, Y. LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics. 2025.
  2. Authors’ LeJEPA implementation.

You May Also Enjoy

The Standard Library Modules Worth Knowing Before You pip install

8 minute read

Published:

Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed, most of what people reach for a package to solve is one import away.

Files and Context Managers: Why with Is Not Optional

9 minute read

Published:

A file object that falls out of scope without with does get closed eventually, but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.

Errors and Exceptions: EAFP, the Hierarchy, and Reading a Traceback

9 minute read

Published:

Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first, which is faster, shorter, and free of race conditions.