Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

Audio-JEPA: Predicting Missing Sound Representations

3 minute read

Published:

TL;DR: Audio-JEPA predicts features of masked spectrogram patches from visible context. A moving-average target encoder supplies the feature targets. The resulting encoder is evaluated across speech, music, and environmental sounds; learning audio representations does not itself produce a transcript or reconstruct a waveform.
First page of Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning
Paper preview: Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning.
Ludovic Tuncay, Etienne Labbé, Emmanouil Benetos, and Thomas Pellegrini
Read the paper (PDF)

A spectrogram has two meaningful axes

Intuition first. A short recording of a door closing contains a sharp onset followed by decaying sound. A spectrogram displays how frequency content changes over time. Hiding some of its patches creates a prediction task in which nearby time and frequency structure can provide useful evidence.

Tuncay et al. adapt I-JEPA to mel-spectrograms using a vision-transformer backbone and random patch masking. This chapter covers their 2025 Audio-JEPA paper, which is distinct from the earlier A-JEPA: Joint-Embedding Predictive Architecture Can Listen by Fei et al.

The target is an embedding of the missing region

Let \(x\) denote spectrogram patches, \(M\) the masked indices, and \(\hat z_j\) the predictor output at a masked position. The objective is

\[ \begin{aligned} z_j&=f_{\mathrm{tgt}}(x)_j,\\ \mathcal L&=\frac{1}{|M|}\sum_{j\in M} \|\hat z_j-\operatorname{sg}(z_j)\|_2^2. \end{aligned} \]

The context branch receives visible patches; the target encoder processes the full spectrogram before the target positions are selected. Its parameters follow an exponential moving average of the context encoder. The predictor receives positional information for the missing patches. Paper, official implementation.

Mask patches in time and frequencyA schematic spectrogram has time on the horizontal axis and frequency on the vertical axis. Grey rectangles hide patches whose representations must be predicted from the coloured visible patches. Time →Frequency →
A schematic mask, not a measured audio example. Both axes carry structure, so patch shape affects what evidence is hidden.

Worked example: count the prediction targets

Consider a \(128\times256\) spectrogram divided into \(16\times16\) patches. This gives

\[ N=(128/16)(256/16)=8\cdot16=128 \]

tokens. At an illustrative 50% masking ratio, the context encoder receives 64 visible tokens and the predictor is scored at 64 masked positions. The target encoder still sees the full spectrogram.

For a single target feature \((1,-1)\) and prediction \((0.8,-0.7)\), the squared feature error is \(0.2^2+0.3^2=0.13\). These coordinates are learned features, not amplitude samples or frequency values. A small feature loss therefore does not directly measure waveform reconstruction quality.

What frozen evaluation actually tests

The paper evaluates frozen representations with downstream classifiers and nearest-neighbour methods across audio tasks. A frozen encoder can still be paired with a classifier trained using labels. This protocol measures accessible task information, rather than an entire speech recognition or audio generation system. Evaluation setup.

Imagine two sounds recorded through the same microphone but describing different events. An encoder that mainly recognises the microphone can appear useful if recording devices correlate with labels. Splitting by recording source, when possible, tests whether the representation transfers beyond that shortcut.

A shared encoder still faces different task demands

Recognising an instrument may depend on timbre, whereas distinguishing similar spoken words may require fine temporal detail. One average benchmark score can hide this difference. Compare task families and inspect the effects of cropping, temporal resolution, and background noise.

Key Insight: Audio-JEPA transfers the feature-prediction idea to sound. Its usefulness depends on which temporal and spectral distinctions survive in the representation and remain accessible to downstream tasks.

Start with I-JEPA for the image formulation, then compare LLM-JEPA for a different way to use paired information in language training.

References

  1. Tuncay, L., Labbé, E., Benetos, E., & Pellegrini, T. Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning. 2025.
  2. Authors’ Audio-JEPA implementation.

You May Also Enjoy

The Standard Library Modules Worth Knowing Before You pip install

8 minute read

Published:

Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed, most of what people reach for a package to solve is one import away.

Files and Context Managers: Why with Is Not Optional

9 minute read

Published:

A file object that falls out of scope without with does get closed eventually, but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.

Errors and Exceptions: EAFP, the Hierarchy, and Reading a Traceback

9 minute read

Published:

Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first, which is faster, shorter, and free of race conditions.