Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

Rapid Motor Adaptation: Inferring the Ground from the Way You Move

3 minute read

Published:

TL;DR: Rapid Motor Adaptation (RMA) trains a policy with privileged simulator information, then replaces that information with a latent estimate inferred from recent robot states and actions. At deployment the context changes through inference, while the network weights remain fixed. This is adaptation of behaviour without an online gradient-training loop.
First page of RMA: Rapid Motor Adaptation for Legged Robots
Paper preview: RMA: Rapid Motor Adaptation for Legged Robots.
Ashish Kumar, Zipeng Fu, Deepak Pathak et al.
Read the paper (PDF)

The same command can reveal a different world

Intuition first. A quadruped commands a step. On firm ground its body advances; on a slippery surface the foot moves but the body hardly follows. That mismatch contains information about the environment, even if the robot has no sensor that directly measures friction.

One instantaneous joint configuration may fit both situations. Recent motion together with the commands that caused it is more informative. The question becomes: what part of that history matters for choosing the next action?

Privileged training, observable deployment

Let \(e_t\) contain simulator-only environmental factors. An encoder \(\mu\) maps them to a compact latent \(z_t\). During initial training, a base policy uses this context. An adaptation network \(\phi\) then learns to predict it from observable history:

\[ \begin{aligned} z_t &= \mu(e_t),\\ \hat z_t &= \phi(x_{t-k:t-1},a_{t-k:t-1}),\\ \mathcal L_{\mathrm{adapt}} &= \mathbb E\|\hat z_t-z_t\|_2^2,\\ a_t &= \pi(x_t,a_{t-1},\hat z_t). \end{aligned} \]

Here \(x_t\) is the observable robot state and \(a_t\) the policy action. Adaptation training includes trajectories generated using the predicted context, exposing the estimator to its own errors. This two-stage procedure is described in Kumar et al..

The latent need not equal friction, mass, or another interpretable physical parameter. Its purpose is to carry information useful to the policy.

Training context and deployment context enter the same policyDuring training, simulator factors pass through an encoder to provide a latent target. At deployment, recent states and actions pass through an adaptation module to estimate that latent for the policy. simulator factorsencoder → zstate/action historyadaptation → ẑbase policytraining targetDeployment uses the lower path.
The privileged encoder supplies a supervised target in simulation. It is not an extra physical sensor available to the deployed robot.

Worked example: why action history matters

Consider a toy scalar system where commanded input \(u_t\) produces displacement \(\Delta x_t=b u_t\). The unknown gain \(b\) summarises how effectively the command moves the system. This is an illustration of identification, not RMA’s physical model.

Given commands \([1,2]\) and measured displacements \([0.4,0.8]\), least squares estimates

\[ \hat b=\frac{\sum_t u_t\Delta x_t}{\sum_t u_t^2} =\frac{1(0.4)+2(0.8)}{1^2+2^2}=0.4. \]

But the same displacements under commands \([0.5,1]\) imply \(\hat b=0.8\). Observed motion alone does not distinguish these gains. The input–response relationship does.

Now set both commands to zero. The denominator vanishes: the history contains no information about how strongly the system would respond. A learned estimator cannot reliably recover distinctions that the observations never reveal.

Adaptation is not necessarily retraining

At deployment, a new history changes \(\hat z_t\), and the same policy weights produce a different action. No gradient step is needed for that change. Calling this online adaptation is appropriate, but calling it online policy optimisation would describe a different mechanism.

A deployment can also update the context less often than the motor policy. The control loop then uses the most recent estimate. This saves computation but introduces a delay between a physical change and the context used for control; the original RMA system uses this asynchronous arrangement. Kumar et al.

Evaluate the transition, not just the final gait

An informative experiment changes a payload or surface during a rollout and measures both immediate degradation and recovery. A good final gait can hide a long period of poor control after the change.

Compare an oracle-context policy, the inferred-context policy, and a fixed-context baseline. Keep command sequences and initial conditions comparable. Log failures as well as recovery times, and specify which perturbations were represented during training.

Key Insight: The adaptation module estimates what the recent interaction reveals. It cannot identify a new surface before the robot has received informative evidence about it.

For another way to incorporate experience into future control, see learned MPC. For learning a direct correction around a controller, see residual RL.

References

  1. Kumar, A., Fu, Z., Pathak, D., & Malik, J. RMA: Rapid Motor Adaptation for Legged Robots. RSS 2021.

You May Also Enjoy

The Standard Library Modules Worth Knowing Before You pip install

8 minute read

Published:

Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed, most of what people reach for a package to solve is one import away.

Files and Context Managers: Why with Is Not Optional

9 minute read

Published:

A file object that falls out of scope without with does get closed eventually, but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.

Errors and Exceptions: EAFP, the Hierarchy, and Reading a Traceback

9 minute read

Published:

Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first, which is faster, shorter, and free of race conditions.