Residual Reinforcement Learning: Teaching a Controller Its Missing Correction

Start from something that already works

Intuition first. A robot can move a peg close to a hole using a conventional position controller. The last few millimetres are harder: alignment errors, friction, and contact can prevent insertion. Asking a neural policy to relearn the entire approach wastes an available solution.

Residual RL decomposes the command into a baseline and a learned correction. Johannink et al. studied this combination for robot control, including physical assembly with contact. The decomposition is an architectural choice; a suitable continuous-action RL algorithm supplies the learning updates.

Define exactly what gets added

A bounded implementation can use

Here \(o_t\) is the observation, \(u_{\mathrm{base}}\) the baseline command, \(\Delta u_\theta\) the learned residual, and \(\alpha\) its scale. The clipping and scaling shown here are design choices, not a stability theorem.

A torque residual adds torque to torque. A Cartesian displacement residual modifies a pose target before an inner tracking controller acts. A normalised policy output needs conversion into the command’s physical units. These interfaces create different learning problems, even if they use the same neural network.

Worked example: a correction measured in newtons

Consider an illustrative one-dimensional force command. The baseline requests \(8\) N, the residual policy outputs \(-0.3\), and the scale is \(\alpha=5\) N per policy unit. Before limiting,

For command bounds \([-10,10]\) N, this remains \(6.5\) N. If the policy instead outputs \(+0.8\), the requested \(12\) N is limited to \(10\) N. Increasing the residual further then produces no additional force command. Saturation changes the action-to-outcome relationship seen by the learner.

These are arithmetic examples, not operating settings for a particular robot. A real force interface also requires its reference frame, control rate, and measured feedback to be specified.

Which action belongs in replay?

Suppose the RL action is the residual \(a_t=\Delta u_t\). Then the environment presented to the learner includes the baseline controller and command limiter. Replay should store that residual action and the resulting transition; the critic estimates the return for residual choices.

If instead replay stores the executed total command, the actor and critic must consistently use that total-command interpretation. Mixing residual actions during optimisation with total commands in replay silently changes the meaning of \(Q(s,a)\).

Keep the baseline fixed during a comparison. If it contains an integrator or another internal state that affects future commands, include sufficient information in the learning observation or acknowledge the resulting partial observability.

How to tell whether the correction helps

Compare the baseline alone, the baseline plus learned residual, and a policy trained without the baseline under the same interaction budget. Record completion rate, peak contact forces, saturation frequency, and residual magnitude. A successful trial that constantly hits actuator limits reveals a different controller from one that uses small corrections.

Zero residual should recover the baseline, provided its commands already satisfy the same limits. That gives a simple interface check before any learning is involved.

For the underlying actor-critic machinery, see TD3. For a different way to reuse existing structure, learned model predictive control learns dynamics and keeps an explicit planning loop.

References

1. Johannink, T., Bahl, S., Nair, A., et al. Residual Reinforcement Learning for Robot Control. ICRA 2019.