TD3: Three Fixes for an Overconfident Actor-Critic
Published:
Scott Fujimoto, Herke van Hoof, David Meger
Read the paper (PDF)
When the actor finds the critic’s mistake
Intuition first. Suppose a robot chooses a motor command by asking a learned critic which command has the highest return. If the critic invents a narrow peak around an unfamiliar command, the actor will move towards it. The replay buffer may eventually correct that error, but meanwhile both policy and bootstrap targets can drift.
TD3 addresses this feedback loop in deterministic actor-critic learning. The actor \(\mu_\theta(s)\) outputs a continuous action; two critics estimate \(Q_{\phi_1}(s,a)\) and \(Q_{\phi_2}(s,a)\). Transitions are sampled from replay, so the method is off-policy. The original method is due to Fujimoto, van Hoof, and Meger.
One target, three changes
Bars denote slowly updated target networks. For action bounds \([a_{\min},a_{\max}]\), form
Here \(d\) marks a genuine terminal transition. Both critics regress towards this detached target. The actor is updated less frequently than the critics, maximising \(Q_{\phi_1}(s,\mu_\theta(s))\). Target-network updates accompany the delayed actor updates. The Spinning Up algorithm description gives the complete update order.
| Mechanism | What it changes | Limitation |
|---|---|---|
| Twin critics | Bootstrap with the smaller estimate | Both critics can share the same error |
| Delayed actor | Give critics more updates between policy changes | Extra updates cannot repair missing information |
| Target smoothing | Evaluate a neighbourhood around the target action over repeated samples | Too much noise can blur useful distinctions |
Worked example: optimism in one number
Take a nonterminal transition with reward \(r=1\), discount \(\gamma=0.9\), and target-critic predictions \(8\) and \(5\) at the same perturbed next action. Using the first critic alone would give
TD3 instead uses
If the true next-action value were \(6\), the exact target would be \(6.4\). The twin target is pessimistic by \(0.9\), whereas the single target is optimistic by \(1.8\). This constructed example explains the tradeoff; it does not prove that the minimum is always closer to the truth.
Three kinds of smoothing that should stay distinct
Exploration noise perturbs actions executed in the environment. Target-policy noise perturbs actions used inside the learning target. Polyak averaging slowly moves target-network parameters towards current parameters. They operate on different objects and need separate settings.
For example, in an action space normalised to \([-1,1]\), a target actor output of \(0.95\) plus noise \(0.10\) becomes \(1.0\) after action clipping. That clipping is separate from limiting the sampled noise itself. Forgetting the outer clip can ask a critic to evaluate actions the actuator could never execute.
What a useful comparison measures
In a small control experiment, keep the environment, interaction budget, replay warm-up, and evaluation episodes fixed. Compare final returns across several seeds, but also record critic disagreement and the scale of executed actions. A high predicted value with flat real returns is a reason to inspect extrapolation, not evidence of progress.
That limitation becomes central when new interaction is unavailable. Implicit Q-learning addresses the fixed-dataset setting; residual RL changes what a robot’s learned actor is asked to control.
References
- Fujimoto, S., van Hoof, H., & Meger, D. Addressing Function Approximation Error in Actor-Critic Methods. ICML 2018.
- OpenAI Spinning Up. Twin Delayed DDPG. Algorithm equations and pseudocode.