Hindsight Experience Replay: Learning from the Goal You Actually Reached
Published:
Marcin Andrychowicz, Filip Wolski, Alex Ray et al.
Read the paper (PDF)
The robot missed, but it moved something
Intuition first. Ask a robot to push a block to the far corner of a table. Its early attempts move the block a few centimetres, earning the same failure reward as doing nothing. Those attempts nevertheless contain evidence about how actions move the block.
Write the policy as \(\pi(a\mid s,g)\), where \(g\) is the desired goal. Let \(h(s)\) extract an achieved goal, such as the block’s position. HER reuses a recorded transition under a different goal \(g'\), often selected from a later state in the same episode. Andrychowicz et al. introduced this replay mechanism for sparse-reward tasks.
What changes in a replay sample?
The action and physical observations are retained. Goal fields in both current and next observations must agree with \(g'\). Any goal-dependent termination flag must also be reconsidered. This assumes that changing the desired goal does not change the physical transition law.
For a simple continuing reaching task, use the reward
Success has reward zero, not one, under this convention. Maximising return encourages reaching the goal quickly because every unsuccessful step costs one.
Worked example: the very same push
Suppose a one-dimensional block starts at \(0.10\) metres and the desired goal is \(0.80\). A recorded trajectory visits \(0.20\), \(0.35\), then \(0.50\). Set the success tolerance to \(0.02\) metres.
Relabel the trajectory with \(g'=0.50\), an achieved future position:
| Transition | Reward for original goal \(0.80\) | Reward for relabelled goal \(0.50\) |
|---|---|---|
| \(0.10\to0.20\) | \(-1\) | \(-1\) |
| \(0.20\to0.35\) | \(-1\) | \(-1\) |
| \(0.35\to0.50\) | \(-1\) | \(0\) |
The early transitions are still failures for the new goal, but bootstrapping can propagate the useful final outcome backwards. We have not pretended that the block reached \(0.80\).
A small replay recipe
Store complete episodes so later achieved goals are available. When drawing a transition, decide whether to keep its original goal or relabel it. For relabelled samples, draw an achieved future goal from that same episode, reconstruct the goal-conditioned observations, and call the task’s reward function again. Then pass the sample to an off-policy learner.
Keeping original-goal samples matters: evaluation still asks for the real task distribution. Relabelled successes alone could emphasise easy goals that early exploration happens to reach.
Where hindsight can mislead
Consider a random disturbance that occasionally throws a block onto a target. Selecting that target afterwards overrepresents lucky outcomes. In stochastic environments, hindsight selection can bias the transition distribution conditioned on the relabelled goal; it is not automatically unbiased experience for the new task.
A second failure is more mundane. If the goal tells the simulator which object to spawn, changing the label also changes the environment that should have generated the transition. The stored physical experience may no longer be valid under the new goal.
Offline RL deals with unsupported actions; HER deals with uninformative goal rewards. A robot-learning system may need to address both problems separately.
References
- Andrychowicz, M., Wolski, F., Ray, A., et al. Hindsight Experience Replay. NeurIPS 2017.