Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

Temporal-Difference Learning: Update a Value from One Transition

2 minute read

Published:

TL;DR: TD learning updates a value estimate towards reward plus an estimated continuation value. It can learn before an episode finishes. Because its target includes another estimate, it bootstraps; terminal states and externally truncated rollouts need different treatment.

Learn from a step before knowing the whole outcome

Intuition first. You predicted that a route would yield return 4. After one step, you receive reward 1 and arrive somewhere currently valued at 5. With discount 0.9, that experience suggests a target of 5.5 for the earlier state.

For transitions collected under a fixed policy, tabular TD(0) uses

\[ \begin{aligned} \delta_t&=R_{t+1}+\gamma V(S_{t+1})-V(S_t),\\ V(S_t)&\leftarrow V(S_t)+\alpha\delta_t. \end{aligned} \]

The TD error \(\delta_t\) measures disagreement between the current estimate and a one-step target. The learning rate \(\alpha\) controls how far to move towards that target. Model-free prediction lecture.

A temporal-difference updateThe target uses the observed reward and an estimated continuation; the learning rate controls the update size. Observe transitionBuild TD targetAdjust old value
The target uses the observed reward and an estimated continuation; the learning rate controls the update size.

Worked example: target, error, update

Take \(V(S_t)=4\), \(R_{t+1}=1\), \(V(S_{t+1})=5\), \(\gamma=0.9\), and \(\alpha=0.2\):

\[ \begin{aligned} \text{target}&=1+0.9(5)=5.5,\\ \delta_t&=5.5-4=1.5,\\ V_{\mathrm{new}}(S_t)&=4+0.2(1.5)=4.3. \end{aligned} \]

The update does not set the estimate equal to the target because \(\alpha<1\). Nor does a positive TD error prove that the action was objectively good: an inaccurate next-state estimate can make the target overoptimistic.

Compare three kinds of backup

MethodInformation used for the targetMain distinction
Dynamic programmingExpected outcomes from a modelSums over possible transitions
Monte Carlo predictionObserved return to episode endWaits for the sampled outcome
TD(0)One reward plus next-state estimateBootstraps immediately

Monte Carlo targets can vary substantially across episodes. TD uses an imperfect continuation estimate and can propagate its errors. Neither method is universally more accurate for a fixed amount of data.

A terminal transition has no continuation

If the transition ends the task, its target is just the received reward. In the same example, a terminal reward of 1 gives \(\delta=1-4=-3\) and updated value \(4+0.2(-3)=3.4\).

An external collection cutoff is different. If the underlying task would continue, its continuation value remains relevant. Use the final observation before reset for that estimate. A reset observation belongs to a different episode. Gymnasium’s termination and truncation guide explains this distinction.

A time limit that is part of the task definition is a genuine finite-horizon boundary; the state should then account for remaining time. The word “timeout” alone does not determine the correct target.

From one step to several

An \(n\)-step target combines several observed rewards before bootstrapping:

\[ G_t^{(n)}=\sum_{k=0}^{n-1}\gamma^kR_{t+k+1} +\gamma^nV(S_{t+n}). \]

If the task terminates earlier, stop at that boundary and omit the continuation. This connects one-step updates to longer returns and prepares the idea behind generalised advantage estimation.

Key Insight: Bootstrapping replaces an unknown future return with a current estimate. It makes early updates possible, while making the target's accuracy and episode boundaries part of the algorithm.

Next, actor–critic basics explains how value estimates help update the action policy.

References

  1. Silver, D. Reinforcement Learning course: Model-Free Prediction.
  2. Farama Foundation. Handling Time Limits.

You May Also Enjoy

The Standard Library Modules Worth Knowing Before You pip install

8 minute read

Published:

Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed, most of what people reach for a package to solve is one import away.

Files and Context Managers: Why with Is Not Optional

9 minute read

Published:

A file object that falls out of scope without with does get closed eventually, but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.

Errors and Exceptions: EAFP, the Hierarchy, and Reading a Traceback

9 minute read

Published:

Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first, which is faster, shorter, and free of race conditions.