Reinforcement Learning Basics: Agents, Rewards, and Returns
Published:
Decisions change the next training example
Intuition first. A robot navigating a corridor chooses whether to turn left or right. That choice changes its next camera image and the decisions that follow. It may only discover several steps later that one route reaches the goal.
The agent needs a rule for choosing actions, called a policy. The environment supplies the consequences: a new observation and a numerical reward. An episode is one run of a task, ending at a terminal condition such as reaching the destination.
Rewards are feedback about outcomes. They do not normally provide the correct action for every observation. The reward definition is part of the problem design: rewarding forward movement alone may encourage driving past a destination.
Reward is one outcome; return includes the future
We write \(R_{t+1}\) for the reward received after action \(A_t\). For an episode ending at time \(T\), the discounted return is
The discount factor \(\gamma\) sets the weight of later rewards. For an infinite discounted sum with bounded rewards, using \(\gamma<1\) makes the return finite. The learning objective is an expectation over possible runs, rather than the outcome of one lucky run. Core RL definitions.
Worked example: accepting a short-term cost
Suppose a three-step route receives rewards \(-1,-1,+10\). With \(\gamma=0.9\),
An alternative that ends immediately with reward \(+2\) has return 2. The first route is preferable under this objective despite its negative initial reward. With \(\gamma=0\), only the first reward counts, and the preference reverses.
This does not mean one discount factor is universally correct. It means discounting is part of what “good behaviour” means in the mathematical problem.
Three objects that recur throughout the book
| Object | Question it answers | Example |
|---|---|---|
| Policy | What action should I take? | A distribution over left and right |
| Value function | How much return should I expect? | Expected return from this junction |
| Dynamics model | What happens after an action? | Distribution of the next location |
An agent may use several of these together. A learned value is not itself a reward, and a model of the next observation is not automatically a good action selector.
Read the foundations in order
- MDPs: states, observations, actions, and transitions.
- Bellman equations: relate present value to future value.
- Exploration: gather information while acting.
- Temporal-difference learning: learn values from sampled transitions.
- Policies and actor–critics: turn estimates into policy updates.
Then follow the paper chapters on IQL, GAE, TD3, and hindsight replay.
References
- OpenAI. Spinning Up: Key Concepts in RL.
- Silver, D. Reinforcement Learning course, introduction and MDP lectures.