Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

Robot Learning Basics: Demonstrations, Rewards, and Models

3 minute read

Published:

TL;DR: Demonstrations provide examples of actions, rewards judge outcomes, and dynamics data teaches a model what follows an action. These are different learning signals. A robot policy must ultimately work on the observations it encounters during its own execution.

Start with what the dataset actually contains

Intuition first. For a cup-picking task, a demonstrator might supply camera images paired with gripper commands. Another dataset might contain trial trajectories labelled by task success. A third records state transitions and actions for learning a predictor. Each supports a different immediate training objective.

A demonstration’s action is an example of what someone did. A reward is an assessment of an outcome. A next-state observation describes what happened. They are not interchangeable labels.

Training signals connect to different predictionsDemonstrations train action prediction, rewards guide return optimisation, and transitions train dynamics. Evaluate the resulting behaviour in a closed loop. Data and objectiveLearned componentRobot execution
Demonstrations train action prediction, rewards guide return optimisation, and transitions train dynamics. Evaluate the resulting behaviour in a closed loop.

Behaviour cloning: predict the demonstrated action

For observation-action pairs \((o_i,a_i)\) and a deterministic continuous-action policy, a simple objective is

\[ \mathcal L_{\mathrm{BC}}(\theta)=\frac{1}{N}\sum_i \|\pi_\theta(o_i)-a_i\|_2^2. \]

This is supervised learning applied to action prediction. Stochastic policies can instead maximise demonstration likelihood. The choice matters when several actions are valid for the same observation. Imitation-learning notes.

Worked example: a reasonable average can be a bad command

Suppose two demonstrations contain effectively identical observations but choose lateral displacement commands of \(-0.1\) and \(+0.1\) metres. A scalar prediction \(u\) has loss

\[ \begin{aligned} \mathcal L(u)&=\tfrac12[(u+0.1)^2+(u-0.1)^2]\\ &=u^2+0.01. \end{aligned} \]

The minimum is \(u=0\). If the demonstrations steer around opposite sides of an obstacle, their average may head into it. This example explains why a small regression error alone does not establish a useful policy, and why history, task conditioning, or a multimodal action distribution may matter.

Reinforcement learning: improve expected return

RL trains using a reward objective and interaction data, either collected during learning or supplied in a fixed dataset. Its basic goal is to choose a policy with high expected accumulated reward. A reward may be sparse, so identifying which earlier actions helped can be difficult.

Read the RL introduction before interpreting an algorithm’s success rate as a statement about its objective. For example, rewarding motion towards a cup can encourage approach without establishing a secure grasp.

Model learning: predict consequences before choosing

A dynamics model learns a relationship such as \(\hat x_{t+1}=f_\phi(x_t,a_t)\). A planner can then compare proposed action sequences. Predicting next state and selecting actions are separate computations; good one-step predictions do not guarantee accurate long rollouts or useful planning costs.

The categories can be combined. Demonstrations can initialise a policy, rewards can refine it, and a learned model can support planning or data generation.

Evaluate complete executions

A cloned policy’s small mistake changes its next observation. Repeated mistakes can lead to situations missing from the demonstrations. This distribution shift is central to imitation learning. Split data by complete trajectories rather than neighbouring frames, then test closed-loop execution from varied starts.

Also preserve action semantics. A displacement in the gripper frame, a position in the base frame, and a joint torque require different labels and interfaces even if each is stored as a vector of numbers.

Key Insight: The learning signal determines what training can directly improve. The robot's task outcome must still be measured through complete executions with the intended observations and action interface.

Next read learned MPC for model-based action selection, ACT for demonstration-based sequences, and residual RL for reward-based corrections.

References

  1. Tedrake, R. Underactuated Robotics: Imitation Learning.
  2. OpenAI. Spinning Up: Key Concepts in RL.

You May Also Enjoy

The Standard Library Modules Worth Knowing Before You pip install

8 minute read

Published:

Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed, most of what people reach for a package to solve is one import away.

Files and Context Managers: Why with Is Not Optional

9 minute read

Published:

A file object that falls out of scope without with does get closed eventually, but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.

Errors and Exceptions: EAFP, the Hierarchy, and Reading a Traceback

9 minute read

Published:

Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first, which is faster, shorter, and free of race conditions.