Policies and Actor–Critics: Turning Value Estimates into Better Actions
Published:
Choosing and judging are different computations
Intuition first. At a junction, an actor decides which route to take. A critic estimates the return associated with the current situation or a candidate choice. The critic supplies a learning signal; it does not necessarily execute actions itself.
For discrete actions, a stochastic policy \(\pi_\theta(a\mid s)\) assigns probabilities that sum to one. A deterministic policy instead produces one action \(\mu_\theta(s)\). Continuous stochastic policies typically specify a probability density over actions, not a finite list of probabilities.
Three values to keep separate
The state value \(V^\pi(s)\) is expected return under the policy. The action value \(Q^\pi(s,a)\) first fixes action \(a\), then follows the policy. Their difference defines advantage:
For continuous actions, replace the sum with an integral. Advantage is relative to a policy in a state, so an action can have positive return and negative advantage.
Worked example: above or below the current baseline
Suppose the policy chooses left with probability 0.75 and right with probability 0.25. Its action values are 6 and 2. Then
The policy-weighted average advantage is zero: \(0.75(1)+0.25(-3)=0\). Right has positive expected return, yet performs below this policy’s baseline. This is why the sign of raw reward and the sign of advantage answer different questions.
How the actor uses a critic
For a simple on-policy stochastic actor, a sample surrogate loss is
The advantages are treated as fixed weights in this actor update. Positive weights encourage the sampled actions; negative weights discourage them locally. The critic has its own fitting objective. This expression illustrates the update mechanism; the exact return objective and sampling convention determine the corresponding policy-gradient estimator. Policy optimisation derivation.
For a single sample with probability 0.25 and estimated advantage 2, the loss contribution is \(-2\log(0.25)\approx2.773\). Increasing that probability to 0.30 lowers this contribution to approximately 2.408. Other actions and states share parameters, so a full update can have broader effects.
A useful gradient is not a performance certificate
Minimising a surrogate repeatedly on old data does not guarantee continued return improvement. The policy changes the data distribution, and the critic’s estimate can be wrong. Evaluate actual episodes separately from the numerical actor and critic losses.
GAE constructs advantage estimates from TD residuals. TD3 uses a deterministic actor and action-value critics, with different update equations. Learning this vocabulary makes those distinctions easier to follow.