Actor-critic methods combine the benefits of value-based and policy-based reinforcement learning methods by learning both a parameterized policy function (the actor) and a value function (the critic) simultaneously.
In value-based approaches, such as Q-learning or SARSA, the agent learns the optimal value function for each state and action, which can then be used to derive a policy. The policy is usually defined as choosing the action with the highest value in each state. In contrast, policy-based methods directly learn a parameterized policy that maps states to actions, without the need for an explicit value function.
Actor-critic methods allow for more flexible and efficient learning by maintaining both a policy and a value function. These methods update the policy parameters based on the expected return from the current state onwards (i.e. the TD error), and update the value function parameters based on the temporal difference (TD) error between the estimated value and the actual return.
Here’s a high-level overview of the actor-critic algorithm:
1. The actor takes the current state as input and outputs a probability distribution over actions.
2. An action is chosen based on this distribution and taken in the environment.
3. The critic observes the current state and the resulting reward, and updates its estimate of the value function for that state.
4. The actor is then updated based on the TD error using policy gradient methods, which encourage actions that lead to higher expected returns.
As an example of actor-critic methods in action, consider the task of training a robot to walk. The actor could be a neural network that learns to output an action vector that controls the robot’s movements, and the critic could be a separate neural network that learns to estimate the value function for each state in the walking task. By maintaining both a policy and a value function, the actor-critic method can more efficiently learn how to control the robot’s movements to achieve high rewards.