On-policy and off-policy algorithms are two types of reinforcement learning algorithms that differ in how they collect and utilize training data.
In on-policy algorithms, like SARSA and A2C, the agent follows the policy that it is currently learning and uses the data generated from that experience to update the policy. These algorithms are generally more sample-efficient since they use the same experiences to learn and improve the policy. However, because the policy is constantly changing, it can be unstable as policy updates may lead to rapid changes in the agent’s behavior. On-policy algorithms are generally better suited for problems where the data is cheap or where the agent can explore the environment quickly.
Off-policy algorithms, like Q-learning and DQN, use a separate behavior policy to collect data while learning a different policy known as the target policy. This approach can be more sample-efficient, as the agent can explore the environment using a more diverse set of actions while still learning the best policy. However, off-policy algorithms can be less stable, as the data they use to train can be less relevant to the target policy. This can lead to slower convergence and suboptimal solutions. An example of an off-policy algorithm used in robotic manipulation tasks is Soft Actor-Critic (SAC), which is able to learn from a diverse set of off-policy experiences and achieve stable performance.
In summary, on-policy algorithms are more sample-efficient but can be unstable, while off-policy algorithms are less sample-efficient but can be more stable. The choice of algorithm depends on the problem at hand, the availability of data, and the trade-off between sample efficiency and stability.