SARSA (State-Action-Reward-State-Action) is an on-policy reinforcement learning algorithm. It is similar to Q-learning in that it learns the optimal action-value function, but it is different in how it chooses actions during training.
In SARSA, the agent selects an action based on its current policy and obtains a reward from the environment. The next state is then observed and the agent selects the next action using the same policy. The update rule for SARSA is:
Q(s_t, a_t) = Q(s_t, a_t) + alpha*(r_t+1 + gamma*Q(s_t+1, a_t+1) - Q(s_t, a_t))where Q(s_t, a_t) represents the action-value function for taking action a_t in state s_t, alpha is the learning rate, gamma is the discount factor, r_t+1 is the reward obtained after taking action a_t in state s_t and transitioning to state s_t+1, and a_t+1 is the next action selected using the current policy.
Q-learning, on the other hand, is an off-policy algorithm. It selects the next action based on the maximum Q-value for the next state, regardless of the current policy. The update rule for Q-learning is:
Q(s_t, a_t) = Q(s_t, a_t) + alpha*(r_t+1 + gamma*max(Q(s_t+1, a)) - Q(s_t, a_t))where max(Q(s_t+1, a)) represents the maximum Q-value for all actions a in the next state.
The main difference between SARSA and Q-learning is that SARSA takes into account the next action the agent will take and updates the action-value function accordingly, while Q-learning assumes the optimal action will be taken in the next state and updates the action-value function based on that assumption. This makes SARSA more stable but also potentially slower to converge compared to Q-learning. SARSA can also be applied to problems where the action space is continuous or stochastic, while Q-learning is typically used for discrete action spaces.