The discount factor (gamma) is a hyperparameter used in reinforcement learning algorithms that determine the trade-off between immediate rewards and future rewards. It’s usually represented by a value ranging between 0 and 1, with 0 indicating that the agent only cares about immediate rewards and 1 indicating that the agent cares equally about immediate and future rewards.
The purpose of gamma is to take into account the impact of future rewards on the decision-making process. RL algorithms typically involve a sequence of state-action pairs, and the goal is to learn the optimal policy that maximizes the expected cumulative rewards over the sequence. By incorporating the discount factor, the algorithm can give more importance to long-term rewards and encourage the agent to pursue actions that might involve some short-term costs but yield greater benefits in the long run.
For instance, imagine a scenario where an agent is trying to navigate a maze to reach a goal. The immediate reward might be a small positive value for reaching a particular cell in the maze, but if the agent reaches the goal, it receives a large positive reward. By setting a high gamma value, the agent will be incentivized to take actions that get it closer to the goal, even if it involves some initial costs.
On the other hand, a low gamma value will make the agent more myopic by prioritizing immediate rewards over long-term rewards, which can lead to suboptimal policies that fail to achieve the ultimate goal. Therefore, choosing an appropriate value for gamma is critical in achieving the desired trade-off between short-term and long-term rewards.