Bootstrapping is a technique commonly used in Temporal Difference (TD) learning algorithms, where current estimates are updated using estimates of future values. This means that the algorithms estimate a current reward based on an estimate of the expected future reward from a current state.
In other words, bootstrapping means updating the estimated value of a state or action using the estimated value of a successor state or action. Unlike Monte Carlo learning, which estimates the value of a state based on the entire return obtained by following a complete episode, TD learning updates estimates incrementally, based on one-step transitions.
For example, consider an agent that is trying to learn to play a game of chess using TD learning. The agent starts at an initial state (i.e., the starting position in the game) and takes an action (i.e., moving a piece) that puts it in a new state. The agent then updates its estimate of the value of the initial state based on the value of the new state plus the reward obtained in the transition.
If the agent wins the game, the reward obtained from the final state will be high, so the value estimates for the states leading up to the final state will increase as a result of this bootstrapping process. Similarly, if the agent loses the game, the reward obtained from the final state will be low, and the value estimates for the states leading up to the final state will decrease as a result of the bootstrapping process.
In summary, bootstrapping is a technique used in TD learning that updates value estimates using estimates of future values, allowing an agent to learn incrementally from one-step transitions. It is a central concept in many reinforcement learning algorithms that involve estimating state values, including Q-learning, SARSA, and TD(lambda).