What are Markov chains and Markov processes?

A Markov chain is a mathematical model that exhibits memoryless properties for stochastic processes. It is a sequence of random variables, often representing events or states, where the probability of transitioning from one state to another depends only on the present state and not on any of the preceding states. This property is called the Markov property, or sometimes Markovian property, and is the distinctive characteristic of a Markov chain. Mathematically, a Markov chain is defined on a state space, and its transition probabilities are represented in a matrix called the transition matrix. In a Markov chain, there are a fixed number of discrete states.

A Markov process, also known as a Markovian process, is a continuous-time extension of a Markov chain. Here, instead of discrete time steps, the time parameter is continuous. The Markov properties still apply, meaning that the probabilities of transitioning between states depend only on the current state and not on the history of the process. A Markov process is typically defined using a stochastic differential equation driven by a Wiener process (a kind of continuous-time random walk) or a diffusion process. Markov processes can be represented in terms of infinitesimal generator matrices.

Let’s review some definitions and notations.

**Markov Chain:**

A discrete-time Markov chain can be defined as a stochastic process {X_n}, where n = 0, 1, 2, …, taking values in a finite or countable state space S = {s₁, s₂, s₃, …}, and satisfying the Markov property:

P(X_n + 1 = s_j|X₀ = s_i0, X₁ = s_i1, …, X_n = s_in) = P(X_n + 1 = s_j|X_n = s_in)

Here, P(X_n + 1 = s_j|X_n = s_in) represents the probability of transitioning to state s_j at time step n + 1, given that the current state is s_in.

The transition probabilities between states can be represented in a matrix, the transition matrix P:

P_ij = P(X_n + 1 = s_j|X_n = s_i)

where i, j ∈ S.

**Markov Process:**

A continuous-time Markov process (or Markov jump process) can be defined as a stochastic process {X_t}, where t ≥ 0, taking values in a finite or countable state space S = {s₁, s₂, s₃, …}, and satisfying the Markov property:

P(X_t + Δt = s_j|X_t0 = s_i0, X_t1 = s_i1, …, X_t = s_it) = P(X_t + Δt = s_j|X_t = s_it)

The transition probabilities can be described by a continuous-time transition rate matrix, the infinitesimal generator matrix Q:

$$Q_{ij} = \lim_{\Delta t \to 0} \frac{P(X_{t + \Delta t} = s_j | X_t = s_i) - \delta_{ij}}{\Delta t}$$

where i, j ∈ S and δ_ij is the Kronecker delta.

In conclusion, Markov chains and Markov processes are mathematical models that represent stochastic processes with the memoryless property, which means the future state only depends on the current state and not on past events or states. Markov chains are discrete-time and discrete-state-space models, while Markov processes extend the idea to continuous time. Their transition probabilities are given by matrices (transition matrix P for Markov chains and infinitesimal generator matrix Q for Markov processes), which allow evaluating the process’s behavior and future state probabilities.