Recurrent Neural Networks (RNNs) are a class of neural networks used for processing sequential data. Unlike traditional feedforward neural networks which only have an input layer and an output layer, RNNs have a hidden layer that allows them to maintain a state or memory of what they have learned in the past. This memory feature makes RNNs very powerful in processing sequential data such as time series, speech, language, and music.
The basic architecture of an RNN involves feeding the input sequence X = {x1, x2, ..., xT} into the network one element at a time, starting with x1. At each time step t, the RNN takes as input the current element xt of the sequence and the previous hidden state ht − 1, and applies a set of learnable parameters to compute a new hidden state ht. The output at time t is then computed as a function of the hidden state ht. The hidden state ht can be considered as a compressed representation of the input sequence up to that time step. The following equations show how the hidden state ht is computed:
$$\begin{aligned}
h_t &= f(W_{hh}h_{t-1} + W_{xh}x_t + b_h)\\
y_t &= g(W_{hy}h_t + b_y)\end{aligned}$$
where f and g are non-linear activation functions, Whh, Wxh, Why are weight matrices, and bh, by are bias vectors.
The most commonly used activation function in RNNs is the hyperbolic tangent function. The output function g can be a softmax function for classification problems, or a linear function for regression problems.
Although RNNs are very powerful, they have some limitations. One of the main limitations is the vanishing gradient problem. Backpropagation through time can cause the gradient to become very small, which prevents the network from learning long-term dependencies. This problem can be partially alleviated by using gating mechanisms, such as the Long Short-Term Memory (LSTM) or Gated Recurrent Unit (GRU) architectures, which selectively allow or discard information from the previous time steps based on the current input. Another limitation is the difficulty in parallelizing the computations due to the sequential nature of the input processing, which can lead to slower training times.