Recurrent Networks and Sequence Modelling

Why sequence models matter

Language, speech, sensor readings, prices, clicks and DNA are all sequences: ordered data where meaning depends on what came before. Recurrent neural networks (RNNs) were the first neural architecture built for this, and from roughly 2014 to 2017 the LSTM variant powered production machine translation, speech recognition and keyboard prediction. Transformers have since taken over most of those jobs, but you cannot really understand transformers — or the recurrent comeback in state-space models — without understanding what RNNs did, why they were hard to train, and which problem attention was invented to solve.

Intuition: reading with a running summary

Imagine reading a novel one word at a time while keeping a single index card of notes. After each word you rewrite the card based on the old card and the new word. You never look back at earlier pages; everything you remember must fit on the card. That card is the RNN's hidden state. It gives the model memory of unlimited span in principle, at constant cost per step. The catch is that the card is small and gets rewritten every step, so details from 200 words ago tend to be overwritten — and, as we will see, the training signal that would teach the model to keep them fades away even faster.

Sequence modelling tasks

Let $x_1, \dots, x_T$ be an input sequence of vectors $x_t \in \mathbb{R}^d$. Common task shapes:

• Many-to-one: classify a whole sequence (sentiment, activity recognition). Use the final hidden state or a pooled summary.
• Many-to-many, aligned: one output per input (part-of-speech tagging, frame-level speech labels).
• Language modelling: predict $x_{t+1}$ from $x_{1:t}$, i.e. model $p(x_1,\dots,x_T) = \prod_t p(x_t \mid x_{<t})$.
• Sequence-to-sequence: map one sequence to another of different length (translation, summarization). Needs an encoder–decoder design.

The vanilla RNN

At each step $t$ the RNN updates a hidden state $h_t \in \mathbb{R}^n$ and emits an output:

Here $W_{xh} \in \mathbb{R}^{n \times d}$ maps the input, $W_{hh} \in \mathbb{R}^{n \times n}$ is the recurrent matrix, $W_{hy}$ maps to outputs, and $h_0$ is usually zeros. The same weights are used at every time step — weight sharing across time, the analogue of a convolution's sharing across space. That is why one RNN can process sequences of any length.

Worked example: a three-step scalar RNN

Take $n = d = 1$, $W_{hh} = w = 0.5$, $W_{xh} = u = 1$, $b_h = 0$, $h_0 = 0$ and inputs $x = (1, 0.5, -1)$.

1. $h_1 = \tanh(0.5 \cdot 0 + 1) = \tanh(1) = 0.7616$
2. $h_2 = \tanh(0.5 \cdot 0.7616 + 0.5) = \tanh(0.8808) = 0.7068$
3. $h_3 = \tanh(0.5 \cdot 0.7068 - 1) = \tanh(-0.6466) = -0.5694$

The state carries information forward: $h_2$ depends on $x_1$ through $h_1$.

Backpropagation through time

Training unrolls the network across the $T$ steps into a deep feed-forward graph with shared weights and applies ordinary backpropagation; this is backpropagation through time (BPTT). With total loss $L = \sum_t L_t$, the gradient for the recurrent weights sums contributions from every step, and each contribution travels back through a chain of Jacobians:

where $1 - h_j^2$ is the derivative of $\tanh$ evaluated elementwise. The gradient that tells step $k$ how it should have behaved to help the loss at step $t$ is multiplied by $t - k$ such Jacobians.

Vanishing and exploding gradients

Taking norms, with $\gamma$ an upper bound on $|\tanh'|$ (which is 1) and $\sigma_{\max}$ the largest singular value of $W_{hh}$:

If $\sigma_{\max} < 1$, the bound shrinks exponentially: long-range gradients vanish. If the largest singular value exceeds 1, gradients can grow exponentially and explode (Pascanu, Mikolov and Bengio, 2013, analyse both conditions; the problem itself was identified by Hochreiter in 1991 and Bengio, Simard and Frasconi in 1994).

Back to our scalar example: $\partial h_3/\partial h_2 = w(1 - h_3^2) = 0.5(1 - 0.3242) = 0.3379$ and $\partial h_2 / \partial h_1 = 0.5(1 - 0.4996) = 0.2502$, so $\partial h_3/\partial h_1 = 0.0845$. After only two steps, 92% of the signal is gone. With a per-step factor of 0.5 for 20 steps the gradient scales by $0.5^{20} \approx 9.5 \times 10^{-7}$. Even a generous factor of 0.9 for 50 steps gives $0.0052$, while 1.1 for 50 steps gives about 117. The model can still represent long-range dependencies; it simply cannot learn them because the training signal never arrives.

Practical remedies:

• Gradient clipping handles explosions: if $\lVert g \rVert > \tau$, rescale $g \leftarrow g \cdot \tau / \lVert g \rVert$. Standard practice for every recurrent model.
• Truncated BPTT: backpropagate through only the last $K$ steps (carrying the hidden state forward but detaching it from the graph). Cuts memory and compute, at the price of not learning dependencies longer than $K$ through gradients.
• Gated architectures (LSTM, GRU) address vanishing gradients structurally.

Long short-term memory (LSTM)

The LSTM (Hochreiter and Schmidhuber, 1997) adds a separate cell state $c_t$ that is updated additively, protected by learned gates — vectors of values in $(0, 1)$ produced by sigmoids that multiply other vectors elementwise, acting as soft switches. The forget gate was added by Gers, Schmidhuber and Cummins (2000); the standard modern form is:

$\sigma$ is the logistic sigmoid and $\odot$ is elementwise multiplication. The key line is the cell update. Along the cell path, $\partial c_t / \partial c_{t-1} = \operatorname{diag}(f_t)$ (ignoring the indirect paths through the gates). If the network learns $f_t \approx 1$ for a unit, the gradient flows back nearly unchanged for many steps — no repeated multiplication by $W_{hh}$ and no squashing derivative. A common initialization sets $b_f$ to 1 or more so the network starts out remembering.

Worked LSTM step (scalar, $c_{t-1} = 0.5$): suppose the pre-activations are $1.0$ for the forget gate, $0.0$ for the input gate, $0.5$ for the candidate and $2.0$ for the output gate. Then $f = \sigma(1) = 0.7311$, $i = \sigma(0) = 0.5$, $\tilde{c} = \tanh(0.5) = 0.4621$, $o = \sigma(2) = 0.8808$. The cell becomes $c_t = 0.7311 \cdot 0.5 + 0.5 \cdot 0.4621 = 0.3655 + 0.2311 = 0.5966$ and the output $h_t = 0.8808 \cdot \tanh(0.5966) = 0.8808 \cdot 0.5346 = 0.4709$. The cell kept 73% of its old content and mixed in half of the new candidate.

Gated recurrent unit (GRU)

The GRU (Cho et al., 2014) merges the cell and hidden state and uses two gates. In the convention used by PyTorch and the original paper:

When $z_t \approx 1$ the unit copies its old state (a skip connection through time); when $r_t \approx 0$ the candidate ignores the past. Some textbooks swap the roles of $z$ and $1 - z$; the model is equivalent but check the convention before comparing code with equations.

Parameter counts. With input size $d$ and hidden size $n$, an LSTM layer has $4(nd + n^2 + n)$ parameters with one bias per gate; PyTorch stores two bias vectors, giving $4n(d + n) + 8n$. A GRU has three gate blocks:$3n(d+n) + 6n$ in PyTorch. For $d = 100$, $n = 256$: LSTM $= 4 \cdot 256 \cdot 356 + 2{,}048 = 366{,}592$; GRU $= 274{,}944$. GRUs are about 25% cheaper and often perform comparably; LSTMs are a slightly safer default on long dependencies.

Training a character-level LSTM in PyTorch

import torch, torch.nn as nn

text = "hello world, hello deep learning. " * 50
chars = sorted(set(text)); stoi = {c: i for i, c in enumerate(chars)}
data = torch.tensor([stoi[c] for c in text])

class CharLSTM(nn.Module):
    def __init__(self, vocab, emb=32, hidden=128):
        super().__init__()
        self.emb = nn.Embedding(vocab, emb)
        self.lstm = nn.LSTM(emb, hidden, batch_first=True)
        self.out = nn.Linear(hidden, vocab)
    def forward(self, x, state=None):
        h, state = self.lstm(self.emb(x), state)    # h: (B, T, hidden)
        return self.out(h), state                   # logits: (B, T, vocab)

model = CharLSTM(len(chars))
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
loss_fn = nn.CrossEntropyLoss()
T, B = 32, 16
for step in range(300):
    idx = torch.randint(0, len(data) - T - 1, (B,))
    x = torch.stack([data[i:i + T] for i in idx])
    y = torch.stack([data[i + 1:i + T + 1] for i in idx])   # next-character targets
    logits, _ = model(x)
    loss = loss_fn(logits.reshape(-1, len(chars)), y.reshape(-1))
    opt.zero_grad(); loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)  # tame explosions
    opt.step()
print(f"final loss {loss.item():.3f}")

Note batch_first=True (inputs shaped batch × time × features; the PyTorch default is time-first), the one-step shift between inputs and targets, and gradient clipping between backward() and step(). nn.LSTM returns per-step outputs plus the final (h, c) state, which you pass back in to continue generating.

Sequence-to-sequence: encoder–decoder

For translation, Sutskever, Vinyals and Le (2014) used one LSTM as an encoder that reads the source sentence into its final state, and a second LSTM as a decoder that starts from that state and generates the target one token at a time:

Training uses teacher forcing: the decoder receives the true previous token as input rather than its own prediction. At inference it must consume its own outputs, a train–test mismatch known as exposure bias. Decoding uses greedy search or beam search, which keeps the $B$ most probable partial outputs at each step. (Cho et al., 2014 proposed a closely related RNN encoder–decoder in the same year.)

The weakness is the bottleneck: the entire source sentence must be squeezed into one fixed-size vector $c$. Translation quality fell off sharply on long sentences. Sutskever et al. found that reversing the source word order helped, because it shortened the distance between the first source and first target words — a clue that path length was the real problem.

Bahdanau attention: the bridge to transformers

Bahdanau, Cho and Bengio (2014) removed the bottleneck. The encoder (a bidirectional RNN) keeps all its hidden states $h_1, \dots, h_T$. At each decoder step $t$, with previous decoder state $s_{t-1}$, the model scores every source position, normalizes the scores with a softmax, and builds a fresh context vector:

$W_a$, $U_a$ and $v_a$ are learned. The weights $\alpha_{t,i}$ form a soft alignment: when generating a French word, the decoder learns to focus on the English words it translates. This is called additive attention; Luong et al. (2015) popularized the cheaper dot-product score $e_{t,i} = s_t^\top h_i$.

Worked example. Three encoder states $h_1 = (1, 0)$, $h_2 = (0, 1)$, $h_3 = (1, 1)$ and scores $e = (2.0, 1.0, 0.1)$. Softmax: $e^{2.0} = 7.389$, $e^{1.0} = 2.718$, $e^{0.1} = 1.105$, sum $11.212$, so $\alpha = (0.659, 0.242, 0.099)$. Context: $c = 0.659(1,0) + 0.242(0,1) + 0.099(1,1) = (0.758, 0.341)$. The decoder now sees a weighted mix concentrated on position 1.

import torch, torch.nn as nn

class AdditiveAttention(nn.Module):
    def __init__(self, enc_dim, dec_dim, attn_dim):
        super().__init__()
        self.W = nn.Linear(dec_dim, attn_dim, bias=False)
        self.U = nn.Linear(enc_dim, attn_dim, bias=False)
        self.v = nn.Linear(attn_dim, 1, bias=False)
    def forward(self, s_prev, enc_h, mask=None):
        # s_prev: (B, dec_dim); enc_h: (B, S, enc_dim); mask: (B, S), True = real token
        e = self.v(torch.tanh(self.W(s_prev).unsqueeze(1) + self.U(enc_h))).squeeze(-1)
        if mask is not None:
            e = e.masked_fill(~mask, float("-inf"))     # never attend to padding
        alpha = torch.softmax(e, dim=-1)                 # (B, S)
        context = torch.bmm(alpha.unsqueeze(1), enc_h).squeeze(1)   # (B, enc_dim)
        return context, alpha

Attention gives every output step a direct, one-hop connection to every input position, so gradients no longer need to travel through dozens of recurrent steps. The transformer's insight (Vaswani et al., 2017) was to ask: if attention provides the long-range paths, do we need the recurrence at all?

Why transformers replaced RNNs

Here $n$ is sequence length and $d$ the model width; the comparison follows Table 1 of Vaswani et al. (2017). The decisive row is the second. An RNN must compute $h_{t-1}$ before $h_t$, so training on a 2,048-token sequence takes 2,048 dependent steps that cannot be spread across thousands of GPU cores. A transformer processes all positions of a training sequence in parallel with large matrix multiplies — exactly what GPUs are built for. Parallel training made it practical to train much larger models on much more data, and the short path length made long-range dependencies learnable. Quadratic cost in $n$ was an acceptable price.

Where RNN-like models still matter

• Streaming and on-device inference. An RNN processes each new input in constant time and memory, while a transformer's key–value cache grows with context. Streaming speech recognition (for example RNN-transducer models), keyword spotting and low-power sensors still use recurrent designs.
• Time series and control with small datasets, where the recurrent inductive bias helps and sequences are modest in length.
• The recurrent comeback. State-space models such as Mamba and linear-attention variants are trained in parallel like transformers but run as recurrences at inference, with a fixed-size state. Several 2025-era production language models mix such layers with attention. These are covered in beyond-transformers.

Common pitfalls

• Forgetting gradient clipping: a single exploding batch produces NaN losses. Clip by global norm (values around 0.5–5 are typical).
• Padding without masking: padded positions leak into final states, attention weights and losses. Use pack_padded_sequence or masks, and ignore_index in the loss.
• Using the last time step of a padded batch as the sequence summary, which may be a padding position for shorter sequences.
• Not detaching hidden state across truncated-BPTT chunks, so the graph grows forever and memory runs out.
• Batch-first confusion: nn.LSTM defaults to (time, batch, feature).
• Expecting an RNN to learn very long dependencies from gradient signal alone. If the dependency is hundreds of steps, use attention or a model designed for it.

In practice

• For new NLP work, start with a pretrained transformer; reach for an LSTM/GRU only for small streaming or edge workloads, or as a quick baseline on short sequences.
• Typical recurrent recipe: 1–3 layers, hidden size 128–1024, dropout between layers, Adam with clipping, forget-gate bias initialized positive, bidirectional layers when the whole sequence is available (not for generation).
• For time series, compare against strong non-neural baselines and gradient-boosted trees before committing to recurrent nets (see time-series-forecasting).
• If you need constant-memory inference over long contexts, evaluate modern state-space or hybrid models rather than classic LSTMs.