Post

LLMs from Scratch Part 1: The Problem with Sequential Models

Posted Updated
By Jayden
11 min read

Before Transformers, researchers spent years working out why sequences are harder than other kinds of data. This post follows that path from the simplest recurrent network to the point where Seq2Seq ran out of room and the field had to try something different.

Table of Contents

0. Why Sequences Are Hard

Consider two versions of the same earnings announcement.

“Revenue missed expectations. Earnings beat.”

“Earnings beat. Revenue missed expectations.”

The words are identical, but the story isn’t. The first version leads with the miss, so the market will probably focus on it. The second leads with the beat, and analysts are more likely to give the company the benefit of the doubt. A model that treats the two sentences as the same input, just because they contain the same words, can’t tell these apart.

That is the core difficulty with sequential data: order carries meaning. A price series of [100, 105, 103, 108] says something very different from [108, 103, 105, 100], even though both contain the same four values. The trajectory holds momentum, volatility, and trend, and all of that disappears once you treat the data as a set instead of a sequence.

Time Path A: Steady Uptrend $108 $100 Steady momentum → bullish Time Path B: Late Spike $108 $100 Late spike → uncertain same price

Figure 1. Two paths arriving at the same price tell completely different stories. A model that ignores order sees them as equivalent and makes the same prediction for both.

A standard feedforward network can’t handle this. It takes a fixed-size input, passes it through its layers, and produces a fixed-size output, with no memory of what came before. Sequential data needs something that can build up context over time.

1. RNN: The First Attempt

The Recurrent Neural Network handles this with a simple idea: a feedback loop. At each time step $t$, the model takes two inputs, the current observation $x_t$ and a summary of everything seen so far stored in a hidden state $h_{t-1}$, and produces an updated hidden state

\[h_t = \tanh(W_h h_{t-1} + W_x x_t + b)\]

That hidden state carries forward to the next step and acts as the network’s working memory. Unrolled across a sequence, an RNN looks like a chain of identical networks sharing the same weights, each passing its state to the next.

Revenue beat Q3 estimates RNN RNN RNN RNN h₀ h₁ h₂ h₃ h₄ y₁ y₂ y₃ y₄ shared weights W across all steps

Figure 2. An unrolled RNN processes one token at a time, passing its hidden state h_t forward as a running summary. All four cells share the same weight matrices; the “recurrence” comes from feeding the state back in at the next step.

Picture a trading model reading the earnings headline word by word. After “Revenue missed expectations”, the hidden state should encode something like revenue disappointed. After “Earnings beat”, it should add but earnings were positive. In principle the RNN builds up this running interpretation as it reads left to right.

The Vanishing Gradient Problem

The trouble shows up in backpropagation.

To train the network we need to know how much each parameter contributed to the final loss. For a sequence of length $T$, the gradient of the loss with respect to an early hidden state $h_1$ is a chain of partial derivatives, one per step

\[\frac{\partial \mathcal{L}}{\partial h_1} = \frac{\partial \mathcal{L}}{\partial h_T} \cdot \prod_{t=2}^{T} \frac{\partial h_t}{\partial h_{t-1}}\]

Each factor involves the weight matrix $W_h$ and the derivative of $\tanh$, which is bounded between 0 and 1. If $|W_h| < 1$, this product shrinks exponentially with sequence length

\[\left\|\frac{\partial \mathcal{L}}{\partial h_1}\right\| \approx \|W_h\|^{T-1} \cdot \left\|\frac{\partial \mathcal{L}}{\partial h_T}\right\|\]

For a sequence of length 50, even $|W_h| = 0.9$ gives $0.9^{49} \approx 0.006$. Almost no gradient signal reaches $h_1$, so the model learns very little from its earliest inputs.

← backpropagation direction t = T ∇ = 1.000 t = T−3 ∇ ≈ 0.243 t = T−7 ∇ ≈ 0.035 t = T−15 ∇ ≈ 0.001 t = T−30 ∇ ≈ 0.000 learns nothing from this step

Figure 3. Gradient magnitude decays exponentially as it backpropagates through time. With a smaller norm (here ||W_h|| = 0.75), the gradient 30 steps back is effectively zero, and the model stops learning from distant context.

In practice, an RNN reading a long earnings call transcript ends up predicting the next word almost entirely from the last few tokens. Its “memory” is very short. For financial NLP, where guidance given in the opening remarks shapes how later questions should be read, that is a serious limitation.

Does “vanishing” mean the gradient becomes exactly zero? No. It becomes exponentially small, so updates driven by early time steps are tiny compared with those from recent steps. The model still trains; it just effectively ignores anything more than a few steps back.

2. LSTM: Remembering What Matters

The LSTM (Long Short-Term Memory) was built to fix the vanishing gradient problem. Its key idea is to separate short-term working memory (the hidden state $h_t$) from long-term storage (a cell state $C_t$), and to control what gets written to and read from that storage with explicit gates. The version used today, shown below, also includes the forget gate, which Gers, Schmidhuber & Cummins (2000) added to the original 1997 design.

The forget gate decides what fraction of the previous cell state to retain

\[f_t = \sigma(W_f \cdot [h_{t-1},\, x_t] + b_f)\]

The input gate and candidate decide what new information to write

\[i_t = \sigma(W_i \cdot [h_{t-1},\, x_t] + b_i), \qquad \tilde{C}_t = \tanh(W_C \cdot [h_{t-1},\, x_t] + b_C)\]

The cell state updates as a combination of forgetting and adding

\[C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t\]

The output gate filters which part of the cell state to expose as the hidden state

\[o_t = \sigma(W_o \cdot [h_{t-1},\, x_t] + b_o), \qquad h_t = o_t \odot \tanh(C_t)\]
C_{t-1} Cell State: long-term memory path Forget Gate σ: erase old info × (erase) Input Gate σ × tanh: write new info + (write) Output Gate σ × tanh(C) → h_t h_{t-1} x_t (current input) h_t

Figure 4. Simplified LSTM structure. The cell state (orange line) is updated only through elementwise scaling and addition, so gradients flow across long sequences much more stably than in a plain RNN.

The key to LSTM’s gradient stability is the additive cell state update

\[C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t\]

Information from 30 steps ago can travel through $C_t$ largely intact, because the backward path through an additive update is much gentler than one through a chain of nonlinear activations. When the forget gate stays close to 1 (“remember everything”), the gradient flows back without exponential decay.

For a model reading a quarterly earnings call, this matters a lot. When an analyst asks a follow-up in minute 45 about revenue guidance given in minute 3, the LSTM at least has a way to hold that guidance in the cell state until it becomes relevant.

Why does LSTM need both $C_t$ and $h_t$? I find it easiest to think of $C_t$ as a long-term scratchpad that accumulates context over the whole sequence, and $h_t$ as the filtered view of that scratchpad exposed as output at each step. The cell state is built for stable gradient flow; the hidden state is what the rest of the network actually reads.

What LSTM Still Doesn’t Fix

LSTM improved long-range memory a lot, but two structural problems remained.

The first is sequential dependency. Each step has to wait for the previous one: $h_t$ can’t be computed until $h_{t-1}$ is ready. A sequence of length $T$ needs $T$ sequential steps no matter how much hardware you have, so GPUs can only parallelize across the batch, not across time. On long documents this is very slow.

The second is the context vector bottleneck, which only became visible once LSTMs were used inside larger sequence-to-sequence models.

3. Seq2Seq: The Encoder-Decoder Idea

By 2014, NLP researchers wanted to do more than predict the next token. They wanted to turn one sequence into another: translate a sentence, summarize a paragraph, generate a report from structured data. These tasks share the same shape: variable-length input, variable-length output, and no requirement that the two lengths match.

Seq2Seq (Sutskever et al., 2014) handled this with two LSTMs. An encoder reads the whole input sequence and compresses it into a single vector, the context vector $c$, taken as the encoder’s final hidden state

\[c = h_T^{enc}\]

A decoder then starts from $c$ and generates the output autoregressively, one token at a time, with each output fed into the next step. In Sutskever et al., $c$ is only the decoder’s initial state; Cho et al. (2014) also feed $c$ into every decoder step, which is the general form written here

\[h_t^{dec} = f(h_{t-1}^{dec},\, y_{t-1},\, c)\]
Encoder LSTM LSTM LSTM LSTM Revenue beat Q3 estimates Context Vector c fixed size Decoder LSTM LSTM LSTM LSTM Le revenu a dépassé

Figure 5. Seq2Seq architecture. The encoder reads the full input and compresses it into a single fixed-size context vector c. The decoder uses c as its initial state and generates the output one token at a time.

For translating financial news from English to French, or producing a structured summary of an earnings transcript, this architecture is a natural fit. The encoder and decoder are separate networks with different weight matrices, trained jointly on a single objective: the log-likelihood of the target sequence. The encoder learns to compress meaning; the decoder learns to expand it.

Do the encoder and decoder share weights? In the original Seq2Seq models they don’t. Tying them would force one set of weights to be good at both reading and generating, which is a harder constraint, so the standard design keeps them separate (later models sometimes share parts, such as embeddings).

4. The Breaking Point

Seq2Seq worked well. On WMT’14 English→French, Sutskever et al. beat a strong phrase-based baseline, and thanks to the trick of reversing the source sentence, their LSTM held up reasonably well on long sentences too. But the design had a basic limit that better engineering couldn’t remove.

The encoder squeezes an input of any length into a fixed-size context vector. A five-word sentence and a five-thousand-word document have to fit into the same $d$-dimensional vector. As inputs get longer, information has to be thrown away.

10-K Filing (80 pages) Q1 revenue guidance: +12%... Supply chain risk factors... Margin expansion initiatives... Executive compensation plan... Litigation disclosures... + 38,000 more words c 512-d bottleneck Generated Summary "Margin initiatives progressing, litigation risk elevated, executive plan unchanged." Q1 revenue guidance: gone

Figure 6. The context vector bottleneck (illustration). An 80-page 10-K filing would have to be compressed into a 512-dimensional vector before the decoder sees any of it. Information from early sections is easily lost, and the summary tends to reflect whatever the encoder saw last.

For a model summarizing SEC filings, this creates a predictable failure. Details from the business overview near the start are unlikely to survive compression when the encoder has just finished reading the exhibits at the end. The summary ends up reflecting the end of the document more than the beginning, simply because the architecture can’t hold all of it.

The numbers showed it. Cho et al. (2014b) and Bahdanau et al. (2015) found that the BLEU score of basic encoder-decoder models dropped sharply as sentences got longer, starting at roughly 20–30 words. Bahdanau et al. showed that adding attention largely removed this drop, which points to the fixed-size context vector as the bottleneck.

What was needed was a model that could look back at any part of the input directly, weighted by relevance, instead of through one compressed vector. The fixed context vector had to go.

References

  • Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8), 1735–1780.
  • Gers, F. A., Schmidhuber, J., & Cummins, F. (2000). Learning to Forget: Continual Prediction with LSTM. Neural Computation, 12(10), 2451–2471.
  • Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to Sequence Learning with Neural Networks. Advances in Neural Information Processing Systems (NeurIPS 2014). arxiv.org/abs/1409.3215
  • Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. EMNLP 2014. arxiv.org/abs/1406.1078
  • Pascanu, R., Mikolov, T., & Bengio, Y. (2013). On the difficulty of training recurrent neural networks. ICML 2013. arxiv.org/abs/1211.5063
  • Cho, K., van Merriënboer, B., Bahdanau, D., & Bengio, Y. (2014b). On the Properties of Neural Machine Translation: Encoder–Decoder Approaches. SSST-8. arxiv.org/abs/1409.1259
  • Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015. arxiv.org/abs/1409.0473
  • Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. deeplearningbook.org
  • Karpathy, A. (2015). The Unreasonable Effectiveness of Recurrent Neural Networks. karpathy.github.io
AI, LLM
rnn lstm seq2seq nlp sequential-models vanishing-gradient encoder-decoder
This post is licensed under CC BY 4.0 by the author.