Post

LLMs from Scratch Part 3: GPT and the Case for Scale

Posted Updated
By Jayden
11 min read

Part 2 ended with the full Transformer: an encoder and a decoder connected by cross-attention, built for sequence-to-sequence tasks like translation. GPT asked what happens if you keep only the decoder. This post follows the GPT series from 2018 to GPT-3, when scale itself started to look like a research strategy.

Table of Contents

0. The Decoder-Only Bet

The original 2017 Transformer was designed for translation, a task with a clear input sequence and a clear output sequence in another language. The encoder read the source, the decoder generated the target, and cross-attention connected them. For translation, that makes sense.

Most language tasks aren’t translation, though. Sentiment classification, question answering, summarization, and text completion all take text in and produce a response. They don’t need two separate sequences in different languages. They need a model that can read a context and continue it sensibly.

GPT’s starting observation was simple: if the goal is to generate text conditioned on earlier text, the decoder half of the Transformer is enough.

Full Transformer Encoder not needed Decoder Masked Self-Attention (cross-attention removed) Feed-Forward GPT (Decoder-only) Masked Multi-Head Self-Attention + Add & Norm Feed-Forward + Add & Norm × 12 layers Token Embedding + Positional Enc. ↑ next token probability no cross-attention, fully parallelizable

Figure 1. GPT removes the encoder and cross-attention. What remains is a stack of masked self-attention and feed-forward layers, which is simpler but can still condition generation on any prior context.

Dropping the encoder also removes the need for paired training data, the aligned source-target pairs a translation model requires. GPT can be trained on raw text alone: any book, article, or document. The training signal is just predicting the next token, which is self-supervised because the labels come from the text itself.

The bet was that this would scale. Unlabeled text exists in huge quantities. If predicting the next word is enough to learn useful representations, training data can grow far beyond any labeled dataset.

1. GPT-1: The Pre-training Hypothesis

GPT-1 (OpenAI, 2018) tested a specific hypothesis: pre-train a large language model on raw text, then fine-tune it on small labeled datasets for specific tasks. The expectation was that pre-training would force the model to build internal representations of language general enough to adapt to downstream tasks with little labeled data.

Pre-training maximizes the likelihood of each token given the previous $k$ tokens

\[L_1(\mathcal{U}) = \sum_i \log P(u_i \mid u_{i-k}, \ldots, u_{i-1};\, \Theta)\]

This objective is standard language modeling. What GPT-1 added was the two-stage protocol. After pre-training on BooksCorpus (about 800M words), the model was fine-tuned on labeled datasets by adding a task-specific output layer and optimizing a supervised objective

\[L_2(\mathcal{C}) = \sum_{(x,y)} \log P(y \mid x^1, \ldots, x^m)\]

Instead of dropping the language modeling objective during fine-tuning, GPT-1 kept it as an auxiliary loss

\[L_3(\mathcal{C}) = L_2(\mathcal{C}) + \lambda \cdot L_1(\mathcal{C})\]
Stage 1: Pre-training 800M words of raw text (BooksCorpus) Objective: predict next token (L₁) 117M parameters, 12 layers no labels required transfer Stage 2: Fine-tuning small labeled dataset (e.g. earnings call sentiment labels) Objective: L₃ = L₂ + λ · L₁ task loss + language model regularizer prevents catastrophic forgetting 9 out of 12 NLP benchmarks: new state-of-the-art

Figure 2. GPT-1’s two-stage training protocol. Pre-training builds general language representations; fine-tuning adapts them to a specific task. The auxiliary L₁ term during fine-tuning prevents the model from forgetting its pre-trained representations.

For a quant fund, the two-stage protocol maps onto a real workflow. Pre-train once on all available financial text (10-K filings, earnings transcripts, analyst reports) to build general financial language representations. Then fine-tune separately on small labeled datasets for each task: sentiment classification for earnings calls, named entity recognition for news, binary classification for trade signals. The expensive pre-training is shared across every downstream task.

Why keep $L_1$ during fine-tuning? Radford et al. say the auxiliary objective improved generalization and sped up convergence, and that it helped more on larger datasets. A common intuition is that it also acts as a regularizer: optimizing only $L_2$ on a small dataset can pull the model away from its pre-trained representations, and keeping the language modeling loss holds it closer to them.

The results supported the hypothesis. With 117M parameters, pre-trained on raw text and fine-tuned on small labeled sets, GPT-1 set the state of the art on 9 of the 12 tasks it was evaluated on in 2018.

2. GPT-2: Fine-tuning Is a Crutch

GPT-1’s protocol assumed you need a labeled fine-tuning dataset for every task. In financial NLP, that means separately curating labeled examples for sentiment, event extraction, risk classification, and so on. That is expensive and slow, small labeled datasets overfit easily, and their label distributions may not generalize.

GPT-2 (2019, 1.5B parameters) pushed further: what if fine-tuning isn’t necessary? What if a large enough language model, trained on diverse enough text, already knows how to do most tasks, and you only need to phrase the input the right way?

The grounding is the usual factorization of a sequence’s probability into a product of conditionals:

\[p(x) = \prod_{n=1}^{N} p(s_n \mid s_1, \ldots, s_{n-1})\]

Almost any NLP task can be written as conditional text generation: predict an output sequence given an input sequence, which is exactly what a language model does. Translation is $p(\text{French} \mid \text{English})$. Summarization is $p(\text{summary} \mid \text{document})$. Question answering is $p(\text{answer} \mid \text{document, question})$. If the model is strong enough and has seen enough varied text, all of these conditionals live inside one language model.

Zero-Shot Task Formulation: same model, different prompts Sentiment: "Q3 revenue missed by 8%." Sentiment of this earnings note: [negative] Translation: "Revenue beat estimates." French: [Le revenu a dépassé les prévisions.] Summary: "[Full 10-K filing text...]" TL;DR: [Strong margins, guidance maintained.] GPT-2 1.5B params no fine-tuning no task labels Output task-specific response

Figure 3. GPT-2’s zero-shot approach. Every task is written as text completion, and the model is never told which task it is doing. The same 1.5B-parameter model handles sentiment, translation, and summarization based only on the prompt format.

GPT-2’s zero-shot results were uneven: strong on some tasks, weak on others. The more lasting contribution was the reframing. Tasks don’t need separate models or labeled data; they need the right prompt. That idea shaped most of what came next.

GPT-2 is also remembered for OpenAI’s staged release: the full model was withheld at launch over concerns about automated disinformation and released in stages over 2019. The concern was reasonable, since the model could produce fluent text at scale. Whether holding back a 2019 model actually delayed that risk is a separate question.

3. GPT-3: Few-Shot Learners

GPT-3 (2020, 175B parameters) scaled GPT-2’s approach by roughly two orders of magnitude and studied more systematically what a large model can do with no weight updates at all, using only examples in the prompt.

The paper compares three evaluation modes. Zero-shot: the model sees only a task description. One-shot: it sees one input-output example before the test query. Few-shot: it sees several examples. In all three, the weights are frozen: no gradient updates, no fine-tuning, no task-specific training.

Zero-Shot Task: classify stock move direction. Input: "Fed raised rates 75bps." Direction: [▼ bearish] no examples One-Shot Example: "CPI beat → ▲ bullish" Input: "Fed raised rates 75bps." Direction: [▼ bearish] 1 example Few-Shot Examples: "CPI beat → ▲ bullish" "Earnings miss → ▼ bearish" "Rate cut → ▲ bullish" Input: "Fed raised rates 75bps." Direction: [▼ bearish] k examples, no weight updates Accuracy improves with more examples and more parameters zero-shot one-shot few-shot

Figure 4. Zero-shot, one-shot, and few-shot prompting on a financial signal classification task. No weights change between examples; the model adapts only through the context window.

GPT-3’s main finding was that few-shot performance improves reliably with model size. Smaller models (1.3B, 6.7B) gain little from in-context examples, while the 175B model gains a lot, with each extra example closing more of the gap to fine-tuned models.

This fits the scaling laws that Kaplan et al. had published a few months earlier (January 2020), and which helped motivate building GPT-3 at that size. They showed that language model loss falls as a smooth power law in the number of parameters $N$, the amount of training data $D$, and compute $C$

\[L(N) \propto N^{-\alpha_N}, \quad L(D) \propto D^{-\alpha_D}\]

with empirical exponents $\alpha_N \approx 0.076$ and $\alpha_D \approx 0.095$. The point is that returns to scale are predictable: you can estimate how much better a 10x larger model will be before training it, which turned scaling from a gamble into a plan.

Are “emergent” abilities, which seem to appear suddenly above some size (Wei et al., 2022), really discontinuous? Schaeffer et al. (2023) argued that much of the jump comes from the metric: all-or-nothing scores like exact match make a smooth improvement look like a sudden one, and continuous metrics often show steady progress instead. The debate isn’t fully settled, but it changed how people evaluate large models.

In the paper, GPT-3 could write news articles that human evaluators struggled to tell apart from real ones, do 2- and 3-digit arithmetic reasonably well (but not larger numbers), and handle translation and question answering from a few examples, all without weight updates. For applications like financial text, that suggested a lot could be done with prompting alone.

4. Why GPT-3 Is Not ChatGPT

GPT-3 did well on benchmarks but was frustrating to use.

Ask it a direct question and it might answer, rephrase the question, or wander off in a different direction depending on small changes in wording. Ask it to summarize a document and you might get a summary, a critique, or more of the document. It had no notion of following instructions. It had learned to predict what text comes next, and the text that follows a question on the internet isn’t always an answer.

GPT-3 User: "Summarize this 10-K for me." GPT-3: "Summarize this 10-K for me. Summarize this 10-K for me. [continues repeating prompt]" follows the distribution, not the intent ChatGPT (InstructGPT) User: "Summarize this 10-K for me." "Here is a summary: The company reported strong Q3 earnings with margin expansion of 200bps..." follows the intent, not just the distribution

Figure 5. The gap between capability and alignment. GPT-3 predicts likely continuations, which sometimes means echoing the prompt. ChatGPT was explicitly trained to follow instructions. The architecture is the same; the behavior is very different.

This gap is usually called alignment. GPT-3 was trained to model internet text, and internet text isn’t a corpus of helpful assistants. It’s human writing of every kind, including pages where a question is followed by more questions, requests by refusals, and prompts by tangents.

So the model learned what text looks like, not what a helpful response looks like.

Fixing that needed a different training signal: human feedback about which responses are better, rather than more raw text. That process and the models it produced are the topic of Part 4.

Most of the distance between GPT-3 and ChatGPT came from that kind of training. ChatGPT was built on the GPT-3.5 series, which also had additional pre-training (including code), but the instruction-following behavior came mainly from fine-tuning on human demonstrations and preferences.

References

  • Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving Language Understanding by Generative Pre-Training. OpenAI Blog. openai.com/research/language-unsupervised
  • Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. OpenAI Blog. openai.com/research/better-language-models
  • Brown, T., Mann, B., Ryder, N., et al. (2020). Language Models are Few-Shot Learners. NeurIPS 2020. arxiv.org/abs/2005.14165
  • Kaplan, J., McCandlish, S., Henighan, T., et al. (2020). Scaling Laws for Neural Language Models. arxiv.org/abs/2001.08361
  • Wei, J., Tay, Y., Bommasani, R., et al. (2022). Emergent Abilities of Large Language Models. TMLR. arxiv.org/abs/2206.07682
  • Schaeffer, R., Miranda, B., & Koyejo, S. (2023). Are Emergent Abilities of Large Language Models a Mirage? NeurIPS 2023. arxiv.org/abs/2304.15004
  • Ouyang, L., Wu, J., Jiang, X., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022. arxiv.org/abs/2203.02155
AI, LLM
gpt gpt-2 gpt-3 language-model pre-training few-shot in-context-learning scaling-laws decoder-only
This post is licensed under CC BY 4.0 by the author.