LLMs from Scratch Part 5: BERT and Reading in Both Directions
GPT kept the decoder and learned to generate. BERT kept the encoder and learned to read in both directions. This post covers why bidirectionality matters, how BERT’s training objectives work, and what the RoBERTa follow-up showed about where the gains actually came from.
Table of Contents
- 0. The Other Half of the Transformer
- 1. Masked Language Model
- 2. Next Sentence Prediction
- 3. Fine-tuning BERT
- 4. What RoBERTa Revealed
- References
0. The Other Half of the Transformer
GPT’s masked self-attention has a hard rule: when processing position $t$, the model can only see earlier positions. That is required for generation, since you can’t look at the token you’re predicting. For understanding a sentence, though, it throws away useful information.
Take the sentence “The company’s guidance remained [MASK] despite the revenue miss.” The left context (guidance remained) tells you an adjective comes next. The right context (despite the revenue miss) tells you which kind: “despite” signals a contrast with bad news, so the word is probably positive, like “strong” or “intact”. You need both sides to fill the blank well.
A GPT-style model only sees the left side. BERT sees both.
Figure 1. GPT processes left to right, so the masked position can only see earlier context. BERT attends to all positions at once and uses both left and right context to fill the blank. For understanding tasks, this is a big advantage.
BERT (Bidirectional Encoder Representations from Transformers) was released by Google in October 2018, the same year as GPT-1. BERT-Base (110M parameters) was deliberately sized to match GPT-1 for comparison, with a larger BERT-Large (340M) alongside it. The architecture is a standard Transformer encoder with no masking in self-attention, so every position can attend to every other one. The interesting choices were in how to train it.
1. Masked Language Model
GPT’s pre-training objective is to predict the next token, which forces left-to-right processing, the exact constraint BERT wanted to remove. But without that constraint, what do you train on?
Predicting every token from all the others doesn’t work: if the model can see the whole sequence, it can just copy the answer. BERT’s solution is to corrupt the sequence first, replacing some tokens with a special [MASK] token, and train the model to recover the originals. That gives a well-defined task that needs context from both directions.
Concretely, 15% of the tokens in each sequence are selected, and the selected tokens are not all treated the same way
\[\text{selected token} \rightarrow \begin{cases} \texttt{[MASK]} & \text{with probability 0.80} \\ \text{random token} & \text{with probability 0.10} \\ \text{unchanged} & \text{with probability 0.10} \end{cases}\]The loss is computed only on the selected 15%; predictions for the other tokens are not scored.
Figure 2. The three treatments applied to the selected 15% of tokens. “grew” is replaced with [MASK] (80%), “12%” is replaced with a random token (10%), “quarter” is left unchanged (10%). BERT must predict the original token in all three cases.
The 80/10/10 split has a reason. [MASK] never appears during fine-tuning, so if every selected token became [MASK], pre-training and fine-tuning would see different kinds of input. Devlin et al. used the mix to reduce that mismatch. Because any token might have been silently swapped (the 10% random case), the model has to keep a good contextual representation of every token, not just the masked ones. The 10% left unchanged pushes those representations toward the word that is actually there.
For financial text, this means the representation of “revenue” in “The company’s revenue [MASK] by 12%” has to encode the surrounding context as well as the word itself: that something happened to revenue, and that 12% is the size of it. That context sensitivity is what makes BERT representations useful downstream.
Why 15%, and not 50%? Masking more gives more training signal per sequence but leaves less context to predict from. BERT simply used 15% without testing other rates, and it became the default. Later work questioned it: Wettig et al. (2023) found that masking around 40% worked better than 15% for BERT-style models, especially larger ones, so 15% is a convention rather than a proven optimum.
2. Next Sentence Prediction
MLM teaches BERT about relationships between words and phrases. Many NLP tasks also need relationships between sentences: does this premise support this hypothesis, does this paragraph follow from the last one, is this answer relevant to this question?
To address this, BERT adds a second pre-training objective, Next Sentence Prediction (NSP). Each training example has two text segments, A and B. Half the time B is the text that actually follows A in the document; the other half, B is a random segment from a different document.
Using a special [CLS] token prepended to every input, the model is trained to classify whether B really follows A
Figure 3. NSP training pairs. IsNext pairs come from consecutive text in the same document; NotNext pairs combine segments from unrelated documents. The model has to tell real document structure from random pairing.
For financial NLP, the hope was that NSP would teach the model how documents are structured. An earnings call has a clear arc (results, discussion, Q&A), and a 10-K has a regulated structure (business overview, risk factors, financials). In principle, NSP teaches the model to track these relationships instead of reading each sentence in isolation.
In practice, NSP’s contribution was less clear. More on that at the end of this post.
3. Fine-tuning BERT
BERT’s pre-training produces a model with rich bidirectional representations, but it doesn’t directly solve any specific task. Fine-tuning adapts the pre-trained weights to a downstream task by adding a small task-specific head on top of the encoder output and training on labeled data.
The [CLS] token is what makes this flexible. It is prepended to every input and has no meaning of its own: its embedding is a learned parameter, and it doesn’t stand for any word. After passing through all 12 (or 24) encoder layers, its output summarizes the whole input, because self-attention at every layer lets [CLS] collect information from every other position. That makes it a natural place to attach a classification head.
Figure 4. Fine-tuning BERT. For classification tasks (sentiment, document type), the [CLS] output goes to a linear classifier. For token-level tasks (NER, SQuAD answer extraction), each token’s output is used directly. The encoder is shared; only the output head changes.
For a financial NLP system, this covers a wide range of tasks from the same pre-trained base.
Sentiment classification: Is this earnings announcement positive or negative? Feed the text to BERT, pass [CLS] through a linear layer with two outputs, fine-tune on labeled examples.
Named entity recognition: Which tokens in this news article are company names, financial figures, or regulatory events? Each token position’s output gets a label from a linear classifier.
Question answering over filings: Given a 10-K and the question “What was the company’s Q3 gross margin?”, BERT is fine-tuned to predict the start and end positions of the answer span within the document. The [SEP] token separates the question from the passage; the model outputs two probability distributions over token positions.
The practical advantage is that the expensive step, pre-training (BERT-Large took 4 days on 16 Cloud TPUs), is done once. Fine-tuning is cheap by comparison. Devlin et al. report that every task in the paper could be fine-tuned in at most an hour on a single Cloud TPU, or a few hours on a GPU. That made BERT usable for teams without large compute budgets.
Can BERT generate text? Not directly. The encoder produces a representation of the input but has no autoregressive decoding mechanism. To generate from a BERT encoder you need to add a decoder and a decoding procedure, at which point you’re rebuilding the original encoder-decoder Transformer. For generation, decoder-only GPT-style models are the natural choice. BERT is strongest when the full input is available at inference time.
4. What RoBERTa Revealed
BERT was a strong result, but it was also under-trained. Facebook’s RoBERTa (Liu et al., 2019) went back through BERT’s training choices one by one and found that a more carefully trained BERT was much stronger, enough to match or beat the newer models of the time.
The architecture stayed the same: same number of layers, heads, and hidden size as BERT-Large. Almost all the changes were in how it was trained. The one other difference is the tokenizer: RoBERTa switched from BERT’s 30K WordPiece vocabulary to a 50K byte-level BPE, which adds some embedding parameters.
More data, longer training. BERT trained on about 16GB of text (BooksCorpus and English Wikipedia). RoBERTa used about 160GB, adding CC-News, OpenWebText, and Stories. It also trained longer with much larger batches. Performance went up across the board, which suggests BERT had been substantially under-trained.
Dynamic masking. The original BERT created masks once during preprocessing. To avoid reusing a single mask, it duplicated the training data 10 times with different masks, so over 40 epochs each sequence was seen with the same mask 4 times. RoBERTa instead generates a new mask every time a sequence is fed to the model. The paper found this comparable to or slightly better than static masking, and used it mainly because it is more efficient for longer training.
Removing NSP. This was the most discussed finding. The original BERT used segment pairs (each segment could contain several sentences) plus NSP. RoBERTa compared this with three alternatives. Using single sentences as the pair with NSP hurt downstream performance, likely because the model couldn’t learn long-range dependencies. Using contiguous full sentences up to 512 tokens without NSP matched or slightly improved on the original. RoBERTa’s explanation for why the BERT paper had found NSP helpful was that its ablation may have removed NSP while keeping the segment-pair input format.
Figure 5. BERT vs RoBERTa. Same architecture and size (apart from the larger vocabulary). The roughly 8-point GLUE improvement comes from training changes: more data, longer training, dynamic masking, and dropping NSP.
The NSP result is worth a closer look. Google published BERT with two pre-training objectives and presented both as contributions. Facebook’s ablation suggested that one of them, the one meant to teach sentence-level understanding, wasn’t pulling its weight once the input format was controlled for.
Why didn’t NSP help, when entailment and QA clearly involve relationships between sentences? The most common explanation, argued in ALBERT (Lan et al., 2020), is that NSP was too easy. A random segment from another document usually has a different topic, so the model can tell IsNext from NotNext from topic cues alone without learning anything about coherence. ALBERT replaced NSP with sentence order prediction (SOP), which uses two consecutive segments and sometimes swaps their order, so topic can’t give the answer away.
RoBERTa’s broader lesson is that strong baselines need careful ablation. It’s easy to credit a new training objective for gains that really came from training longer on more data, and papers still sometimes mix the two up.
What BERT and its variants left behind is a template: pre-train a large encoder on raw text with a self-supervised objective, then fine-tune on small labeled datasets for specific tasks. The pre-trained representations transfer well. A model trained on Wikipedia and books can be fine-tuned for financial sentiment, medical record classification, or legal documents with a modest amount of domain-specific labeled data. That transferability matters more than the particular objectives that produced it.
References
- Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019. arxiv.org/abs/1810.04805
- Liu, Y., Ott, M., Goyal, N., et al. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arxiv.org/abs/1907.11692
- Lan, Z., Chen, M., Goodman, S., et al. (2020). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. ICLR 2020. arxiv.org/abs/1909.11942
- Wettig, A., Gao, T., Zhong, Z., & Chen, D. (2023). Should You Mask 15% in Masked Language Modeling? EACL 2023. arxiv.org/abs/2202.08005
- Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. EMNLP 2016. arxiv.org/abs/1606.05250
- Wang, A., Singh, A., Michael, J., et al. (2018). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. arxiv.org/abs/1804.07461
- Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention Is All You Need. NeurIPS 2017. arxiv.org/abs/1706.03762
- Alammar, J. (2018). The Illustrated BERT, ELMo, and co. jalammar.github.io