Deep Learning Optimization Part 1: From Information Theory to Cross-Entropy Loss
Most of us call Cross-Entropy Loss without asking why it works. This post follows that question back from the loss function you use every day to Shannon’s 1948 communication theory.
Table of Contents
- 0. The Problem We’re Trying to Solve
- 1. Probability vs Likelihood
- 2. Information Theory
- 3. KL Divergence
- 4. Cross-Entropy Loss
- 5. References
0. The Problem We’re Trying to Solve
Imagine you’re building a model to predict whether a stock will go up or down tomorrow. You train it, evaluate it, and at some point you call nn.CrossEntropyLoss() without thinking twice. But why Cross-Entropy? Why not mean squared error? Why not something you invent yourself?
The answer isn’t arbitrary. Cross-Entropy Loss has a precise justification in information theory, a field built in the 1940s for a completely different purpose.
Training a neural network is fundamentally an optimization problem. You have a model parameterized by $\theta$, and you want to find the configuration that makes predictions as close to correct as possible. To search in any meaningful direction, you need a function that collapses the difference between prediction and ground truth into a single number.
\[\theta^* = \arg\min_\theta\ \mathcal{L}(\theta)\]That function, the loss function, is not a minor implementation detail. It defines what the model is actually optimizing. If it’s wrong, the model learns the wrong thing no matter how good the architecture is.
1. Probability vs Likelihood
The Same Formula, Two Different Questions
Say a quant model assigns a 70% probability to a stock rising tomorrow, and the stock rises. What does that actually tell us?
There are two completely different ways to interpret that 70%, and they are easy to mix up.
\[P(x \mid \theta)\]Probability treats the model as fixed and asks what data it would generate. Given that we believe in this particular model $\theta$, how likely is it that we’d observe outcome $x$?
Likelihood flips the question entirely. The data is already observed, and you’re asking which model best explains it. Given that the stock rose, how plausible is the model that assigned 70% to that outcome versus one that assigned 30%?
\[L(\theta \mid x) = P(x \mid \theta) \quad \leftarrow\ \text{same formula, } x \text{ is now fixed}\]Figure 1. The same expression P(x|θ) reads differently depending on what is fixed. Probability reasons from model to data; Likelihood reasons from data to model.
| Probability | Likelihood | |
|---|---|---|
| Fixed | Model $\theta$ | Observed data $x$ |
| Varies | Data $x$ | Parameters $\theta$ |
| Question | What data does this model predict? | Which model best explains this data? |
| Direction | Model → Data | Data → Model |
Same formula, opposite reading. This matters because training a deep learning model is a likelihood problem: we observe data and ask which parameters explain it best.
Maximum Likelihood Estimation
Suppose our quant model observes a sequence of daily market outcomes: up, up, down, up, down. We want to find the model parameters that make this sequence most probable. This is Maximum Likelihood Estimation.
For $n$ independent observations, the likelihood is
\[L(\theta) = \prod_{i=1}^{n} P(x_i \mid \theta)\]and MLE finds the parameters that maximize it
\[\hat{\theta}_{MLE} = \arg\max_\theta\ L(\theta)\]In practice, multiplying thousands of small probabilities underflows almost immediately, so we take the log first. Because log is monotonically increasing, the maximizer stays the same; we just work in a numerically safer space.
\[\log L(\theta) = \sum_{i=1}^{n} \log P(x_i \mid \theta)\]Deep learning frameworks are built around minimization, not maximization, so we flip the sign and minimize the Negative Log-Likelihood (NLL)
\[\text{NLL} = -\sum_{i=1}^{n} \log P(x_i \mid \theta)\]This NLL is where the story connects to Cross-Entropy. But to see exactly how, we need to take a detour through information theory.
2. Information Theory
Shannon’s Original Problem
In 1948, Claude Shannon was working on a practical engineering problem at Bell Labs: how much information can a noisy channel reliably carry? His answer, published as “A Mathematical Theory of Communication”, ended up applying far beyond telephone lines.
Figure 2. Shannon’s general communication system. The same framework describes a neural network estimating an unknown distribution from noisy observed data.
Shannon’s starting question sounds simple:
How much “information” does a message carry?
Self-Information: Why Rare Events Matter More
Think about financial news. “Markets open today” carries almost no information, because it happens every weekday. “Exchange trading halted due to systemic failure” carries a lot, precisely because it almost never happens.
We want a function $I(x)$ that satisfies three conditions.
Condition 1. Rare events carry more information than common ones.
\[P(x) \downarrow \implies I(x) \uparrow\]Condition 2. A certain event carries no information.
\[I(1) = 0\]Condition 3. Information from independent events should add up.
\[I(x,\ y) = I(x) + I(y)\]Condition 3 is the key one. Independent events multiply their probabilities, but we want their information to add. Among continuous functions, the only one that turns multiplication into addition is the logarithm. Adding Condition 1 gives the negative log
\[\boxed{I(x) = -\log P(x)}\]The minus sign confused me at first. $\log P(x)$ is always zero or negative because $P$ is between 0 and 1, so why negate it? Because we want information to increase as probability decreases, and $\log P(x)$ moves the other way. Flipping the sign makes rare events map to large positive values.
Figure 3. Self-information $I(x) = -\log P(x)$. As probability approaches zero, information grows without bound. Certain events carry zero information.
| Event | Probability | Information $-\log P$ |
|---|---|---|
| Major index closes within ±2% of the prior day | 0.95 | 0.05 |
| A stock moves more than 5% in a day | 0.10 | 2.30 |
| A blue-chip stock drops 30% in a single session | 0.001 | 6.91 |
Entropy: Average Surprise Across a Distribution
Self-information tells us how surprising a single event is. But in machine learning, we’re usually dealing with entire distributions. What we want is a measure of how unpredictable a distribution is on average.
\[H(X) = \mathbb{E}_{x \sim P}[I(x)] = -\sum_{x} P(x) \log P(x)\]This is a weighted average of self-information, with each event’s surprise weighted by how often it happens. A stock that always goes up has zero entropy. A stock whose daily direction is a coin flip has maximum entropy.
For a binary distribution with parameter $q$:
\[H(q) = -q \log q - (1-q)\log(1-q)\]Figure 4. Binary entropy $H(q)$ in nats (natural log, as everywhere else in this post). A model that is completely uncertain about tomorrow’s direction (q=0.5) has maximum entropy. Certainty in either direction brings entropy to zero.
| $q$ | Scenario | Entropy |
|---|---|---|
| 0.01 | Model is almost certain the stock will fall | Near 0 (very predictable) |
| 0.50 | Model has no idea | Maximum (completely uncertain) |
| 0.99 | Model is almost certain the stock will rise | Near 0 (very predictable) |
Low entropy doesn’t mean the model is right, only that it’s confident. A model can be confidently wrong, which is why the loss function has to punish confident mistakes harder than uncertain ones. Section 4 shows this concretely.
3. KL Divergence
Measuring the Gap Between Two Distributions
We now have tools to describe a single distribution’s uncertainty. But what we actually need in deep learning is a way to measure the distance between two distributions: the true label distribution $P$ and the model’s predicted distribution $Q$.
Entropy alone can’t do this, since it describes one distribution rather than the relationship between two. KL Divergence fills that gap.
\[D_{KL}(P \| Q) = \mathbb{E}_{x \sim P}\left[\log \frac{P(x)}{Q(x)}\right] = \sum_x P(x)\left[\log P(x) - \log Q(x)\right]\]Think of it as the extra cost incurred when a portfolio manager uses an approximate model $Q$ to make decisions about a market that actually follows $P$. When $Q = P$, there’s no extra cost. The further $Q$ drifts from $P$, the higher the cost.
Key Properties
| Property | Description |
|---|---|
| $D_{KL}(P | Q) \geq 0$ | Always non-negative |
| $D_{KL}(P | Q) = 0$ | If and only if $P = Q$ |
| $D_{KL}(P | Q) \neq D_{KL}(Q | P)$ | Asymmetric: order matters |
The asymmetry is easy to forget. KL Divergence is tempting to treat as a distance, but it isn’t one, because a distance metric has to be symmetric. $D_{KL}(P | Q)$ measures the cost of approximating $P$ with $Q$, and that is a different quantity from the reverse.
Forward KL vs Reverse KL
Figure 5. Forward KL (left) forces Q to cover every region where P has mass (mean-seeking). Reverse KL (right) lets Q collapse onto one mode and ignore the rest (mode-seeking).
Forward KL, $D_{KL}(P | Q)$, penalizes any region where $P > 0$ but $Q \approx 0$. To minimize it, $Q$ must put non-zero probability everywhere $P$ does. This leads to mean-seeking (mass-covering) behavior.
Reverse KL, $D_{KL}(Q | P)$, penalizes any region where $Q > 0$ but $P \approx 0$. The minimizer concentrates on one high-probability region of $P$ and ignores the rest. This leads to mode-seeking behavior.
In supervised classification, minimizing Cross-Entropy Loss is minimizing the Forward KL, so the model is pushed to cover every case where the true label has mass.
4. Cross-Entropy Loss
Where Everything Connects
Cross-Entropy is the sum of the true distribution’s entropy and the KL Divergence between them:
\[H(P, Q) = H(P) + D_{KL}(P \| Q)\]Expanding and simplifying:
\[H(P, Q) = -\mathbb{E}_{x \sim P}[\log P(x)] + \mathbb{E}_{x \sim P}[\log P(x) - \log Q(x)]\] \[\boxed{H(P, Q) = -\sum_x P(x) \log Q(x)}\]The $\log P(x)$ terms cancel, leaving only $Q$, the model’s prediction. That is what makes Cross-Entropy usable as a loss function: we never need to know $P$’s entropy.
Are Cross-Entropy and KL Divergence the same thing? No, but during training they are equivalent objectives. $H(P)$ is a constant (the true labels don’t change), so minimizing Cross-Entropy is the same as minimizing KL Divergence:
\[\min_\theta H(P, Q) = \underbrace{H(P)}_{\text{constant}} + \min_\theta D_{KL}(P \| Q)\]Minimizing Cross-Entropy = closing the gap between the model’s predictions and the true distribution.
One-Hot Labels and the Connection to NLL
In practice, classification labels are one-hot vectors. For a model predicting three market regimes (bull, bear, sideways), if the true state is “bull”:
\[P = [1,\ 0,\ 0]\]Plugging into the Cross-Entropy formula:
\[H(P, Q) = -(1 \cdot \log Q(\text{bull}) + 0 \cdot \log Q(\text{bear}) + 0 \cdot \log Q(\text{sideways}))\] \[= -\log Q(\text{bull})\]All the zero terms drop out, and only the log-probability of the correct class remains. That is exactly the NLL from Section 1. With one-hot labels, Cross-Entropy Loss and NLL are the same quantity, reached from two different starting points.
Softmax + Cross-Entropy
Raw model outputs (logits) are unconstrained real numbers. Softmax converts them into a valid probability distribution:
\[\text{Softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}\]Figure 6. The standard pipeline: raw logits → Softmax → probabilities → Cross-Entropy Loss.
Binary vs Categorical Cross-Entropy
Binary Cross-Entropy applies when the output is a single probability, for example whether a stock will rise or fall:
\[H = -\left[y \log q + (1-y)\log(1-q)\right]\]Categorical Cross-Entropy applies to multi-class problems such as bull, bear, or sideways:
\[H = -\sum_{c=1}^{C} y_c \log q_c\]Binary Cross-Entropy is simply the two-class special case of Categorical Cross-Entropy.
Why the Logarithm Matters
The shape of the log curve is what makes this work.
Figure 7. Cross-Entropy loss $-\log(q)$. The loss is near zero when the model correctly assigns high probability to the true class, but explodes when the model is confidently wrong.
| Predicted Probability | Loss $= -\log q$ | Verdict |
|---|---|---|
| 0.99 (confident and correct) | 0.01 | Good |
| 0.50 (uncertain) | 0.69 | |
| 0.10 (leaning wrong) | 2.30 | |
| 0.01 (confident and wrong) | 4.60 | Bad |
Take two models predicting a market crash. Model A gives it 51% and is wrong; Model B gives it 99% and is wrong. Both missed, but B should be punished much more for its false confidence. The log does exactly that: the loss grows faster and faster as confidence in the wrong answer increases.
This ties back to the information-theory view. A confident wrong prediction means the true outcome was very surprising under the model, so it carries a lot of information and gets a large loss.
5. References
- Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(3), 379–423.
- Kullback, S., & Leibler, R. A. (1951). On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1), 79–86.
- Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.
- Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. deeplearningbook.org
- Li, F.-F., Karpathy, A., et al. CS231n: Convolutional Neural Networks for Visual Recognition. Stanford University. cs231n.github.io
- Murphy, K. P. (2022). Probabilistic Machine Learning: An Introduction. MIT Press.