Deep Learning Optimization Part 2: Backpropagation, Gradient Descent, and Modern Optimizers
In Part 1, we derived Cross-Entropy Loss from first principles. The next question is how to actually minimize it. This post walks through the training loop step by step: Forward Propagation, Backpropagation, Gradient Descent, modern Optimizers, and the skip-connection, the architectural change that made very deep networks practical to train.
Table of Contents
- 0. The Problem We’re Trying to Solve
- 1. Loss Landscape
- 2. Forward and Backpropagation
- 3. Gradient Descent and SGD
- 4. Optimizers
- 5. Skip-Connection and ResNet
- 6. References
0. The Problem We’re Trying to Solve
In Part 1, we worked through how a neural network measures its own error. Starting from Shannon’s information theory, we derived Cross-Entropy Loss, a principled way to quantify the gap between what a model predicts and what actually happens.
But defining the gap is only half the problem.
A quant model that knows it’s wrong by 4.6 nats still needs to figure out what to do about it. Which parameters are responsible for the error? By how much should each one change? In which direction?
This is the optimization problem, and it is harder than the update formula makes it look.
The Loss Landscape (the surface that maps every possible set of parameters to a loss value) is rarely a smooth bowl with one obvious minimum. It looks more like a mountain range, with valleys, ridges, flat plateaus, and shallow dips that look like the bottom but aren’t. Gradient Descent tells us which way to step, Backpropagation makes computing that direction affordable, and modern Optimizers are the refinements that make the whole process reliable at scale.
This post covers all three, in the order they happen during training.
1. Loss Landscape
What the Loss Surface Actually Looks Like
Every set of model parameters $\theta$ maps to a scalar loss value $\mathcal{L}(\theta)$. Even a rough picture of this mapping shows why optimization is non-trivial.
Figure 1. Three obstacles in the loss landscape: local minima (trapped at suboptimal solutions), saddle points (gradient vanishes), and plateaus (no useful gradient signal).
These obstacles explain most of what follows: why we need Backpropagation, why vanilla Gradient Descent isn’t enough, and why modern optimizers are built the way they are.
2. Forward and Backpropagation
The Training Loop
Before diving into the mechanics, it helps to see the full training loop in one place.
Figure 2. The training loop. Forward propagation produces a prediction; loss computation measures the error; backpropagation computes gradients; parameter update applies them.
Forward Propagation
Forward propagation is the straightforward part. Input data flows through each layer sequentially, with each layer applying a linear transformation followed by a nonlinear activation:
\[a^{(l)} = \sigma\left(W^{(l)} a^{(l-1)} + b^{(l)}\right)\]For a stock prediction model, raw market features (price history, volume, volatility) enter as $x$, pass through hidden layers that learn increasingly abstract representations, and come out of the Softmax as a probability distribution over up, down, or sideways.
The output of forward propagation is a prediction. The gap between that prediction and the true label is the Cross-Entropy Loss we computed in Part 1. The question is now: which parameters caused the error, and by how much?
Backpropagation and the Chain Rule
Backpropagation answers this question by computing the gradient of the loss with respect to every parameter in the network. With hundreds of millions of parameters this sounds expensive, but the Chain Rule makes it tractable.
The Core Idea
Consider a simplified two-layer computation:
\[x \xrightarrow{f} u \xrightarrow{g} \mathcal{L}\]We want $\frac{\partial \mathcal{L}}{\partial x}$, but there’s no direct path. The Chain Rule gives us:
\[\frac{\partial \mathcal{L}}{\partial x} = \frac{\partial \mathcal{L}}{\partial u} \cdot \frac{\partial u}{\partial x}\]The gradient flows backwards through each computation, from the loss to the output, through the hidden layers, and down to the input weights, multiplying local gradients at each step.
Node-Level Backpropagation
Every operation in a neural network is a node. Each node receives a gradient from the right (from the loss), computes its local gradient, and passes the result to the left. The type of operation determines how the gradient is transformed.
Figure 3. Backpropagation through a simple computation graph. Forward pass computes values left-to-right; backward pass propagates gradients right-to-left using local derivatives.
Two local gradient rules cover most of what happens in practice:
Addition node. The incoming gradient passes through unchanged to both inputs. If the loss gradient arriving at the addition node is $-4$, both $x$ and $y$ receive $-4$.
Multiplication node. The incoming gradient is scaled by the other input’s value. In $q = x \cdot z$, the gradient flowing to $x$ is multiplied by $z$, and vice versa.
Why This Matters at Scale
In a deep network with $L$ layers, the gradient for parameters in layer 1 involves a product of $L$ local gradients:
\[\frac{\partial \mathcal{L}}{\partial W^{(1)}} = \frac{\partial \mathcal{L}}{\partial a^{(L)}} \cdot \frac{\partial a^{(L)}}{\partial a^{(L-1)}} \cdots \frac{\partial a^{(2)}}{\partial a^{(1)}} \cdot \frac{\partial a^{(1)}}{\partial W^{(1)}}\]Each factor is a local gradient. If those factors are consistently smaller than 1 (common with saturating activations like sigmoid and tanh), the product shrinks exponentially with depth. This is the Vanishing Gradient problem. Section 5 looks at a related but different issue in very deep networks.
3. Gradient Descent and SGD
The Update Rule
Once we have gradients, the parameter update is straightforward. Move each parameter in the direction that reduces loss:
\[\theta_{t+1} = \theta_t - \alpha \cdot \nabla_\theta \mathcal{L}(\theta_t)\]where $\alpha$ is the learning rate, the step size at each iteration.
Figure 4. One gradient descent step. The gradient points uphill; we move in the opposite direction by a distance proportional to the learning rate α.
The Learning Rate Problem
The learning rate $\alpha$ matters more than it looks. Set it too small, and training takes forever or stalls on a plateau. Set it too large, and updates overshoot the minimum, so the loss oscillates or diverges.
Figure 5. The effect of learning rate. Too small: convergence is painfully slow. Just right: efficient descent. Too large: updates overshoot, causing oscillation or divergence.
Batch GD vs SGD vs Mini-Batch
The update rule raises a practical question: which data do we use to compute the gradient?
Batch Gradient Descent computes the gradient over the entire dataset before taking a single step. For a market model trained on ten years of daily returns, that means processing 2,500+ data points before updating any weight. The gradient is accurate, but each step is slow, and the memory requirement can be prohibitive.
Stochastic Gradient Descent (SGD) computes the gradient from a single randomly selected data point and updates immediately. It is fast and memory-efficient, but the gradient estimate is noisy: one trading day might say little about the broader market regime.
Mini-Batch Gradient Descent, which is what most practitioners mean when they say “SGD”, splits the data into small batches (typically 32–512 samples) and updates once per batch. This balances estimation accuracy against speed, and the noise from batch sampling has a useful side effect: it can help the optimizer escape shallow local minima.
| Method | Data per Update | Gradient Quality | Practical Use |
|---|---|---|---|
| Batch GD | Full dataset | Exact | Rarely (too slow) |
| SGD | 1 sample | Very noisy | Rarely (too unstable) |
| Mini-Batch GD | 32–512 samples | Good approximation | Standard practice |
4. Optimizers
Why Vanilla SGD Isn’t Enough
Vanilla SGD has three well-known failure modes that become apparent when training deep models on complex data.
Slow convergence in flat regions. When the loss surface is nearly flat, gradients are small, and updates are tiny regardless of how far we are from the minimum.
Oscillation in narrow valleys. When the loss surface curves sharply in one direction and gently in another, which is common in real networks, SGD bounces back and forth across the steep axis while making slow progress along the gentle one.
Sensitivity to learning rate. A fixed learning rate that works well early in training often becomes too large as the model approaches a minimum. Manually tuning a learning rate schedule is tedious and problem-specific.
Each optimizer below addresses one or more of these problems.
Momentum
Momentum adds a velocity term, an exponentially decaying average of past gradients, to smooth out the update trajectory.
\(v_{t+1} = \beta v_t + \alpha \nabla_\theta \mathcal{L}(\theta_t)\) \(\theta_{t+1} = \theta_t - v_{t+1}\)
The hyperparameter $\beta$ (typically 0.9) controls how much of the previous velocity is retained. Think of it as a ball rolling down a hill: it accumulates speed in consistent directions and resists being deflected by noisy gradients.
Figure 6. SGD (left) oscillates across the narrow axis of an elliptical loss surface. Momentum (right) accumulates velocity in the consistent direction, producing a smoother and faster path to the minimum.
For a stock prediction model trained on multivariate time series, momentum is particularly valuable: market features often have very different scales and correlations, producing exactly the kind of anisotropic loss surface where SGD oscillates badly.
Nesterov Accelerated Gradient (NAG)
NAG is a refinement of Momentum with one key modification: instead of computing the gradient at the current position, it computes the gradient at the anticipated next position.
\(v_{t+1} = \beta v_t + \alpha \nabla_\theta \mathcal{L}(\theta_t - \beta v_t)\) \(\theta_{t+1} = \theta_t - v_{t+1}\)
The term $\theta_t - \beta v_t$ is a lookahead: where would we be if we applied the current velocity? By computing the gradient there, NAG corrects its course before overshooting rather than after. Nesterov’s guarantee of faster convergence holds for convex problems; in deep learning the gain over standard Momentum is usually modest and not guaranteed.
Adagrad
Momentum adapts the trajectory but uses the same learning rate for all parameters. Adagrad takes a different approach: it adapts the learning rate per parameter, based on the history of its gradients.
\(G_t = G_{t-1} + \left(\nabla_\theta \mathcal{L}\right)^2\) \(\theta_{t+1} = \theta_t - \frac{\alpha}{\sqrt{G_t + \epsilon}} \cdot \nabla_\theta \mathcal{L}\)
Parameters that receive large gradients frequently (common features) get smaller learning rates. Parameters that receive small gradients rarely (rare features) get larger learning rates. For a market model, this means high-frequency signals like daily volume naturally receive smaller updates than rare regime indicators.
The problem with Adagrad is that $G_t$ only grows over time. The learning rate shrinks monotonically and can eventually become so small that the model effectively stops learning.
RMSprop
RMSprop fixes Adagrad’s accumulation problem by using an exponentially decaying average of squared gradients rather than a cumulative sum:
\(E[g^2]_t = \gamma E[g^2]_{t-1} + (1-\gamma)\left(\nabla_\theta \mathcal{L}\right)^2\) \(\theta_{t+1} = \theta_t - \frac{\alpha}{\sqrt{E[g^2]_t + \epsilon}} \cdot \nabla_\theta \mathcal{L}\)
The decay factor $\gamma$ (typically 0.9) ensures that old gradients are forgotten gradually. The learning rate remains adaptive but doesn’t collapse over time.
Adam
Adam (Adaptive Moment Estimation) combines the ideas behind Momentum and RMSprop. It keeps a first moment estimate (a running mean of gradients, like Momentum) and a second moment estimate (a running mean of squared gradients, like RMSprop; this is the uncentered second moment, not the variance):
\(m_t = \beta_1 m_{t-1} + (1 - \beta_1) \nabla_\theta \mathcal{L}\) \(v_t = \beta_2 v_{t-1} + (1 - \beta_2) \left(\nabla_\theta \mathcal{L}\right)^2\)
Both estimates are biased toward zero early in training (since they’re initialized at zero), so Adam applies bias correction:
\[\hat{m}_t = \frac{m_t}{1 - \beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}\]The final update:
\[\theta_{t+1} = \theta_t - \frac{\alpha}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t\]Default values of $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon = 10^{-8}$ work well across a wide range of problems, which is a big part of why Adam became the default optimizer for most deep learning work.
Figure 7. Conceptual convergence comparison. Adam typically reaches low loss faster than SGD or Momentum, thanks to adaptive learning rates and momentum combined.
| Optimizer | Adapts LR? | Uses Momentum? | Key Limitation |
|---|---|---|---|
| SGD | No | No | Slow, sensitive to LR |
| Momentum | No | Yes | Fixed LR for all params |
| NAG | No | Yes (lookahead) | Same as Momentum |
| Adagrad | Yes | No | LR decays to zero |
| RMSprop | Yes | No | No first moment |
| Adam | Yes | Yes | Can generalize worse than SGD; L2 penalty couples with the adaptive LR (fixed by AdamW) |
5. Skip-Connection and ResNet
When Depth Becomes a Liability
Stacking more layers should, in principle, let a network learn more complex representations. In practice, plain networks beyond a certain depth get worse, and not only on test data: their training error goes up too.
He et al. (2016) showed this directly: on CIFAR-10, a 56-layer plain network had higher training and test error than a 20-layer one. Since training error also rose, this isn’t overfitting. The authors called it the degradation problem.
A natural first guess is vanishing gradients. Recall from Section 2 that the gradient for early layers is a product of $L$ local gradients. If each factor were around 0.9, a 56-layer network would give
\[0.9^{56} \approx 0.0027\]which is a shrinkage of roughly 365×. But He et al. argue that this is not the main cause in their experiments. Their plain networks used Batch Normalization, and they checked that forward and backward signals stayed healthy. The deeper network is simply harder to optimize: in theory it could copy the 20-layer solution and set the extra layers to the identity, but stacked nonlinear layers struggle to learn an identity mapping.
Figure 8. Conceptual picture of gradient flow across layers. In a plain deep stack the backward signal can weaken toward early layers; an identity shortcut gives the gradient a direct path. (Illustrative, not measured data.)
The Skip-Connection Solution
ResNet’s fix is simple: add a direct path, a skip-connection or identity shortcut, that carries the input $x$ around each block of layers and adds it to the block’s output.
\[\text{output} = \mathcal{F}(x) + x\]Figure 9. Plain network (left) vs ResNet block (right). The identity shortcut bypasses the weight layers and adds x directly to F(x), so gradients always have a direct path backward.
Why It Works
The gradient of the skip-connection output with respect to the input is:
\[\frac{\partial}{\partial x}\left(\mathcal{F}(x) + x\right) = \frac{\partial \mathcal{F}(x)}{\partial x} + 1\]The $+1$ term (in matrix form, the identity matrix $I$ added to the Jacobian) means the gradient always has a path that does not pass through the block’s weights. This does not mathematically guarantee that the gradient is at least 1, since $\frac{\partial \mathcal{F}}{\partial x}$ can be negative or cancel part of it. In practice, though, this shortcut keeps the backward signal usable even in networks with hundreds of layers.
The name “residual” comes from reframing what the block learns. Instead of learning the full transformation from input to output, the block learns the residual, the difference between the desired output and the identity:
\[\mathcal{F}(x) = \text{desired output} - x\]If the optimal mapping for a block is close to the identity, pushing $\mathcal{F}(x)$ toward zero is much easier than learning the identity from scratch with a stack of nonlinear layers. That is exactly the case the degradation problem exposed.
Effect on the Loss Landscape
Skip-connections also change the shape of the loss landscape. Li et al. (2018) visualized this: a 56-layer network without skip-connections has a chaotic, jagged surface with many sharp local minima, while the same network with skip-connections (ResNet-56) is much smoother. Denser connections (as in DenseNet) smooth it further.
This matters for the optimizers from Section 4, which work best on smooth, well-conditioned surfaces. Skip-connections turn the loss landscape into the kind of terrain gradient-based optimizers were designed for.
6. References
- Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536.
- Kingma, D. P., & Ba, J. (2014). Adam: A Method for Stochastic Optimization. ICLR 2015.
- He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep Residual Learning for Image Recognition. CVPR 2016.
- Duchi, J., Hazan, E., & Singer, Y. (2011). Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. JMLR.
- Polyak, B. T. (1964). Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 4(5), 1–17.
- Tieleman, T., & Hinton, G. (2012). Lecture 6.5, RMSProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning.
- Loshchilov, I., & Hutter, F. (2019). Decoupled Weight Decay Regularization. ICLR 2019.
- Nesterov, Y. (1983). A method for solving the convex programming problem with convergence rate O(1/k²). Soviet Mathematics Doklady.
- Li, H. et al. (2018). Visualizing the Loss Landscape of Neural Nets. NeurIPS 2018.
- Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. deeplearningbook.org
- Li, F.-F., Karpathy, A., et al. CS231n: Convolutional Neural Networks for Visual Recognition. Stanford University. cs231n.github.io