Walking Downhill: What Gradient Descent Actually Does

Imagine you are dropped, blindfolded, somewhere on a vast hillside in fog. You cannot see the valley floor, but you can feel the slope under your feet. A sensible plan is to feel which way the ground tilts most steeply, take a short step in the downhill direction, and repeat. Do this faithfully and you will, eventually, end up somewhere low. This is, almost literally, how every large neural network alive today is trained.

The hillside is a loss surface: a function $L(\theta)$ that assigns to every setting of the parameters $\theta$ a single number measuring how badly the model fits the data. Training is the search for a point where $L$ is small. The procedure that does the searching — gradient descent — is the subject of Plate VIII, and it is worth understanding not as a black box but as a precise consequence of one fact from calculus.

The gradient points uphill

At a point $\theta$ , the gradient $\nabla L(\theta)$ is the vector of partial derivatives $\left(\partial L/\partial \theta_1, \dots, \partial L/\partial \theta_n\right)$ . Its defining property is directional. If you move a tiny step $\epsilon$ in a unit direction $u$ , the change in loss is, to first order,

$L(\theta + \epsilon u) - L(\theta) \approx \epsilon\, \nabla L(\theta) \cdot u.$

The dot product $\nabla L \cdot u = \lVert \nabla L \rVert \cos\alpha$ is largest when $\alpha = 0$ , i.e. when $u$ points along the gradient.This is exactly why the gradient is the natural object: among all directions, it answers “which way increases the function fastest, and how fast?” in a single stroke. So the gradient is the direction of steepest ascent. To go downhill as fast as possible, we walk in the opposite direction, $-\nabla L$ . That single sign flip is the whole reason for the minus in the update rule:

$\theta \leftarrow \theta - \eta\, \nabla L(\theta).$

Each iteration evaluates the slope where we currently stand, then takes a step against it. The scalar $\eta > 0$ is the step size, called the learning rate.

The learning rate is the size of your stride

The linear approximation above is only trustworthy nearby. The gradient tells you the slope at one point; take too large a step and the terrain may have changed completely by the time you land. This is the central tension in choosing $\eta$ .

If $\eta$ is too small, every step barely moves you, and reaching the valley takes an enormous number of iterations — the optimizer crawls. If $\eta$ is too large, you can overshoot the minimum and land on the far wall higher than you started; the next step overshoots again, and the loss oscillates outward and diverges. There is a useful exact picture for a quadratic bowl $L(\theta) = \tfrac{1}{2} a\theta^2$ : the update becomes $\theta \leftarrow (1 - \eta a)\theta$ , which converges only when $|1 - \eta a| < 1$ , that is $0 < \eta < 2/a$ . Below $1/a$ you approach monotonically; between $1/a$ and $2/a$ you approach while bouncing across the minimum; above $2/a$ you fly apart.For a general loss, $a$ is replaced by the largest curvature (top eigenvalue of the Hessian), which is why the safe learning rate is governed by the steepest direction even if most directions are flat. The right stride is large enough to make progress, small enough to respect the curvature.

Convex valleys and everything else

When the loss is convex — a single bowl with no false bottoms — gradient descent is well behaved: any point where $\nabla L = 0$ is the global minimum, and with a sensible learning rate the method provably converges to it. Convexity is the friendly case, and much of classical optimization theory lives there.

Neural network losses are emphatically non-convex. The surface is riddled with features where the gradient vanishes but the point is not a global minimum. A local minimum is a basin lower than its surroundings but higher than the true floor. A saddle point is more insidious: the surface curves up along some directions and down along others, like a mountain pass, so $\nabla L = 0$ even though you are not at any minimum at all. In high dimensions saddles vastly outnumber local minima, and a naive optimizer can stall near one because the gradient there is tiny in every direction. Part of the art of modern training is escaping these flat regions rather than settling into them.

Why stochastic, why mini-batches

Computing $\nabla L$ exactly means summing the gradient contribution of every training example — millions or billions of them — for a single step. That is ruinously expensive. Stochastic gradient descent (SGD) replaces the full gradient with an estimate computed from a small random mini-batch of examples. Each estimate is noisy, but it is unbiased: on average it points the same way as the true gradient, and it is hundreds of times cheaper to compute.

The trade is overwhelmingly worth it. With a mini-batch you take many approximate steps in the time one exact step would cost, and many rough steps beat one perfect step. The noise is not purely a cost, either: the random jitter can shake the optimizer out of sharp local minima and off saddle points, acting as a mild, free form of regularization. The gradients themselves, for each example, are produced by backpropagation, which is just the chain rule applied efficiently across the network’s layers.

Momentum and the zig-zag valley

Plain gradient descent has a famous failure mode, visible whenever the valley is long, narrow, and curved — the classic test case is the Rosenbrock function, whose minimum lies at the bottom of a banana-shaped trough. Across the narrow trough the walls are steep; along the trough the floor descends gently. The gradient is dominated by the steep cross-direction, so each step lurches across the valley toward the opposite wall, with only a sliver of progress along it. The path becomes a slow zig-zag, ricocheting between walls while inching toward the goal.This is the geometric face of ill-conditioning: the curvature differs wildly between directions, so no single learning rate suits both at once.

Momentum fixes much of this. Instead of stepping along the raw gradient, accumulate a running velocity:

$v \leftarrow \beta v - \eta\, \nabla L(\theta), \qquad \theta \leftarrow \theta + v.$

The velocity is an exponential average of past gradients. In the cross-direction the gradient flips sign each step, so successive contributions cancel and the oscillation is damped. Along the valley the gradient points consistently the same way, so contributions reinforce and the optimizer accelerates, like a ball rolling downhill that builds speed in the direction of persistent descent. Adaptive methods such as Adam extend this idea by also rescaling each coordinate by a running estimate of its gradient magnitude, so flat and steep directions get effectively different learning rates.

Everything here flows from the opening image. The gradient feels the steepest tilt; the minus sign turns ascent into descent; the learning rate sets the stride; non-convexity scatters traps across the slope; sampling makes each step affordable; and momentum smooths the walk so a curved valley no longer sends you bouncing wall to wall.

The gradient points uphill

The learning rate is the size of your stride

Convex valleys and everything else

Why stochastic, why mini-batches

Momentum and the zig-zag valley

Further reading