Deep Learning

Attention is a soft, differentiable dictionary lookup—dot-product scores, a softmax, and a weighted sum of values—that gives every position a global receptive field in a single layer.

Gradient descent trains almost every neural network by repeatedly taking a small step opposite the direction of steepest ascent — a procedure whose successes and pathologies both follow from that single geometric idea.