What Attention Actually Computes
Attention is a soft, differentiable dictionary lookup—dot-product scores, a softmax, and a weighted sum of values—that gives every position a global receptive field in a single layer.
Read entryDeep Dives [7]
Extended pieces that follow one idea all the way down — from deep learning and probability to the structures the field is built on.
Attention is a soft, differentiable dictionary lookup—dot-product scores, a softmax, and a weighted sum of values—that gives every position a global receptive field in a single layer.
Read entryGradient descent trains almost every neural network by repeatedly taking a small step opposite the direction of steepest ascent — a procedure whose successes and pathologies both follow from that single geometric idea.
Read entryThe sample mean of many independent finite-variance draws becomes Gaussian, and its spread shrinks like σ/√n — a rate that follows from how variance adds, and that breaks precisely when its hypotheses do.
Read entryWhy fully deterministic systems—weather, pendulums, a tumbling die—can still defy long-range prediction, and how a positive Lyapunov exponent turns finite-precision measurement into vanishing knowledge of the future.
Read entryAny reasonable periodic function is a sum of rotating vectors, and the recipe for the radii is a sequence of inner products.
Read entryA sunflower head packs its seeds by placing each one a fixed angle of about 137.5° from the last, and that exact angle falls out of the golden ratio being the hardest number to approximate by fractions.
Read entryThe final abstraction layer of any intelligent system is limited to the language English. Why???
Read entry