What Attention Actually Computes
Attention is a soft, differentiable dictionary lookup—dot-product scores, a softmax, and a weighted sum of values—that gives every position a global receptive field in a single layer.
Read entryI study deep learning and probability through notes, small experiments, and interactive visualizations that make abstract structure easier to inspect.
dx/dt = sigma(y - x)
dy/dt = x(rho - z) - y
dz/dt = xy - beta z
A live field of primes in polar coordinates, built to make the hidden diagonal arms feel physical.
Polar / Ulam / Sacks mappingsA strange attractor as a slow ribbon of instability, where tiny shifts in start become visible structure.
3D trajectory / sensitivityDraw a digit and watch the signal move through a compact network, layer by layer, until it becomes a guess.
Inference / live activationsA chain of rotating circles summing into a square, sawtooth, or triangle wave — add harmonics and the corners sharpen.
Epicycles / harmonicsFollow the slope of a Rosenbrock valley to its floor; drop the ball anywhere and watch the classic zig-zag.
Steepest descent / momentumAverage n values from a lopsided source, repeat, and watch the means pile into a Gaussian — whatever the source.
Sampling / convergenceThe boundary of bounded orbits of z ↦ z² + c, rendered as an ink mezzotint you can zoom into.
Escape-time / zoomOne local rule over a row of cells; 256 possible worlds, from Sierpiński order to genuine chaos.
Elementary rules 0–255Seven hundred verses arranged in semantic space, exposing thematic neighborhoods and philosophical adjacency.
Embeddings / 3D semantic mapjinxed myself, got a fracture, was on the verge of a displacement fracture but avoided the worst, and now I am back to my hometown. everything that seemed to be going great — being physically active, enjoying my work, and planning for an upcoming trip — feels in vain. now I need to take a step back, revise my current scenario, and move accordingly.
life’s is awesome lately, had an awesome workout session, took steam and sauna bath, enjoyed one of the best dosa place in the city and now back to trying a new hot chocolate recipe (let’s see if cinnamon adds any cherry on top) while having elon and nikhil kamath by my side :)
Attention is a soft, differentiable dictionary lookup—dot-product scores, a softmax, and a weighted sum of values—that gives every position a global receptive field in a single layer.
Read entryGradient descent trains almost every neural network by repeatedly taking a small step opposite the direction of steepest ascent — a procedure whose successes and pathologies both follow from that single geometric idea.
Read entryThe sample mean of many independent finite-variance draws becomes Gaussian, and its spread shrinks like σ/√n — a rate that follows from how variance adds, and that breaks precisely when its hypotheses do.
Read entryWhy fully deterministic systems—weather, pendulums, a tumbling die—can still defy long-range prediction, and how a positive Lyapunov exponent turns finite-precision measurement into vanishing knowledge of the future.
Read entryAny reasonable periodic function is a sum of rotating vectors, and the recipe for the radii is a sequence of inner products.
Read entryA sunflower head packs its seeds by placing each one a fixed angle of about 137.5° from the last, and that exact angle falls out of the golden ratio being the hardest number to approximate by fractions.
Read entryThe final abstraction layer of any intelligent system is limited to the language English. Why???
Read entry