Walking Downhill: What Gradient Descent Actually Does
Gradient descent trains almost every neural network by repeatedly taking a small step opposite the direction of steepest ascent — a procedure whose successes and pathologies both follow from that single geometric idea.
Read entry