Attention

Attention is a soft, differentiable dictionary lookup—dot-product scores, a softmax, and a weighted sum of values—that gives every position a global receptive field in a single layer.