Attention

The mechanism is a lookup that's soft instead of exact. Think of a library where you ask a question:

• Your query (Q): what you're looking for — "I'm the word 'it'; who could I be referring to?"
• Every word's key (K): its label on the shelf — "I'm 'ball', a small physical object."
• Every word's value (V): the actual content it will contribute if selected.

In a normal lookup you'd match the query to exactly one key and grab one value. Attention instead scores the query against every key, converts the scores into percentages (softmax), and takes a weighted blend of all the values — maybe 85% of "ball", 10% of "robot", 5% of everything else. Blending instead of picking is what makes the whole thing differentiable, so it can be learned by gradient descent.

Do this for every word simultaneously, with several independent "heads" each learning to look for different things (one head tracks pronoun references, another tracks syntax, another tracks nearby adjectives…), and you have multi-head self-attention — the core layer of a transformer.

Where you actually meet attention in practice:

• Every token, every layer. A model like the ones behind ChatGPT or Claude runs attention dozens of times (layers) with dozens of heads per layer, for every single token it reads or writes.
• The O(n²) cost is the context-window bottleneck. Doubling your prompt length roughly quadruples attention compute. This is why long documents cost more and why the context window is a headline spec.
• KV cache. During generation, the keys and values of all previous tokens are stored so they aren't recomputed for every new token — this cache is the main memory hog at inference time (see KV Cache).
• FlashAttention and similar kernels compute exact attention much faster by being clever about GPU memory movement — a good example that hardware-aware software matters as much as the math.
• Attention maps are a debugging window. You can visualize which words a head attends to — one of the few places you can literally look inside a model and see interpretable structure.

Watch the whole pipeline run once — then the prose below will feel obvious:

Each token's embedding is projected through three learned weight matrices into a query q, key k, and value v vector. For token i:

1. Score: compute the dot product of qᵢ with every kⱼ — one relevance number per token pair.
2. Scale: divide by √d (d = key dimension) so scores don't blow up as dimensions grow, which would saturate the softmax into hard one-hot picks and kill gradients.
3. Normalize: softmax the scores across j — now they're positive weights summing to 1.
4. Blend: output for token i = the weighted sum Σ softmax-weightⱼ · vⱼ.

In self-attention, queries, keys, and values all come from the same sequence. In cross-attention (encoder–decoder models, and how image generators condition on your text prompt), queries come from one sequence and keys/values from another. In causal (masked) attention — used by all GPT-style decoder models — token i is only allowed to attend to tokens ≤ i, so the model can't peek at the future it's supposed to predict.

Multi-head: instead of one attention with big vectors, split into h heads with smaller vectors, run them independently, concatenate the results, and mix with one more learned matrix. Heads specialize; the model gets several simultaneous "kinds of looking."

• Every frontier LLM (GPT, Claude, Gemini, Llama, DeepSeek…) is a stack of attention layers — this is not one technique among many, it is the architecture.
• Beyond text: Vision Transformers treat image patches as tokens; Whisper does it for audio; AlphaFold used attention over amino-acid pairs to crack protein folding; image/video generators use cross-attention to bind your prompt words to image regions.
• Serving economics: attention's compute and KV-cache memory drive the actual cost-per-token you pay from an API, and are why prompt caching and context-window pricing tiers exist.
• Variants in production now: grouped-query attention (GQA) and multi-head latent attention (MLA) shrink the KV cache; sliding-window attention bounds cost on long inputs; FlashAttention-style kernels are standard everywhere.

The O(n²) wall is the active battlefront:

• Sub-quadratic architectures — state-space models (Mamba) and linear-attention families process sequences in linear time; current frontier practice is hybrids that keep a few full-attention layers and replace the rest.
• Effectively-infinite context — million-token windows via ring attention, compression, and retrieval; the research question is not just fitting long context but actually using it (models still miss things buried in the middle).
• Interpretability — attention heads are a main object of study in reverse-engineering what models compute (circuit analysis: induction heads, etc.).
• Hardware co-design — chips increasingly shaped around attention's memory-access pattern; the KV cache is a first-class hardware design constraint now.

Here's the whole machine, drawn — and where this page's concept sits inside it:

One sentence of history: before 2017, models read text in order, one word at a time, passing a memory forward — and long-range meaning kept washing out. The transformer's bet, right in its title ("Attention Is All You Need"), was to throw away the in-order reading entirely and let every word look directly at every other word.

That one swap did two things at once: it made models better at language (constant path between any two words) and — the underrated part — made training massively parallelizable, so you could throw thousands of GPUs at it. Parallelism unlocked scale, scale revealed scaling laws, scaling laws justified the giant training runs, and the whole modern AI era followed.

So when you read any AI article, here's the mental anchor: attention is the engine; almost everything else you'll read about — KV caches, context windows, quantization, FlashAttention, Mamba, inference costs — is either feeding that engine, speeding it up, or trying to replace it.