The Geometry of Attention
When a person reads the sentence “The trophy didn’t fit in the suitcase because it was too big,” they immediately know that “it” refers to the trophy, not the suitcase. This inference draws on context the meaning of nearby words shapes the interpretation of the ambiguous pronoun. For decades, building this kind of contextual sensitivity into neural networks was one of the central unsolved problems in natural language processing.
The self-attention mechanism, introduced in the 2017 paper “Attention Is All You Need,” solved this problem in an elegant and computationally powerful way. At the core of self-attention is a trio of concepts borrowed from information retrieval: the key, the query, and the value. Understanding this framework is not just a technical curiosity it is the conceptual foundation for how every modern large language model processes and understands language.
What the Key, Query, and Value Actually Represent
The terminology key, query, value is deliberately borrowed from the world of databases and search engines. In a traditional key-value store, you submit a query, the system looks up the matching key, and returns the associated value. Self-attention generalizes this idea dramatically: instead of looking for exact matches, it computes soft matches across every position in a sequence simultaneously.
Here is the intuition. When processing a token (say, a word or subword unit), the model generates three distinct vectors from that token’s representation:
The query vector represents what this token is looking for. It encodes the question: “What information do I need from the rest of the sequence to be understood correctly?”
The key vector represents what this token offers to others. It encodes the answer to: “What kind of information can I provide to other tokens searching for context?”
The value vector represents the actual content that will be passed along if this token is selected as relevant.
In mathematical terms, given an input matrix X (where each row is a token’s embedding), three learned weight matrices project the input into the query, key, and value spaces:
Q = X · W_Q (queries)
K = X · W_K (keys)
V = X · W_V (values)
The attention scores between all token pairs are then computed as:
Attention(Q, K, V) = softmax( Q · K^T / sqrt(d_k) ) · V
where d_k is the dimensionality of the key vectors
The three weight matrices W_Q, W_K, and W_V are learned during training. They are not the same matrix. This means the model learns three different projections of the same token: one for asking, one for advertising, and one for delivering. The separation of these roles is what gives attention its flexibility.
Walking Through an Example
Consider the sentence: “The bank by the river was steep.”
When the transformer processes the word “bank,” it needs to determine whether “bank” refers to a financial institution or a riverbank. Self-attention lets “bank” gather evidence from every other word to resolve this ambiguity.
Here is a simplified conceptual trace:
Token positions: ["The", "bank", "by", "the", "river", "was", "steep"]
For "bank" (position 1):
- Its query vector Q_bank asks: "Am I a financial or geographical entity?"
- It computes dot products with all key vectors
- After softmax, the attention distribution might look like:
The --> 0.03
bank --> 0.05
by --> 0.04
the --> 0.03
river --> 0.72 (high attention)
was --> 0.05
steep --> 0.08
- The output for "bank" is a weighted sum of all value vectors,
dominated by V_river and V_steep
No hard lookup or rule triggers this disambiguation. It emerges entirely from the learned geometry of the query-key dot products. The model has learned, through exposure to vast text, that “river” and “bank” co-occur in geographical contexts, so their query-key interactions produce high similarity scores.
Why the Scaling Factor Matters
The division by the square root of d_k — the scaling factor — deserves its own explanation because it has a non-obvious but critical effect.
When query and key vectors are high-dimensional (as they are in practice often 64 to 128 dimensions per head), their dot products can become very large in magnitude. If the dot products are large, the softmax function saturates: it assigns nearly all of the probability mass to the single highest-scoring token and near-zero to everything else. This is equivalent to a hard lookup, and it destroys the gradient signal needed for training the model can no longer learn to refine its attention patterns.
By scaling down by sqrt(d_k), the dot products are brought back into a range where the softmax remains sensitive and differentiable. Concretely, if query and key vectors have components with unit variance, their dot product has variance equal to d_k. Dividing by sqrt(d_k) restores unit variance to the scores.
Think of it this way, imagine scoring a set of candidates from 1 to 10, versus 1 to 1000. In the second case, small differences in the top score dominate everything else. The scaling factor is the mechanism that keeps the “scoring range” sensible.
The scaled dot-product design is not arbitrary. It is a principled solution to a training stability problem that arises directly from the geometry of high-dimensional vector spaces.
The Significance of Learned Projections
A critical property of the KQV framework one that is often glossed over is that the three weight matrices are learned independently and differently for each attention head in a multi-head attention layer. This means the model can learn different ways of “asking” and “advertising” information simultaneously, capturing different kinds of relationships in parallel.
For example, one attention head might learn to track syntactic dependencies (which word is the subject of which verb), while another tracks semantic co-reference (which pronoun refers to which noun), and a third might track positional proximity. None of these specializations are programmed in they emerge from the training objective.
This is fundamentally different from older architectures. In a recurrent neural network, information from distant positions is passed through a chain of intermediate states, and the signal degrades over distance. In the KQV attention framework, every token can directly query every other token, regardless of distance. The complexity is quadratic in sequence length O(n²) but the payoff is a fully connected information exchange at each layer.
The learned separation of Q, K, and V gives the model the ability to disentangle what to look for from what to offer from what to deliver. This three-way decomposition is what makes self-attention so expressive compared to simpler weighted averaging schemes.
The key-query-value framework is, at its core, a learned soft retrieval system. Every token in a sequence simultaneously acts as a searcher (query), an advertisement (key), and a content provider (value). Through the dot-product similarity, scaling, and softmax normalization, the model computes how much each token should inform the representation of every other token dynamically, in parallel, and without hard-coded rules.
This mechanism is why transformers can resolve ambiguous pronouns, maintain long-range dependencies, and build rich contextual representations from raw token sequences. It is also why attention weights have become a subject of interpretability research (they offer a window into which relationships the model has learned to treat as significant).
If you want to understand why large language models behave the way they do why context matters so profoundly, why they can track referents across long passages, and why they sometimes fail when context is noisy or ambiguous the key-query-value framework is the right place to start. Everything else in the transformer architecture is built on top of this one elegant idea.
Key, Query, and Value Framework was originally published in AI Evergreen on Medium, where people are continuing the conversation by highlighting and responding to this story.