LLMS & GENERATIVE AI · QUESTION 13
How does self-attention work in a Transformer?
On this page
Self-attention lets each token build a representation using other tokens in the same input. The model creates query, key, and value vectors, scores how relevant each token is to the others, and combines their values using those scores. Multiple attention heads learn different relationships in parallel. In a decoder-only language model, a causal mask prevents a token from looking at future tokens.
LET'S MAKE IT SIMPLE
First, the idea in plain English
A word often gets its meaning from the words around it. In “the server returned an error because it was overloaded,” a model needs to connect “it” with “server.” Self-attention gives each token a way to weigh information from other tokens in the input and update its representation. The same mechanism runs repeatedly through Transformer layers, helping the model represent context rather than treating every word in isolation.
A USEFUL COMPARISON
Picture it this way
Imagine each person in a meeting asking, “Who has information relevant to my part of the discussion?” They listen more closely to some contributions than others, then update what they think. Attention works with learned numeric vectors rather than people consciously deciding what matters, and every token’s scores are computed by the model.
FOLLOW THE JOURNEY
Each token gathers context through attention
Follow how token representations produce relevance weights and a context-aware output.
For each token, the query is compared with all visible keys. The resulting weights mix the corresponding values. A causal mask removes future positions during autoregressive generation.
Input tokens create query, key, and value vectors; query-key scores are normalized into weights that combine value vectors into contextual token representations.
How to read this visual
For each token, the query is compared with all visible keys. The resulting weights mix the corresponding values. A causal mask removes future positions during autoregressive generation.
UNDER THE HOOD
How it works, step by step
Create three views
Project each token representation into a query, key, and value.
Why it matters: These learned views let the model compare context and carry useful information separately.
Score relationships
Compare a token’s query with other tokens’ keys, then scale the scores.
Why it matters: The scores estimate which positions can contribute relevant context.
Normalize and combine
Apply softmax to obtain weights and use them to combine value vectors.
Why it matters: The output becomes a weighted summary of visible context.
Repeat across heads and layers
Run several learned attention heads and process their outputs through more layers.
Why it matters: Different heads and layers can represent different kinds of relationships.
OUTSIDE THE TEXTBOOK
A practical example
In “Maya gave Priya her notebook,” the pronoun “her” could refer to either person. Attention gives the representation at “her” a way to incorporate signals from the rest of the sentence. The model’s final interpretation is learned and context-dependent; attention weights alone are not a complete explanation of the model’s reasoning.
GO A LITTLE DEEPER
Technical details that matter
- Scaled dot-product attention divides query-key scores by the square root of the key dimension before softmax, helping keep scores at a useful scale.
- Multi-head attention applies separate learned projections, allowing different heads to capture different patterns before their outputs are combined.
- Self-attention over a sequence has pairwise interactions, so its compute and memory can grow quadratically with sequence length in standard implementations.
- Causal masking is used for next-token generation; encoder-style tasks can allow each token to attend to both earlier and later input positions.
MAKE A GOOD CHOICE
When is this useful?
- Building context-sensitive token representations in language models.
- Connecting distant parts of a sequence when a task depends on their relationship.
- Understanding the central operation in Transformer-based language and multimodal models.
Recurrent networks process sequences step by step and can be useful when streaming or compact state is important. Convolutional sequence models use local windows. The right architecture depends on sequence length, latency, available compute, and the task.
WATCH OUT FOR
Common misunderstandings
- Treating attention weights as a definitive explanation of a model’s decision.
- Assuming every Transformer uses the same masking pattern.
- Describing attention as simply “remembering” earlier words without explaining the learned weighted combination.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Standard full attention can become expensive for long contexts.
- Attention patterns are learned correlations and do not guarantee correct reasoning or factual output.
- The model’s positional information and masking choices affect which relationships it can represent.
WHAT TO REMEMBER
Key takeaways
- 1Self-attention builds context-aware token representations.
- 2Queries and keys determine weights; values supply the information being combined.
- 3Multiple heads and layers learn different representations.
- 4Causal masks prevent a generated token from seeing the future.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
What are query, key, and value vectors?
They are learned projections of token representations. Queries and keys determine how strongly positions relate, while values are combined according to those weights.
Why use multiple attention heads?
Different heads can learn different projections and relationship patterns in parallel. The outputs are combined into the next representation.
Why is a causal mask needed?
When predicting the next token, training must not reveal later tokens. The mask blocks those positions so the model learns the same information flow it will use at generation time.
SAY IT OUT LOUD
How to explain this in an interview
“Self-attention lets each token use relevant context from other tokens. A Transformer projects token representations into queries, keys, and values. It compares queries with keys to get attention weights, then uses those weights to combine values into contextual representations. Multiple heads can learn different relationships. For generation, a causal mask blocks future tokens. This mechanism is powerful, though standard attention can be costly for long sequences.”
Connect this question to the roadmap topic it supports.