LLMS & GENERATIVE AI · QUESTION 33
What is tokenization, and why does it matter for LLM applications?
On this page
Tokenization splits text into model-specific units called tokens, which may be words, word pieces, or punctuation. The model processes token IDs rather than raw text, so tokenization affects context-window usage, latency, and often API cost. Different languages, code, and unusual strings can use tokens differently; estimate with the model’s tokenizer instead of treating a token as a fixed number of characters.
LET'S MAKE IT SIMPLE
First, the idea in plain English
Before text reaches a language model, a tokenizer converts it into pieces from a vocabulary and then into numbers. A familiar word might be one token, while an uncommon word can be several pieces. Because models have limits and services may charge by token, the number of tokens in a prompt matters even when its character count seems small.
A USEFUL COMPARISON
Picture it this way
Think of packing a paragraph into labeled tiles before sending it through a machine. Common words may fit on one tile, while rare or specialized words need several. The tiles are not meaningful word boundaries in every case; each model uses its own vocabulary and encoding rules.
FOLLOW THE JOURNEY
Text becomes token IDs before model processing
Show text segmentation and the practical effects of token counts.
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Oversized requests may be rejected or need truncation.
Raw text is split into vocabulary tokens, mapped to token IDs, processed within a context limit, and decoded back into text.
How to read this visual
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Oversized requests may be rejected or need truncation.
UNDER THE HOOD
How it works, step by step
Input text
Start with the exact prompt, code, or document content.
Why it matters: Whitespace and unusual characters can affect segmentation.
Split into tokens
Apply the tokenizer vocabulary to create subword or other units.
Why it matters: Token boundaries are model-specific and not always words.
Map to IDs
Convert each token into a numeric identifier.
Why it matters: The model operates on numerical representations.
Check context budget
Count input and reserved output tokens against the model limit.
Why it matters: Oversized requests may be rejected or need truncation.
OUTSIDE THE TEXTBOOK
A practical example
A support application sends a system instruction, retrieved policy passages, chat history, and a user question. The combined token count—not just the newest question—uses the context budget. Measuring with the deployed model’s tokenizer helps decide how much history to keep and whether long passages need to be shortened.
GO A LITTLE DEEPER
Technical details that matter
- Subword tokenization can represent unseen words by composing known pieces, but unusual names or code can consume many tokens.
- Tokenizers differ across model families; token counts and embeddings should not be assumed interchangeable.
- Prompt compression and truncation can change meaning, so evaluate answer quality after changing token budgets.
MAKE A GOOD CHOICE
When is this useful?
- Estimating context size and request cost.
- Designing document chunking and conversation-history policies.
- Debugging why code, multilingual, or symbol-heavy prompts exceed limits.
Character and word counts are useful rough indicators, but use the target model tokenizer for accurate budgeting.
WATCH OUT FOR
Common misunderstandings
- Assuming one token equals one word or a fixed number of characters.
- Counting only the user message and ignoring system instructions or generated output.
- Truncating text without preserving the context needed to interpret it.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Token counts do not measure semantic importance.
- Tokenizer behavior differs among models and versions.
- Reducing tokens can remove necessary qualifications or evidence.
WHAT TO REMEMBER
Key takeaways
- 1Tokenization maps text to model-readable IDs.
- 2Token count affects context, latency, and cost.
- 3Use the tokenizer for the exact model and preserve important context when trimming.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
Why do uncommon words use multiple tokens?
A tokenizer often represents text by composing vocabulary pieces rather than storing every possible word. This supports open-ended text but makes counts variable.
How do you reduce token use safely?
Remove redundant history, retrieve focused passages, and shorten repeated instructions, then evaluate whether the essential evidence and behavior remain.
SAY IT OUT LOUD
How to explain this in an interview
“Tokenization converts input text into model-specific pieces and numeric IDs. A token may be a whole word or part of one, so character and word counts are only rough estimates. Token use affects context limits and often cost. In production I would count with the deployed model’s tokenizer and manage history and retrieved context without dropping information the answer depends on.”
Connect this question to the roadmap topic it supports.