Breaking Down Language for Machines
At the heart of every natural language processing system lies a fundamental challenge: how do we bridge the gap between human language and machine computation? Computers don’t understand words, sentences, or paragraphs the way humans do. They process numbers, vectors, and mathematical operations. This is where tokenization becomes crucial — the process of breaking down text into smaller, manageable units that machines can process and learn from.
Tokenization is far more than simply splitting text at spaces. It’s a sophisticated process that directly impacts how well AI models understand language, how efficiently they process information, and even how much computational power they require. The choice of tokenization strategy can mean the difference between a model that handles multiple languages gracefully and one that struggles with anything beyond English, or between a system that processes text efficiently and one that wastes valuable computational resources.
In this article, we’ll explore the different approaches to tokenization, understand their mechanisms through practical examples, and examine the real-world implications of these choices for modern AI systems.
What Is Tokenization?
Tokenization is the process of converting raw text into a sequence of discrete units called tokens. These tokens serve as the basic building blocks that language models use to understand and generate text. Think of tokenization as breaking down a complex sentence into digestible pieces, similar to how we might break down a large project into smaller, manageable tasks.
Example:
- Input text: “I love natural language processing!”
- Tokens (simple approach): [“I”, “love”, “natural”, “language”, “processing”, “!”]
In this basic example, each word and punctuation mark becomes a separate token. The model would then convert each of these tokens into numerical representations (embeddings) that it can process mathematically.
However, this simple approach raises immediate questions: What about compound words? What about languages without clear word boundaries? What about rare words the model has never seen? These questions lead us to explore more sophisticated tokenization methods.
Character-Level Tokenization
Character-level tokenization represents the simplest approach: every individual character becomes a token. This includes letters, numbers, punctuation marks, and even spaces.
Example:
- Input text: “AI is cool”
- Tokens: [“A”, “I”, “ “, “i”, “s”, “ “, “c”, “o”, “o”, “l”]
The text is broken into 10 individual characters. Each letter, including the spaces, becomes a separate token. The model must learn to combine these characters into meaningful words and then those words into meaningful sentences.
Advantages:
- Small vocabulary size: Only needs to handle the finite set of characters in a language (typically 26–100 for most writing systems)
- No unknown tokens: Any word, even made-up ones like “supercalifragilisticexpialidocious,” can be represented
- Handles spelling variations naturally: Typos and creative spellings are just different character sequences
Disadvantages:
- Very long sequences: A simple sentence becomes dozens or hundreds of tokens, requiring more computational resources
- Loses word boundaries: The model must learn from scratch that certain character sequences form meaningful units
- Inefficient: Processing “the” requires three separate tokens instead of one
Character-level models are rarely used in modern large language models because they’re computationally expensive. However, they’re useful for specialized tasks like spell-checking or text generation where character-level control is important.
Word-Level Tokenization
Word-level tokenization treats each complete word as a single token. This is closer to how humans naturally think about language.
Example:
- Input text: “The quick brown fox jumps over the lazy dog.”
- Tokens: [“The”, “quick”, “brown”, “fox”, “jumps”, “over”, “the”, “lazy”, “dog”, “.”]
Each word becomes one token, with punctuation typically separated. Notice that “The” and “the” are treated as different tokens because of capitalization. This creates 10 tokens total, much more efficient than character-level tokenization would be.
Handling Unknown Words:
Example:
- Input text: “I love pneumonoultramicroscopicsilicovolcanoconiosis.”
- Tokens: [“I”, “love”, “[UNK]”, “.”]
If the model’s vocabulary doesn’t include this extremely rare medical term, it gets replaced with an “[UNK]” (unknown) token. The model loses all information about what this word actually is, it could be any unknown word.
Advantages:
- Intuitive: Aligns with human understanding of language
- Shorter sequences: More efficient than character-level approaches
- Preserves word meaning: Each word unit is kept intact
Disadvantages:
- Massive vocabulary: English alone has hundreds of thousands of words
- Unknown word problem: Rare words, proper nouns, and new terms become [UNK]
- Morphological blindness: Can’t see relationships between words like “run,” “running,” “ran”
- Poor for morphologically rich languages: Languages like Turkish or Finnish that build complex words from smaller parts struggle with this approach
Early neural models used word-level tokenization, but its limitations led to the development of more sophisticated approaches that balance efficiency with flexibility.
Subword Tokenization
Subword tokenization represents the modern standard for most language models. It strikes a balance between character-level and word-level approaches by breaking words into meaningful subunits.
Byte Pair Encoding (BPE)
BPE starts with individual characters and iteratively merges the most frequent pairs of tokens to create new tokens. It builds a vocabulary based on statistical patterns in the training data.
Example — How BPE learns:
Training data: “low”, “lower”, “lowest”, “newer”, “wider”
Iteration 1: Most common pair is “e” + “r” → create token “er”
- Vocabulary grows: […, “e”, “r”, “er”]
Iteration 2: Most common pair is “l” + “o” → create token “lo”
- Vocabulary grows: […, “l”, “o”, “lo”]
Iteration 3: Most common pair is “lo” + “w” → create token “low”
- Vocabulary grows: […, “lo”, “w”, “low”]
Final tokenization example:
- Input: “lowest”
- Tokens: [“low”, “est”]
The word is split into “low” (a learned common subword) and “est” (a common suffix). The model can now understand that “est” indicates a superlative form, even when applied to new words.
Example with unknown word:
- Input: “unhappiness”
- Tokens (BPE): [“un”, “happ”, “iness”]
Even if the model never saw “unhappiness” as a complete word, it breaks it into recognizable parts: the prefix “un” (meaning not), the root “happ” (relating to happy), and the suffix “iness” (forming an abstract noun). The model can infer meaning from these components.
WordPiece
WordPiece, used by models like BERT, is similar to BPE but chooses merges that maximize the likelihood of the training data rather than simple frequency.
Example:
- Input: “playing”
- Tokens (WordPiece): [“play”, “##ing”]
WordPiece uses “##” to indicate continuation of a word. The token “##ing” means this is a suffix that attaches to the previous token. This helps the model understand word structure.
Example with compound word:
- Input: “unforgettable”
- Tokens (WordPiece): [“un”, “##forget”, “##table”]
The word is decomposed into a negative prefix “un”, a root “forget”, and an adjective suffix “table”. Each part contributes meaning to the whole.
SentencePiece
SentencePiece treats text as a raw stream and doesn’t require pre-tokenization. It’s language-agnostic and works directly on Unicode characters.
Example with multiple languages:
- Input (English): “Hello world”
- Tokens: [“▁Hello”, “▁world”]
- Input (Japanese): “こんにちは世界”
- Tokens: [“▁”, “こん”, “にちは”, “世界”]
The “▁” symbol represents a space. SentencePiece doesn’t assume spaces separate words, making it effective for languages like Japanese or Chinese that don’t use spaces. Each language is tokenized based on its statistical patterns.
Advantages of Subword Tokenization:
- Balanced vocabulary size: Typically 30,000–50,000 tokens — manageable but expressive
- Handles rare words: Decomposes them into known subwords rather than [UNK]
- Captures morphology: Recognizes prefixes, suffixes, and root words
- Language flexibility: Works across different languages and writing systems
- Efficient: Shorter sequences than character-level, more flexible than word-level
Disadvantages:
- Not perfectly intuitive: Tokenization boundaries may seem arbitrary to humans
- Language bias: More efficient for languages well-represented in training data
- Inconsistent segmentation: Same word might tokenize differently in different contexts
Almost all modern large language models — GPT, BERT, T5, Claude, and others — use subword tokenization. It’s the reason these models can handle new words, understand morphology, and work across multiple languages.
Practical Implications of Tokenization Choices
Computational Efficiency
Example comparison for the sentence: “The researcher analyzed the experimental data.”
- Character-level: 45 tokens (including spaces)
- Word-level: 6 tokens
- Subword (BPE): ~8 tokens [“The”, “research”, “er”, “analyz”, “ed”, “the”, “experimental”, “data”]
Computational cost scales with sequence length. Character-level tokenization requires processing 7.5x more tokens than word-level for the same sentence. This means more memory, more processing time, and higher costs for both training and inference.
Handling Rare and Novel Words
Example: Processing a scientific term
- Input: “The mitochondria is the powerhouse of the cell.”
- Word-level tokens: [“The”, “[UNK]”, “is”, “the”, “powerhouse”, “of”, “the”, “cell”, “.”]
- Subword tokens: [“The”, “mito”, “chond”, “ria”, “is”, “the”, “power”, “house”, “of”, “the”, “cell”, “.”]
With word-level tokenization, “mitochondria” becomes [UNK] and all information is lost. With subword tokenization, the word is broken into meaningful parts. The model might recognize “mito-” as a prefix related to cellular biology, “chond” as relating to granules or cartilage, giving it at least partial understanding.
Multilingual Capabilities
Example: Processing code-switched text
- Input: “I’m learning 机器学习 today” (mixing English and Chinese for “machine learning”)
- Character-level: Works but inefficient
- Word-level: Struggles with Chinese characters
- Subword tokens: [“I”, “‘m”, “learning”, “机器”, “学习”, “today”]
Subword tokenization handles the language mixing gracefully. Chinese characters are tokenized based on their frequency patterns in the training data, allowing the model to process multilingual text naturally.
Token Limits and Context Windows
Example: Understanding model capacity
If a model has a 4,096 token context window:
- With character-level tokenization: ~800–1,000 words
- With word-level tokenization: ~3,500–4,000 words
- With subword tokenization: ~2,500–3,500 words
Result explained: The tokenization method directly affects how much text a model can process at once. This impacts applications like document summarization, long-form question answering, and maintaining context in conversations.
Vocabulary Size and Model Parameters
Example: Memory footprint
A model with 768-dimensional embeddings:
- Character vocabulary (~100 tokens): 76,800 parameters
- Subword vocabulary (~50,000 tokens): 38,400,000 parameters
- Word vocabulary (~500,000 tokens): 384,000,000 parameters
The vocabulary size directly affects model size. However, this is offset by efficiency gains — subword models need fewer layers to achieve the same performance as character-level models, and avoid the unknown word problem of word-level models.
Special Tokenization Considerations
Handling Numbers
Example:
- Input: “The temperature is 98.6 degrees”
- Inefficient tokenization: [“The”, “temperature”, “is”, “9”, “8”, “.”, “6”, “degrees”]
- Efficient tokenization: [“The”, “temperature”, “is”, “98”, “.”, “6”, “degrees”]
Good tokenizers keep common number patterns together. Breaking “98.6” into individual digits forces the model to learn number meaning from scratch, while keeping larger numbers as units helps the model understand numerical concepts better.
Code and Technical Text
Example — Python code:
- Input: def calculate_sum(a, b):
- Tokens: [“def”, “calculate”, “_”, “sum”, “(“, “a”, “,”, “b”, “):”]
Code has different statistical patterns than natural language. Specialized tokenizers for code (like those in Codex) learn to keep meaningful code units together, such as function names and common operators, while splitting at semantic boundaries.
Whitespace and Special Characters
Example:
- Input: “Hello world!!!”
- Basic tokenization: [“Hello”, “world”, “!”, “!”, “!”]
- Advanced tokenization: [“Hello”, “▁▁▁▁”, “world”, “!!!”]
Advanced tokenizers can preserve information about spacing and repeated punctuation, which may carry meaning (emphasis, formatting) in certain contexts.
The Future of Tokenization
Recent developments are pushing tokenization in new directions:
Byte-Level Tokenization
Models like GPT-4 use byte-level BPE, which operates on raw bytes (UTF-8 encoded) rather than Unicode characters.
Example:
- Input (with emoji): “I love AI 🤖”
- Byte-level tokens: Can represent the emoji as a sequence of bytes without needing a special emoji token
This approach can handle any text in any language or writing system, including emojis, rare Unicode characters, and binary data, without ever encountering truly unknown tokens.
Learned Tokenization
Some research explores end-to-end learning where the model learns its own optimal tokenization strategy during training rather than using a fixed tokenizer.
Potential benefit: The model could develop tokenization strategies specifically optimized for its tasks, possibly discovering patterns humans haven’t explicitly programmed.
Tokenization sits at the crucial intersection between human language and machine learning, serving as the translator that makes modern AI language understanding possible. What might seem like a simple technical detail — how we split text into pieces — has profound implications for everything from computational efficiency to a model’s ability to understand new words, process multiple languages, and scale to billions of parameters.
We’ve seen how different approaches offer different trade-offs. Character-level tokenization provides maximum flexibility but at steep computational costs. Word-level tokenization aligns with human intuition but struggles with vocabulary size and unknown words. Subword tokenization, the current standard, achieves a practical balance that enables today’s remarkable language models.
The evolution from simple word splitting to sophisticated subword methods like BPE, WordPiece, and SentencePiece reflects the field’s maturity. These approaches allow models to handle rare words by decomposing them into meaningful parts, process multiple languages without language-specific rules, and maintain manageable vocabulary sizes while preserving expressiveness.
Understanding tokenization helps us appreciate both the capabilities and limitations of AI language models. When a model struggles with a particular word or seems more efficient in English than other languages, tokenization is often part of the explanation. When we see token limits in API documentation or context windows in model specifications, we now understand these are real constraints that affect what the model can process.
As the field continues to advance, tokenization methods will evolve alongside model architectures. Future developments may blur the line between tokenization and model learning, creating systems that dynamically adapt their text segmentation strategies. But regardless of how sophisticated these methods become, the fundamental principle remains: to understand language, machines must first break it into pieces they can process, and how we choose to break it matters immensely.
Understanding Tokenization: was originally published in AI Evergreen on Medium, where people are continuing the conversation by highlighting and responding to this story.