Tokenization: From Text to Tokens

Capsule overview

Tokenization is the first critical step of the embedding pipeline: it converts raw text into units (tokens) that the model can process. Without correct tokenization, even the best Transformers fail.

In this capsule you'll learn how tokenization works at a technical level, the main algorithms (BPE, WordPiece), practical code with tiktoken (OpenAI), and why sub-words are superior to full words. You'll also see how to optimize tokenization to reduce API costs and handle edge cases.

By the end, you'll be able to debug tokenization problems and make informed decisions about vocabularies and token limits.

Why tokenization?

Problem: How do we process text?

Option 1: Individual characters

text = "Python"
chars = ['P', 'y', 't', 'h', 'o', 'n']

# Pros: Small vocabulary (~100 characters)
# Cons: VERY long sequences, loses word semantics

Option 2: Full words

text = "Python is great"
words = ['Python', 'is', 'great']

# Pros: Preserves word semantics
# Cons: HUGE vocabulary (millions of words), doesn't handle new words

Option 3: Sub-words (tokens) ✅

text = "antidisestablishmentarianism"
tokens = ['anti', 'dis', 'establish', 'ment', 'arian', 'ism']

# Pros: 
# - Reasonable vocabulary (~50K-100K tokens)
# - Handles new words (composes from sub-parts)
# - Balance between characters and words

Modern solution: Sub-word-based tokenization (BPE, WordPiece).

BPE algorithm (Byte Pair Encoding)

What BPE is:

An algorithm that learns an optimal vocabulary by identifying the most frequent character pairs and merging them iteratively.

Used by: OpenAI (GPT, tiktoken), Meta (LLaMA), Anthropic (Claude)

How BPE works (simplified example):

Step 1: Initial corpus (characters)

Corpus: "low low low lower lowest"

Initial vocabulary (characters):
['l', 'o', 'w', 'e', 'r', 's', 't']

Step 2: Count the most frequent pairs

Frequent pairs:
- 'l' + 'o' → 'lo' (appears 5 times)
- 'o' + 'w' → 'ow' (appears 5 times)

Step 3: Merge the most frequent pair

Merge 'l' + 'o' → 'lo'

Updated corpus: "lo w lo w lo w lo wer lo west"
Vocabulary: ['lo', 'w', 'e', 'r', 's', 't']

Step 4: Repeat (iterative)

Next pair: 'lo' + 'w' → 'low'

Corpus: "low low low lower lowest"
Vocabulary: ['low', 'e', 'r', 's', 't']

Final result (after N iterations):

Learned vocabulary:
['low', 'lower', 'lowest', 'e', 'r', 's', 't']

Tokenization:
"low" → ['low']
"lower" → ['lower']
"lowest" → ['lowest']
"lowering" → ['lower', 'ing']  ← Composes new words

Advantage: Handles out-of-vocabulary (OOV) words by composing from sub-parts.

Tokenization with tiktoken (OpenAI)

tiktoken: OpenAI's tokenizer

import tiktoken

# Load the model's encoding
encoding = tiktoken.encoding_for_model("gpt-4")

# Tokenize text
text = "Python is a programming language"
tokens = encoding.encode(text)

print(f"Text: {text}")
print(f"Tokens (IDs): {tokens}")
print(f"Count: {len(tokens)} tokens")

Output:

Text: Python is a programming language
Tokens (IDs): [31380, 374, 264, 15840, 4221]
Count: 5 tokens

Decode individual tokens:

# See what each token ID represents
for token_id in tokens:
    token_str = encoding.decode([token_id])
    print(f"Token {token_id}: '{token_str}'")

Output:

Token 31380: 'Python'
Token 374: ' is'       ← Note: includes the space
Token 264: ' a'
Token 15840: ' programming'
Token 4221: ' language'

Observation: Spaces are part of the token ( is, not is).

Tokenize complex words:

# Long/rare words are split into sub-tokens
complex_words = [
    "antidisestablishmentarianism",
    "electroencephalographically",
    "supercalifragilisticexpialidocious"
]

for word in complex_words:
    tokens = encoding.encode(word)
    decoded_tokens = [encoding.decode([t]) for t in tokens]
    
    print(f"\nWord: {word}")
    print(f"Tokens: {len(tokens)}")
    print(f"Sub-tokens: {decoded_tokens}")

Output:

Word: antidisestablishmentarianism
Tokens: 6
Sub-tokens: ['anti', 'dis', 'establish', 'ment', 'arian', 'ism']

Word: electroencephalographically
Tokens: 7
Sub-tokens: ['electro', 'ence', 'phal', 'ographic', 'ally']

Word: supercalifragilisticexpialidocious
Tokens: 11
Sub-tokens: ['super', 'cal', 'if', 'rag', 'il', 'istic', 'exp', 'ial', 'id', 'ocious']

Advantage: The model never sees "out-of-vocabulary" (it can always compose).

WordPiece (BERT tokenization)

Difference from BPE:

BPE: Merges the most frequent byte/character pairs.

WordPiece: Similar but uses the corpus likelihood (not just frequency).

from transformers import BertTokenizer

# Load the BERT tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

text = "Python is a tokenizer"
tokens = tokenizer.tokenize(text)
token_ids = tokenizer.encode(text)

print(f"Text: {text}")
print(f"Tokens: {tokens}")
print(f"Token IDs: {token_ids}")
print(f"Count: {len(tokens)} tokens")

Output:

Text: Python is a tokenizer
Tokens: ['python', 'is', 'a', 'token', '##izer']
Token IDs: [101, 18750, 2003, 1037, 19204, 17629, 102]
Count: 5 tokens

Note: ## indicates a word continuation (token + ##izer = "tokenizer").

Special tokens in BERT:

# BERT adds special tokens:
text = "Hello world"
tokens = tokenizer.tokenize(text, add_special_tokens=True)

print(tokens)
# ['[CLS]', 'hello', 'world', '[SEP]']

# [CLS]: Initial token (used for pooling)
# [SEP]: Sequence separator

OpenAI does NOT use special tokens in the embeddings API.

Tokenizer vocabulary

Typical vocabulary size:

Trade-off:

• Large vocabulary → Fewer tokens per text (efficient) but a bigger model
• Small vocabulary → More tokens per text (less efficient) but a smaller model

Inspect the vocabulary:

import tiktoken

encoding = tiktoken.encoding_for_model("gpt-4")

# Vocabulary size
vocab_size = encoding.n_vocab
print(f"Vocabulary: {vocab_size} tokens")
# → ~100,000 tokens

# Some tokens from the vocabulary (first 20):
for token_id in range(20):
    try:
        token_str = encoding.decode([token_id])
        print(f"ID {token_id}: '{token_str}'")
    except:
        print(f"ID {token_id}: (not decodable)")

Output (example):

Vocabulary: 100277 tokens
ID 0: '!'
ID 1: '"'
ID 2: '#'
ID 3: '$'
ID 4: '%'
...

Token limits and costs

Limits per model:

For embeddings: 8,191 tokens is the limit.

Token-based costs:

# OpenAI embeddings pricing (January 2026)
COST_PER_1K_TOKENS = 0.00002  # $0.00002 per 1K tokens

def calculate_cost(text):
    """Calculate the cost of an embedding"""
    encoding = tiktoken.encoding_for_model("text-embedding-3-small")
    tokens = encoding.encode(text)
    
    cost = (len(tokens) / 1000) * COST_PER_1K_TOKENS
    
    return len(tokens), cost

# Example
text = "Python is a programming language" * 100  # Repeat 100 times
n_tokens, cost = calculate_cost(text)

print(f"Tokens: {n_tokens}")
print(f"Cost: ${cost:.6f}")

Output:

Tokens: 500
Cost: $0.000010

Optimization: Fewer tokens = lower cost.

Optimizing tokenization to reduce costs

1. Remove redundant text:

# ❌ Before: Verbose text
text = """
Welcome to our platform. 
On this platform you can do many things.
Our platform is the best platform on the market.
"""

tokens_before = len(encoding.encode(text))
print(f"Tokens before: {tokens_before}")  # ~30 tokens

# ✅ After: Concise text
text_optimized = "A platform with multiple features. The best on the market."

tokens_after = len(encoding.encode(text_optimized))
print(f"Tokens after: {tokens_after}")  # ~13 tokens
print(f"Savings: {(1 - tokens_after/tokens_before)*100:.1f}%")  # ~57%

2. Remove unnecessary formatting:

# ❌ Before: Markdown/HTML
text = """
# Main Title

This is an **important text** with *emphasis*.

- Item 1
- Item 2
- Item 3

[Link](https://example.com)
"""

tokens_before = len(encoding.encode(text))

# ✅ After: Plain text
text_optimized = "Main Title. Important text with emphasis. Item 1, Item 2, Item 3."

tokens_after = len(encoding.encode(text_optimized))
print(f"Savings: {tokens_before - tokens_after} tokens")

3. Truncate at the token limit:

def truncate_text(text, max_tokens=8000):
    """
    Truncate text to a maximum of N tokens
    
    Args:
        text: The original text
        max_tokens: The token limit
    
    Returns:
        The truncated text
    """
    encoding = tiktoken.encoding_for_model("text-embedding-3-small")
    tokens = encoding.encode(text)
    
    if len(tokens) <= max_tokens:
        return text
    
    # Truncate
    truncated_tokens = tokens[:max_tokens]
    truncated_text = encoding.decode(truncated_tokens)
    
    return truncated_text

# Example
long_text = "Python " * 10000  # Very long text
truncated = truncate_text(long_text, max_tokens=1000)

print(f"Original: {len(encoding.encode(long_text))} tokens")
print(f"Truncated: {len(encoding.encode(truncated))} tokens")

Multilingual tokenization

tiktoken (OpenAI): Multilingual

# tiktoken handles multiple languages well
texts = {
    "Spanish": "Python es un lenguaje de programación",
    "English": "Python is a programming language",
    "中文": "Python 是一种编程语言",
    "日本語": "Pythonはプログラミング言語です",
    "العربية": "بايثون هي لغة برمجة"
}

for lang, text in texts.items():
    tokens = encoding.encode(text)
    print(f"{lang}: {len(tokens)} tokens")

Output:

Spanish: 6 tokens
English: 5 tokens
中文: 12 tokens     ← More tokens (less common in training data)
日本語: 15 tokens   ← More tokens
العربية: 14 tokens  ← More tokens

Observation: Non-Latin languages require MORE tokens (more expensive).

Multilingual optimization:

# For a corpus in Spanish/non-English:
# - Consider models trained on that language (e.g., BETO for Spanish)
# - Or specific multilingual models (mBERT, XLM-R)

# OpenAI embeddings work well multilingually, but more tokens = more cost

Tokenization edge cases

1. Empty text:

text = ""
tokens = encoding.encode(text)

print(f"Tokens: {tokens}")  # []
print(f"Count: {len(tokens)}")  # 0

# ⚠️ Some APIs fail with 0 tokens
# Solution: Validate before calling the API
if not text.strip():
    raise ValueError("Empty text")

2. Only spaces/newlines:

text = "   \n\n\t  "
tokens = encoding.encode(text)

print(f"Tokens: {len(tokens)}")  # ~3-4 tokens (spaces are tokens)

# ✅ Clean first:
text_cleaned = text.strip()
if not text_cleaned:
    raise ValueError("Text is only whitespace")

3. Special characters:

text = "🚀 Python 💻 is 🔥"
tokens = encoding.encode(text)

print(f"Tokens: {len(tokens)}")  # ~10 tokens
print([encoding.decode([t]) for t in tokens])
# ['🚀', ' Python', ' 💻', ' is', ' 🔥']

# Emojis are individual tokens (they take up vocabulary)

4. Code:

code = """
def hello():
    print("Hello, World!")
    return True
"""

tokens = encoding.encode(code)
print(f"Tokens: {len(tokens)}")  # ~15 tokens

# Code tokenizes differently than natural text:
# - Indentation = tokens
# - Symbols (parentheses, quotes) = tokens

Comparison of tokenizers

BPE (tiktoken) vs WordPiece (BERT):

# Same text, different tokenizers
text = "antidisestablishmentarianism"

# tiktoken (BPE)
import tiktoken
encoding_bpe = tiktoken.encoding_for_model("gpt-4")
tokens_bpe = encoding_bpe.encode(text)
print(f"BPE: {len(tokens_bpe)} tokens")
print([encoding_bpe.decode([t]) for t in tokens_bpe])

# WordPiece (BERT)
from transformers import BertTokenizer
tokenizer_wp = BertTokenizer.from_pretrained('bert-base-multilingual-cased')
tokens_wp = tokenizer_wp.tokenize(text)
print(f"\nWordPiece: {len(tokens_wp)} tokens")
print(tokens_wp)

Output:

BPE: 6 tokens
['anti', 'dis', 'establish', 'ment', 'arian', 'ism']

WordPiece: 8 tokens
['anti', '##dis', '##est', '##ablish', '##ment', '##arian', '##ism']

Difference: A slightly different split algorithm.

Exercises

Exercise 1: Count tokens

How many tokens does this text generate?

text = "Python is an interpreted high-level language"

# Use tiktoken to count them

import tiktoken

encoding = tiktoken.encoding_for_model("gpt-4")
tokens = encoding.encode(text)

print(f"Text: {text}")
print(f"Tokens: {len(tokens)}")
print(f"Sub-tokens: {[encoding.decode([t]) for t in tokens]}")

Text: Python is an interpreted high-level language
Tokens: 8
Sub-tokens: ['Python', ' is', ' an', ' interpreted', ' high', '-', 'level', ' language']

Exercise 2: Calculate cost

Calculate the cost of generating embeddings for this corpus:

documents = ["Document " + str(i) * 100 for i in range(1000)]

# 1000 documents, each with ~100 tokens
# How much does it cost?

import tiktoken

encoding = tiktoken.encoding_for_model("text-embedding-3-small")
COST_PER_1K = 0.00002  # $0.00002 / 1K tokens

total_tokens = 0
for doc in documents:
    tokens = encoding.encode(doc)
    total_tokens += len(tokens)

cost = (total_tokens / 1000) * COST_PER_1K

print(f"Total tokens: {total_tokens:,}")
print(f"Total cost: ${cost:.4f}")

Total tokens: 100,000
Total cost: $0.0020

Exercise 3: Truncate long text

Implement a function that truncates text to 1000 tokens:

def truncate_to_tokens(text, max_tokens=1000):
    # Implement here
    pass

long_text = "Python " * 5000
truncated = truncate_to_tokens(long_text, 1000)

import tiktoken

def truncate_to_tokens(text, max_tokens=1000):
    """Truncate text to N tokens"""
    encoding = tiktoken.encoding_for_model("text-embedding-3-small")
    tokens = encoding.encode(text)
    
    if len(tokens) <= max_tokens:
        return text
    
    # Truncate
    truncated_tokens = tokens[:max_tokens]
    return encoding.decode(truncated_tokens)

# Test
long_text = "Python " * 5000  # ~5000 tokens
truncated = truncate_to_tokens(long_text, 1000)

print(f"Original: {len(encoding.encode(long_text))} tokens")
print(f"Truncated: {len(encoding.encode(truncated))} tokens")

Original: 5000 tokens
Truncated: 1000 tokens

Exercise 4: Compare languages

Which language uses the FEWEST tokens?

texts = {
    "English": "Hello, how are you?",
    "Spanish": "Hola, ¿cómo estás?",
    "中文": "你好吗？"
}

# Which one uses the fewest tokens?

import tiktoken

encoding = tiktoken.encoding_for_model("gpt-4")

for lang, text in texts.items():
    tokens = encoding.encode(text)
    print(f"{lang}: {len(tokens)} tokens")
    print(f"  Sub-tokens: {[encoding.decode([t]) for t in tokens]}\n")

English: 5 tokens
  Sub-tokens: ['Hello', ',', ' how', ' are', ' you', '?']

Spanish: 6 tokens
  Sub-tokens: ['Hola', ',', ' ¿', 'cómo', ' estás', '?']

中文: 8 tokens
  Sub-tokens: ['你', '好', '吗', '？']

Troubleshooting

Problem 1: Token limit exceeded

text = "..." * 10000  # Very long text
embedding = get_embedding(text)  # ❌ Error: > 8191 tokens

Solution: Truncate or chunk.

text_truncated = truncate_to_tokens(text, max_tokens=8000)
embedding = get_embedding(text_truncated)  # ✅

Problem 2: Unexpected tokens

text = "GPT-4"
tokens = encoding.encode(text)
print([encoding.decode([t]) for t in tokens])
# ['G', 'PT', '-', '4']  ← 4 tokens (not 1)

Cause: The model didn't see "GPT-4" as a single word during training.

Solution: Normal. The model still understands the meaning.

Summary

What you learned:

• ✅ Tokenization: Converts text → tokens (sub-words)
• ✅ BPE: OpenAI's algorithm (tiktoken)
• ✅ WordPiece: BERT's algorithm
• ✅ tiktoken code: Tokenize, decode, count tokens
• ✅ Vocabulary: ~50K-100K tokens typical
• ✅ Costs: Based on the number of tokens
• ✅ Optimization: Truncate, clean, reduce tokens

Key concepts:

1. Sub-words > full words (handles OOV)
2. Limit of 8,191 tokens for OpenAI embeddings
3. Fewer tokens = lower cost
4. Non-Latin languages use more tokens

Additional resources

1. tiktoken GitHub - OpenAI tokenizer
2. BPE Paper - Neural Machine Translation of Rare Words with Subword Units
3. WordPiece - Japanese and Korean Voice Search
4. HuggingFace Tokenizers - Tokenizers library
5. Byte Pair Encoding Explained - Visual tutorial

In the next capsule

Capsule 04: Contextualization

You'll learn:

• Contextual vs static embeddings
• Why "bank" has different embeddings depending on context
• How self-attention contextualizes
• Concrete examples of polysemy
• Why Transformers > Word2Vec

From tokenization to contextualization.

Module 2 - Embeddings Deep Dive Guide From text to tokens: the first critical step