Module 5: LLMs (Large Language Models) - GPT, Claude and More

3. Tokenization: How Text Becomes Numbers

Next →

Description

LLMs don't process text directly; they process tokens (numbers). In this lesson you'll understand:

  • What tokens are (the basic units an LLM processes).
  • How tokenization works (text → tokens).
  • Why tokens ≠ words (especially in non-English languages).
  • Why it matters (costs are calculated per token, not per word).

What Tokens are

Token: The basic unit an LLM processes. It can be:

  • A complete word (e.g. "hello").
  • Part of a word (e.g. "running" → "run" + "ning").
  • A special character (e.g. "!", "?").
  • A space or punctuation mark.

Rule of thumb:

  • English: 1 token ≈ 0.75 words (or ~4 characters).
  • Spanish/French: 1 token ≈ 0.6 words.
  • Chinese/Japanese: 1 character ≈ 2-3 tokens.

Example: "Hello world" vs "Hola mundo"

English:

Text: "Hello world"
Tokens: ["Hello", " world"]
Count: 2 tokens

Spanish:

Text: "Hola mundo"
Tokens: ["Hola", " mundo"]
Count: 2 tokens

In this case, English and Spanish use the same number of tokens. But that isn't always so.

A Complex Example: Non-English Languages

English:

Text: "Hello, how are you today?"
Tokens: ["Hello", ",", " how", " are", " you", " today", "?"]
Count: 7 tokens

Spanish:

Text: "Hola, ¿cómo estás hoy?"
Tokens: ["Hola", ",", " ¿", "c", "ómo", " est", "ás", " hoy", "?"]
Count: 9 tokens

Chinese:

Text: "你好，今天怎么样？" (equivalent to "Hello, how are you today?")
Tokens: Each Chinese character → 2-3 tokens
Count: ~15-20 tokens

Moral: Non-English languages (Spanish, Chinese) use more tokens than English to express the same thing → more expensive.

How Tokenization Works: BPE (Byte Pair Encoding)

The algorithm used by GPT, Claude, Llama: BPE (Byte Pair Encoding).

Process (simplified):

  1. Start with characters: Each character is a token.
  2. Find frequent pairs: Look for pairs of tokens that appear together frequently (e.g. "t" + "h" → "th").
  3. Merge the pairs: Create a new token for the pair (e.g. "th" is now a token).
  4. Repeat: Until you have a vocabulary of ~50K-100K tokens.

Result: A vocabulary that balances:

  • Complete common words: "hello", "world", "the" (1 word = 1 token).
  • Subwords: "running" → "run" + "ning" (1 word = 2 tokens).
  • Rare characters: "你" (a Chinese character) → multiple tokens.

Why Tokens ≠ Words

Common words (English) → 1 token

"the" → 1 token
"hello" → 1 token
"world" → 1 token

Long or rare words → multiple tokens

"unbelievable" → 2-3 tokens (e.g. "un", "believ", "able")
"antidisestablishmentarianism" → 8-10 tokens

Non-English languages → more tokens

"你好" (Chinese, "hello") → 4-6 tokens
"こんにちは" (Japanese, "hello") → 8-10 tokens

Why: The tokenizer was trained mainly on English → it knows complete English words, but not complete Chinese/Japanese characters.

Why It Matters: Costs and Limits

1. Costs are calculated per token

OpenAI pricing (GPT-4):

  • Input: $0.03 per 1K tokens
  • Output: $0.06 per 1K tokens

Example 1 (English):

Prompt (English): "Explain quantum physics" (3 words, ~3 tokens)
Response (English): 500 words → ~666 tokens

Cost:
- Input: 3 tokens × $0.03 / 1000 = $0.00009
- Output: 666 tokens × $0.06 / 1000 = $0.04
- Total: ~$0.04

Example 2 (Spanish):

Prompt (Spanish): "Explica la física cuántica" (4 words, ~5 tokens)
Response (Spanish): 500 words → ~833 tokens (Spanish uses more tokens)

Cost:
- Input: 5 tokens × $0.03 / 1000 = $0.00015
- Output: 833 tokens × $0.06 / 1000 = $0.05
- Total: ~$0.05 (25% more expensive than English)

Moral: If you work in Spanish/Chinese, your costs will be 20-50% higher than in English.

2. The context window is measured in tokens

GPT-4:

  • Context window: 128K tokens.
  • In English: ~96K words (~300 pages).
  • In Spanish: ~76K words (~240 pages).
  • In Chinese: ~30K-40K characters (~100 pages).

Moral: The context window is smaller in non-English languages.

Tools for Counting Tokens

1. OpenAI Tokenizer (web)

URL: https://platform.openai.com/tokenizer

Use:

  1. Paste the text.
  2. See the tokens generated.
  3. Count the total number of tokens.

2. The tiktoken library (Python)

import tiktoken

# Use GPT-4's tokenizer
encoding = tiktoken.encoding_for_model("gpt-4")

# Count tokens
text = "Hello, how are you today?"
tokens = encoding.encode(text)
print(f"Tokens: {len(tokens)}")  # Output: 7

# See the individual tokens
print(tokens)  # Output: [9906, 11, 1268, 527, 499, 3432, 30]

Why this matters for an AI Engineer

1. Optimizing costs

Before optimizing:

A verbose prompt (English, 50 words, ~66 tokens):
"I would like you to please explain to me, in great detail and with many examples, what quantum physics is all about and how it works."

Cost per request: ~$0.002 (input) + $0.04 (output) = $0.042

After optimizing:

A concise prompt (English, 10 words, ~10 tokens):
"Explain quantum physics with examples."

Cost per request: ~$0.0003 (input) + $0.04 (output) = $0.0403
(saving: 5% in this case, but it can be 20-30% if the prompts are very verbose)

Moral: Concise prompts reduce costs.

2. Calculating limits

Your app: A chatbot that processes documents.

Document: 200 pages in Spanish (~60K words → ~100K tokens).

Problem: GPT-4's context window is 128K tokens → the complete document + the prompt + the response → it exceeds the limit.

Solution:

  • Split the document into chunks of ~30K tokens each.
  • Use RAG (search for the relevant sections, pass only those).
  • Use Claude 3 (200K tokens).

3. Debugging

Problem: The request gives a "context_length_exceeded" error.

Diagnosis:

# Count the tokens of the prompt + document
prompt_tokens = encoding.encode(prompt)
document_tokens = encoding.encode(document)
total = len(prompt_tokens) + len(document_tokens)

print(f"Total tokens: {total}")  # Output: 135,000

if total > 128000:
    print("Exceeds GPT-4's context window (128K tokens)")

Solution: Reduce the length of the document or the prompt.

Common Mistakes

1. Assuming that 1 token = 1 word

Mistake: Calculating costs assuming 1 token = 1 word.

Reality: In English, 1 token ≈ 0.75 words. In Spanish/Chinese, far less.

Consequence: Costs are 30-100% higher than expected.

2. Not counting tokens in non-English languages

Mistake: Designing an app in Spanish without considering that it uses more tokens.

Reality: An app in Spanish costs 20-50% more than in English (for the same number of words).

Summary

Tokens:

  • The basic unit an LLM processes (not words).
  • They can be complete words, subwords, or characters.

Tokenization:

  • The process of converting text → tokens.
  • Algorithm: BPE (Byte Pair Encoding).

Why tokens ≠ words:

  • Common words (English) → 1 token.
  • Rare or long words → multiple tokens.
  • Non-English languages → more tokens per word.

Why it matters:

  • Costs: They're calculated per token (not per word).
  • Context window: It's measured in tokens.
  • Optimization: Concise prompts reduce costs.

Next step: Lesson 04: Embeddings — Vector representations of meaning, semantic search.

← Previous2. What is an LLM? Definition, Scale and CapabilitiesNext →4. Embeddings: Vector Representations of Meaning