Tokenization Demystified - a detailed review

 




In AI and Large Language Models (LLMs), a token is the foundational unit of data used to process and generate language. AI does not read text word-by-word or letter-by-letter. Instead, it breaks text down into these smaller mathematical chunks.
Think of tokens as the building blocks or the "atomic level" of an AI's vocabulary.
1. How Text Becomes Tokens (Tokenization)
Before an AI can understand your prompt, a piece of software called a tokenizer chops your text into pieces.
  • The Core Rule: As a general rule of thumb, 1 token equals roughly 4 characters of English text, or about 0.75 words.
  • The Math: 100 tokens are approximately equal to 75 words.
Tokens do not always split cleanly at the space between words. They can be:
  • Whole words: cat, the, apple
  • Parts of words (sub-words): The word unbelievable might be split into un, believ, and able.
  • Single characters: Punctuation marks (?, !), spaces, or emojis.
 Visual Example
If you pass the sentence: "Tokenization is easy." to an AI, the tokenizer might break it down like this:
  • Token (Token 1)
  • ization (Token 2)
  • _is (Token 3)
  • _easy (Token 4)
  • . (Token 5)
2. Why AI Uses Tokens Instead of Words
AI models are massive mathematical calculators. They cannot process raw text; they can only process numbers. Tokenization bridges this gap through a three-step concept:
  1. The Vocabulary Index: The AI has a fixed master dictionary (usually containing between 32,000 and 100,000+ unique tokens). Every token in this dictionary has a specific number ID.
  2. Vector Embeddings: Once text is turned into token IDs, the AI replaces those numbers with high-dimensional math vectors. These vectors map the meaning and relationships of words (e.g., the AI mathematically knows that the token king is closely related to queen).
  3. Efficiency: Sub-word tokenization allows the AI to handle a massive variety of words without needing an infinitely large dictionary. If it encounters a brand-new word like AI-driven, it can just piece it together using the existing tokens AI, -, dri, and ven.
3. The Challenges and Limitations of Tokens
Understanding tokens clarifies why AI models behave the way they do:
  • The Multilingual Tax: Non-English languages are much less efficient. Because AI models are trained predominantly on English, words in Spanish, Mandarin, or Hindi are often broken down into many more, smaller tokens. Writing a 500-word essay in Spanish will use significantly more tokens (and cost more) than writing the exact same essay in English.
  • Math and Spelling Blindness: Because the AI sees the word apple as a single token (e.g., Token ID #4502), it doesn't inherently know that it contains two "p"s. This is why LLMs sometimes struggle with simple letter-counting riddles or spelling backward—they don't see the individual letters unless they are forced to break the token down.
  • Code and Spaces: In programming languages, indentation and repeating spaces take up a lot of tokens. This makes long scripts highly token-expensive.
 4. Why Tokens Matter to Users: Limits and Costs
When interacting with AI systems, you will constantly encounter tokens in two major contexts:
Context Windows
Every AI model has a "context window," which is the maximum number of tokens it can read and write at a single time.
  • If a model has a 128k token context window, it can hold roughly 96,000 words in its active memory (about the length of a 300-page book).
  • Once your conversation exceeds this limit, the AI begins to "forget" the earliest messages in the chat.
Pricing and Billing
If you use developer APIs (like OpenAI, Anthropic, or Google Cloud), you do not pay by the month or by the query. You pay per million tokens.
  • Input Tokens: The prompt you type. (Cheaper)
  • Output Tokens: The text the AI generates back to you. (More expensive, because it requires more computing power).  
In deep: 
A tokenizer relies on mathematical and statistical principles to convert raw text into token IDs. Rather than relying on simple whitespace rules, modern LLMs utilize sub-word tokenization algorithms. This allows them to strike a balance between maintaining a compact vocabulary and avoiding the issue of out-of-vocabulary (OOV) words. 
Byte-Pair Encoding (BPE)
Originally designed as a data compression algorithm, BPE is utilized by OpenAI's GPT models and Meta's LLaMA models. 
  • How it works: It begins by treating every individual character and byte as a base token. It iteratively scans the text corpus, counts adjacent pairs of tokens, and merges the most frequently occurring pair into a single new token. 
  • Example: The characters t + h frequently appear together and are merged into th. Then, th + e are merged into the. This loop repeats until the model reaches its target vocabulary size (e.g., 100,000 unique tokens). 
WordPiece
Developed by Google and utilized by models like BERT, WordPiece operates similarly to BPE but handles merges differently. 
  • How it works: Instead of choosing the most frequent character pair, WordPiece evaluates all potential merges and selects the one that maximizes the likelihood of the training data according to a probabilistic language model.
  • Visual Marker: It uses special prefix symbols (like ##) to denote sub-words that must be attached to a preceding token (e.g., un + ##believ + ##able).
SentencePiece
Mainly used by Google's Gemini models and various open-source LLMs, SentencePiece is an open-source tokenization wrapper. 
  • How it works: Traditional tokenizers require language-specific rules to isolate words before splitting them. SentencePiece treats the input string raw—including spaces—as a continuous byte stream. It replaces spaces with a dedicated meta-character (_) and runs BPE or Unigram segmentation directly on it. This makes it highly effective for languages like Japanese or Chinese that do not use whitespace to separate words. 
 Calculating Token Usage via Python
To estimate, track, and budget API costs without running live API queries, developers can calculate token counts locally. 
OpenAI provides an open-source library called tiktoken, while Hugging Face provides transformers for open-source models and Anthropic. The following Python code demonstrates how to calculate precise token counts and map text back to individual token strings: 
python
import tiktoken

def analyze_tokens(text: str, model_encoding: str = "cl100k_base") -> None:
    """
    Encodes text into token IDs, calculates the total count,
    and decodes individual token pieces to see how the model splits them.
    """
    # 1. Get the correct tokenizer encoding mapping
    encoding = tiktoken.get_encoding(model_encoding)
    
    # 2. Convert text to numerical Token IDs
    token_ids = encoding.encode(text)
    token_count = len(token_ids)
    
    print(f"--- Token Analysis Report ---")
    print(f"Raw Text length: {len(text)} characters")
    print(f"Total Token Count: {token_count} tokens\n")
    
    # 3. Decode individual token IDs back into bytes/strings to see the breakdown
    print(f"{'Token ID':<10} | {'Decoded Piece (Bytes)':<25}")
    print("-" * 40)
    for t_id in token_ids:
        # We use .decode_single_token_bytes() because some tokens are raw, incomplete UTF-8 bytes
        token_bytes = encoding.decode_single_token_bytes(t_id)
        print(f"{t_id:<10} | {token_bytes}")

# Example Usage:
sample_text = "Tokenization is unbelievable!"
analyze_tokens(sample_text, model_encoding="cl100k_base")
Why Running This Script Matters:
If you pass "unbelievable!", the script will reveal how BPE shards the root word into sub-words and separates the punctuation mark ! into its own unique numerical ID. 
Comprehensive Model Cost Breakdown
AI model providers structure API billing around Prices per 1 Million Tokens (1M MTok). Output tokens command a premium price because auto-regressive generation requires computing a full forward pass of the neural network for every single generated token. 
1. OpenAI Ecosystem Pricing
OpenAI's pricing spans a wide spectrum, separating standard foundational workloads, deep-reasoning systems, and highly optimized low-cost engines: 
Model ClassContext WindowInput Cost (per 1M)Output Cost (per 1M)Core Target Use Case
GPT-5.6 Sol1.1MVariable (Promo)VariableUltra-long context reasoning & advanced physical-AI agent simulation.
GPT-5.6 Terra128K$2.00$12.00Flagship production tier balancing complex multi-step orchestration.
GPT-5.6 Luna128K$0.20$1.20Hyper-efficient, high-volume classification, extraction, and fast automation.
GPT-5.4 Nano128K$0.20$1.25Low-overhead edge pipelines and simple chatbot fallback logic.
GPT-4o128K$2.50$10.00Legacy general enterprise workhorse.
GPT-4o mini128K$0.15$0.60High-volume basic tasks.
2. Anthropic Claude Ecosystem Pricing
Anthropic targets multi-agent software infrastructure and software development, competing on prompt caching and coding logic:
Model ClassContext WindowInput Cost (per 1M)Output Cost (per 1M)Core Target Use Case
Claude Fable 51M$10.00$50.00Frontier complex reasoning, mathematical proofs, and logic verification.
Claude Opus 51M$5.00$25.00Premium software agent development and unstructured research.
Claude Sonnet 51M$2.00$10.00General development workhorse; target Sweet spot for code production.
Claude Haiku 4.5128K$1.00$5.00High-speed structured logs processing and lightweight tool execution.
Advanced Infrastructure Optimization: Slashing Token Overhead
To control costs at enterprise scale, developers look beyond raw baseline token prices and leverage platform mechanics to optimize token efficiency: 
  • Prompt Caching: Anthropic and OpenAI allow developers to cache long, static instructions (like a 50,000-word corporate policy or codebase framework). If consecutive queries reference this exact block, reading it from the cache yields up to a 90% discount. For example, Claude Sonnet 5 drops from $2.00/M input down to $0.20/M for cached reads. 
  • Batch APIs: For non-real-time jobs (e.g., overnight data parsing or batch transcript processing), developers submit asynchronous requests via Batch Endpoints to receive a flat 50% discount across all input and output token channels. 

Comments

Popular posts from this blog

IBM goes big to keep quantum computers cold

IBM - avances en mantener adecuadamente fríos computadores cuánticos.

Estructura multinivel de los sistemas de agentes de IA