Posts

Showing posts from August, 2026

Tokenization Demystified - a detailed review

Image
  In AI and Large Language Models (LLMs), a token is the foundational unit of data used to process and generate language. AI does not read text word-by-word or letter-by-letter. Instead, it breaks text down into these smaller mathematical chunks. Think of tokens as the building blocks or the "atomic level" of an AI's vocabulary. 1. How Text Becomes Tokens (Tokenization) Before an AI can understand your prompt, a piece of software called a tokenizer chops your text into pieces. The Core Rule: As a general rule of thumb, 1 token equals roughly 4 characters of English text , or about 0.75 words . The Math: 100 tokens are approximately equal to 75 words. Tokens do not always split cleanly at the space between words. They can be: Whole words: cat , the , apple Parts of words (sub-words): The word unbelievable might be split into un , believ , and able . Single characters: Punctuation marks ( ? , ! ), spaces, or emojis.  Visual Example If you pass the sentence: "T...