Tokens, Not Words: How LLMs Actually Read Your Prompt

When you type a prompt, you see words. The model does not. Before a single layer of the network runs, your text is chopped into tokens and converted to a list of integers. Everything the model knows about language, it learned in terms of those tokens. Understanding tokenization is one of the fastest ways to stop being surprised by odd model behavior, from strange spelling mistakes to why some languages cost more to process.

What a token actually is

A token is a chunk of text, usually a common word, a piece of a word, or a single character. Common short words like 'the' or 'and' are typically one token each. Longer or rarer words get split into several pieces. The word 'tokenization' might break into 'token' and 'ization', while a made up string like 'zxqwv' could become one token per character. On average, English text runs close to 4 characters per token, so a useful rule of thumb is that 100 tokens is roughly 75 words.

Each token maps to an integer ID from a fixed vocabulary. The model's entire input is that sequence of numbers. It has no direct access to the letters inside a token, which is the root cause of several behaviors people find baffling.

How byte pair encoding builds the vocabulary

Most modern models use a method called byte pair encoding, or BPE. It starts from the raw bytes of text, so the base vocabulary is just the 256 possible byte values. That guarantees any character on earth, including emoji and scripts the model rarely sees, can always be represented without an 'unknown' token.

From there, training scans a huge corpus and repeatedly finds the most frequent adjacent pair of tokens, then merges it into a new single token. Merge the most common pair, add it to the vocabulary, and repeat. Frequent letter combinations become subwords, frequent subwords become whole words, and this continues until the vocabulary reaches a target size. The result is that common text compresses into few tokens while rare text stays fragmented.

Why vocabulary size is a tradeoff

Vocabulary size is a deliberate design choice. Early GPT-2 used a vocabulary of around 50,000 tokens. Later OpenAI tokenizers grew to roughly 100,000, and newer ones reach about 200,000. Bigger vocabularies pack more text into each token, so a given passage becomes a shorter sequence, which means less compute per forward pass and more room inside the context window.

The cost is on the other side. A larger vocabulary means a larger embedding table and a larger output layer, so the model spends more parameters just on knowing its tokens. Designers balance these forces, which is why different model families tokenize the same sentence into different token counts.

Why tokenization makes models stumble

Several classic model failures trace straight back to tokenization. Ask a model how many times the letter 'r' appears in 'strawberry' and it may get it wrong, because 'strawberry' arrives as a few tokens, not as individual letters. The model literally cannot see the letters inside its own tokens without extra effort.

Arithmetic suffers for a related reason. Numbers get split into token pieces in inconsistent ways, so '1234' and '12345' may not share a clean structure the model can reason over digit by digit. Whitespace matters too: a token that includes a leading space is different from the same word without one, which is why a stray trailing space in a prompt can nudge the output.

Language is the most expensive surprise. Tokenizers are trained mostly on English and English heavy data, so English compresses efficiently. Text in many non Latin scripts breaks into far more tokens for the same meaning. That inflates cost, eats context budget, and can slightly degrade quality for those languages, all before the model reasons about anything.

What this means for your prompts

A few practical habits follow directly. Token count, not word count, drives both price and how much fits in the context window, so trimming filler saves real money at scale. If you work in a non English language, expect higher token counts and budget accordingly. Do not lean on a model for character level tasks like counting letters or reversing strings unless you give it a way to work step by step. And when a prompt behaves strangely around numbers or spacing, suspect the tokenizer before you suspect the model's intelligence.

None of this requires you to compute tokens by hand. It just helps to remember that the model reads a different text than you wrote. Browse the PromptingIndex library for prompts that are already tight, specific, and tested by the community.