How LLM Tokenization Actually Works (And Why Your Token Count Is Always Wrong)

Every LLM API bills you per token, every context window is measured in tokens, and almost nobody can predict how many tokens a piece of text will be. The usual heuristic — “about four characters per token” — is a population average that can be off by 4× on text that isn’t English prose, which is exactly the text people tend to be surprised by.

Here’s what a token actually is.

The problem tokenization solves

A language model needs a fixed vocabulary — a finite list of symbols it can read and emit. Two obvious choices both fail:

• One token per word. Vocabulary is unbounded. You will always meet a word you’ve never seen: a typo, a product name, a German compound, a variable name like getUserPreferences. Unknown words become <UNK> and the information is simply gone.
• One token per character. Vocabulary is tiny and nothing is ever unknown, but sequences become enormous. “tokenization” is 12 steps instead of 1, and since attention cost grows with sequence length, you’ve made every model vastly more expensive to run.

Byte-pair encoding is the compromise: learn a vocabulary of subword pieces from the training corpus, so common words are single tokens and rare words decompose into fragments. Nothing is ever unknown, and sequences stay short.

BPE: it’s a merge list

The training algorithm is genuinely simple. Start with a vocabulary of individual characters. Then repeat: find the most frequent adjacent pair in the corpus, merge it into a new single symbol, add it to the vocabulary. Stop when you hit your target vocabulary size.

Sketching it on a toy corpus of low, lower, lowest:

start:   l o w   l o w e r   l o w e s t
merge 1: "l"+"o" → "lo"      lo w   lo w e r   lo w e s t
merge 2: "lo"+"w" → "low"    low   low e r   low e s t
merge 3: "e"+"s" → "es"      low   low e r   low es t
merge 4: "es"+"t" → "est"    low   low e r   low est

After four merges, low and est are single tokens. The learned artifact is just an ordered list of merge rules, and encoding new text means applying those rules in order. That ordering matters: BPE is deterministic and greedy, applying earlier (more frequent) merges first.

Modern tokenizers — the ones behind GPT-2 onward — use byte-level BPE. The base vocabulary isn’t characters, it’s the 256 possible byte values. This guarantees any input whatsoever is encodable, including emoji, invalid UTF-8, and scripts the model has barely seen. There is no <UNK> token. There is no such thing as text this tokenizer can’t read.

The consequences that surprise people

Everything below follows mechanically from “the vocabulary is learned from a corpus by frequency.”

Leading spaces are part of the token

This is the single most common source of confusion. In byte-level BPE, whitespace isn’t stripped — it’s absorbed into the following token. " hello" (with the space) and "hello" (without) are different tokens with different IDs.

That’s an efficiency win: since most words in running text are preceded by a space, baking the space in halves the token count versus emitting a separate space token every time. But it means "hello world" tokenizes as ["hello", " world"] — two tokens — while "hello" and "world" concatenated without the space is something else entirely.

It also explains a class of prompt-engineering bug: a trailing space at the end of your prompt can genuinely change the model’s output, because it changes which token the model has to continue from.

Common words are cheap; rare words are shrapnel

Frequency in the training corpus decides everything. Roughly:

• the, and, of — 1 token each
• Most common English words — 1 token
• Longer or less common words — 2–3 pieces
• Proper nouns, technical jargon, typos — often 3–6 pieces
• A long random string or hash — close to 1 token per few characters

This is why token counts on code, UUIDs, and base64 blobs blow past every estimate. A UUID is high-entropy hex with no repeated substructure for BPE to have learned, so it costs far more than its 36 characters suggest.

Non-English text costs multiples more

The training corpora behind these vocabularies are English-dominated. English words earned their own merge rules; other languages often didn’t, so their text falls back toward per-character or even per-byte encoding.

The effect is compounded by UTF-8: a Latin character is 1 byte, but CJK characters are 3 bytes each and many emoji are 4. If a Japanese character doesn’t have a learned merge covering it, you’re paying multiple tokens for one character.

Practically, the same meaning conveyed in Japanese, Korean, Thai, or Hindi can cost two to four times more tokens than the English version. Since billing and context windows are both counted in tokens, non-English users are structurally charged more for the same task and get less usable context. Newer tokenizers with larger vocabularies have narrowed this gap considerably, but they haven’t closed it.

Numbers tokenize badly

There’s no principled reason for a model’s vocabulary to align with place value, and it doesn’t. Depending on the tokenizer and the specific digits, 1234 might be one token, or 12+34, or 1+234. Some tokenizers deliberately split all numbers into fixed-size digit groups for exactly this reason.

This is a decent part of why LLMs have historically been shaky at arithmetic: the digits of a number don’t necessarily arrive as separate, positionally meaningful units.

Why the same text has different counts on different models

The tokenizer is not universal. It’s a specific artifact trained alongside a specific model family, and it changes between generations.

Encoding	Approx. vocab size	Used by
r50k_base / gpt2	~50k	GPT-2, GPT-3 (davinci)
p50k_base	~50k	Codex, older instruct models
cl100k_base	~100k	GPT-4, GPT-3.5 Turbo, text-embedding-ada-002
o200k_base	~200k	GPT-4o, GPT-4.1, o-series

A larger vocabulary means fewer tokens for the same text — more merges were learned, so longer pieces are single tokens. Moving from cl100k_base to o200k_base typically reduces token counts on the same input, and the reduction is largest on non-English text, which is where the extra 100k vocabulary slots mostly went.

The practical implication: a token count is only meaningful with respect to a named encoding. “This prompt is 1,200 tokens” is an incomplete statement. Our token counter runs the real o200k_base and cl100k_base encodings side by side so you can see the difference on your own text rather than trusting a ratio.

A note on Claude and other non-OpenAI models

Anthropic’s tokenizer is not publicly distributed the way tiktoken’s encodings are, and neither are most other vendors’. Any tool showing you a “Claude token count” locally is estimating — ours is explicit about it, using a characters-per-token divisor of 3.5 and labeling the result as an approximation rather than dressing it up as exact.

If you need an exact count for a non-OpenAI model, the reliable route is that vendor’s own token-counting API endpoint. For budgeting and staying under a context limit, an estimate with headroom is fine. For anything where being off by 10% matters, don’t estimate.

Practical takeaways

• Don’t trust characters ÷ 4 on anything but English prose. Check code, structured data, and non-English text against a real tokenizer.
• Watch structured formats. JSON and XML spend a real fraction of their tokens on braces, quotes, and indentation. Minifying a JSON payload before putting it in a prompt is a free reduction — our JSON formatter will do it.
• Repeated boilerplate is repeated cost. A system prompt sent on every request is billed on every request, unless you’re using prompt caching.
• Output tokens count too, and are typically billed at a higher rate than input. “Be concise” is a cost control.
• Turn counts into dollars. The token counter’s cost calculator prices the same prompt across current GPT, Claude, and Gemini API rates — per call and per 1,000 calls, cheapest first.
• Count before you hit the limit, not after. Context-window errors surface at the worst possible moment.

Try it

• Token Counter — exact o200k_base and cl100k_base counts with a colored token-boundary view, so you can see exactly where the splits land. Runs entirely in your browser.
• Word Counter — for when characters and words are what you actually need.
• JSON Formatter — minify structured payloads before they go into a prompt.