Tokens and tokenization: why LLMs can't count the r's in strawberry
A token is the unit of text an LLM reads and writes: often a whole common word, sometimes part of a word, a space plus a word, a punctuation mark or a single byte. A tokenizer splits your text into tokens and maps each one to a number, usually with an algorithm called byte pair encoding (BPE), and the model only ever sees those numbers. That is why models struggle to count letters, why cost and context limits are measured in tokens instead of words, and why the same sentence costs more in Hindi or Japanese than in English.
Why models need tokens
A model can't read text. It can only do math on numbers, so before anything else, text has to become numbers.
The simplest way already exists in every computer. Each key on your keyboard has a code: in ASCII, capital A is 65, capital B is 66, small a is 97, and a space is 32. Those numbers can be converted to binary, the zeros and ones a machine works with, by dividing by 2 and keeping the remainders, so 10 becomes 1010.
But a number for each character doesn't carry meaning. 65 tells the machine it is looking at an A, and nothing about what the word "India" or the sentence "where is my order" means. So language processing moved from characters, to words, to the subword tokens that LLMs use today, and then to embeddings, which give each token a vector that can carry meaning, as the embeddings page explains.
Three ways to split text
There are three basic approaches, and the reason LLMs use the third becomes clear when you compare them.
Character tokens
Every character is a token. The vocabulary is tiny, a few hundred characters, and there is never an unknown word. But sequences become very long, because every word becomes 5 to 10 tokens, and each token carries almost no meaning on its own, so the model has to work much harder to learn anything.
Word tokens
Every word is a token. Each token carries a lot of meaning, but English alone has hundreds of thousands of words, plus names, typos, code, other languages and new words every year. The vocabulary would be enormous, and any word not in it becomes "unknown", so the model can't read it at all.
Subword tokens
Common words are single tokens, and rare words are split into common pieces. "Tokenization" might become "Token" and "ization", and a word the tokenizer has never seen can still be built from smaller pieces, down to single bytes if needed.
This keeps the vocabulary at a manageable size, usually between about 30000 and 250000 tokens, keeps sequences reasonably short, and never fails on unknown text. That balance is why almost every modern LLM uses subword tokenization.
Byte pair encoding
The most common subword method is byte pair encoding (BPE). Its idea is simple: start with the smallest units and repeatedly merge the pair that appears together most often.
Training a BPE tokenizer works like this:
- Convert the training text into bytes, so every possible character is covered from the start.
- Count every pair of neighbouring tokens in the text.
- Merge the most frequent pair into a new token, and add it to the vocabulary.
- Repeat steps 2 and 3 until the vocabulary reaches the size you want.
The merges are based only on how often pieces appear next to each other, not on meaning. "th" becomes a token because it is frequent in English, and later " the" becomes one token for the same reason. Meaning comes later, in the embeddings and the transformer layers.
To encode new text, the tokenizer applies the same merges in the same order. To decode, it maps each token ID back to its bytes and joins them.
Here is a minimal BPE in plain Python, enough to see the mechanism.
from collections import Counter
def count_pairs(ids: list[int]) -> Counter:
return Counter(zip(ids, ids[1:]))
def merge(ids: list[int], pair: tuple[int, int], new_id: int) -> list[int]:
out, i = [], 0
while i < len(ids):
if i < len(ids) - 1 and (ids[i], ids[i + 1]) == pair:
out.append(new_id)
i += 2
else:
out.append(ids[i])
i += 1
return out
def train_bpe(text: str, num_merges: int) -> dict[tuple[int, int], int]:
ids = list(text.encode("utf-8"))
merges = {}
for step in range(num_merges):
pairs = count_pairs(ids)
if not pairs:
break
top = pairs.most_common(1)[0][0]
new_id = 256 + step
ids = merge(ids, top, new_id)
merges[top] = new_id
return merges
def encode(text: str, merges: dict[tuple[int, int], int]) -> list[int]:
ids = list(text.encode("utf-8"))
for pair, new_id in merges.items():
ids = merge(ids, pair, new_id)
return ids
corpus = "the tokenizer splits the text into tokens and the model reads the tokens " * 20
merges = train_bpe(corpus, num_merges=30)
sentence = "the tokenizer reads the text"
print(len(sentence.encode("utf-8")), "bytes ->", len(encode(sentence, merges)), "tokens")
After 30 merges on this small corpus, the 28 byte sentence becomes 13 tokens, because frequent pieces like "the " and "token" have been merged into single tokens. Production tokenizers do the same thing on billions of words, with extra rules, for example to stop merges across spaces and punctuation in odd ways.
What real tokens look like
You can see real tokenization with OpenAI's open-source tiktoken library or any provider's online tokenizer tool.
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
for text in ["This is an apple", "This is an apple.", "Tokenization", "strawberry"]:
ids = enc.encode(text)
pieces = [enc.decode([i]) for i in ids]
print(len(ids), pieces)
A few patterns show up immediately.
- Common words are one token each, and the space before a word is usually part of the token, so " is" and "is" are different tokens.
- Adding a full stop adds a token.
- Longer or rarer words are split into pieces.
- Every piece has a fixed ID in the vocabulary.
As a rough rule for English, one token is about 4 characters or about three quarters of a word, so 1000 tokens is around 750 words. Use it only for quick estimates, because the real number depends on the tokenizer and the text, and code, numbers and other languages behave very differently.
Tokenizer, token IDs and embeddings
These three things are often mixed up, so it helps to separate them.
The tokenizer turns text into token IDs, which are just integers, like 3923 for " apple". The model then looks up each ID in its embedding table, a learned matrix with one row per token in the vocabulary, and gets a vector, a long list of floating point numbers. Those vectors are what the transformer layers process.
The vector from the embedding table is the same for a token every time, whatever the sentence. " bank" starts with the same vector in "river bank" and in "bank account". It is the attention layers that mix in information from the surrounding tokens, so by the end of the model, the two "bank" tokens have very different representations. Older word embeddings like Word2Vec and GloVe stop at that first step, one fixed vector per word, while transformer models like BERT and GPT produce contextual representations.
Why models can't count the r's in strawberry
Ask many LLMs how many times the letter r appears in "strawberry", and for a long time a common answer was 2. The reason is tokenization.
The model never sees the letters s, t, r, a, w, b, e, r, r, y. It sees two or three token IDs, something like "str", "aw" and "berry". Information about which letters are inside each token has to be learned indirectly, and it often isn't learned reliably.
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
word = "strawberry"
ids = enc.encode(word)
print("model sees:", [enc.decode([i]) for i in ids], ids)
print("python counts r:", word.count("r"))
Python counts the letters correctly, because it works on characters. The model works on token IDs.
Newer models often get this particular question right, because they reason step by step, spell the word out first, or were trained on this exact example. But the underlying limit is still there, and it shows up in related tasks: reversing words, counting characters, working with exact spellings, rhymes and anagrams.
The engineering lesson is not "LLMs are bad at spelling". It is: when a task needs exact operations on characters or numbers, do it in code, and give the model a tool for it.
Numbers and arithmetic
Numbers are tokenized too, and not always in a way that matches place value. Depending on the tokenizer, "12345" might be one token, or "123" and "45", or individual digits, and two numbers that look similar can be split differently.
That is one reason models make arithmetic mistakes, especially with long numbers, even when their reasoning about the problem is correct. For any calculation that matters, like prices, totals, tax or dates, have the model call a calculator or write code, and use the result, rather than trusting the number it writes.
Why non-English text costs more
Most tokenizers are trained on text that is mostly English. English words therefore get efficient single tokens, while words in many other languages are split into many more pieces, especially in scripts like Devanagari, Tamil, Thai or Amharic.
The same meaning can take several times more tokens in Hindi than in English. Research by Petrov and others in 2023 found differences of up to 15 times between languages for the same content with some tokenizers.
That matters in three ways:
- Cost: you pay per token, so the same conversation costs more.
- Context: the same context window holds less text.
- Speed: more output tokens take longer to generate.
What to do about it:
- Measure. Count tokens for real samples of your users' text with the tokenizer of the model you plan to use, before you estimate costs.
- Compare models. Newer tokenizers with larger vocabularies are much more efficient for many languages than older ones, so token counts for the same Hindi text can vary a lot between providers.
- Consider where translation fits. Some systems translate the input into English, process it and translate back, which saves tokens but adds a step that can lose meaning, so test it on your real data before adopting it.
Different models, different tokenizers
Every model family has its own tokenizer and vocabulary. The same text can be 100 tokens for one model and 130 for another, so token counts, limits and prices are not directly comparable between providers.
Count tokens with the tokenizer of the model you will use. Most providers offer a token counting endpoint or a library for their models, and the API response tells you how many tokens each call actually used, which is the number you should log.
Special tokens and chat templates
Tokenizers also have special tokens that don't represent text. An end-of-sequence token tells the model to stop generating. Chat models use tokens that mark where the system message, the user's message and the assistant's reply start and end.
When you send a list of messages to a chat API, the provider converts them into one sequence of tokens with these markers. That is why a short message costs a few more tokens than its text alone, and why, when you run open models yourself, using the correct chat template matters, because a model given the wrong markers often produces strange output.
Build it yourself
The best way to understand tokenization is to build three tokenizers and compare them on the same text.
- A character tokenizer: build a vocabulary of every character in a small corpus, with
encodeanddecodefunctions. - A word tokenizer: split on spaces and punctuation with a regular expression, build a word vocabulary, and see what happens with a word that isn't in it.
- A BPE tokenizer: train it on the same corpus with the code above.
For each, check the round trip, decode(encode(text)) == text, measure the vocabulary size and the number of tokens per sentence, and compare with tiktoken.
Then run the strawberry check and count tokens for the same sentence in English and in another language you know.
From the floor
Build30 day 3 was tokenization, and we built exactly this: a character tokenizer, a word tokenizer, a small byte pair encoder, a tiktoken lab, the strawberry demo and a multilingual cost comparison, all logged with the token tracker from day 1. In the multilingual comparison, "Good morning, how are you today?" took 7 tokens in English, and the same sentence in Hindi took around 40 tokens in our comparison, which is the clearest way I know to show a builder why a client with Hindi-speaking users needs a different cost estimate.
In the same session, a builder asked a precise question: when a token is first embedded, before the transformer, does its vector already contain context, or is context only added in the attention layers? Another builder answered with the bank example, that Word2Vec gives "bank" the same vector in every sentence while BERT gives it different representations depending on the sentence, and that is the right way to think about it: the first lookup is context-free, and attention adds the context.