How LLM tokens work - vocabularies, encoding and cost
- YT :: https://www.youtube.com/watch?v=nKSk_TiR8YA
- Original title :: Most devs don't understand how LLM tokens work
Tokens are the currency of LLMs: input text is split into the largest chunks found in a model's vocabulary, each chunk is mapped to a number, and the model computes and bills in those numbers. Pocock explains why the same prompt costs a different number of tokens on different providers, with TypeScript demos of real and toy tokenizers.
Why it matters
He taught an AI workshop in Poland where only about a third of the room knew what a token is. Input text is broken into tokens and billed at one rate per 1K tokens; the model's output tokens are billed at a different rate. Total cost is input tokens plus output tokens, each times its rate.
With the AI SDK, the same "Hello world" prompt reports 11 input and 20 output tokens on Claude 3.5 Haiku, but 4 input and 11 output tokens on Gemini 2.0 Flash Lite. Eleven input tokens for two words looks odd until you know how tokenizers work.
What a token is
Every LLM has its own vocabulary of words, subwords and characters, each assigned a number, and that number is the token. Encoding splits the text into the largest vocabulary pieces ("Hello", " world", "!") and looks up their numbers.
Using js-tiktoken with the o200k_base tokenizer (GPT-4o), a text of about 2,300 characters becomes fewer than 500 tokens, and "Hello world" (12 characters) becomes 3 tokens. The LLM works entirely on these numbers, not on text. The output tokens are decoded back to text by looking up the chunks and joining them.
How vocabularies are built
A tokenizer is trained on a corpus, usually the same terabytes of text the model is trained on. His toy corpus is the single line "the cat sat on the mat".
- Character-level tokenizer :: the vocabulary is just the unique characters (10 tokens), so token count always equals character count. That is the worst case, since more tokens means more work for the model.
- Subword-level tokenizer :: groups that occur together ("th", "he", "at") get their own tokens, so an 11-character input becomes 8 tokens. His demo only goes one level deep (two-character groups); real tokenizers keep merging groups of groups, e.g. "th" + "e" into "the".
Vocabulary size trade-off
A bigger vocabulary gives larger chunks and fewer tokens per text. "Understanding" might be five tokens at a 1K vocabulary, three at 50K and two at 200K. But a larger vocabulary needs a bigger model to house it and more memory to run, so providers pick different trade-offs, which explains the differing counts above.
Unusual words and languages
Rare words are split into more tokens: Lewis Carroll's made-up "frabjous" becomes four tokens in o200k. The same holds for languages that are scarce in the training data, including programming languages. Common languages are cheaper, so 20 lines of JavaScript cost fewer tokens than 20 lines of Haskell, another advantage for popular languages in the AI era.
Summary
- You are charged by the token, and providers tokenize the same input differently.
- Encoding turns text into the largest known chunks, then into numbers; decoding reverses it.
- The LLM thinks in tokens: encode, compute, produce output tokens, decode.