Measuring a tokenizer

Day 1 trained the same small model twice for the same 40 steps, once on character tokens and once on BPE tokens, and evaluated both on the same held-out file. If you read it by loss or perplexity, the character model wins by a lot. If you read it by bits per byte, BPE wins. And only one of those is actually a fair comparison.

Two models, three metrics, opposite answers

All six numbers come from runs/day1-character.jsonl and runs/day1-bpe.jsonl at step 40.

A token is a different amount of text in each model

The character model predicts one byte per step. The BPE model predicts almost two bytes per step on this file, because a BPE token covers 1.90 bytes on average. Predicting two bytes at once is harder than predicting one, so obviously the loss per prediction is higher. So comparing per-token loss across tokenizers is comparing predictions of different sizes, which doesn't make sense.

Bits per byte takes the size out. Take the total loss across the whole file, which is the information the model needed for that file, and divide by the file's bytes:

Both models describe the same bytes, so this asks both of them the same question, how many bits does the model need per byte of this text ?

Plug in the character model, one byte per token, so bits per byte is just the loss divided by $\ln 2$. For BPE the per-token loss is bigger, but each token pays for almost two bytes, and the division brings it under the character model.

So what happened ?

• At the measured losses, perplexity prefers the character model and bits per byte prefers BPE. The verdict line tells you which.
• Lower the BPE loss until its perplexity beats the character model. The BPE loss has to fall below the character model's loss, and by then its bits per byte is about half the character model's.
• The token count is where the whole difference lives. BPE used about half as many predictions for the same text.

Compare tokenizers only in bits per byte

After 40 steps, BPE improved bits per byte by 15.4 percent over the character model. That kept BPE for the Day 1 model and also made a rule that every later comparison follows. Perplexity is fine inside one tokenizer, but across tokenizers only bits per byte counts. AGENTS.md carries that rule for the rest of the project.

The comparison has limits, and the note says them. Forty steps on a tiny model and a tiny file gives you the direction of the effect, not the size of it at scale. It doesn't pick a vocabulary size either, that needs two full training runs, which is why Day 4 took 8,192 entries from the TinyStories paper's 10K and wrote it down as a choice, not a result.

Bytes per token, the other number to track

Bytes per token measures compression, and it changes with the text, not just the tokenizer.

The Day 2 row is the inverse of the 0.431 tokens per byte that notes/day2.md records. The Day 4 vocabulary is four times bigger but compresses TinyStories less than Day 2's compressed its corpus. That's because TinyStories has a small, simple vocabulary and the words are short, more merges can't make a three-letter word shorter than one token.

Better compression is good for speed, because a context window holds more text. But it's not the same as a better model. Bits per byte is the model's number, bytes per token is the tokenizer's.

What we didn't measure, and why

• The final vocabulary size. Two planning docs suggested 16K to 32K. A choice like that needs trained models at each size, and Day 1 only measured what its small corpus could support.
• Per-token perplexity across tokenizers. It's recorded in the runs and never used to compare them.