Characters as tokens

A model reads integers, not text. So a tokenizer is basically the pair of functions that turns text into a list of integer IDs and back again. And its design decides a lot, how long every sequence is, what the model can even represent, and what one prediction actually means.

Day 1 built two tokenizers. This post is the first and simplest one, which is also on purpose the baseline for the second one.

One character, one ID

The character tokenizer collects every distinct character in the training text, sorts them and numbers them. Four special tokens come first:

Encoding looks each character up. Decoding looks each ID up and joins the characters. That's the whole algorithm, like literally:

def encode(self, text: str, add_special_tokens: bool = False) -> list[int]:
    lookup = self.token_to_id
    tokens = [lookup.get(character, 3) for character in text]
    return [self.bos_id, *tokens, self.eos_id] if add_special_tokens else tokens

lookup.get(character, 3) is the important bit. A character outside the vocabulary becomes ID 3, and there's no way back from that. Decoding ID 3 gives you <|unk|>, not the original character.

A vocabulary learned from one file

EXP-003 trained the character tokenizer on the project's own PLAN.md. The playground below learns its vocabulary from today's PLAN.md at build time, same way.

So what happened ?

• English and Python encode cleanly, because PLAN.md is English prose with code-ish names in it. Every token is one character, so the token count equals the character count.
• The Japanese preset is almost all unknown. PLAN.md has no Japanese in it, so those characters just don't have an ID.
• The emoji are unknown for the same reason. One emoji is one character here, but it's four UTF-8 bytes.

What we measured

Problem, we needed a tokenizer to compare BPE against, trained on the same text.

The character tokenizer learned 86 entries from PLAN.md, 4 specials and 82 characters. On the held-out file, day-wise.md, it produced 13,939 tokens for 13,939 bytes. That file is plain ASCII, so one character is one byte is one token.

On an unseen Unicode sample, 77.8 percent of the characters mapped to <|unk|>. That's bad.

Then a small model trained on each tokenizer's output for the same 40 steps. With the character tokenizer, validation loss got to 3.4011 nats per token, perplexity 29.998, and 4.9068 bits per byte. Those three numbers come from runs/day1-character.jsonl. Measuring a tokenizer puts them next to BPE.

Decision, keep it as the baseline. It was never supposed to be the model's actual tokenizer.

Why characters hurt, especially on code

Day 1's paper exercise asked for three reasons character tokens hurt a model that reads code. The note has these:

1. Sequences get long. Every character is a position. A 4-space indent is 4 positions, return is 6. Attention cost grows with the square of the length, and the context window fills up with syntax.
2. The model spends its capacity spelling. To predict values after sum(, a character model has to predict v, then a, then l, each one a separate decision. A tokenizer that has values as one token makes that one decision.
3. Unknown characters destroy information. The vocabulary is fixed at training time. A new symbol, an accented name or an emoji in a string literal becomes <|unk|>, and the model can't read or write it.

The small vocabulary does have one real advantage, every ID gets a lot of training data. The next tokenizer keeps that for its base alphabet and fixes all three problems.

What every octlm tokenizer has to preserve

AGENTS.md sets rules that both tokenizers follow, and there are tests for each one:

• Exact round trip. Decoding the encoded text gives back the same string. No lowercasing and no Unicode normalization, so é written as one code point and é written as e plus a combining accent stay different.
• Tabs, indentation and CRLF survive. Code depends on whitespace.
• Stable IDs. Special-token IDs never move between versions, and loading a tokenizer file with different specials fails.
• Ordinary text never becomes a special token. If someone literally types <|eos|> into a document, it's encoded as characters, never as ID 2. Otherwise a document could end itself early, which is a pretty bad bug.

What we didn't build, and why

• A bigger character vocabulary. Training on more text would shrink the unknown rate, but no amount of training text covers every character in Unicode. Bytes do, and BPE is built on bytes.
• Word-level tokens. A word vocabulary has the unknown-token problem way worse than characters, and code is full of names no dictionary has.