What a language model predicts

Everything octlm builds serves one task. Given the tokens so far, output a probability for every token in the vocabulary as a guess for the next one. Generation, training and every metric on this site follow from that.

A probability for every possible next token

Text is first split into tokens, small pieces with integer IDs. The Day 1 tokenizer has 1,024 of them, the Day 4 one has 8,192. Characters vs bytes covers how the split works. For now a token is a number between 0 and V − 1.

The model reads tokens $x_1, \dots, x_t$ and produces one score per vocabulary entry, a vector of $V$ numbers called logits. Softmax turns the scores into probabilities:

The exponent makes every probability positive, and the division makes them sum to 1. A higher score means a higher probability, and only the differences between scores matter. Adding 5 to every score changes nothing.

That is the whole interface. The model does not output a word. It outputs a distribution, and something outside the model picks from it. It takes the most likely token for greedy decoding, or a random draw for sampling, which the sampling post covers.

Scoring one guess

What happened?

• Raising the score for " mat" takes probability from every other word at once, because the six probabilities must sum to 1.
• Cross entropy is 0 only when p is 1, and it grows without bound as p approaches 0. A confident wrong guess costs far more than an unsure one.
• Perplexity is $1/p$ for one prediction. At p = 0.25 it is 4, so the model is as unsure as a fair four-way choice.
• Bits per byte divides the bits by how many bytes of text the token covered. Move the bytes slider and only that number changes.

Cross entropy, averaged over every position

Training adjusts the model's weights to raise the probability of the token that actually came next. The loss for one position is the negative log of that probability:

Over a whole text of $N$ predictions, the loss is the average:

Why a logarithm? The probability of the whole text is the product of the per-token probabilities, and the log turns that product into a sum. Minimizing the average negative log probability is the same as maximizing the probability of the training text. It also has a meaning from information theory. $-\log_2 p$ is the number of bits an ideal compressor using the model would spend on that token. A language model is a compressor, and its loss is its compressed size.

The natural log gives the loss in nats. Dividing by $\ln 2$ converts to bits. PyTorch's cross_entropy uses nats, and so does every loss field in runs/.

A useful reference point is the loss of a model that knows nothing and spreads probability evenly, which is $\ln V$. For the Day 1 vocabulary that is $\ln 1024 \approx 6.93$ nats. A fresh model should start near there. The baseline block shows what happened when ours did not.

One sequence is many training examples

A decoder predicts at every position in parallel. Feed it $T$ tokens and it returns $T$ distributions. The target for position $t$ is the token at $t + 1$, so the targets are the inputs shifted left by one.

What happened?

• A sentence of a dozen tokens gives a dozen predictions, all computed at once. This is why training a decoder is efficient. One pass gives one loss per position.
• Position $t$ must not see token $t + 1$, which sits in the same input. Causal attention enforces that with a mask.
• The last input token has a target, <|eos|>. Nothing predicts <|bos|>, because nothing comes before it.

Perplexity is the loss in a friendlier unit

Perplexity is $e^{\mathcal{L}}$, the exponent of the average loss. It is the geometric mean of $1/p$ across positions:

A perplexity of 30 means the model is, on average, as unsure as a fair choice among 30 tokens. A uniform model has perplexity $V$.

Perplexity has a trap. It depends on what a token is. A tokenizer whose tokens cover more text makes each prediction harder, because each token carries more information, and so its perplexity is higher even when the model is better. The Day 1 results hit this directly: the BPE model's perplexity was 8 times the character model's, and it was the better model. Measuring a tokenizer works through the numbers.

A unit every tokenizer shares

Every tokenizer in octlm turns text into tokens and back without loss, so the underlying text is the same whatever the split. Bits per byte measures the model against that text:

The numerator is the total information in nats over all predictions. The denominator is the number of bytes those predictions covered, times $\ln 2$ to turn nats into bits. The result is how many bits the model needs per byte of text. Plain UTF-8 English costs 8 bits per byte. A good compressor like gzip reaches about 2 to 3. A trained language model goes lower.

Special tokens like <|eos|> count toward the loss but cover 0 bytes. PAD targets count toward neither.

octlm computes all three numbers in one pass over the validation blocks:

def evaluate(model: Decoder, blocks: TokenBlocks, pad_id: int) -> dict[str, float]:
    model.eval()
    device = next(model.parameters()).device
    nll_sum = 0.0
    token_count = 0
    byte_count = 0
    with torch.inference_mode():
        for inputs, targets, byte_lengths in zip(
            blocks.inputs, blocks.targets, blocks.target_bytes, strict=True
        ):
            inputs, targets = inputs.to(device), targets.to(device)
            byte_lengths = byte_lengths.to(device)
            logits = model(inputs.unsqueeze(0))
            losses = F.cross_entropy(
                logits.flatten(0, 1), targets, ignore_index=pad_id, reduction="none"
            )
            valid = targets != pad_id
            nll_sum += losses[valid].sum().item()
            token_count += valid.sum().item()
            byte_count += byte_lengths[valid].sum().item()
    model.train()
    nll = nll_sum / max(1, token_count)
    return {
        "bits_per_byte": nll_sum / (math.log(2) * max(1, byte_count)),
        "loss": nll,
        "perplexity": math.exp(nll),
        "tokens": token_count,
    }

nll_sum adds up the loss over real targets. byte_count adds up how many bytes those targets decode to, from a per-token table the tokenizer provides. Perplexity comes from the average, bits per byte from the total.

What we did not measure, and why

• Accuracy. The share of positions where the top token is right ignores how much probability the model gave the right answer when it was wrong. Loss uses all of it.
• Downstream benchmarks. A model this small scores at chance on standard benchmarks. Bits per byte on held-out text is the number that moves. The harness eval on Day 7 is the task measure.