What a language model predicts

Everything octlm builds is for one task. Given the tokens so far, output a probability for every token in the vocabulary as a guess for the next one. That's it. Generation, training and every metric on this site come out of that one thing.

A probability for every possible next token

Text first gets split into tokens, small pieces with integer IDs. The Day 1 tokenizer has 1,024 of them, the Day 4 one has 8,192. Characters vs bytes covers how the split works, for now just think of a token as a number between 0 and V − 1.

The model reads tokens $x_1, \dots, x_t$ and produces one score per vocabulary entry, so a vector of $V$ numbers called logits. Softmax turns those scores into probabilities:

The exponent makes every probability positive and the division makes them sum to 1. Higher score means higher probability, and only the differences between scores matter, if you add 5 to every score nothing changes.

And that's the whole interface. The model doesn't output a word. It outputs a distribution, and something outside the model picks from it, either the most likely token for greedy decoding, or a random draw for sampling, which the sampling post covers.

Scoring one guess

So what happened ?

• Raising the score for " mat" takes probability away from every other word at once, because the six probabilities have to sum to 1.
• Cross entropy is 0 only when p is 1, and it grows without limit as p goes to 0. So a confident wrong guess costs way more than an unsure one.
• Perplexity is $1/p$ for one prediction. At p = 0.25 it's 4, meaning the model is as unsure as a fair four-way choice.
• Bits per byte divides the bits by how many bytes of text the token covered. Move the bytes slider and only that number changes.

Cross entropy, averaged over every position

Training nudges the model's weights to raise the probability of the token that actually came next. The loss for one position is the negative log of that probability:

Over a whole text of $N$ predictions, the loss is just the average:

Why a log though ? The probability of the whole text is the product of the per-token probabilities, and the log turns that product into a sum. So minimizing the average negative log probability is the same thing as maximizing the probability of the training text. And there's an information theory reading too, $-\log_2 p$ is the number of bits an ideal compressor using this model would spend on that token. So a language model is basically a compressor, and its loss is its compressed size.

Natural log gives the loss in nats, divide by $\ln 2$ and you get bits. PyTorch's cross_entropy uses nats, and so does every loss field in runs/.

A useful reference point is the loss of a model that knows nothing and spreads probability evenly, which is $\ln V$. For the Day 1 vocabulary that's $\ln 1024 \approx 6.93$ nats, so a fresh model should start somewhere near there. The baseline block shows what happened when ours didn't.

One sequence is many training examples

A decoder predicts at every position in parallel. Feed it $T$ tokens and it gives back $T$ distributions. The target for position $t$ is the token at $t + 1$, so the targets are just the inputs shifted left by one.

So what happened ?

• A sentence of a dozen tokens gives a dozen predictions, all computed at once. That's why training a decoder is efficient, one pass gives one loss per position.
• Position $t$ must not see token $t + 1$, which is sitting in the same input. Causal attention enforces that with a mask.
• The last input token has a target, <|eos|>. Nothing predicts <|bos|>, because nothing comes before it.

Perplexity is the loss in a friendlier unit

Perplexity is $e^{\mathcal{L}}$, the exponent of the average loss, which works out to the geometric mean of $1/p$ across positions:

A perplexity of 30 means the model is, on average, as unsure as a fair choice among 30 tokens. A uniform model has perplexity $V$.

But perplexity has a trap, it depends on what a token is. If a tokenizer's tokens cover more text, each prediction is harder because each token carries more information, so its perplexity goes up even when the model is better. The Day 1 results hit exactly this, the BPE model's perplexity was 8 times the character model's, and it was the better model. Measuring a tokenizer works through the numbers.

A unit every tokenizer shares

Every tokenizer in octlm turns text into tokens and back without losing anything, so the underlying text is the same no matter how it's split. Bits per byte measures the model against that text:

The top is the total information in nats over all predictions. The bottom is the number of bytes those predictions covered, times $\ln 2$ to turn nats into bits. So the result is how many bits the model needs per byte of text. Plain UTF-8 English costs 8 bits per byte, a good compressor like gzip gets to about 2 to 3, and a trained language model goes lower than that.

Special tokens like <|eos|> count toward the loss but cover 0 bytes. PAD targets count toward neither.

octlm computes all three numbers in one pass over the validation blocks:

def evaluate(model: Decoder, blocks: TokenBlocks, pad_id: int) -> dict[str, float]:
    model.eval()
    device = next(model.parameters()).device
    nll_sum = 0.0
    token_count = 0
    byte_count = 0
    with torch.inference_mode():
        for inputs, targets, byte_lengths in zip(
            blocks.inputs, blocks.targets, blocks.target_bytes, strict=True
        ):
            inputs, targets = inputs.to(device), targets.to(device)
            byte_lengths = byte_lengths.to(device)
            logits = model(inputs.unsqueeze(0))
            losses = F.cross_entropy(
                logits.flatten(0, 1), targets, ignore_index=pad_id, reduction="none"
            )
            valid = targets != pad_id
            nll_sum += losses[valid].sum().item()
            token_count += valid.sum().item()
            byte_count += byte_lengths[valid].sum().item()
    model.train()
    nll = nll_sum / max(1, token_count)
    return {
        "bits_per_byte": nll_sum / (math.log(2) * max(1, byte_count)),
        "loss": nll,
        "perplexity": math.exp(nll),
        "tokens": token_count,
    }

nll_sum adds up the loss over real targets. byte_count adds up how many bytes those targets decode to, using a per-token table from the tokenizer. Perplexity comes from the average, bits per byte from the total.

What we didn't measure, and why

• Accuracy. The share of positions where the top token is right ignores how much probability the model gave the right answer when it was wrong. Loss uses all of that.
• Downstream benchmarks. A model this small scores at chance on standard benchmarks. Bits per byte on held-out text is the number that actually moves. The harness eval on Day 7 is the task measure.