Multi-token prediction at 26 million parameters

So say the model just read “Lily found a”. The usual head guesses the next piece, “ key”. Now a second head tries to guess one piece further, “ in”, and it has to do that from the same hidden state. Slide through the example below, the two target rows move together until the second target runs off the end of the block.

One trunk, two different guesses

The normal model predicts $x_{t+1}$ from hidden state $h_t$. Depth 2 asks a second head to predict $x_{t+2}$ from that same state. The training loss averages the valid targets of both heads, the second head has no target at the last input position, and PAD targets are excluded. For validation we only report the next-token head against the single-head control, because if we scored the average two-head loss against a one-head loss we'd be answering a different question.

The second head needs its own transformation, and this one is easy to get wrong. If it used the same tied output matrix on the same hidden state it would produce the exact same logits, so it would just repeat the next-token prediction. So octlm adds one bias-free 512 × 512 linear adapter and then reuses the tied output matrix. That's 262,144 extra parameters, about 1 percent of the modern trunk. The paper's independent future heads use a full Transformer layer, octlm's adapter is cheaper and tests a smaller change, so I shouldn't call this a paper reproduction, it isn't one.

The earlier run was never evidence

Day 2's 3.3M models trained on just 819,200 tokens. Day 3 planned more tokens but still way less than a data-efficient run. We built and tested the MTP implementation, and then the architecture lab stopped before its GPU stage. So there is no Day 3 quality result or draft-agreement result to carry forward, nothing. The paper says MTP gains grow with model size, and that's a reason to test at 26M, not a prediction that this model will win.

Day 5 uses the three modern seeds from EXP-069 as the control. MTP runs the same three seeds, same tokenizer, same TinyStories token stream, 512-token context, 6,000 steps and the same held-out blocks. The only planned change is mtp_depth = 2. EXP-069 finished on 2026-09-30 and froze the threshold before any MTP run started, modern's control mean is 0.5539 bits per byte and the larger seed spread is 0.0050. So MTP has to beat 0.5539 by more than 0.0050 to count. We fix the number first so we can't pick a threshold after seeing the MTP results.

def stage_mtp(args: argparse.Namespace) -> None:
    base, tokenizer, train, valid, dataset_hash = inputs(args)
    controls, completed = read_runs(
        args.spread_in, base, dataset_hash, tokenizer.fingerprint, ("baseline", "modern")
    )
    if any(
        (variant, seed) not in completed
        for variant in ("baseline", "modern")
        for seed in SEEDS[: args.seeds]
    ):
        raise ValueError("EXP-069 must finish before EXP-070 starts")
    threshold = max(spread(controls, variant)["spread"] for variant in ("baseline", "modern"))
    existing, completed = read_runs(args.out, base, dataset_hash, tokenizer.fingerprint, ("mtp",))
    for seed in SEEDS[: args.seeds]:
        if ("mtp", seed) in completed:
            continue
        row = train_variant(base, tokenizer, train, valid, dataset_hash, "mtp", seed, args.device)
        _write(args.out, row)
        existing.append(row)
        completed.add(("mtp", seed))
    if all(("mtp", seed) in completed for seed in SEEDS[: args.seeds]):
        print(
            json.dumps(
                {
                    "control_mean": spread(controls, "modern")["mean"],
                    "mtp_mean": spread(existing, "mtp")["mean"],
                    "minimum_detectable_effect": threshold,
                },
                sort_keys=True,
            )
        )

Quality and draft agreement are different questions

Bits per byte of the next-token head asks, did the shared trunk learn a better language model ? The report also includes parameters and seconds per training step, because the extra loss and the extra projection aren't free. Depth-2 top-1 accuracy asks something else, how often is the head's best guess the true token two positions ahead on held-out blocks. A high rate could justify looking into draft verification, but it doesn't make the current generator faster by itself. octlm has no speculative verifier and no speculative speed result, just to be clear.

def mtp_agreement(model: Decoder, blocks: TokenBlocks, pad_id: int) -> float:
    """How often the depth-2 head's argmax is the true t+2 token. The Phase 5 draft signal."""
    if model.config.mtp_depth < 2:
        return 0.0
    model.eval()
    device = next(model.parameters()).device
    hits = total = 0
    with torch.inference_mode():
        for inputs, targets in zip(blocks.inputs, blocks.targets, strict=True):
            stack = model(inputs.unsqueeze(0).to(device), all_depths=True)
            predicted = stack[1, 0, :-1].argmax(dim=-1)
            actual = targets[1:].to(device)
            valid = actual != pad_id
            hits += int((predicted[valid] == actual[valid]).sum())
            total += int(valid.sum())
    model.train()
    return hits / max(1, total)

We can't use raw token perplexity across different tokenizers for this. Both Day 5 variants use the same tokenizer anyway, but bits per byte also ties this experiment to the earlier reports without switching units.

Three trained seeds lose beyond the noise

octlm.day5 mtp refuses to run if any requested EXP-069 control seed is missing. It checks the input hashes, trains only missing seeds, and reports both means against the frozen threshold. All three T4 runs finished on 2026-09-30 with matching config, data, and tokenizer hashes and 98,304,000 tokens per run.

MTP averages 0.572688 bits per byte against the control's 0.553899. The regression is 0.018788, 3.77 times the frozen threshold. Every MTP seed is worse than every control seed. The second head adds 262,144 parameters and costs 10.95% more training time per step. Its mean depth-2 accuracy is 73.08%, which measures the true future token and proves no speculative decoding speedup.

Decision: keep mtp_depth = 1 as the default. The measured adapter stays behind the depth-2 flag. This negative result applies to the linear adapter at this scale and budget; the paper uses Transformer heads.

Stop condition, if the mean next-token improvement is not bigger than the frozen spread, we record it as unresolved or negative and leave the head off by default. A draft head also needs a measured agreement rate before we even think about serving work.