Can six seeds settle an architecture choice?

Okay so, train one model three times and only change the random seed. You don't get one validation score, you get three. Do the same for a second model and the two groups can overlap even when one average is lower. This is exactly what hit us on Day 2, most of the architecture gaps were smaller than the spread between copies of the same model. So Day 5 repeats the comparison, but with a much bigger training budget this time.

Three runs per stack, one threshold

Each dot is one possible validation score. The modern improvement slider moves all three modern dots left, and left is better, because lower bits per byte means the model put more probability on the held-out text. The three-seed spread slider pushes the dots in each row apart. Whichever row has the bigger range, that range becomes Day 5's minimum detectable effect, and if the gap between the two means is smaller than that, we call it unresolved. Try a gap of 0.02 with a spread of 0.04, and then shrink the spread to 0.01 without touching the gap. Same mean difference, but now it actually means something, because the runs got more consistent.

What does a seed actually change ? The initial weights, the order of the training blocks, and the other random draws. All six real runs use the same tokenizer, the same held-out blocks, batch shape, step count and training procedure. So this range is a practical noise floor for this exact setup, not some universal confidence interval.

On Day 2 the baseline range on code was 0.043 bits per byte. Those models saw 819,200 tokens each, roughly 0.25 tokens per parameter, which is basically nothing. The grid shows why those decimals couldn't settle most choices. And the planned Day 3 budget would still have left the models badly undertrained, so the plan changed.

Same data and same budget for both stacks

The baseline stack uses learned positions, LayerNorm, GELU and eight separate key-value heads. The modern stack uses RoPE, RMSNorm, SwiGLU and two grouped key-value heads. Both use PyTorch's scaled dot product attention, and we keep that kernel the same on purpose, so an attention backend difference can't sneak in and pretend to be an architecture result.

The baseline has 29,639,680 parameters and modern has 26,255,872, the baseline is bigger because of full multi-head attention. So to be honest this is a stack comparison, not a parameter-matched ablation. It tells us which stack does better under this budget, it can't tell us which of the four changes caused the gap. For that you'd go back to Day 2's single-change experiments, with their noise problem attached.

Each run trains for 6,000 steps × 32 sequences × 512 tokens = 98,304,000 tokens. 512-token contexts, an 8,192-entry vocabulary, the same fixed validation blocks. Three seeds per variant, so six runs. The budget is one quarter of Day 4's main run, long enough to compare at a trained scale and short enough that six runs fit in the GPU time we had. Whether it's long enough to actually resolve a gap, that's one of the things the experiment finds out.

def variant_config(base: ProjectConfig, variant: str, seed: int) -> ProjectConfig:
    model = base.model
    if variant == "baseline":
        model = replace(
            model, position="learned", norm="layernorm", feed_forward="gelu", kv_heads=0
        )
    elif variant == "mtp":
        model = replace(model, mtp_depth=2)
    elif variant != "modern":
        raise ValueError(f"unknown variant: {variant}")
    return replace(base, model=model, training=replace(base.training, seed=seed))

One trained model is a baseline, not a seed range

Day 4's modern model trained for 393 million tokens and got to 0.4839 validation bits per byte. That gives Day 5 a throughput and quality reference, but it is not one of the matched 98.3 million-token controls, if we mix budgets the architecture comparison means nothing. The full curve is here so you can see the difference between one training trajectory and a seed comparison.

The curve was still falling slowly at the end. That's one of the reasons Day 5 uses a matched shorter budget for both stacks, instead of comparing a short baseline against this longer modern run.

Finish all six runs before comparing means

octlm.day5 spread trains baseline and modern at seeds 1337, 1338 and 1339. Every finished run writes its config hash, data hash, tokenizer hash, model size, device, PyTorch version, validation loss, bits per byte and seconds per step. If a Kaggle session dies halfway, and they do, the runner reads the output, checks those hashes and only resumes the missing seeds. It rejects duplicate seeds and rows that came from different inputs.

def stage_spread(args: argparse.Namespace) -> None:
    base, tokenizer, train, valid, dataset_hash = inputs(args)
    existing, completed = read_runs(
        args.out, base, dataset_hash, tokenizer.fingerprint, ("baseline", "modern")
    )
    for variant in ("baseline", "modern"):
        for seed in SEEDS[: args.seeds]:
            if (variant, seed) in completed:
                continue
            row = train_variant(
                base, tokenizer, train, valid, dataset_hash, variant, seed, args.device
            )
            _write(args.out, row)
            existing.append(row)
            completed.add((variant, seed))
    for variant in ("baseline", "modern"):
        if all((variant, seed) in completed for seed in SEEDS[: args.seeds]):
            print(json.dumps({"variant": variant, **spread(existing, variant)}, sort_keys=True))

The result reports each variant's mean and max-minus-min range, then compares the gap between the means with the larger range. If the first run diverges or doesn't fit in the session, the stop condition is to revise the budget from the measured throughput. If the gap doesn't clear the range, the decision is unresolved. A lower average alone is not enough, that was the whole lesson of Day 2.

Six runs on a T4, and the gap is almost seven times the noise

All six runs finished on one Kaggle T4 with PyTorch 2.10 in float16. Every row has the same data hash, tokenizer hash and 98,304,000 training tokens. Nothing diverged, every validation curve fell at every 1,000-step check.

The bigger range is modern's, 0.0050 bits per byte, so that becomes the minimum detectable effect for the rest of Day 5. The means differ by 0.0338, which is 6.8 times that threshold, and even the worst modern seed beats the best baseline seed by 0.0295. On Day 2 the baseline range was 0.043 on code after 0.8M tokens, here it's 0.0020 on TinyStories after 98.3M. The corpus changed too, so it's not a clean before-and-after, but the decision is not sitting inside the noise anymore.

Decision: keep modern as the default stack. It gets 5.8% lower bits per byte with 11% fewer parameters, per token of training.

Per token, not per second

And here's the catch. Modern took 0.625 seconds per step against the baseline's 0.283, so 2.2 times as long, 62 minutes per run against 28. We matched tokens, not wall-clock time. If you gave the baseline the same hour it would see about twice the data, and this run doesn't tell us where it would land.

The likely cause is the attention kernel. PyTorch 2.10's SDPA documentation says grouped-query attention only works with the Flash and math kernels. Flash needs a newer GPU than the T4, so modern's two KV heads probably push it onto the slow math kernel, while the baseline's eight heads get to keep the memory-efficient one. SwiGLU and RoPE add a bit of cost too. But that's a lead, not a measurement, I haven't profiled it. Day 4's 0.614 seconds per step would carry the same cost.

EXP-070 now has its control mean of 0.5539 and its 0.0050 threshold.