Why the plan changed

On 2026-09-23 the project stopped before its Day 3 runs and rewrote the plan. Two facts forced it. The Day 2 models had seen about 1 percent of the data they needed, and no model this lab can pretrain could ever drive a tool-use harness.

The first plan was to pretrain a 50 to 150 million parameter model and grow it, over 64 experiments, into a local coding and writing assistant. Day 3 had four experiments built and ready for a GPU. And right before the first run I stopped and asked, okay, is this actually the right track ? Working through our own numbers said no, it wasn't.

Fact 1

The Day 2 models were starved of data

Hoffmann et al. found that for a fixed compute budget, the best loss comes from training on about 20 tokens per parameter. Fewer tokens means the model is bigger than the budget can use.

The Day 2 grid trained models of about 3.3 million parameters for 400 steps of 8 blocks of 256 tokens, so 819,200 tokens each. That's about 0.25 tokens per parameter, 1 percent of the compute-optimal budget. The Day 3 plan raised the steps fivefold, to 4.1 million tokens, which is still only about 1.1 tokens per parameter. Its long-context config reached 8.2 million tokens, and the whole Day 2 corpus holds 2.96 million, so it would have repeated data too.

And that explains the Day 2 grid. A seed spread of 0.043 bits per byte, bigger than most of the effects we were measuring, is just what an undertrained model looks like. Where it ends up depends a lot on where it started. Stage 0 would have measured that spread again, at a budget that's still about 18 times too small.

Playground

Training compute is about 6 × parameters × tokens, 2 for the forward pass and 4 for the backward pass, per parameter per token. Time assumes one T4 at a sustained throughput. Presets are the runs this project did or considered.

So what happened ?

The Day 2 grid is at about 1 percent of its optimal token budget. The Day 3 plan is at about 5 percent.
The Day 4 main run, 26 million parameters on 393 million tokens, gets to about 15 tokens per parameter. That's close enough to actually read results.
Switch the throughput between the planned 15 TFLOPs and the 4.1 that EXP-066 measured on a real T4. At the measured rate the Day 4 run takes about 4 hours, a 50M model at 1 billion tokens takes about 20, and a 150M model is just out of reach.

That last row is why pretraining stops around 20 million parameters.

Fact 2

The goal needs a model that follows instructions

My goal with this is to learn ML properly, like in depth, for placement season, and to end up with a harness that can control my own model. A harness shows the model a task and some tools, parses each tool call it emits, runs the tool and feeds back the result. That needs a model that follows instructions and produces well-formed tool calls. No model you can pretrain on free Colab does either of those.

The decision

Keep the from-scratch model, then move the same code to Qwen

So the new plan has two halves:

Train the from-scratch model once, properly. The resulting model has 26.3 million parameters, on TinyStories, with the Day 2 modern stack. TinyStories is a dataset where models under about 30 million parameters write coherent English, and the run fits the measured compute. Day 4 trained it, and Day 5 compares and speeds it up.
Load Qwen into the same model code. octlm/model.py already has RoPE, RMSNorm, SwiGLU and grouped-query attention, the same parts Qwen uses. Match its logits, write LoRA, fine-tune it for tool calls, and build the harness and its eval around it. That's Day 6 to Day 9.

Two alternatives got rejected. Continuing the first plan spends the remaining time on variants the harness never uses, and ends with a model too weak to drive it. Fine-tuning Qwen with an off-the-shelf trainer gives the harness a working model, but then there's nothing to explain when an interviewer asks how attention, the KV cache or LoRA actually work. And that's kind of the whole point.

What was dropped

Where each first-plan experiment went

Every first-plan experiment got a new number, a merge into another experiment, or a reason it was dropped. New numbers start at EXP-065, so no number ever means two things. The table is read from notes/day3.md when the site builds.

Map

Show

First plan		Now
Stage 0 noise floor	→	Replaced by EXP-069 at the new scale
EXP-016 multi-token prediction	→	Code kept. Runs as EXP-070
EXP-017 sparse attention	→	Code kept behind default-off flags. Not scheduled
EXP-018 compressed attention and copy probe	→	Code kept behind default-off flags. Not scheduled
EXP-019 MLA	→	Code kept behind default-off flags. Not scheduled
EXP-020 to EXP-025, MoE, combined run, asymmetric compute, Muon, mHC	→	Dropped
EXP-026 mixed precision	→	EXP-066, float16 only
EXP-027 to EXP-029, batch size, schedule, checkpointing	→	Folded into EXP-066 and EXP-067 where the run needs them
EXP-030 scaling check	→	Dropped
EXP-031 to EXP-034, DDP, FSDP, tensor and pipeline parallelism	→	Dropped. One GPU
EXP-035 and EXP-036, naive generation and KV cache	→	EXP-071, then EXP-076 on Qwen
EXP-037 and EXP-038, prefill split and `torch.compile`	→	Folded into EXP-071 and EXP-076 if time allows
EXP-039 quantization	→	EXP-072, int8 only
EXP-040 to EXP-042, batching, prefix cache, speculative decoding	→	Dropped. Cache reuse across turns lives in EXP-078
EXP-043 SFT	→	EXP-082
EXP-044 LoRA	→	EXP-080
EXP-045 and EXP-046, DPO and reward model	→	Dropped
EXP-047 and EXP-048, GRPO and RLVR	→	Day 9 option
EXP-049 to EXP-053, evals and regression gate	→	Replaced by the EXP-079 harness eval and bits per byte
EXP-054 task classifier	→	Dropped
EXP-055 and EXP-056, hybrid retrieval	→	Dropped. The harness has a `grep` tool
EXP-057 tool loop	→	EXP-078
EXP-058 tool-call fine-tune	→	EXP-081 and EXP-082
EXP-059 writing path, EXP-060 cache stack	→	Dropped
EXP-061 router and cost log	→	Day 9 option
EXP-062 to EXP-064, long context, stability run, model card	→	Dropped

Every first-plan experiment and where it went, read from notes/day3.md. Filter by outcome.

The dropped groups share reasons:

MoE, MLA, sparse and compressed attention, Muon, mHC and the DeepSeek-style runs don't help the harness. The built Day 3 code stays behind default-off flags.
DDP, FSDP, tensor and pipeline parallelism need more than one GPU, and the free tiers give you one.
Pretraining at 50 to 150 million parameters, and the scaling check, are ruled out by the compute table above.
DPO and the reward model. Supervised fine-tuning on harness traces is the post-training the eval can actually measure.
Retrieval, the writing assistant, the cache stack, batching and speculative decoding. The harness uses grep for retrieval and serves one request at a time.

The rules

Conflicts the change created, and how they were recorded

AGENTS.md says don't start a day before every exit check of the previous day passes. None of Day 3's passed. But the user's request outranks the plan, so the Day 3 checks were recorded as withdrawn, not failed, and Day 4 starts from Day 2's passed checks.

AGENTS.md also says every tokenizer learns its merges from our own data. Qwen's tokenizer ships with its weights. So the rule keeps applying to every tokenizer octlm trains, and for Qwen the checkpoint loader applies the hash check to tokenizer.json instead.