Modern stack
0.5539bits per byte against 0.5877 for the baseline, three seeds each
Day 09 closed · GRPO found nothing to learn from
Built from thefirst byte.
octlm is a nine-day lab in PyTorch. It writes the tokenizer and decoder from scratch, trains a 26M model on TinyStories, loads Qwen3-0.6B into the same code, and teaches it to fix bugs with a LoRA written by hand. Every number on this page comes from a logged run.
Fig. 0 · agent eval, 120 runs eachday7 + day8 evals
Day 0426.3Mparameters trained from scratch
Day 04393MTinyStories tokens, 4.1 h on one T4
Day 040.4839bits per byte on held-out stories
Day 062.65e-4largest logit gap to Transformers, Qwen3-0.6B
Day 080.33 → 0.98share of tool calls that parse and validate
Days 01–0938experiments in the plan, each with an exit check
No number without a baseline.
Each experiment opens with a problem, a hypothesis, a baseline, a measurement and a stop condition. It closes with keep or revert. Failed ideas stay in the notes with their numbers. The code leans on the standard library and PyTorch, and on nothing it can avoid.
- 01
38 experiments, measured
Three seeds set the noise floor. A result counts only when its gap beats the spread.
- 02
Every choice cites a source
Papers, model reports and official code, each named in the day note next to the decision it caused.
- 03
Written here, not imported
BPE, attention, RoPE, KV cache, int8, LoRA, a safetensors reader and the agent harness all live in octlm.
From raw bytes to an agent that fixes bugs.
Each day has one note, a set of experiments and an exit check. Hover a day to see what it built and the number that closed it.
Day 08 · EXP-080 to 083
LoRA and tool-use fine-tuning
Rank-16 adapters on all seven projections, trained on 74 scripted traces for about five minutes. The merged int8 model keeps the pass rate and halves the stored block weights.
Read 4 postsDone0.467pass^1, up from 0.025
26 million parameters that tell stories.
One decoder, trained once and trained properly. 24,000 steps of 32 sequences of 512 tokens, float16 with a gradient scaler, cosine decay to a tenth of the peak rate. Validation loss is the same 131,072 held-out tokens at every checkpoint.
- Layers
- 8
- Width
- 512
- Heads
- 8 query · 2 KV
- Context
- 512 tokens
- Vocabulary
- 8,192 byte-level BPE
- Positions
- RoPE
- MLP
- SwiGLU
- Norm
- RMSNorm, pre-norm
- Precision
- float16 + GradScaler
- Compute
- one T4, 4.1 hours
Spec sheet · Day 04 model
Loss over 24,000 stepsday4-metrics.json
validationtrain batch
Hover the chart to read values.
Live on a laptop CPU
The Day 4 model writing a story, recorded with the real demo command. Each token is colored by the probability the model gave it.
≥ 0.9≥ 0.5≥ 0.2< 0.2
Story time · pick a first line, the model writes the restday4-samples.json
Once upon a time, there was a little girl named Lily.
1 of 85 four-word runs repeatSampling keeps the story moving.
Every idea had to beat the noise.
Three seeds of each stack on 98.3M tokens set the floor. The six dots below are those runs. Anything smaller than the spread inside a group is not a result.
Bits per byte, three seeds each, lower is betterseed-spread.json
SwiGLU
−0.082bits per byte on code at matched parameters
Two KV heads
16 → 4 MiBKV cache at 4K context, quality within noise
Post-norm
+1.00bits per byte worse on code than pre-norm
Multi-token prediction
+0.0188bits per byte, against a 0.0050 noise threshold
Int8 weights
−42%model bytes for a 0.000025 bits-per-byte change, 20% slower on a T4
KV cache
Same tokensas naive decoding at temperature 0, at 64, 256 and 512 tokens
Merged int8 Qwen
0.375pass^1 unchanged after merging the LoRA and quantizing
Same decoder file. Real weights.
Largest logit gap2.65×10⁻⁴
Largest float32 logit difference between octlm and Hugging Face Transformers, Qwen3-0.6B on a T4. Tokenizer, chat template and greedy tokens match exactly.
┌──────────┬────────────────────┬─────────────────────────┐ │ u64 n │ JSON header (n B) │ raw tensor bytes ··· │ └──────────┴────────────────────┴─────────────────────────┘ struct json.loads memoryview → torch
The weights load through struct and json from the standard library. No safetensors package.
- Layers28
- Width1,024
- Heads16 query · 8 KV
- ExtrasQK-norm, RoPE base 10⁶
- Vocabulary151,936
- Parameters596M
Same task, same seed. One adapter apart.
The harness shows the model a failing repository and five tools. It parses each call, runs it in a sandbox and feeds the result back until the model answers or hits eight turns. Both recordings run the real eval loop on a laptop CPU.
task ──▶ model ──▶ <tool_call> ──▶ parse + validate ──▶ sandbox
▲ │
└──────────────── tool result ◀─────────────────┘
tools list_files · read_file · grep · write_file · run_testsStockAnswers in prose. No tool call.
LoRARuns tests, reads, fixes, reruns.
One CPU run is one sample. It shows the result, it does not measure it. The measurement is the grid below.
40 tasks × 3 sampling seeds, on a T4day7 + day8 evals
pass^1
Qwen3-0.6B, stock0.025
LoRA, seed 00.375
LoRA, seed 10.467
Seed 0, merged int80.375
valid tool calls
Qwen3-0.6B, stock0.333
LoRA, seed 00.976
LoRA, seed 10.985
Seed 0, merged int80.954
mean turns
Qwen3-0.6B, stock1.18
LoRA, seed 04.62
LoRA, seed 15.06
Seed 0, merged int85.42
Every run, clickable
Learn from the test results, when they disagree.
SFT taught the tool format but not debugging. Day 9 samples six rollouts per task, scores each with the task check, and pushes up the ones that beat their group. Groups where every rollout agrees carry no gradient and get dropped. On the T4, 13 of 16 groups agreed, so the run stopped before its first update. Click the rollouts to see why a uniform group teaches nothing.
Group-relative advantage, a playground
Clone it. Watch it work.
Python 3.14, uv and the CPU build of PyTorch. The demos need their weights in artifacts/; the README lists which files.
# Python 3.14 and uv$git clone https://github.com/whynotramaa/octlm.git$cd octlm && uv sync$uv run python -m unittest # the test suite, on CPU
$uv run python -m octlm.demo agent # Qwen + LoRA on bug-low-stock$uv run python -m octlm.demo agent --base # same task, no adapter$uv run python -m octlm.demo agent --list # the 40 eval tasks$uv run python -m octlm.demo agent --task cart-clear
$uv run python -m octlm.train --config configs/day1.toml --dry-run # config hash, params, logits shape$uv run python -m octlm.tokenizer --vocab-sizes 512 1024 2048$uv run python -m octlm.train --tokenizer bpe --overfit # the memorization gate$python -m octlm.day4 train # EXP-067, on a T4
$python -m octlm.harness eval # 40 tasks × 3 seeds → JSONL$python -m octlm.day8 # traces, LoRA training, eval$python -m octlm.day9 probe # GRPO signal check
Python 3.14PyTorch 2.14uvKaggle T4Colab T4Qwen3-0.6BTinyStoriesAstro + Preact site
43 posts · days 01–09
Every playground runs in your browser.
Each post explains one piece, what it measured, and what it chose not to build. Tests check the playground math against the Python code.