Building a small language modeland checking every step.
octlm is a decoder written from scratch in PyTorch. These posts explain each piece: what it does, how we built it, what it measured, and what we chose not to build. Playgrounds let you test the ideas; tests compare model-math playgrounds with the Python code.
Day 00
Start here
- 00.1What octlm is, and how it measuresWhat we are building, the rules every experiment follows, the laptop it started on, and the first two experiments, which built the lab before there was even a model.
- 00.2What a language model predictsA language model gives a probability to every possible next token. Cross entropy, perplexity and bits per byte are three ways of reading how much probability it gave the right one.
Day 01
Foundations and the Generation 0 Transformer
- 01.1Characters as tokensThe simplest tokenizer gives every character its own ID. It's easy to build and easy to check, and it shows exactly what a better tokenizer has to fix.
- 01.2Byte-level BPE, merge by mergeStart from the 256 byte values, then keep gluing the most frequent adjacent pair into a new token. Nothing is ever unknown, and common fragments become single tokens.
- 01.3Measuring a tokenizerThe BPE model got a perplexity eight times worse than the character model, and it was still the better model. Bits per byte is the unit that makes the two comparable.
- 01.4Embeddings and positionsA token ID becomes a vector by looking up one row of a learned table. Attention can't see order on its own, so a second table, or a formula, adds the position.
- 01.5Causal attention, by handHow each position in a sequence decides which earlier positions to read, traced step by step through octlm's attention layer with the weights PyTorch actually produced.
- 01.6The baseline blockA Transformer block is attention and a feed-forward layer, each wrapped in a norm and a residual connection. This post puts the Day 1 decoder together from those parts, including the init bug that stopped it from learning.
- 01.7A training loop that resumes exactlyBatches, AdamW, gradient clipping, a warmup and cosine schedule, and checkpoints that restore every random generator, so a run split in two ends with the exact same weights as a run done in one go.
Day 02
Modern decoder core
- 02.1A corpus big enough to measureDay 1 trained on 111 KB of planning notes. Day 2 needed held-out code and prose to compare architectures, so it built a 7 MB corpus from the Python standard library and six public-domain books.
- 02.2RoPE, position as a rotationRotary position embedding turns each query and key by an angle proportional to its position, so attention scores depend on how far apart two tokens are. Day 2 measured how far past its training length it actually holds up.
- 02.3RMSNorm against LayerNormRMSNorm drops LayerNorm's mean subtraction and its bias. On Day 2 it matched LayerNorm's quality inside the noise and ran slower, for a reason that had nothing to do with arithmetic.
- 02.4SwiGLU, a gated feed-forward layerSwiGLU replaces the feed-forward layer's single activation with a product of two projections, one of them passed through a smooth gate. It was the only single change on Day 2 that beat the noise.
- 02.5Grouped-query attention and the KV cacheGeneration stores every past key and value, and that cache grows with the number of key and value heads. Grouped-query attention shares them across query heads. Day 2 measured the saving exactly, and the quality cost not at all, because it was below the noise.
- 02.6SDPA and FlashAttention's tilingPlain attention builds a T by T score matrix in memory. FlashAttention computes the same result a block at a time with a running maximum and sum, and never stores the square. Day 2 wrote the tiling by hand and measured PyTorch's kernels against the plain path.
- 02.7Where the norm goesThe 2017 Transformer normalized after each residual addition. Modern decoders normalize before each block instead. Day 2 swapped the order back and lost a full bit per byte.
- 02.8Reading a grid through seed noiseEight architectures, three seeds each, one budget. Before you read any gap between two rows, read how far one row moves when only the random seed changes. On Day 2 that one number decided almost everything.
Day 03
Multi-token prediction and the attention variants
- 03.1Multi-token predictionTrain extra heads to predict the token two and three steps ahead from the same hidden state. The code is built and tested, the plan predicted no gain at 3.3M parameters, and the run moved to Day 5.
- 03.2Sparse attentionA sliding window lets each token read only its recent neighbors, and strided columns add a few long-range links. The masks are built and tested, and the run got dropped when the plan changed.
- 03.3Compressed attention and the copy probeKeep the recent keys exact and replace older ones with block averages, for a fraction of the cache. Day 3 built it, fixed a subtle problem with RoPE, and designed a probe that a 3M model could actually pass.
- 03.4Multi-head latent attentionMLA caches one small shared vector per token instead of per-head keys and values. Against multi-head attention that's a big saving. Against the two-KV-head model octlm already had, the arithmetic said it was small, and the note wrote that down before building it.
- 03.5Why the plan changedOn 2026-09-23 the project stopped before its Day 3 runs and rewrote the plan. Two facts forced it. The Day 2 models had seen about 1 percent of the data they needed, and no model this lab can pretrain could ever drive a tool-use harness.
Day 04
Train the from-scratch model properly
- 04.1TinyStories and a 6x faster encoderThe Day 4 model needs about 400 million tokens, and I expected the pure-Python BPE encoder to be way too slow for 2.2 GB of text. One dictionary from chunk to IDs made it six times faster, and the whole training split encoded in 16 minutes.
- 04.2Mixed precision on a T4Running the matrix math in 16-bit floats nearly doubled training speed with no measurable loss. The first attempt picked bfloat16, which the T4 only emulates, and it ran slower than float32.
- 04.3The main run26 million parameters, the Day 2 modern stack, 393 million TinyStories tokens, 4.1 hours in float16 on a Kaggle T4. Validation loss fell from 1.25 to 0.756, 0.484 bits per byte, and the curve was still sloping a little when the budget ran out.
- 04.4SamplingA model outputs a distribution, and something has to pick a token from it. Greedy decoding repeats itself on stories. Temperature reshapes the distribution and top-k cuts its tail, one line each. On the trained Day 4 model both decoders wrote coherent stories, and sampling traded the loops for weirder plot turns.
Day 05
Compare at the new scale and make generation fast
- 05.1Can six seeds settle an architecture choice?Six dots, two model stacks, and one question, is the gap between the averages bigger than the noise between seeds ? Move the dots, then look at the six real Day 5 runs. Modern wins by almost seven times the seed spread.
- 05.2Multi-token prediction at 26 million parametersA second head predicts two tokens ahead. Three matched T4 runs made next-token quality worse by 0.0188 bits per byte and cost 11% more training time, so the default model keeps one head.
- 05.3A KV cache that actually generatesStep through generation with the cache on and off. On the trained T4 model the cached path kept every greedy token, used 256 KiB to 2 MiB, and ran 10.6 to 15.8% faster in single timings.
- 05.4What int8 weight-only quantization savesFollow one float weight into an int8 code and back. The trained checkpoint got 41.7% smaller with almost the same bits per byte, but this PyTorch decode path ran 20.1% slower.
Day 06
Run Qwen through octlm
- 06.1A real model in our own decoderWhy a 26M story model can't run a tool harness, what Qwen3-0.6B brings, and how octlm reads its 1.5 GB safetensors file byte by byte into the same model.py. Five config switches were enough.
- 06.2Proving it's the same modelFluent text doesn't prove the weights loaded right. Octlm compared every logit of Qwen3-0.6B against Transformers on ten prompts. The worst gap was 2.65e-4 on the T4 against a limit of 1e-3, fixed before the run, and the top token matched at every position.
- 06.3Speaking Qwen's languageRight weights with wrong token IDs still give garbage. Octlm loads Qwen's own tokenizer, keeps literal control strings from turning into commands, and renders the native chat template character for character, tool calls and thinking mode included.
- 06.4Generating with Qwen through the Day 5 cacheThe same generate_cached that sped up the story model now runs Qwen3-0.6B. On a T4 in float16 it matched every greedy token of the reference at 24 to 25 tokens per second, and it showed the model rambling past its own end-of-turn marker.
Day 07
Build the harness and its eval
- 07.1Reading tool calls out of free textA harness is the program around a model: it shows tools, parses calls, runs them and feeds results back. Qwen writes calls as JSON inside <tool_call> tags. octlm's parser checks each block on its own, so one broken call becomes an error message, not a crash.
- 07.2The agent loop and a KV cache that survives turnsThe harness renders the conversation, generates until <|im_end|>, runs the tools inside a sandbox and repeats. Qwen's template rewrites past turns, so the cache keeps the longest shared token prefix and prefills only what changed.
- 07.3An eval Day 8 has to beatForty tasks in a small fixture repository, each with a check a script can decide, run under three sampling seeds. Run 1 exposed two checks a wrong answer could pass. The fixed v2 checks give stock Qwen3-0.6B its baseline, and it mostly answers without calling a tool.
Day 08
LoRA and tool-use fine-tuning
- 08.1LoRA from scratchFine-tuning all 600M Qwen weights needs about 9.7 GB of optimizer state before a single activation. LoRA freezes them and trains two thin matrices per projection instead. octlm's version trains 1.7 percent of the parameters and matches the base model's logits exactly at step 0.
- 08.2Training data from a scripted expertFine-tuning needs transcripts of the model doing the job right. The base model solves 1 task in 40, and the terms of a stronger API model forbid training on its output. octlm writes the transcripts with a scripted expert that replays hand-written solutions through the real tools.
- 08.3Fine-tuning Qwen to use its toolsFive minutes of LoRA training on 223 scripted examples takes Qwen3-0.6B from pass^1 0.025 to about 0.4 on the unchanged Day 7 eval. Valid tool calls go from a third to almost all. Two training seeds differ by more than the sampling seeds do, and tasks that need debugging stay unsolved.
- 08.4Merging the adapter and storing it in int8Merging folds LoRA into the base weights so inference runs one matmul per projection. The first merge check failed, because its tolerance was smaller than one float16 rounding step. With a correct gate, the merged int8 model halves the stored projections, keeps pass^1, and runs faster than the unmerged adapter.
Day 09
One optional extension
- 09.1GRPO with a pass-or-fail rewardSFT taught Qwen the tool format but not how to fix a bug. GRPO samples several attempts at one task, scores each with the task check, and pushes up the attempts that beat their group. It only learns from groups that mix passes and fails, and a probe before training found too few of them.
- 09.2The GRPO run that never took a stepThe first GRPO step sampled 96 rollouts in 25 minutes, found 3 mixed groups where it needed 4, and stopped. The saved adapter is the SFT adapter, bit for bit. Its eval matches SFT to the third decimal, which makes it a clean measure of GPU noise and a failed exit check.