What octlm is, and how it measures

octlm is a small decoder language model that I'm writing from scratch in PyTorch. The whole point is to actually learn how a language model works, like down to the arithmetic, and to be able to defend every piece of it with a number that came from running it, not from a blog post.

This series follows the build in the order it happened. Day 0 is this post and the next one, so what the project is, how it measures things, and what a language model is even trying to do. Day 1 builds the first working model. Day 2 swaps its parts for the ones current open models use. Day 3 builds four research ideas and then stops. Day 4 trains the model for real.

Build it, train it once, then move to a real model

The plan has three stages.

1. Write a decoder from scratch, around 20 million parameters, and train it on children's stories until it writes coherent ones.
2. Load Qwen, a real open model, into the same code and check that it gives the same logits as the reference implementation. Done on Day 6.
3. Fine-tune Qwen with LoRA, written inside octlm, so it emits tool calls, and build a harness that runs it against a fixed set of tasks.

The first stage proves the internals. The second proves they're the same internals a production model uses. The third gives us something we can measure on top of them. The plan used to be way bigger, and why the plan changed covers what got cut and why.

Every experiment gets written down before it runs

Each experiment gets an ID, EXP-001 onward, and a section in that day's note with the same five parts:

• Problem. One sentence on what's wrong or what we don't know.
• Hypothesis. What we expect, written so a result can prove it wrong.
• Baseline. The thing we compare against, measured the same way.
• Measurement. The exact numbers that will decide it.
• Stop condition. When to give up or change direction.

After the run the section records the result, what failed, and a keep or revert decision. Failed ideas stay in the note with their numbers. A bunch of posts in this series are about things that didn't work, and honestly those taught me the most.

Two more rules shape basically everything:

• Change one thing per comparison. If two parts change at once and the number gets better, you can't credit either of them.
• No optimization claim without a baseline. "Faster" means a measured time against a measured time on the same machine, nothing else.

A lab that can repeat its own results

Before any model, the repo had to be able to produce the same numbers twice. EXP-001 set that up:

• Python 3.14 and PyTorch 2.14 for CPU, pinned with uv.lock, so a fresh checkout installs the exact same versions.
• One TOML file per day holds every setting that varies. A SHA-256 of those settings becomes the config fingerprint, and a checkpoint refuses to resume under a different fingerprint.
• A dry run builds the model and prints the parameter count and output shape without training. For the Day 1 config it reports 541,952 parameters and logits shaped [1, 128, 1024], so one sequence, 128 positions, and a score for each of 1,024 vocabulary entries.
• Every measurement is one JSON line in runs/. The charts on this site read those files, and a test fails the site build if a file is broken.

The parameter count is simple enough to do by hand, and the counter below does the same sums octlm does. A test checks it against parameter_count on the real PyTorch model for five configs.

So what happened ?

• The token embedding is the biggest single part at Day 1 size, 1,024 entries of 128 numbers. The output head reuses the same matrix, so it adds nothing. The baseline block explains why.
• Each layer costs the same, so doubling the layers roughly doubles everything except the embedding.
• Switch positions to RoPE and the position table just disappears, RoPE has no parameters.

How far can a laptop go ?

The machine is an Intel i5-12450H with 12 logical CPUs, 15 GiB of RAM and no usable GPU. EXP-002 asked, at which context length does a single forward pass of the Day 1 model shape stop being practical on this CPU ?

octlm.bench builds the model at each length, runs one untrained forward pass on a sequence of zeros, and records the time and the process's peak memory.

def measure(context_length: int, device: torch.device) -> dict[str, object]:
    torch.manual_seed(0)
    config = DecoderConfig(
        vocab_size=512,
        context_length=context_length,
        d_model=128,
        n_heads=4,
        n_layers=2,
    )
    model = Decoder(config).eval().to(device)
    inputs = torch.zeros((1, context_length), dtype=torch.long, device=device)
    with torch.inference_mode():
        if device.type == "cuda":
            model(inputs)  # first call loads kernels, so time the second
            torch.cuda.synchronize()
        started = time.perf_counter()
        model(inputs)
        if device.type == "cuda":
            torch.cuda.synchronize()
        elapsed = time.perf_counter() - started
    count = parameter_count(model)
    token_vectors = F.normalize(model.token_embedding.weight.detach(), dim=1)
    position_vectors = F.normalize(model.position_embedding.weight.detach(), dim=1)
    token_neighbor = int((token_vectors[0] @ token_vectors[1:].T).argmax()) + 1
    position_neighbor = (
        int((position_vectors[0] @ position_vectors[1:].T).argmax()) + 1
        if context_length > 1
        else None
    )
    return {
        "batch_size": 1,
        "config_sha256": None,
        "context_length": context_length,
        "decode_tokens_per_second": None,
        "device": str(device),
        "dtype": "float32",
        "elapsed_seconds": elapsed,
        "experiment_id": "EXP-002",
        "kv_cache_bytes": 0,
        "loss": None,
        "model_bytes": count * 4,
        "nearest_position_to_zero": position_neighbor,
        "nearest_token_to_zero": token_neighbor,
        "parameter_count": count,
        "peak_rss_bytes": resource.getrusage(resource.RUSAGE_SELF).ru_maxrss * 1024,
        "perplexity": None,
        "python": platform.python_version(),
        "run_id": f"{device.type}-context-{context_length}",
        "schema": "octlm-bench-v1",
        "time_to_first_token_ms": None,
        "tokenizer_sha256": None,
        "tokens_processed": context_length,
        "torch": torch.__version__,
    }

So what happened ?

• From 128 to 1,024 tokens the time grows about as fast as the length. The feed-forward layers and projections cost the same per token, and at these lengths they dominate.
• From 1,024 to 2,048 the time goes 4x, from 28.7 ms to 115.2 ms. Attention compares every position with every earlier one, so its cost grows with the square of the length, and at 2,048 you start to see it. Causal attention and SDPA come back to this square.
• Peak memory hit 504 MB by the end of the process.

These are smoke measurements from one run, and the note says that. They set 2,048 as the longest context Day 1 would claim to support, and nothing more than that.

Training moved off the laptop

Day 2 ran a grid of 24 training runs. Maxing out every core for most of an hour overheated the laptop, like actually overheated, so from Day 2 onward all training runs on a cloud T4 GPU, first on Colab and from Day 4 on Kaggle. The laptop keeps the short stuff, the unit tests, a dry run, and measurements that don't need a GPU. Every result on this site names the machine it ran on.

What we didn't build, and why

• A configuration framework. One TOML file per day and a frozen dataclass cover every setting. A framework would add code without adding a single setting.
• Plots in the Python code. octlm writes JSON lines, this site draws them.
• A GPU from day one. Day 1 models trained in seconds on the CPU. The GPU came in only when a measured run made the laptop overheat.