A corpus big enough to measure

Day 2 compares six changes to the decoder, one at a time, against the Day 1 baseline. And for a comparison you need held-out text big enough that the difference between two models is bigger than the noise of evaluating them. The project's own planning docs, around 111 KB, were nowhere near that. So before any Day 2 experiment ran, Day 2 built a corpus.

Python code and nineteenth-century prose

The plan wanted a model that would eventually read code and edit prose, so the corpus has both, kept in separate categories so we can evaluate each one alone.

• Code is every top-level module of the Python standard library installed on the machine, sorted, skipping test files and dunder files. It's Python-licensed.
• Prose is six Project Gutenberg books, all public domain in the United States. Four are narrative and two are expository, so the prose isn't just one author's voice.

By chunk, not by book

octlm/corpus.py cuts every source into chunks of about 64 KB on line boundaries, so no chunk ends in the middle of a line. A seeded shuffle then sends 10 percent of the chunks to validation.

def chunk(text: str, limit: int = CHUNK_BYTES) -> list[str]:
    """Split on line boundaries so no chunk ends mid-line."""
    chunks, current, size = [], [], 0
    for line in text.splitlines(keepends=True):
        current.append(line)
        size += len(line.encode())
        if size >= limit:
            chunks.append("".join(current))
            current, size = [], 0
    if current:
        chunks.append("".join(current))
    return chunks

Splitting by chunk instead of by whole book keeps every source in both splits. Two neighboring chunks of one book share a style but share no text, so validation never has a sentence the model trained on. Each chunk stays a separate document in the JSONL file, so the BPE trainer never merges across a chunk boundary.

The build every result uses

The corpus was first built on the laptop. When training moved to Colab the corpus had to move too, because the code half is read from the running machine's standard library, and Colab runs Python 3.13 while the laptop runs 3.14. Different Python, different corpus. The Colab build is the one every Day 2 training result uses:

The mix is 60 percent code and 40 percent prose by bytes. data/manifest.json records a SHA-256 fingerprint of each split, and the note tells every later reader to check them before comparing a new run, because Colab upgrades its Python image without telling anyone.

The tokenizer was retrained on this corpus at 2,048 entries, from every tenth training document to keep the pair-recount cost down. It stopped at 1,787 merges and got to 0.431 tokens per byte. Cut into 256-token blocks, the training split is 11,508 blocks, about 2.96 million tokens, and validation is 1,528 blocks, 782 of code and 745 of prose.

Remember that 2.96 million. Why the plan changed comes back to it. The Day 2 models had 3.3 million parameters, and a model that size wants about 20 times more tokens than this.

What the note says it doesn't cover

• The code is Python only. The TypeScript and JavaScript the first plan targeted aren't there.
• The prose is nineteenth-century literature, not modern writing.
• The standard-library file set depends on the Python install. The fingerprints pin what was used, they don't promise another machine rebuilds the same corpus.
• The plan asked for 50 percent code, 30 percent prose and 20 percent instruction data. There was no instruction data, and the code side ran out before its budget, which is the whole gap between 3 to 2 and 5 to 3.

Day 2 needed a corpus big enough to separate architectures, not the final data mix. Whether it was big enough, that's the question the rest of Day 2 answers.