A real model in our own decoder

Day 4's model writes "Lily found a red ball and smiled." It can't answer "what is 2 + 2", and it can't ask to read a file. That skill comes from scale and from post-training, and free GPUs can pay for neither. So Day 6 borrows a model that has both, Qwen3-0.6B, and runs it through octlm's own model.py. It doesn't use Hugging Face's model code.

The surprise is how little had to change. Qwen is the same kind of decoder we built on Day 2. It has more layers, wider vectors and a few extra details. Five new config switches and one weight loader were enough.

What Qwen has that we can't train

Our Day 4 model has 26,255,872 parameters and saw about 400 million tokens of children's stories. Qwen3-0.6B has 596,049,920 parameters, about 23 times as many. Alibaba trained it on trillions of tokens and then post-trained it on examples of chat, instructions and tool calls. We can't afford that compute. The rest of the plan needs a model that follows instructions, so borrowing one is the only way forward.

What we can do is run it through our own code. If octlm's decoder gives the same outputs as the official implementation, that proves our attention, norms, RoPE and cache are the real thing and not a toy version. The next post measures that match.

Same design, bigger numbers, five switches

Both use RMSNorm before each sublayer, SwiGLU feed-forward layers, grouped-query attention, RoPE and a tied input and output embedding. That's the Day 2 modern stack. The differences are five switches, all off by default, so every earlier config and its fingerprint stays unchanged.

1. head_dim. Octlm used to assume head width = width ÷ heads. Qwen has 16 heads of 128, so attention works in 2048 dimensions while the residual stream is only 1,024 wide. The query and output projections are rectangular now.
2. ff_hidden. The feed-forward width comes straight from the config (3,072) instead of Day 2's two-thirds rule.
3. rope_base. The config sets 1,000,000 instead of 10,000. The playground below shows why.
4. rope_split. This is the subtle one, also below.
5. qk_norm. Qwen3 applies a small RMSNorm to each query head and each key head before RoPE. It keeps attention scores from growing too large.

def qwen_config(data: dict) -> DecoderConfig:
    supported = {
        "model_type": "qwen3",
        "hidden_act": "silu",
        "attention_bias": False,
        "attention_dropout": 0.0,
        "rope_scaling": None,
        "use_sliding_window": False,
        "tie_word_embeddings": True,
        "rms_norm_eps": 1e-6,
    }
    for key, value in supported.items():
        if data.get(key) != value:
            raise ValueError(f"unsupported Qwen setting: {key}")
    config = DecoderConfig(
        vocab_size=positive_int(data, "vocab_size"),
        context_length=positive_int(data, "max_position_embeddings"),
        d_model=positive_int(data, "hidden_size"),
        n_heads=positive_int(data, "num_attention_heads"),
        n_layers=positive_int(data, "num_hidden_layers"),
        kv_heads=positive_int(data, "num_key_value_heads"),
        head_dim=positive_int(data, "head_dim"),
        ff_hidden=positive_int(data, "intermediate_size"),
        rope_base=float(data["rope_theta"]),
        position="rope",
        norm="rmsnorm",
        feed_forward="swiglu",
        attention="sdpa",
        rope_split=True,
        qk_norm=True,
    )
    config.validate()
    return config

Which channels rotate together

The RoPE post rotates channel pairs by an angle that grows with position. Octlm paired neighbors, channel 0 with 1, 2 with 3 and so on. Hugging Face's Qwen code pairs the first half with the second half, channel 0 with 64, 1 with 65. Both are valid RoPE. But Qwen's query and key weights were trained with split halves. Load those weights and rotate with the other layout, and every angle is correct while the wrong channels get rotated together. The model would produce fluent-looking garbage.

The split version is three lines in apply_rope, and the cached path shares it, so generation got it for free. A parity test checks the JavaScript port above against the Python function.

A safetensors file is a JSON header and raw bytes

The weights ship as one 1.40 GiB model.safetensors file. The format is simple enough to parse with the standard library:

1. The first 8 bytes are a little-endian integer, the header length.
2. The next 35,552 bytes are JSON. For each tensor it lists a dtype, a shape and a start and end offset.
3. Everything after that is raw tensor bytes, packed back to back.

read_header doesn't trust the file. It rejects a header length larger than the file, duplicate JSON keys, unknown dtypes, a byte count that doesn't match the shape, and offsets that overlap, leave a gap or fail to cover the payload. A truncated download or a corrupt file fails here, before any tensor reaches the model.

def read_header(stream, file_size: int) -> tuple[dict, int]:
    prefix = stream.read(8)
    if len(prefix) != 8:
        raise ValueError("truncated safetensors header length")
    length = struct.unpack("<Q", prefix)[0]
    if not 2 <= length <= min(100_000_000, file_size - 8):
        raise ValueError("invalid safetensors header length")
    header = json.loads(stream.read(length), object_pairs_hook=unique_keys)
    if not isinstance(header, dict):
        raise ValueError("safetensors header must be an object")
    metadata = header.pop("__metadata__", {})
    if not isinstance(metadata, dict) or any(not isinstance(v, str) for v in metadata.values()):
        raise ValueError("invalid safetensors metadata")
    cursor = 0
    entries = {name: tensor_spec(entry) for name, entry in header.items()}
    for _, _, start, end in sorted(entries.values(), key=lambda x: (x[2], x[3])):
        if start != cursor:
            raise ValueError("tensor offsets overlap or leave a gap")
        cursor = end
    if cursor != file_size - 8 - length:
        raise ValueError("tensor offsets do not cover the complete payload")
    return entries, 8 + length

Build empty, then stream one tensor at a time

A naive loader builds the model with random weights, reads the whole file into a dictionary, then copies. That holds two full copies in memory. Octlm builds the model on PyTorch's meta device, which records shapes and allocates nothing. to_empty() allocates real storage once, and the generator hands over one tensor at a time. Each tensor gets its name mapped, its shape checked and its values copied, then it's freed.

def load_qwen(
    directory: Path, device: str = "cpu", dtype: torch.dtype = torch.float32
) -> tuple[Decoder, dict]:
    manifest = verify_snapshot(directory)
    config = qwen_config(read_json(directory / "config.json"))
    with torch.device("meta"):
        model = Decoder(config)
    model = model.to(dtype=dtype).to_empty(device=resolve_device(device))
    model.lm_head.weight = model.token_embedding.weight
    state, names, seen = model.state_dict(), weight_names(config), set()
    with torch.no_grad():
        for name, tensor in safetensors(directory / "model.safetensors"):
            if name not in names:
                raise ValueError(f"unexpected checkpoint tensor: {name}")
            target = state[names[name]]
            if tensor.shape != target.shape:
                raise ValueError(f"checkpoint tensor shape mismatch: {name}")
            tied = {"model.embed_tokens.weight", "lm_head.weight"}
            if name in tied and seen & tied:
                if not torch.equal(target, tensor.to(device=target.device, dtype=target.dtype)):
                    raise ValueError("checkpoint tied weights differ")
            target.copy_(tensor)
            seen.add(name)
    missing = set(names) - seen - {"lm_head.weight"}
    if missing:
        raise ValueError(f"missing checkpoint tensors: {sorted(missing)}")
    return model.eval(), manifest

Two things the real file taught us

The duplicate output matrix. The config says the input embedding and output matrix are tied, one shared matrix. The file still stores both model.embed_tokens.weight and lm_head.weight, 311 entries instead of 310. A tiny random test model had omitted the duplicate, so only the real download exposed this. The loader now accepts the duplicate and fails if the two copies differ.

A mask nobody used. Octlm's attention layer used to store a square causal mask at the full context length, even for SDPA, which never reads it. At Qwen's 40,960 positions that's about 1.68 GB per layer. The layer now keeps a one-element placeholder for full SDPA and builds masks only for the attention types that need them. This is a loading fix, not a speed claim.

Everything arrives, nothing extra

The pinned snapshot loads all 311 BF16 entries into 596,049,920 unique parameters. Config, tokenizer and weight hashes pass before any tensor is read, and a forward pass runs. The same load ran on the laptop and on the Kaggle T4.

What this doesn't prove is that the numbers come out right. A single wrong transpose would still load without an error. That's the next post.

Not built

• Sharded checkpoints. Larger Qwen models split weights across several files with an index. The 0.6B model is one file, so the loader reads one.
• Qwen3.5. The newer small Qwen mixes a different layer type (Gated DeltaNet) into the stack. That would be a second architecture, not a switch, so the plan's Qwen3 fallback won.
• BF16 compute. The file is BF16. The laptop runs float32 and the T4 runs float16, because the T4 has no BF16 hardware.