Generating with Qwen through the Day 5 cache

Loading weights and matching logits proves one forward pass. A chat model runs thousands of them, one per generated token, each reusing the KV cache from the last. So the final Day 6 check runs Day 5's cached generator on Qwen and compares its greedy tokens against Transformers.

The Day 5 cache path, unchanged

generate_cached didn't need any Qwen-specific code. It calls forward_cached, which calls each attention layer's cached path, and those use the same projections, QK-norm and split RoPE as the full forward pass. The only thing that changed is the model it runs.

What does change is the size of the cache. Per token, the cache stores one key and one value for every layer and KV head:

With 28 layers, 8 KV heads, 128 values per head and 2 bytes per float16 value, that's 114,688 bytes per token, 28 times the Day 4 model's 4,096 bytes per token. A full 40,960-token context would need 4.4 GiB for one sequence. Grouped-query attention already halves this. With 16 KV heads instead of 8, every number would double.

Same prompts, same tokens, timed

Each run renders two chat prompts with the native template, warms both paths up, then generates 32 greedy tokens three times with each implementation. It synchronizes CUDA around every timer and reports the median. After timing, it rebuilds the cache and checks that its actual byte count equals the formula.

Two choices keep the comparison fair. Both sides use the same dtype on the same device. And the reference switches from eager attention, which the parity test needed, to SDPA for timing. Timing octlm's SDPA against eager attention would make octlm look faster than it is.

def generation_benchmark(
    model: Decoder, reference, tokenizer: QwenTokenizer, count: int, repeats: int, out: Path
) -> bool:
    device, all_equal = next(model.parameters()).device, True
    for index, messages in enumerate(CHATS[:2]):
        prompt = torch.tensor(
            [tokenizer.encode(tokenizer.chat(messages), allow_special=True)], device=device
        )
        model.generate_cached(prompt, 2)
        reference_generate(reference, prompt, 2)
        ours_times, reference_times, equal = [], [], True
        for _ in range(repeats):
            ours, elapsed = timed(lambda: model.generate_cached(prompt, count), device)
            expected, reference_elapsed = timed(
                lambda: reference_generate(reference, prompt, count), device
            )
            ours_times.append(elapsed)
            reference_times.append(reference_elapsed)
            equal &= check_generation(model, reference, ours, expected, index, out)
        _, cache = model.forward_cached(ours[:, :-1])
        cache_bytes = sum((k.numel() + v.numel()) * k.element_size() for k, v in cache)
        expected_bytes = kv_cache_bytes(
            model.config, ours.shape[1] - 1, next(model.parameters()).element_size()
        )
        if cache_bytes != expected_bytes:
            raise RuntimeError("actual Qwen cache bytes differ from the formula")
        _write(
            out,
            {
                "type": "qwen_generation",
                "case": index,
                "device": str(device),
                "dtype": str(next(model.parameters()).dtype),
                "new_tokens": count,
                "seconds": ours_times,
                "reference_seconds": reference_times,
                "tokens_per_second": count / statistics.median(ours_times),
                "reference_tokens_per_second": count / statistics.median(reference_times),
                "cache_bytes": cache_bytes,
                "reference_attention": reference.config._attn_implementation,
                "greedy_equal": equal,
                "generated_ids": ours[:, prompt.shape[1] :].tolist(),
                "text": tokenizer.decode(ours[0, prompt.shape[1] :].tolist()),
            },
        )
        all_equal &= equal
    return all_equal

What counts as a mismatch in float16

The T4 has no BF16 support, so the GPU run uses float16. Two correct float16 implementations can round differently. Over 32 greedy steps, one near tie between the top two candidates can flip a token, and after that the two sequences go their separate ways. Exact equality in float16 partly tests luck.

So before the T4 run, the note changed the stop rule. Float32 still requires exact equality. In float16, a mismatch writes a record of where the sequences split, the reference's gap between its top two logits there, and octlm's logit error at the same spot. The run fails if any logit isn't finite, or if the gap is bigger than the error, since then rounding can't explain the flip.

def check_generation(
    model: Decoder, reference, ours: torch.Tensor, expected: torch.Tensor, case: int, out: Path
) -> bool:
    if torch.equal(ours, expected):
        return True
    strict = next(model.parameters()).dtype == torch.float32
    row = {"type": "generation_divergence", "case": case, "strict": strict}
    row |= divergence(model, reference, ours, expected)
    row |= {"actual_ids": ours.tolist(), "reference_ids": expected.tolist()}
    _write(out, row)
    if strict or not row["finite"] or row["reference_margin"] > row["logit_error"]:
        raise RuntimeError("cached generation differs from the reference")
    return False

The rule never fired. Every token matched on every repeat.

Every greedy token matched

The T4 rows generate 32 tokens and the CPU rows 4, with two threads to keep the laptop cool. Both include prefill. Every cache size matched the formula.

On the T4, octlm is 6 and 3 percent slower than the reference. The seed-spread post already has a lead on why. PyTorch's documentation says SDPA supports grouped-query attention only in its flash and math kernels on CUDA, and the flash kernel needs a newer GPU than the T4. So octlm's enable_gqa=True probably runs the slow math kernel. That's a lead, not a measurement. Decode speed isn't a Day 6 goal, so it stays open. On the CPU octlm is slightly faster, by a margin too small to claim.

The model doesn't know when to stop, the loop does

Look at the generated text. The model answers ("2 + 2 = 4."), emits <|im_end|>, and then keeps going. It writes <|endoftext|> and invents a new "Human:" turn. That isn't a bug in octlm. The benchmark asks for exactly 32 tokens so both sides do the same work, and nothing tells the loop to stop.

A harness can't work like that. It would read the invented turn as part of the answer. Day 7's loop must stop at <|im_end|>, ID 151645, and treat everything before it as the reply.

Decision: keep the Qwen path in model.py. All four Day 6 exit checks pass.

Not built yet

• A stop token in the benchmark. The benchmark keeps its fixed 32 tokens for parity. After Day 6, generate_cached accepts stop IDs, and reply stops at either end ID listed in the pinned generation_config.json, <|im_end|> or <|endoftext|>.
• Greedy decoding for the harness. The Qwen3-0.6B model card warns against greedy decoding in thinking mode because of endless repetition. So the Day 7 harness samples in non-thinking mode with the card's temperature 0.7, top-p 0.8 and top-k 20, and runs each task under several seeds. Greedy decoding stays for parity checks only.
• Faster GQA on the T4. Timing repeat_interleave keys against enable_gqa would test the kernel lead.