Proving it's the same model

The last post loaded all 596,049,920 of Qwen's parameters into octlm. Now the question is whether it computes the same function as the official code. Generating nice text isn't proof. A model with one transposed matrix, or the wrong RoPE layout, can still produce sentences that look fine, just worse ones. The honest test compares the raw numbers.

Every logit, every position

For each input position, the model outputs one score per vocabulary entry, called a logit. Qwen has 151,936 entries, so the 188-token tool-call prompt gives 28.6 million logits. The test runs the same token IDs through both implementations and takes the element-wise difference. It records three numbers per prompt:

• Maximum absolute error. The single worst logit gap anywhere. This is the gate.
• Mean absolute error. How far off a typical logit is.
• Argmax agreement. The share of positions where both pick the same top token.

A max error is strict. One bad position in one prompt fails the whole run, even if everything else matches.

Remove every source of noise first

The reference is Hugging Face Transformers 4.57.6, loaded from the same pinned files. The run removes everything that could make two correct implementations disagree:

• Float32 on both sides. Half precision rounds after every operation, so errors pile up in different places.
• Eager attention in the reference. It computes the softmax step by step, with no fused kernel choosing its own order.
• TF32 off on the GPU. Ampere-class GPUs can quietly do float32 matrix products at lower precision. The T4 doesn't have TF32, but the flag stays off anyway.
• Eval mode. No dropout.

Then the test checks two paths through octlm. The full path runs the whole prompt in one call. The cached path splits the prompt in half, runs the first half, keeps its KV cache and feeds the second half on top. The cached path must match too, otherwise the cache would generate from slightly wrong numbers.

@torch.inference_mode()
def logits_parity(model: Decoder, reference, cases: list[list[int]], out: Path) -> None:
    device = next(model.parameters()).device
    for index, ids in enumerate(cases):
        tokens = torch.tensor([ids], device=device)
        expected = reference(tokens, use_cache=False).logits.float()
        torch.save(
            {"ids": ids, "logits": expected.cpu()}, out.parent / f"reference-logits-{index}.pt"
        )
        full = model(tokens)
        split = max(1, tokens.shape[1] // 2)
        first, cache = model.forward_cached(tokens[:, :split])
        second, _ = model.forward_cached(tokens[:, split:], cache)
        cached = torch.cat((first, second), dim=1)
        for path, actual in (("full", full), ("cached", cached)):
            delta = (actual.float() - expected).abs()
            maximum = delta.max().item()
            row = {
                "type": "logit_parity",
                "case": index,
                "path": path,
                "max_absolute_error": maximum,
                "mean_absolute_error": delta.mean().item(),
                "argmax_agreement": (actual.argmax(-1) == expected.argmax(-1))
                .float()
                .mean()
                .item(),
                "tolerance": LOGIT_TOLERANCE,
            }
            _write(out, row)
            if not torch.isfinite(actual).all() or maximum > LOGIT_TOLERANCE:
                raise RuntimeError(f"float32 {path} logit parity failed on case {index}: {maximum}")

Pick the tolerance before looking

The note fixed the limit at a max absolute error of 1.00e-3 before the first real run, with a rule that it would not change after seeing results. That rule matters. If you pick the limit after seeing the numbers, any result can be made to pass.

Why 1.00e-3? In the saved reference logits, the gap between a position's highest and lowest logit has a median of 29 on the story prompt and 42 on the tool-call prompt. A mapping bug, such as a swapped projection or the wrong RoPE pairing, changes what every layer computes, so it shows up at the scale of those ranges. Float32 rounding across 28 layers stays near 1e-4, which is what both real runs measured. The limit sits between the two.

Both devices pass, with room to spare

Every one of the 20 T4 comparisons passed, and so did every CPU comparison. The T4's worst gap is larger than the CPU's. A GPU adds the same numbers in a different order, in parallel blocks, and floating-point addition isn't associative. So a larger gap on a different device is expected. A gap near 1 would mean a bug.

One detail in the raw data looks like a failure but isn't. On the tool-call prompt, the T4 reports argmax agreement of 0.99999994 instead of 1. That prompt has 188 positions. A single wrong top token would show as 187 ÷ 188 = 0.9947. The tiny shortfall comes from how CUDA computes a mean, as a sum times 1/188, which rounds in float32. All 188 positions agree.

Decision: the octlm decoder computes the same function as Qwen3's reference implementation. EXP-074 passes. Everything after this, the harness and the fine-tune, can trust model.py.

Not measured

• Half-precision parity. Float16 logits drift further, and the drift says more about rounding than about the implementation. The generation post covers what float16 does to greedy tokens.
• Long prompts. The longest test prompt is 188 tokens. RoPE at base 1,000,000 is exact at any position, but the test doesn't go near Qwen's 40,960-token limit.
• A second checkpoint. Only the pinned 0.6B commit was tested.