What int8 weight-only quantization saves

Take one weight, −0.35. Store it as a small integer plus a scale that says what one integer step means. When the model uses that layer, multiply the integer by the scale and you get an approximate float back. Now move the largest weight below and watch what happens to −0.35, one outlier makes the scale coarser for the whole row.

The Day 4 model uses float16 on a T4, so every float16 projection weight takes two bytes in an inference checkpoint. Quantization asks, can we store one-byte integer weights instead and make the checkpoint smaller without hurting held-out story prediction too much ? It might change generation speed too, but the current PyTorch implementation rebuilds a float weight on every linear call. So a smaller file doesn't mean a faster decoder, those are two separate things.

Float16 was already a compression step

The mixed-precision experiment first moved matrix operations from float32 to float16. The T4 ran it faster and the validation loss matched the float32 control, and it halved the bytes per floating-point weight. Int8 starts from that float16 inference model and tries to halve the projection-weight payload again. This is an inference storage change, not a training recipe, the optimizer and gradients stay in the training checkpoint.

The model has 26,255,872 parameters. Its attention and feed-forward linear layers hold 22,052,864 weights in 36,224 output rows. The tied token embedding and output matrix holds 4,194,304 weights, and octlm leaves that one in float16, because quantizing a tied matrix opens a whole separate quality and implementation question. Norm gains also stay floating point.

One scale per output row

For a row of weights $w$, take its largest absolute value and map it to signed int8 value 127. The row's scale and each stored integer are

On a forward pass the approximate weight is $\hat w_i = s q_i$. The scale is float32 and the integer is signed int8. The $10^{-8}$ floor is just so a row of all zeros doesn't blow up. This is called symmetric per-output-channel weight-only quantization, input activations aren't quantized and every output row gets its own scale. If we used a single scale for the whole matrix, one big row would reduce the precision of every other row.

Read the conversion

• With a small maximum, the integer levels represent the other weights pretty closely.
• A big outlier pushes the scale up for the whole row, and then several small, different float values can land on the same integer.
• The storage example is five one-byte weights plus one four-byte scale, so 9 payload bytes instead of 10 float16 weight bytes. Short rows barely save anything, but real model rows are hundreds or thousands of weights, so the scale cost is tiny there.

Replace the projections, leave the tied output alone

quantize_int8 walks only the decoder blocks and swaps their linear projections for Int8Linear. It requires eval mode and doesn't touch the tied token embedding and output weight. Int8Linear stores integer rows and scales as buffers. Its forward casts the integer weights to the input dtype, multiplies by the scales, and then calls a normal floating-point linear.

class Int8Linear(nn.Module):
    def __init__(self, source: nn.Linear) -> None:
        super().__init__()
        weight = source.weight.detach().float()
        scale = weight.abs().amax(dim=1).clamp_min(1e-8) / 127
        self.register_buffer(
            "qweight", (weight / scale[:, None]).round().clamp(-127, 127).to(torch.int8)
        )
        self.register_buffer("scale", scale)
        self.register_buffer("bias", None if source.bias is None else source.bias.detach().clone())

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        weight = self.qweight.to(x.dtype)
        weight.mul_(self.scale.to(x.dtype).unsqueeze(1))
        return F.linear(x, weight, None if self.bias is None else self.bias.to(x.dtype))

This makes the storage format easy to inspect and compare, but it is not a fused int8 matrix multiply. Rebuilding the float weights for every token can cost more time and temporary memory than reading the smaller stored weights saves. TorchAO has specialized inference paths, but here I deliberately measure the plain PyTorch path we already have before deciding whether a new dependency or kernel is worth it.

Just from parameter counts, the 22,052,864 projection weights take about 42.1 MiB in float16. Their integer values plus 36,224 float32 row scales take about 21.2 MiB, so the payload drops by roughly 20.9 MiB. The final .pt files also hold the untouched embedding, norm weights and serialization overhead, so the file comparison below uses actual saved bytes. And file size by itself says nothing about peak GPU memory during dequantization.

Size, quality and decode speed have to be measured together

octlm.day5 quant loads the trained Day 4 checkpoint, evaluates the same held-out blocks before and after conversion, and saves two inference-only state dictionaries. Then it measures 256 generated tokens with the same KV-cached generator, prefill included, for both variants. The output records model bytes, tokens per second and the bits-per-byte change.

def stage_quant(args: argparse.Namespace) -> None:
    if args.out.exists():
        raise FileExistsError(f"{args.out} already contains a measurement")
    model, tokenizer = load_model(args)
    device = next(model.parameters()).device
    if device.type == "cuda":
        model.half()
    valid_tokens = token_file(args.data / "valid.bin")
    valid = validation_blocks(valid_tokens, tokenizer, model.config.context_length)
    baseline_bpb = evaluate(model, valid, tokenizer.pad_id)["bits_per_byte"]
    model.eval()
    quantized = quantize_int8(copy.deepcopy(model))
    quantized_bpb = evaluate(quantized, valid, tokenizer.pad_id)["bits_per_byte"]
    quantized.eval()
    args.out.parent.mkdir(parents=True, exist_ok=True)
    baseline_path = args.out.with_suffix(".float.pt")
    quantized_path = args.out.with_suffix(".int8.pt")
    torch.save(model.state_dict(), baseline_path)
    torch.save(quantized.state_dict(), quantized_path)
    prompt = torch.tensor([[tokenizer.bos_id]], device=device)
    model.generate_cached(prompt, 2)
    quantized.generate_cached(prompt, 2)
    _, baseline_seconds = timed_generation(model, prompt, args.quant_tokens, True)
    _, quantized_seconds = timed_generation(quantized, prompt, args.quant_tokens, True)
    _write(
        args.out,
        {
            "baseline_bits_per_byte": baseline_bpb,
            "baseline_model_bytes": baseline_path.stat().st_size,
            "baseline_tokens_per_second": args.quant_tokens / baseline_seconds,
            "bits_per_byte_delta": quantized_bpb - baseline_bpb,
            "device": str(device),
            "dtype": str(next(model.parameters()).dtype),
            "generated_tokens": args.quant_tokens,
            "int8_bits_per_byte": quantized_bpb,
            "int8_model_bytes": quantized_path.stat().st_size,
            "int8_tokens_per_second": args.quant_tokens / quantized_seconds,
            "pytorch": torch.__version__,
            "type": "int8_benchmark",
        },
    )

The earlier one-step CPU command only proved that the files and the code path worked on a random model. The finished Kaggle run gives the actual trained-checkpoint answer. It used one Tesla T4, PyTorch 2.10.0+cu128, float16 activations, the same held-out blocks and one timed 256-token generation per variant after warmup.

The float16 file was 52,542,613 bytes and int8 was 30,652,347 bytes, so 41.66 percent smaller. Held-out bits per byte went from 0.48392883 to 0.48395385, a +0.00002503 change, which is far below the +0.01 stop threshold. But cached generation dropped from 104.31 to 83.30 tokens per second, 20.14 percent slower. The path rebuilds float weights on every linear call, which is the likely cost, but one timing can't isolate the cause or give us a speed distribution.

Decision, keep int8 as an opt-in smaller checkpoint and for the Day 8 merge experiment, and keep float16 as the default inference path, because it decoded faster and had marginally better bits per byte. So this int8 implementation is a storage tradeoff, not a speed optimization.