Merging the adapter and storing it in int8

The fine-tuned adapters work, but they cost something at inference. Every adapted projection runs three matmuls instead of one: the base weight, then A, then B. This post folds the adapter into the base weights, checks the fold, and stores the result with Day 5's int8 path.

One matmul instead of three

The adapter's contribution is linear in $x$, so it folds into the weight once:

After the merge, the model has the base model's shapes and speed. octlm computes $\frac{\alpha}{r} B A$ in float32, adds it to $W_0$ in float32, and casts the sum back to float16, as PEFT does.

-2.090  3.308  0.900  3.416  -3.605  2.846
-1.112  -3.575  -3.450  3.871  3.782  0.250
-0.385  0.383  0.840  2.219  2.238  -0.655
-0.207  2.817  -1.100  -0.148  -3.264  1.993

def merge_lora(model: Decoder) -> None:
    for parent, name in projections(model):
        setattr(parent, name, getattr(parent, name).merged())

The first gate was tighter than float16

The plan said to stop if merged and unmerged logits differed by more than 0.05, "the float16 rounding scale". The first T4 run measured 0.157 on a 516-token prompt and stopped before the int8 eval.

That looked like a broken merge. A float32 check on the laptop said otherwise:

In float32 the merge is exact to about $10^{-4}$. Rounding only the merged weights to float16 already costs 0.042, and float16 activations across 28 blocks add the rest. The 0.05 was a guess. It was never measured, and it broke the project's own rule that a tolerance comes from a measured noise floor.

The largest logit on the prompt is about 38. Near 38, adjacent float16 values are $2^{5-10} = 0.031$ apart. Any tolerance near 0.05 sits within two rounding steps of the largest logits, so ordinary rounding trips it.

The new gate asks a question rounding can't fake. Do the merged and unmerged models pick the same top-1 token at every position of the prompt? It still logs the max difference. The float32 table is the proof that the math is right.

Same top token at all 516 positions

Kaggle wouldn't take the errored run's adapter as an input, and uploading the 40 MB file from the laptop stalled. So the rerun trained seed 0 again on the T4, then merged, quantized and ran the full eval.

The rerun trained a different adapter

The rerun's adapter hash is {hash(env)}. The first run's was {hash(eval0[0])}. Same code, same data, same seed, different bits. The two loss curves agree within 0.001 for the first ten steps, then drift apart. Step 42 is 0.077 in the first run and 0.045 in the rerun.

Two things differ between the runs. Some CUDA kernels PyTorch uses in training are nondeterministic unless you ask for deterministic ones, and the training traces contain a random sandbox path and unittest timings that change from run to run. So the int8 eval below runs a different adapter from the float16 eval it's compared with.

pass^1 holds at 0.375

The mean is identical. Run by run, 35 of 120 runs pass in both evals, 10 pass only in float16, and 10 pass only in int8. The sets of solved runs overlap a lot but aren't the same, which is what two samplers over two similar models should produce.

The turn-limit stops doubled, from 26 to 55. Most of the rise is answer tasks, 1 in float16 and 22 in int8. The int8 model keeps reading files after it has the value, yet answers correctly as often. This eval can't say whether the new adapter or the int8 weights cause the looping, because both changed at once.

Speed went up, from 13.9 to 17.1 tokens per second. The stock model ran at 18.3. Merging removes 14 adapter matmuls per block, and int8 dequantization costs less than they did. Day 5 found int8 slower than float16 on the 20M model, so the gain comes from the merge, not from int8. A float16 merged eval would measure the split. It wasn't run.

Decision: keep. The merge is exact in float32 and passes the top-1 gate. int8 halves the stored block weights, and pass^1 matches the float16 adapter. A fair comparison still needs the float16 eval of adapter {hash(env)}, one more 20-minute T4 run.

Not built

• A float16 merged eval. It would split the speed gain between the merge and int8, and remove the adapter confound. The exit check didn't need it.
• int8 activations. The weights are int8 but the matmuls run in float16 after dequantizing. True int8 matmuls need kernels PyTorch's eager mode doesn't offer on a T4.
• Deterministic training. torch.use_deterministic_algorithms would make the reruns match, at a speed cost. It would hide nothing the eval needs, so it can wait until a result depends on bit-exact reruns.