LoRA from scratch

Day 7 ended with a clear diagnosis. Stock Qwen3-0.6B answers without reading the repository, and when it does write a file, the JSON breaks. Day 8 fine-tunes the model on examples of doing it right. Before any of that, we need a way to train a 600M-parameter model on a free T4 with 15 GB of memory.

Full fine-tuning doesn't fit

Plain fine-tuning updates every weight. With mixed precision and AdamW, each parameter costs 16 bytes of state: a float32 master copy, a float32 gradient, and two float32 Adam moments. Qwen3-0.6B has 606,142,464 parameters in octlm's decoder, so that's 9.7 GB before activations. Activations for a 1,000-token example come on top. A T4 has 15 GB, and full fine-tuning would leave almost none of it for activations.

Train a small correction, not the weights

LoRA freezes the original weight matrix $W_0$ and learns a low-rank update next to it. For a projection with $d_\text{in}$ inputs and $d_\text{out}$ outputs:

The rank $r$ is small, 16 here, against $d_\text{in} = {qwen.config.d_model}$. So $A$ and $B$ together hold $r(d_\text{in} + d_\text{out})$ numbers, while $W_0$ holds $d_\text{in} d_\text{out}$. The bet is that the change fine-tuning needs lives in a few directions, so a rank-16 update can carry it.

Two details make it work:

• B starts at zero. $A$ gets the usual random nn.Linear initialization, and $B$ is all zeros, so $BA = 0$. At step 0 the adapted model is exactly the base model. Training starts from Qwen's behavior, not from noise.
• The scale is α/r. With $\alpha = {alpha}$ and $r = {rank}$, the update is multiplied by 2. Fixing α while changing r keeps the update's size roughly constant, so the learning rate needs less retuning.

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, rank: int, alpha: float, generator: torch.Generator):
        super().__init__()
        self.base, self.scale = base, alpha / rank
        a = torch.empty(rank, base.in_features)
        nn.init.kaiming_uniform_(a, a=math.sqrt(5), generator=generator)
        device = base.weight.device
        self.lora_a = nn.Parameter(a.to(device))
        self.lora_b = nn.Parameter(torch.zeros(base.out_features, rank, device=device))

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        update = x.to(self.lora_a.dtype) @ self.lora_a.T @ self.lora_b.T
        return self.base(x) + (update * self.scale).to(x.dtype)

    def merged(self) -> nn.Linear:
        weight = self.base.weight
        linear = nn.Linear(
            self.base.in_features,
            self.base.out_features,
            bias=False,
            device=weight.device,
            dtype=weight.dtype,
        )
        delta = self.lora_b.float() @ self.lora_a.float() * self.scale
        linear.weight.data.copy_(weight.float() + delta)
        return linear

All seven projections, not just attention

The LoRA paper adapted only the query and value projections. Later work disagreed. QLoRA found that query and value adapters alone could not match full fine-tuning, and that "LoRA on all linear transformer block layers are required". Thinking Machines' LoRA Without Regret found attention-only LoRA "significantly underperforms" MLP-only LoRA. octlm adapts all seven projections in every block: query, key, value and output in attention, and gate, up and down in the SwiGLU feed-forward.

The feed-forward projections dominate the count. Each is 1,024 by 3,072, so they hold most of the weights the adapters can reach. Dropping them cuts the adapter by more than half and skips the layers QLoRA and Thinking Machines found matter most.

Exactly zero at step 0

The plan set one stop condition before the run. Because $B$ starts at zero, any nonzero logit difference at step 0 means the code is wrong. The training stage runs the base model and the freshly adapted model on the longest training example, 1,157 tokens, and compares their logits.

The difference is exactly 0.0 in all 3 training runs, including the later rerun of seed 0. It's not merely small. The adapter output is cast to float16 before the add, and $0 \times$ anything is an exact zero, so the sum is bit-identical. Peak memory stayed near half the T4.

Decision: keep. The adapters fit with room to spare, and step 0 is provably the base model.

Not built

• QLoRA's 4-bit base. Storing $W_0$ in 4-bit NormalFloat would save more memory. At 0.6B parameters the float16 base already fits, so the extra code buys nothing here.
• The PEFT library. Hugging Face PEFT does all of this. octlm writes LoRALinear itself, under 30 lines, so every line is visible and the merge in the last Day 8 post can be checked by hand. octlm copies PEFT's initialization and its float32 merge.
• LoRA dropout and DoRA. Both are common options. Neither was needed to pass the exit check, and each would be another setting to tune.