Day 08EXP-082

Fine-tuning Qwen to use its tools

Five minutes of LoRA training on 223 scripted examples takes Qwen3-0.6B from pass^1 0.025 to about 0.4 on the unchanged Day 7 eval. Valid tool calls go from a third to almost all. Two training seeds differ by more than the sampling seeds do, and tasks that need debugging stay unsolved.

The scripted traces show Qwen reading before it answers and writing valid JSON. Now we train on them with LoRA and rerun the Day 7 eval with nothing changed. It uses the same 40 tasks, the same three sampling seeds and the same limits.

Setup

Settings frozen before the run

The plan fixed every setting before the first step, starting from QLoRA's 7B recipe:

Setting	Value	Why
Rank and α	16 and 32	QLoRA found rank barely matters once every layer is adapted
Learning rate	0.0002, constant	QLoRA's 7B rate. LoRA wants about ten times the full fine-tuning rate
Examples per optimizer step	8	LoRA loses more than full fine-tuning as the batch grows
Epochs	3	223 examples, so 84 optimizer steps
Max gradient norm	0.3	QLoRA's setting
Optimizer	AdamW	betas (0.9, 0.999), no weight decay
Precision	float16 autocast	The T4 has no bfloat16, so a GradScaler guards against underflow

Two adapters train with seeds 0 and 1. The seed sets A's initialization and the example order. Comparing the two shows how much of the result is luck.

octlm/day8.pyline 183

def stage_train(args: argparse.Namespace) -> None:
    device = resolve_device(args.device)
    model, manifest = load_qwen(args.model, str(device), dtype_for(device))
    tokenizer = QwenTokenizer(args.model)
    data = [row for _, messages in traces(args.tasks) for row in examples(messages, tokenizer)]
    longest = max(data, key=lambda row: len(row[0]))[0]
    _write(args.out, step_zero(model, longest, device, args.seed))
    trainable = [p for p in model.parameters() if p.requires_grad]
    optimizer = torch.optim.AdamW(trainable, lr=LEARNING_RATE, weight_decay=0.0)
    scaler = torch.amp.GradScaler(device.type, enabled=device.type == "cuda")
    generator = torch.Generator().manual_seed(args.seed)
    if device.type == "cuda":
        torch.cuda.reset_peak_memory_stats(device)
    model.train()
    started, step = time.perf_counter(), 0
    for epoch in range(EPOCHS):
        order = torch.randperm(len(data), generator=generator).tolist()
        for first in range(0, len(order), BATCH):
            batch = [data[i] for i in order[first : first + BATCH]]
            total = sum(len(ids) - start for ids, start in batch)
            loss = 0.0
            for ids, start in batch:
                value = example_loss(model, ids, start, total, device)
                scaler.scale(value).backward()
                loss += value.item()
            scaler.unscale_(optimizer)
            norm = torch.nn.utils.clip_grad_norm_(trainable, MAX_GRAD_NORM).item()
            scaler.step(optimizer)
            scaler.update()
            optimizer.zero_grad(set_to_none=True)
            step += 1
            row = {"type": "sft_step", "step": step, "epoch": epoch, "loss": loss}
            _write(args.out, row | {"grad_norm": norm, "target_tokens": total})
    seconds = time.perf_counter() - started
    adapters = {k: v.detach().cpu() for k, v in model.state_dict().items() if ".lora_" in k}
    torch.save(
        {
            "adapters": adapters,
            "rank": RANK,
            "alpha": ALPHA,
            "seed": args.seed,
            "manifest": manifest,
            "tokenizer_hash": tokenizer.fingerprint,
            "tasks_hash": file_hash(args.tasks),
        },
        args.adapter,
    )
    _write(
        args.out,
        {
            "type": "sft_summary",
            "seed": args.seed,
            "examples": len(data),
            "steps": step,
            "seconds": seconds,
            "tokens_per_second": EPOCHS * sum(len(ids) for ids, _ in data) / seconds,
            "peak_memory_bytes": torch.cuda.max_memory_allocated(device)
            if device.type == "cuda"
            else None,
            "adapter_hash": file_hash(args.adapter),
            "settings": [RANK, ALPHA, LEARNING_RATE, BATCH, EPOCHS, MAX_GRAD_NORM],
        },
    )

Training

The loss goes to almost zero

Loss

Loss on target tokens at every optimizer step, both seeds. Each point is one batch of 8 examples, so it jumps with which examples landed in the batch.

Losses this low on 7,287 target tokens mean the adapters memorized the traces. There's no held-out loss to say otherwise, because the eval is the held-out test. Each run took about 268 seconds at 1,777 tokens per second.

Result

pass^1 goes from 0.025 to 0.421

Measure	Day 7 base	Seed 0	Seed 1
pass^1 mean	0.025	0.375	0.467
pass^1 per sampling seed	0.050, 0.025, 0.000	0.350, 0.375, 0.400	0.450, 0.475, 0.475
pass^3	0.000	0.350	0.450
Valid-call rate	0.333	0.976	0.985
Success, answer	0.024	0.333	0.595
Success, file	0.074	0.815	0.704
Success, hidden	0.000	0.250	0.500
Success, tests	0.000	0.111	0.000
Mean turns	1.18	4.62	5.06
Runs stopped by the turn limit	2	26	46
Generated tokens per second	18.3	13.9	14.1

The exit check, written before the run, asks whether the gain beats the spread between training seeds. The mean of the two adapters is 0.421. Minus the base 0.025, that's a gain of 0.396, against a gap of 0.092 between the adapters. The check passes by a factor of 4.

Every run

One cell per run, as in the Day 7 eval. Switch between the base model and each adapter. Dashed cells changed result against the base model, or against seed 0 when you view the base.

Both Day 7 failures are gone

Day 7 found two problems. The model answered without calling a tool, and its write_file JSON broke. The valid-call rate rose from 0.333 to above 0.976. pass^1 counting only runs with a valid call is 0.375 and 0.467, equal to plain pass^1, so every pass came from a run that used a tool. The lucky guesses of the base model are gone.

Debugging is still out of reach

tests tasks plant a bug and ask the model to fix it. The adapters passed 3 of 54 such runs. Seed 1 hit the turn limit in 46 runs, including all 27 of its tests runs. It loops on reading files and running tests instead of writing a fix. The traces showed the right sequence, run the tests, read, fix, run again, but 14 examples of a bug fix aren't enough to teach what the fix should be.

The training seed matters more than the sampling seed

Within one adapter, the three sampling seeds span 0.050 for seed 0 and 0.025 for seed 1. Between the adapters, pass^1 moves by 0.092. Two adapters trained on the same data with the same settings behave differently, and the per-kind rows show it: seed 1 is better at answer and hidden, and seed 0 is better at file. One training run would have hidden that.

Decision: keep. The fine-tune passes the exit check, and both Day 7 failure modes are gone.

Skipped

Not built

A held-out training loss. All 74 traces went into training. The eval on a different repository is the held-out test, and it's the number that matters.
More training seeds. Two seeds meet the exit check. With a seed gap of 0.092, a finer comparison between settings would need more.
Tuning. Rank, learning rate and epochs stayed at the frozen values. Changing them after seeing the eval would turn the eval into a training signal.