The GRPO run that never took a step
The first GRPO step sampled 96 rollouts in 25 minutes, found 3 mixed groups where it needed 4, and stopped. The saved adapter is the SFT adapter, bit for bit. Its eval matches SFT to the third decimal, which makes it a clean measure of GPU noise and a failed exit check.
The previous post built GRPO and probed the task pool. Here is the run. It's short.
Settings frozen before the run
| Setting | Value |
|---|---|
| Start | SFT seed 1, retrained in the same notebook |
| Learning rate | 0.00001, constant |
| Rollouts per group | 6 |
| Mixed groups per step | 4, from at most 16 tried |
| Steps | 30, or 5 hours |
| Max gradient norm | 1 |
| KL penalty | none |
The learning rate is ten times the 1e-6 of DeepSeekMath, DAPO and Search-R1, following LoRA Without Regret, which finds LoRA wants about ten times the full fine-tuning rate.
Step 1 kept 3 groups and stopped
The first step tried 16 tasks, 96 rollouts in all, which took 1501 seconds. 3 groups came back mixed. The batch needed 4, so the run hit its sparse stop before computing a single gradient. The log says "steps": {done.steps}, "stop": "{done.stop}".
The saved adapter has the same 392 tensors as the SFT adapter, bit for bit. Its file hash differs only because the metadata now records zero GRPO steps and the pool it didn't train on.
The probe had already said so
At the probe's pass rates, a group of 6 from this pool is mixed with chance 0.179. Across 16 tries that's 2.86 kept groups on average. The run got 3. The outcome was the expected one, not bad luck.
The mistake was in the stop condition. It checked that a fifth of tasks were mixed in groups of 4. The batch rule needed a quarter of groups of 6 to be mixed. Those are different numbers, and the probe cleared the first while missing the second.
The time budget was out of reach as well. One rollout took 15.6 seconds on the T4. 30 steps at the measured keep rate need about 17 hours of sampling. Five hours buys about 9 steps.
Held-out eval: 0.000
The notebook ran the unchanged Day 7 eval on both adapters, the same 40 tasks and three sampling seeds.
| Measure | Day 8 seed 1 | SFT, retrained | GRPO, 0 steps |
|---|---|---|---|
| pass^1 mean | 0.467 | 0.517 | 0.517 |
| pass^1 per sampling seed | 0.450, 0.475, 0.475 | 0.500, 0.500, 0.550 | 0.500, 0.500, 0.550 |
| pass^3 | 0.450 | 0.475 | 0.475 |
| Valid-call rate | 0.985 | 0.982 | 0.981 |
| Success, answer | 0.595 | 0.595 | 0.595 |
| Success, file | 0.704 | 0.926 | 0.926 |
| Success, hidden | 0.500 | 0.500 | 0.500 |
| Success, tests | 0.000 | 0.000 | 0.000 |
| Runs stopped by the turn limit | 46 | 35 | 35 |
| Tool calls | 533 | 488 | 485 |
The exit check asked for a gain above 0.092, the gap between the two Day 8 training seeds. The gain is 0.000. The check fails.
What identical weights measure
Since the weights match, the two evals differ only by GPU noise. 2 of 120 runs made different calls, took different turns or gave a different answer. None changed its pass or fail. The pool probe, rerun on the same weights after "training", moved 1 task: {probeMoves[0]?.task} went from 0 to 1 passes of 4, so mixed tasks went from 7 to 8. Same seeds on the same T4 give nearly the same rollouts, not exactly the same.
Retraining is noisier. The Day 8 seed-1 adapter scored 0.467. The same code and seed, retrained here in 286 seconds, scored 0.517. T4 training isn't bit-reproducible, so one retrain moved pass^1 by 0.050. That's why the exit check compares GRPO with an SFT start evaluated in the same notebook, not with the best number from an earlier day.
What a rerun would change
None of this ran. In order of cost:
- Size the batch from the probe. With 7 of 33 tasks mixed, accept 2 groups per step or try 24.
- Train on the mixed tasks only. The pool shrinks to 7 tasks, so the eval would have to rule out memorization.
- Give partial credit on bug fixes, such as the fraction of hidden tests passed. A bug the model never fixes would still give different rewards inside a group.
Only the third touches tests tasks, where success is still 0.000. The first two would train on answer and hidden tasks that SFT already passes half the time.
Decision: Day 9 closes with a failed exit check. GRPO did not change the model, and the Day 8 SFT adapter stays the best agent.
Not built
- A second run with looser settings. Changing the batch rule after seeing the result would be tuning against the outcome. The rerun belongs in a new experiment with its own frozen settings.
- Routing failed tasks to an external API. The plan allowed one Day 9 extension, and this was the other one.