GRPO with a pass-or-fail reward

The SFT adapter calls tools correctly almost every time. It still can't fix a bug. On the tests tasks of the Day 7 eval, it reads the file, runs the tests, and runs out of turns. The traces showed it what a fix looks like. They didn't show it which fix.

Reinforcement learning is the obvious next try. Let the model attempt the task, check the result, and make whatever it did on the passing attempts more likely. The harness already has the check. Every task ends in a script that says pass or fail.

The reward is the task check

A rollout is one full run of the agent loop on one task. The model writes, a tool answers, the model writes again, until it gives a final answer or hits the turn limit. Then the task's check runs, and the reward is 1 for a pass and 0 for a fail. Nothing else counts. No points for a valid call, no penalty for length.

That reward is sparse but honest. A shaped reward, say a point for running the tests, invites the model to collect points without fixing anything. The check can't be gamed that way, because it's the same check the eval uses.

Group-relative advantage

Policy-gradient methods push up the log-probability of each action in proportion to its advantage, how much better it did than expected. PPO learns a second network, the value model, to say what to expect. GRPO drops it. It samples a group of $G$ rollouts for the same task and uses the group's mean reward as the expectation:

The DeepSeekMath paper also divides by the group's standard deviation. Dr. GRPO shows that the division makes the update depend on how hard the task is. A group with one pass in six gets a much larger push than a group with three. octlm leaves it out.

The widget shows the catch. If all 6 rollouts pass, or all fail, every advantage is zero and the group teaches nothing. DAPO drops such groups and samples more until the batch is full of mixed groups. octlm does the same. It keeps sampling until it has 4 mixed groups, and gives up after 16 tries.

Which tokens get the gradient

A rollout holds tokens the model wrote and tokens it didn't: the system prompt, the task, every tool result. Only the model's own tokens are actions. Following Search-R1, the loss covers the tokens the model generated in each assistant turn and nothing else:

$T$ counts every generated token in the batch, the DAPO choice, so a long rollout weighs more than a short one. There's no clipping. PPO clips the ratio between the new policy and the one that sampled, but with one update per batch those are the same policy, the ratio is 1, and the clip never fires. There's no KL penalty to a reference model either. DAPO drops it, and dropping it saves a second copy of the model on a 15 GB T4.

def backward_group(model, group: list, temperature: float, total: int, scaler) -> float:
    mean = statistics.mean(reward for reward, _, _ in group)
    loss = 0.0
    for reward, turns, _ in group:
        advantage = reward - mean
        for prompt, new in turns:
            log_probs = reply_log_probs(model, prompt + new, len(prompt), temperature)
            value = -advantage * log_probs.sum() / total
            scaler.scale(value).backward()
            loss += value.item()
    return loss

Every turn of a rollout gets the same advantage. A pass that wasted a turn on a useless file read still pushes that read up. That's coarse, and it's what outcome-only rewards give you.

Fitting it on a T4

Qwen's vocabulary has 151,936 entries. A float32 log-softmax over every position of a 5,600-token prompt would take 3.4 GB on its own. The loss only needs the reply tokens, so reply_log_probs runs the decoder blocks with gradient checkpointing and applies the output head only where a reply token is predicted. A CPU check against the plain forward pass matched within 5e-7.

def reply_log_probs(model: Decoder, ids: list[int], start: int, temperature: float):
    device = model.token_embedding.weight.device
    tokens = torch.tensor([ids], device=device)
    config = model.config
    rope = rope_tables(len(ids) - 1, config.head_size, config.rope_scale, device, config.rope_base)
    with torch.autocast(device.type, dtype=torch.float16, enabled=device.type == "cuda"):
        x = model._embed(tokens[:, :-1])
        for block in model.blocks:
            x = checkpoint(block, x, rope, use_reentrant=False)
        logits = model.lm_head(model.final_norm(x[:, start - 1 :]))
    log_probs = (logits.float() / temperature).log_softmax(-1)
    return log_probs.gather(-1, tokens[:, start:, None]).squeeze(-1)

The probe: is there anything to learn from?

Before any training, the notebook ran the SFT adapter 4 times on each task and counted passes. A task that passes 0 or 4 times out of 4 is likely to give uniform groups and no gradient.

The plan said to run GRPO on the SFT training tasks. The adapter memorized them, so 11 of 12 give uniform groups. That's why Day 9 built a new pool of 33 tasks with new bugs and questions on the same repository.

The pool gave 7 mixed tasks, 21 percent. The hypothesis asked for a third. The stop condition was a fifth, so training went ahead.

Not one mixed bug fix

The 13 tests tasks split cleanly. 5 passed 4 of 4 and 8 passed 0 of 4. None was mixed. The model either always fixes a bug or never does, and a pass-or-fail reward can't tell a near miss from a hopeless attempt. On the skill this day set out to teach, the reward gave no signal at all.

The widget's numbers also predict the training run. At groups of 6, a group from this pool is mixed with chance 0.179. In 16 tries that's 2.86 kept groups on average, and the batch needed 4. The next post shows what happened.

Not built

• A value model. GRPO's group mean replaces it. A value model the size of Qwen would not fit next to the policy on a T4.
• A KL penalty and reference model. DAPO argues the penalty isn't needed, and it would cost a second model in memory.
• Several updates per batch. One update means no clipping and no stored sampling probabilities. With several, the ratio would leave 1 and the clip would matter.
• Partial credit. A reward for the fraction of hidden tests passed would give tests tasks a signal. It also changes what the experiment measures, so it's the next experiment, not part of this one.