An eval Day 8 has to beat

Day 8 will fine-tune Qwen to use tools. To know whether that worked, we need a fixed test that a script can grade, plus the stock model's score on it. This post builds that test and records the baseline.

Forty small jobs in a toy shop

The fixture is a small Python package called shop, with prices, a cart, stock counts, text helpers, a config file, a CSV of customers and 12 passing unittest tests. Every task starts from a fresh copy. There are 40 tasks of four kinds, and each kind has its own check:

Each tests task first plants one bug with an exact string replacement, such as percent / 100 becoming percent / 10. The check also requires every original test file to be byte-identical, which blocks the obvious cheat of deleting or weakening the failing test. The hidden kind borrows SWE-bench's idea: grade new code with a test the model never sees.

Every task proves it can fail and pass

A check that always passes, or never can, quietly ruins an eval. SWE-bench runs every task's gold patch before using the task. octlm does the same with a validate stage that needs no model. Each task has a reference solution written as tool calls. For every task, validate confirms the check fails on the untouched repository. It then runs the reference calls through the real parser and tools, and confirms the check passes. For answer tasks it also confirms some tool output contains the answer, except two counting tasks whose answer no tool prints directly.

40 of 40 tasks passed validation, on the laptop and again on Kaggle before each eval.

def validate(args: argparse.Namespace) -> None:
    tasks, fixtures = load_tasks(args.tasks), args.tasks.parent
    for task in tasks:
        with sandbox(task, fixtures) as root:
            before = passes(task, root, "", fixtures)
        with sandbox(task, fixtures) as root:
            results = replay(task, root)
            after = passes(task, root, task["solution"]["answer"], fixtures)
        check = task["check"]
        found = (
            check["kind"] != "answer"
            or check.get("derived")
            or any(check["expected"].casefold() in result.casefold() for result in results)
        )
        row = {"type": "task_validation", "task": task["id"], "kind": check["kind"]}
        _write(args.out, {**row, "fails_before": not before, "passes_after": after, "found": found})
        if before or not after or not found:
            raise RuntimeError(f"task {task['id']} failed validation")
    _write(args.out, {"type": "validation_summary", "tasks": len(tasks), "valid": True})

Sampling makes one run a coin flip

The harness samples, as Qwen's model card recommends, so the same task can pass on one seed and fail on the next. Each task runs under seeds 0, 1, 2, for 120 runs in all. Two numbers summarize them. pass^1 is the plain success rate. pass^k comes from tau-bench and gives the chance that k independent tries all succeed. For a task with c successes in n tries:

pass@k, the HumanEval metric, rewards a model that gets something right once. pass^k rewards one that's reliable. If a fine-tune only makes the model more consistent, pass^k moves first.

110 of 120 runs never called a tool

Asked which port the server uses, the model doesn't read config/settings.ini. It replies "The server uses port 8080." The config says 8042. It does this on nearly every answer and tests task. It only uses tools when the prompt names a file to write, and even then most of its code arrives as broken JSON.

Read the passing transcripts

Run 1 scored 6 passes. Reading those transcripts turned up two bad checks:

• lamp-stock passed with "The number of lamps in stock is 123." The answer check tested for a substring, and "123" contains "12".
• add-customer passed twice after the model overwrote data/customers.csv with a header and the one new row, deleting the other seven customers. The file check only looked for the new line.

The plan written before the run said the task set freezes, and that any change means a new version and a new run, not an edit. So run 1 stays on record and v2 fixes both. An answer must now match as a whole token: no letter, digit, dot or dash just before it, and no letter, digit, dash or decimal part just after. File edits now also list original lines that have to survive. Run 2 scored the same tasks with the v2 checks. 3 runs changed from pass to fail.

That comparison is unusually clean. 120 of 120 transcripts in run 2 match run 1 token for token. Fixed seeds reproduce sampled generation across two separate T4 sessions, so the scoring is the only thing that changed.

Even v2 has a known blind spot. lamp-stock on seed 0 passed without a single tool call. The model guessed a number and happened to be right. Only 2 of the 3 passes are grounded in a tool result. Every row stores its transcript, so Day 8 can report grounded success next to the raw rate without rerunning anything.

Decision: v2 is the Day 8 baseline. The fine-tuned model has to beat pass^1 0.025 by more than the seed spread of 0.025, on the same tasks, seeds and limits.

Not built

• An LLM judge. Every check is a script. A judge could grade open-ended answers, but it brings its own biases, and this task set doesn't need one.
• A bigger task set. With 40 tasks, one task is 0.025 of pass^1, as large as the seed spread. A stronger model would need more tasks to separate small gains.
• Training traces. Day 8 collects them from a separate fixture repository, so eval tasks can't leak into training.