Reading tool calls out of free text

A model only produces text. To make it act, you need a program around it that offers tools, spots when the model asks for one, runs it, and hands the result back. That program is the harness. Day 7 builds one for Qwen3-0.6B. This first post covers the bit where text becomes an action: the tool-call format and its parser.

Qwen already knows a format

We don't invent one. Qwen's chat template, the same one Day 6 matched byte for byte, prints the tool schemas into the system turn and tells the model how to call them:

For each function call, return a json object with function name and arguments
within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call>

The model learned this format in post-training, so using it costs nothing. A homemade format such as ReAct's Action: lines would fight that training. Qwen's own guide also warns that stop-word formats break with reasoning models, because the model can write the stop word while it's thinking.

The harness offers 5 tools: read_file, list_files, grep, write_file, run_tests. Every argument is a string, so a well-formed call is easy to check.

One block at a time

The parser does two jobs. First it finds the blocks with the same regular expression vLLM uses for Qwen:

<tool_call>(.*?)</tool_call>|<tool_call>(.*)

The second branch catches a final block with no closing tag, which happens when a turn hits its token limit.

Then it checks each block on its own, using the same checks as the Berkeley Function Calling Leaderboard: valid JSON, an object with exactly name and arguments, a known tool name, every required argument present, no argument the schema doesn't list, and string values. A valid call runs. Anything else becomes an error string that goes back to the model as that call's result.

vLLM works differently. If one block has a JSON error, it drops every call in the turn and returns the raw text. octlm still runs the good calls in a mixed turn and reports the bad one. Either way, a malformed call never crashes the loop. The model just gets a message saying what went wrong.

def validate_call(block: str) -> dict | str:
    try:
        call = json.loads(block)
    except json.JSONDecodeError as error:
        return f"error: tool call is not valid JSON: {error}"
    if not isinstance(call, dict) or set(call) != {"name", "arguments"}:
        return 'error: a tool call must be a JSON object with exactly "name" and "arguments"'
    schema = SCHEMAS.get(call["name"]) if isinstance(call["name"], str) else None
    if schema is None:
        return f"error: unknown tool {call['name']!r}"
    arguments = call["arguments"]
    if not isinstance(arguments, dict):
        return "error: arguments must be a JSON object"
    missing = set(schema["required"]) - set(arguments)
    unknown = set(arguments) - set(schema["properties"])
    if missing or unknown:
        return f"error: missing arguments {sorted(missing)}, unknown arguments {sorted(unknown)}"
    if any(not isinstance(value, str) for value in arguments.values()):
        return "error: every argument must be a string"
    return call

A turn with no block at all is the model's final answer, and the loop stops. That rule is simple, but it has a cost. A model that never calls a tool just answers straight away, and the harness can't tell a grounded answer from a guess.

A third of the calls were valid

In the baseline, stock Qwen3-0.6B ran 40 tasks under 3 seeds, 120 runs in all. It emitted 24 tool-call blocks. 8 were valid, a rate of 0.333. Every invalid block was a write_file call with code in its content argument:

• 8 contained a raw newline inside a JSON string. JSON requires \n there, and Python's strict json.loads rejects a literal newline.
• 8 were cut off because the turn hit its 512-token limit while the model repeated the same boolean expression.

No exception ever ended a run, which was the experiment's stop condition. The feedback reached the model, but it didn't help. In the two runs that hit the 8-turn limit, the model read "not valid JSON" and sent the same broken call again, every turn.

The bigger story is calls that never happened. Only 10 of 120 runs made any call at all. The eval post shows what that does to the score.

Decision: keep the per-block parser.

Not built

• Lenient JSON. json.loads(strict=False) would accept the raw newlines and turn 8 errors into valid calls. That hides a format error Day 8's fine-tune can fix, so the parser stays strict.
• Constrained decoding. A grammar that masks every token which would break the JSON makes invalid calls impossible. It changes the sampler, not the model, and it would hide what Day 8 is supposed to measure.
• Streaming parse. The harness parses a turn only after it ends. It never acts on a half-written call.