The agent loop and a KV cache that survives turns
The harness renders the conversation, generates until <|im_end|>, runs the tools inside a sandbox and repeats. Qwen's template rewrites past turns, so the cache keeps the longest shared token prefix and prefills only what changed.
The parser turns text into calls. This post is about everything around it: the loop that alternates between the model and the tools, the sandbox that keeps the tools contained, and a KV cache that carries over from one turn to the next.
Render, generate, parse, run, repeat
The transcript is just a list of messages. Following mini-swe-agent, there's no hidden state and no persistent shell. Each turn appends to the list, and each tool call runs on its own. That keeps runs easy to replay, and Day 8 can use the transcripts as training data without converting them.
def run_task(
model: Decoder,
tokenizer: QwenTokenizer,
settings: dict,
task: dict,
root: Path,
seed: int,
turns: list | None = None,
watch: Callable[[str, object], None] | None = None,
) -> dict:
generator = torch.Generator().manual_seed(seed)
device = model.token_embedding.weight.device
messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": task["prompt"]}]
state = {"ids": [], "cache": []}
reserved = [token.content for token in tokenizer.native.get_added_tokens_decoder().values()]
record = {
"stop": "turns",
"turns": 0,
"calls": 0,
"valid_calls": 0,
"tool_errors": 0,
"call_errors": [],
"truncated_turns": 0,
"generated_tokens": 0,
"rendered_tokens": 0,
"prefilled_tokens": 0,
"seconds": 0.0,
"answer": None,
}
for _ in range(MAX_TURNS):
text = tokenizer.chat(messages, tools=TOOLS, thinking=settings["thinking"])
prompt = tokenizer.encode(text, allow_special=True)
if len(prompt) + MAX_NEW_TOKENS > CONTEXT_LIMIT:
record["stop"] = "context"
break
(new, keep), seconds = timed(
lambda: generate_turn(model, prompt, state, settings, generator, watch), device
)
ended = new[-1] in settings["stop_ids"]
if turns is not None:
turns.append((prompt, new))
record["turns"] += 1
record["truncated_turns"] += not ended
record["generated_tokens"] += len(new)
record["rendered_tokens"] += len(prompt)
record["prefilled_tokens"] += len(prompt) - keep
record["seconds"] += seconds
reply = tokenizer.decode(new[:-1] if ended else new)
messages.append({"role": "assistant", "content": reply})
calls = parse_calls(reply)
if not calls:
record["stop"], record["answer"] = "answer", reply
break
for call in calls:
result = respond(root, call, record, reserved)
messages.append({"role": "tool", "content": result})
if watch:
watch("tool", result)
return {**record, "messages": messages}Five tools, one directory
Every path goes through one function. It joins the path to the sandbox root, resolves .. and symlinks, and refuses anything that lands outside the root. That covers both ../../etc/passwd and an absolute path. Grep and list skip __pycache__.
run_tests starts python -m unittest in its own process session. Python's subprocess timeout only kills the direct child, so a test that starts a process of its own could outlive the 30-second limit. On timeout, the harness kills the whole process group instead.
One more guard follows from Day 6. The rendered conversation gets encoded with Qwen's control tokens switched on, so a file that contained the literal text <|im_end|> would turn into a real end-of-turn token. Any tool result containing one of the tokenizer's added-token strings is replaced by an error.
This is path isolation, not a security boundary. run_tests executes Python the model wrote. That's why the eval runs on a throwaway Kaggle machine.
Why appending tokens is wrong
Day 5 built a KV cache for one prompt. A tool loop has many prompts, and each one starts with the previous one. The obvious move is to keep the cache and append each new turn's tokens. Qwen's template makes that wrong.
In non-thinking mode, every generation prompt ends with an empty think block, <think>\n\n</think>\n\n. When the same assistant turn later appears in the history, the template renders it without that block. So the history the model would see by appending differs from the history the template produces, and the template is what the model was trained on.
So the harness re-renders the whole conversation every turn and compares token IDs:
LCP is the length of the longest common prefix. The cache gets cut to keep positions, and only the rest is prefilled. The minus one ensures at least one token is prefilled, because the next token's logits come from the last prompt position. Day 6's parity test already showed that a cache built in two chunks matches a full forward pass, so a cut cache needs no new proof.
@torch.inference_mode()
def generate_turn(
model: Decoder,
prompt: list[int],
state: dict,
settings: dict,
generator: torch.Generator,
watch: Callable[[str, object], None] | None = None,
) -> tuple[list[int], int]:
keep = min(common_prefix(state["ids"], prompt), len(prompt) - 1)
cache = [(k[:, :, :keep], v[:, :, :keep]) for k, v in state["cache"]] if keep else None
device = model.token_embedding.weight.device
logits, cache = model.forward_cached(torch.tensor([prompt[keep:]], device=device), cache)
new = []
for step in range(MAX_NEW_TOKENS):
token = sample_token(
logits[:, -1].float(),
settings["temperature"],
settings["top_k"],
generator,
settings["top_p"],
)
new.append(token.item())
if watch:
watch("token", new[-1])
if new[-1] in settings["stop_ids"] or step + 1 == MAX_NEW_TOKENS:
break
logits, cache = model.forward_cached(token, cache)
state["ids"], state["cache"] = prompt + new[:-1], cache
return new, keepMost of each prompt came from the cache
| Runs | Count | Prefilled share of rendered tokens |
|---|---|---|
| All | 120 | 0.72 |
| Two turns or more | 10 | 0.29 |
| Three turns or more | 2 | 0.21 |
The hypothesis was that tasks of three or more turns would save more than half their prefill. They saved 79 percent. One-turn runs save nothing, since their single prompt is all new. That's why the overall share is so high. The comparison needs no second run, because full prefill is just the sum of rendered prompt lengths, which the harness logs every turn.
The long-run row rests on 2 runs, because the stock model rarely takes a second turn. Day 8's fine-tuned model should produce many more.
Generation ran at 18.3 tokens per second on the T4. That includes each turn's prefill, so it sits below Day 6's decode-only 24.0 and 24.7.
Decision: keep prefix reuse.
Not built
- Real isolation. A container or a seccomp jail would contain model-written code. The throwaway VM does the job for now.
- Parallel tool calls. Calls in one turn run in order, which is what Qwen's guide describes.
- A cache shared across tasks. Every task starts with the same system prompt and schemas, so their cache could be computed once and copied. It would save 495 prefill tokens per task, a small win next to generation time.