Speaking Qwen's language

A model only understands the token IDs it was trained on. Give Qwen the right weights and our Day 4 tokenizer, and "Once upon a time" becomes a list of numbers that mean something else entirely to it. So Day 6 needs Qwen's exact tokenizer, and the exact text format its post-training used for chat.

<|im_start|>user
Reply with one short sentence about a robot.<|im_end|>
<|im_start|>assistant
<think>

</think>

Load Qwen's tokenizer, don't rebuild it

Day 1's BPE learns merges from our own data, and AGENTS.md requires that for models we train. Qwen is the exception. Its vocabulary of 151,936 entries ships with its weights in tokenizer.json, and the weights only make sense with that exact vocabulary.

Could octlm's BPE code read that file? Not without changes. Qwen maps bytes to characters differently, splits text into pre-tokens with a different regex, and handles added tokens its own way. Rewriting all of that would duplicate a tokenizer that already exists. So octlm adds one dependency, Hugging Face's tokenizers package, which loads tokenizer.json directly. The plan allowed either route and said to decide by exact ID match.

Ordinary text must never become a command

Qwen marks conversation structure with added tokens such as <|im_start|>, <|im_end|> and <|endoftext|>. Each has a single ID. By default, the tokenizer turns the literal text <|im_end|> into that ID wherever it appears.

That's a problem for a harness. Suppose a file the model reads contains the string <|im_end|>. Encoded the default way, the model would see a real end-of-turn marker in the middle of a tool result. A user could end the model's turn or fake a new one just by writing those characters. So octlm keeps two modes:

• Literal, for any text that comes from outside: files, tool output, user messages. Control strings split into ordinary characters.
• Native, only for text octlm rendered itself from the trusted chat template. Control strings become their single IDs.

Case 4 in the playground shows the difference. The same string gives 25 IDs in literal mode and 13 in native mode, because the three tags collapse into three control IDs.

class QwenTokenizer:
    def __init__(self, directory: Path) -> None:
        self.native = Tokenizer.from_file(str(directory / "tokenizer.json"))
        self.literal = Tokenizer.from_str(self.native.to_str())
        self.literal.encode_special_tokens = True
        self.literal.normalizer = None
        self.fingerprint = file_hash(directory / "tokenizer.json")
        self.settings = read_json(directory / "tokenizer_config.json")
        environment = ImmutableSandboxedEnvironment(
            trim_blocks=True, lstrip_blocks=True, extensions=["jinja2.ext.loopcontrols"]
        )
        environment.filters["tojson"] = template_json
        self.template = environment.from_string(self.settings["chat_template"])
        self.special_ids = {
            token["id"]
            for token in json.loads(self.native.to_str())["added_tokens"]
            if token["special"]
        }

    def encode(self, text: str, *, allow_special: bool = False) -> list[int]:
        encoder = self.native if allow_special else self.literal
        return encoder.encode(text, add_special_tokens=False).ids

    def decode(self, ids: list[int]) -> str:
        return self.native.decode(ids, skip_special_tokens=False)

    def chat(
        self, messages: list[dict], *, tools: list[dict] | None = None, thinking: bool = False
    ) -> str:
        if not messages or any(
            message.get("role") not in ("system", "user", "assistant", "tool")
            for message in messages
        ):
            raise ValueError("chat needs nonempty messages with supported roles")
        return self.template.render(
            messages=messages,
            tools=tools,
            add_generation_prompt=True,
            enable_thinking=thinking,
            **{key: self.settings[key] for key in ("bos_token", "eos_token", "pad_token")},
        )

Qwen's tokenizer changes your text

The real tokenizer.json contains an NFC normalizer. NFC merges a letter followed by a combining accent into one precomposed character. So cafe plus a combining acute accent (two code points at the end) becomes café with one code point. The IDs match Qwen's reference, but decoding no longer gives back the exact input.

AGENTS.md says never to normalize Unicode. Exact input text matters in code, where two strings that look the same can differ. But Qwen was trained with NFC, so native encoding has to keep it. Octlm doesn't edit the file. The literal mode turns normalization off and round-trips every test string exactly, while the native mode keeps NFC for template text. Each mode is compared against a reference configured the same way.

In the playground, 2 of the 4 test strings give different IDs in the two modes: case 3 because of the accent, case 4 because of the control strings.

Render Qwen's own template, exactly

A chat model expects its conversation in the exact layout it saw in post-training. Qwen's tokenizer_config.json ships that layout as a Jinja template, a small program that turns a list of messages into one string. It handles the system prompt, the tool list, tool calls and results, and the thinking switch.

Octlm renders that same template instead of writing its own chat format. It uses Jinja2's sandboxed environment, because a template is code shipped inside a downloaded file, and the sandbox blocks it from reaching Python internals. Two settings match what Transformers does. A tojson filter keeps non-ASCII characters, and the loop-control extension lets templates use break and continue.

Thinking mode is the most visible switch. With thinking on, the prompt ends at <|im_start|>assistant, and the model may write its reasoning inside <think> tags before answering. With thinking off, the template adds an empty <think></think> block, so the model skips straight to the answer.

Every ID and every character matches

The test strings cover tabs, indentation, CRLF, a NUL byte, emoji with a skin-tone modifier, a combining accent, Japanese, Russian and literal control strings. The conversations cover one user turn, a system prompt, and a full tool call with its result, each with thinking on and off. Octlm and Transformers matched on both the laptop and the T4.

Decision: keep the tokenizers dependency and both modes. Day 7 must encode every tool result in literal mode.

Not built

• A generic chat renderer. Octlm renders only the template shipped with the model. Another model family would bring its own.
• Octlm's BPE reading tokenizer.json. Possible, but it would copy a tokenizer that already exists, for no measured gain.
• Template error messages. Qwen's template never calls raise_exception. A template that does would fail with a Jinja error instead of its own message.