Where the norm goes

A Transformer block has two parts, attention and a feed-forward layer, and each one is wrapped in a residual addition and a norm. And the order of those three operations actually has a name. The 2017 paper added first and normalized after, that's post-norm. GPT-2 and almost everything since normalize first, inside the branch, that's pre-norm. EXP-015 tested whether that choice matters at our scale.

Norm inside the branch, or on the main path

What changes

• In pre-norm, the residual stream is the embedding plus every block's correction. Nothing on the main path rescales it, so the gradient of the loss reaches the first layer through additions alone.
• In post-norm, every sum gets normalized before it moves on. The main path goes through $2L$ norms, and the gradient has to pass back through every one of them.

Both are one line in octlm:

class TransformerBlock(nn.Module):
    def __init__(self, config: DecoderConfig) -> None:
        super().__init__()
        self.pre_norm = config.residual == "pre"
        self.attention_norm = make_norm(config)
        self.attention = CausalSelfAttention(config)
        self.feed_forward_norm = make_norm(config)
        self.feed_forward = FeedForward(config)

    def forward(self, x: Tensor, rope: tuple[Tensor, Tensor] | None = None) -> Tensor:
        if self.pre_norm:
            x = x + self.attention(self.attention_norm(x), rope)
            return x + self.feed_forward(self.feed_forward_norm(x))
        x = self.attention_norm(x + self.attention(x, rope))
        return self.feed_forward_norm(x + self.feed_forward(x))

    def forward_cached(
        self,
        x: Tensor,
        rope: tuple[Tensor, Tensor] | None,
        cache: tuple[Tensor, Tensor] | None,
    ) -> tuple[Tensor, tuple[Tensor, Tensor]]:
        if self.pre_norm:
            attended, updated = self.attention.forward_cached(self.attention_norm(x), rope, cache)
            x = x + attended
            return x + self.feed_forward(self.feed_forward_norm(x)), updated
        attended, updated = self.attention.forward_cached(x, rope, cache)
        x = self.attention_norm(x + attended)
        return self.feed_forward_norm(x + self.feed_forward(x)), updated

Why post-norm needs warmup

Xiong et al. analyzed both layouts at initialization. In post-norm, the gradients of the parameters near the output are large, no matter the depth. A big learning rate at step 0 on big gradients makes the first updates unstable, which is why the original Transformer needed a learning-rate warmup to train at all. In pre-norm the gradient scales with $1/\sqrt{L}$ and stays well behaved, and the paper shows pre-norm can train without warmup.

The note took a rule out of this for how we run experiments. Removing warmup is a separate change, so it never goes together with a residual change. EXP-015 kept Day 1's warmup for both layouts, only the placement moved.

One bit per byte worse

Hypothesis, at four layers with warmup, post-norm should still train, because that's the regime where the paper says it works. The question was how much it costs.

So what happened ?

• Post-norm is 1.00 bits per byte worse on code and 0.70 worse on prose.
• Its seed spread is small, 0.0047 on code. So it's not unstable across seeds, it just reliably lands somewhere worse.
• Parameter count is identical, 3,740,160, because the same norms exist in both layouts. Step time is about the same.

So the prediction held, in the weak sense. It trained, badly. Four hundred steps with a 40-step warmup wasn't enough for post-norm to catch up, and nothing in the grid suggests more steps would close a full bit.

Decision, revert. The Day 1 pre-norm placement stays in every model.

What we didn't build, and why

• Removing warmup from pre-norm. The theory says pre-norm can drop it. The note put the question off, because changing warmup would invalidate every Day 2 comparison. It never came back as its own experiment, and the Day 4 model keeps a 500-step warmup.
• Newer residual variants. day-wise.md listed "newer residual variants" like DeepNorm and hyper-connections. mHC was a Day 3 experiment and got dropped with the rest of Day 3b.