SwiGLU, a gated feed-forward layer

The feed-forward layer holds about two thirds of a Transformer block's parameters. Day 1's version widens the vector, applies GELU to each hidden unit, and projects back. SwiGLU, which LLaMA, Qwen and most current decoders use, changes what happens in the middle.

Two projections, one of them a gate

SwiGLU computes two projections of the input instead of one. One goes through SiLU, also called Swish, and the result multiplies the other one elementwise:

Think of each hidden unit as a valve. The gate projection decides how open the valve is, and the up projection decides what flows through. With GELU the same number decides both, whether a unit fires and how strongly. With a gate the model can learn those two things separately, and the product lets a unit respond to a combination of input features instead of a threshold on one.

So what happened ?

• SiLU looks a lot like GELU, near 0 for big negative inputs, near the identity for big positive ones, with a small dip below 0 in between.
• A gate value near −4 closes the unit no matter what the up value is. A big gate passes the up value almost unchanged, sign included. GELU can only pass positive values through.
• At every width, SwiGLU's hidden size is about two thirds of GELU's, and the two parameter counts differ by well under 1 percent.

Three matrices, so two thirds of the width

SwiGLU has three weight matrices where GELU has two. At the same hidden width it would have 50 percent more feed-forward parameters, and then any improvement could just come from the extra weights. So the GLU-variants paper cuts the hidden width to $\tfrac{2}{3} \cdot 4C$. octlm rounds that to a multiple of 8, which GPU matrix kernels like:

@property
def hidden_size(self) -> int:
    """SwiGLU spends three matrices instead of two, so 2/3 holds the parameter count."""
    if self.ff_hidden:
        return self.ff_hidden
    full = self.d_model * self.ff_multiplier
    if self.feed_forward == "gelu":
        return full
    return max(8, round(full * 2 / 3 / 8) * 8)

At $C = 256$ that gives 680 instead of 1,024, and at $C = 512$ it gives 1,368 instead of 2,048. EXP-011 compared at matched parameter count for exactly this reason, and the note says it. If we matched the width instead, we'd only be measuring that we added weights.

The layer itself is the same class as Day 1's, with the gate branch turned on:

class FeedForward(nn.Module):
    def __init__(self, config: DecoderConfig) -> None:
        super().__init__()
        hidden = config.hidden_size
        self.gated = config.feed_forward == "swiglu"
        self.up = nn.Linear(config.d_model, hidden, bias=False)
        self.gate = nn.Linear(config.d_model, hidden, bias=False) if self.gated else None
        self.down = nn.Linear(hidden, config.d_model, bias=False)
        self.dropout = nn.Dropout(config.dropout)

    def forward(self, x: Tensor) -> Tensor:
        if self.gate is None:
            return self.dropout(self.down(F.gelu(self.up(x))))
        return self.dropout(self.down(F.silu(self.gate(x)) * self.up(x)))

The one swap that cleared the noise

Hypothesis, SwiGLU improves bits per byte at equal parameters. The paper reports a perplexity of 1.944 against GELU's 1.983 after 65,536 steps.

So what happened ?

• On code, SwiGLU beats the baseline by 0.0818 bits per byte. The baseline's own seed spread is 0.0434, so the gain is about twice the noise.
• On prose it's better by 0.0196, a smaller gain against a much tighter prose spread.
• Its own seeds agree closely, with a code spread of 0.0085, a fifth of the baseline's.
• It costs 11 percent more time per step. Three smaller matrix multiplies are slower than two bigger ones at this size, and there's one more elementwise product.

Decision, keep. Of the six single-component changes on Day 2, this is the only one whose effect sat outside the seed noise. SwiGLU is part of the modern stack and of the Day 4 model.

What's known, and what's not

The GLU-variants paper tests several gates and reports that the gated forms beat the ungated ones. On why, its conclusion famously offers no explanation and credits the success to "divine benevolence". A common reading is that the multiplicative interaction adds expressiveness per parameter, a unit can compute something like "feature A, but only when feature B", which a single threshold needs more units to approximate. But that's an intuition, not something this project measured.

What we didn't build, and why

• Other GLU variants. GEGLU and ReGLU use GELU or ReLU as the gate. The paper finds them close to SwiGLU, and the models Day 6 has to load use SwiGLU.
• A width sweep. Only the parameter-matched width was run.