Mixture of Depths Explained for AI Product Managers

The Core Problem: Not Every Token Needs Every Layer

Standard transformers process every token through every layer. A 32-layer model applies all 32 layers to the word "the" in your prompt with the same compute cost as it applies to "transformer architecture." That is wasteful. Common function words, punctuation, and simple tokens carry less information and arguably need less processing.

Mixture of Depths, introduced in the Google DeepMind paper published in April 2024, addresses this directly. The core idea: at each transformer layer, a learned routing mechanism decides which tokens continue through that layer and which ones skip it entirely. Tokens that are routed out of a layer pass through it unchanged, carrying their representation forward without any compute being spent on them at that step.

Standard transformer

Every token is processed by every layer. A 32-layer, 7B-parameter model applies 32 full attention plus feed-forward operations to every single token in your input and output.

Mixture of Depths transformer

Each layer has a capacity budget. Only the top-k tokens (ranked by a learned routing score) are processed by that layer. The rest skip it and rejoin downstream. Total compute drops significantly with minimal quality loss.

The routing signal

A lightweight learned router (essentially a linear projection) assigns each token a scalar score at each layer. Tokens scoring above the layer's threshold are processed; those below it skip. The threshold is set by the target capacity, not a fixed rule.

The key insight is that depth (number of layers processed) can be dynamic and input-dependent rather than fixed. A model can apply its full representational power where the input demands it and conserve compute everywhere else.

MoD vs Mixture of Experts: What's Actually Different

If you've heard of Mixture of Experts (MoE), you might wonder whether MoD is the same thing with a different name. They're related but operate on different axes.

Mixture of Experts (MoE)

Routes tokens across different expert feed-forward networks at a single layer. Width routing: the model has many experts side by side, and each token is sent to a few of them. GPT-4 and Mixtral use this. Reduces active parameters per token without reducing layer count.

Mixture of Depths (MoD)

Routes tokens to skip or include entire layers. Depth routing: the model has a fixed set of layers stacked vertically, and each token decides how many of those layers to pass through. Reduces total operations per token by skipping whole layers.

MoE + MoD combined

The DeepMind paper proposes combining both. An MoE model that also uses MoD gets width sparsity (fewer active parameters per layer) AND depth sparsity (fewer layers per token). The theoretical compute savings multiply.

Why this matters for model selection

When a model provider says '50% faster at equivalent quality,' read the fine print. MoE savings come from reduced parameter activation. MoD savings come from reduced layer traversal. Both reduce compute, but they have different quality profiles on different tasks.

How the Routing Mechanism Works

The routing decision at each layer is learned end-to-end during training, not hand-designed. Here's the mechanics without the math:

Step 1: Token scoring

What happens: Before each transformer layer, a tiny linear function (a scalar projection) scores each token in the sequence. Higher score = more informative, harder to process, more in need of this layer's computation.

PM note: The model learns this scoring from the data. It isn't told which tokens matter more. It discovers that certain positions and contexts demand more processing.

Step 2: Top-k selection

What happens: Only the top-k tokens by score pass through the full layer (attention plus feed-forward). The remaining tokens skip the layer entirely, carrying their current representation forward unchanged. k is set by a capacity hyperparameter chosen at training time.

PM note: Capacity is a dial you (or the model provider) sets. Higher capacity means fewer skips, more compute, higher quality. Lower capacity means more skips, less compute, slightly lower quality. It's a tunable tradeoff.

Step 3: Residual pass-through

What happens: Skipped tokens don't lose their accumulated representation. They pass through the layer via a residual connection, so their state from all prior layers is preserved. They simply don't get updated at this layer.

PM note: This is what makes MoD practical rather than dangerous. Skipping a layer doesn't delete information. The token just doesn't get processed at that depth.

Step 4: Reassembly

What happens: After the layer, all tokens (processed and skipped) are merged back into the sequence in their original positions. The next layer receives the full sequence again and applies its own routing decision independently.

PM note: Each layer makes its own routing choice independently. A token that is skipped at layer 5 might be selected at layer 6. Routing patterns vary layer by layer.

Go Deeper in the AI PM Masterclass

The masterclass covers how architecture decisions like MoD translate directly into cost, latency, and product strategy. Taught live by a Salesforce Sr. Director PM.

The Performance Numbers: What the Research Actually Shows

The original DeepMind paper trained isoFLOP baselines (MoD models vs standard transformers at matched training compute) and compared quality. The headline results:

12.5% capacity MoD matches dense baseline quality

A model that routes only 12.5% of tokens through each layer matches a dense transformer trained on the same compute budget. The simple tokens are carrying far less signal than their compute cost implied.

50% compute reduction at matched quality is achievable

By choosing the right capacity setting, MoD models can run at roughly half the inference compute of a dense equivalent while maintaining benchmark scores. This directly translates to halved inference cost or halved latency at equivalent output quality.

Quality degrades gracefully under extreme sparsity

Push capacity too low (routing almost all tokens to skip most layers) and quality does fall. The degradation is smooth rather than cliff-like, which means capacity is a tunable dial rather than a binary good/bad threshold.

Tasks with complex reasoning benefit less

On tasks requiring multi-step reasoning, careful reading of long contexts, or difficult factual recall, the quality gap between MoD and dense widens slightly. The routing is learned and generally accurate but not perfect.

Research citation

Raposo, D. et al. (2024). Mixture of Depths: Dynamically allocating compute in transformer-based language models. Google DeepMind. The paper trains models up to 6B parameters and demonstrates quality-compute tradeoffs across standard language modeling benchmarks.

What MoD Means for AI Product Decisions

MoD is an architecture technique baked into a model at training time. You can't apply it to an existing model you're using via API. But understanding it changes how you evaluate models and plan costs.

Model benchmarks tell only part of the story

A model advertised as 'X% faster at equivalent quality' may use MoD. That claim is true on average across all input types. On your specific task, especially if it involves complex reasoning or long-context recall, the quality difference may be larger than the benchmark average suggests.

Cost modeling for MoD-based models

MoD models are more compute-efficient on simple inputs than complex ones. If your product handles a mix of simple and complex queries, you may see higher-than-expected per-token costs on complex prompts relative to the model's advertised average.

Latency profile is non-uniform

Because routing is conditional on token content, MoD models may show more latency variance than dense models. Simple inputs complete faster; complex inputs take longer. Design your UX for the P95 latency, not the average.

Fine-tuning preserves the routing

When you fine-tune an MoD model on your domain, the routing mechanism is fine-tuned too. The model learns which domain-specific tokens deserve more compute. This generally improves routing accuracy for your use case.

On-device deployment becomes more viable

MoD reduces inference memory pressure and FLOP count. For products targeting edge deployment (mobile, embedded), MoD models are a significant enabler. A 7B MoD model may run faster than a dense 3B at comparable task quality.

Evaluating 'fast' model variants

When model providers launch 'mini', 'flash', or 'lite' variants, MoD is one of the techniques likely behind the speed gains alongside quantization and distillation. Ask specifically how the efficiency was achieved before relying on benchmark quality claims.

MoD in the Broader Architecture Landscape

MoD is part of a broader movement toward conditional compute in transformers. The pattern: apply more compute where the model needs it, less where it doesn't. Several parallel developments point in the same direction.

Early exit networks

Models that can produce an output after fewer layers if confidence is high enough. Close cousin to MoD, but the decision is made for the whole sequence rather than individual tokens.

Adaptive computation

The broader class of methods that dynamically allocate compute based on input complexity. MoD, MoE, and early exit are all variants. The unifying principle: static compute allocation is inefficient.

Speculative decoding

A related technique where a small draft model generates candidate tokens and a large verification model checks them. Speeds up inference through a different form of conditional compute allocation.

Test-time compute scaling

Reasoning models (o1, o3) that spend more compute at inference time for harder problems. Conceptually similar: harder tasks deserve more compute. MoD operates at the token level; test-time scaling operates at the task level.

The practical takeaway for AI PMs: efficiency claims from model providers are increasingly based on these conditional compute techniques rather than brute-force distillation or quantization. Understanding which technique drives a given efficiency gain helps you predict where quality will hold and where it will erode on your specific use case.