Multi-Head Latent Attention (MLA) Explained: What DeepSeek's Architecture Means for AI Inference Costs
TL;DR
Multi-Head Latent Attention (MLA) is an attention variant introduced in DeepSeek-V2 that reduces the KV cache memory requirement by up to 93% compared to standard multi-head attention, without meaningful loss in model quality. It works by storing a compressed latent vector instead of full key and value tensors, then reconstructing the needed K and V data at inference time. The practical consequences for AI PMs: cheaper long-context inference, wider deployment options for memory-constrained environments, and a signal about why frontier model inference costs have been falling faster than scaling would predict. MLA is now a cornerstone of the DeepSeek-V3 architecture and is being referenced as an influence in broader frontier model design discussions in 2026.
The AI PM Minute
One tactic to make you a sharper AI PM, twice a week. 60 seconds to read. Free.
No fluff. Unsubscribe anytime.
The Problem: Why the KV Cache Is Expensive
To understand MLA, start with why the KV cache exists and what makes it costly. When a transformer generates output tokens, it needs to compute attention between the current token and every previous token in the context. Recomputing all prior token representations from scratch at each step would be prohibitively slow. The KV cache solves this by storing the Key and Value vectors for every token as they are processed. Future tokens can then look up the stored K and V tensors instead of recomputing them.
KV cache grows linearly with context length
A 128K token context with a large model can require 100-400 GB of GPU memory just for the KV cache, depending on the number of layers, the number of attention heads, and the head dimension. This is often more memory than the model weights themselves at long contexts.
KV cache must live in GPU HBM (high-bandwidth memory)
HBM is scarce and expensive. Every GPU allocated to serving a model must hold not just the model weights but the KV cache for all active concurrent requests. At high concurrency with long contexts, the KV cache is often the binding constraint on how many users a deployment can serve simultaneously.
Decode is memory-bandwidth-bound, not compute-bound
During autoregressive generation, the GPU's bottleneck is reading the KV cache from memory, not doing matrix math. Reducing the KV cache size reduces memory reads per decode step, which directly improves throughput (tokens per second per GPU) and lowers latency.
Long-context pricing reflects this cost
When providers charge more per token for long context windows, they are reflecting the KV cache cost. A 128K input context roughly quadruples attention compute and multiplies KV memory significantly. MLA architectures change this math.
Standard MHA and GQA: The Landscape Before MLA
Two prior approaches addressed KV cache memory before MLA arrived. Understanding them helps you see what MLA does differently.
Multi-Head Attention (MHA)
How it works: Standard attention used in GPT-2, early LLaMA models, and many others. Each attention head maintains its own full K and V tensors. A model with 96 heads stores 96 independent K tensors and 96 independent V tensors for every token in the context. High quality but very large KV cache.
Tradeoff: Best model quality per head. Highest KV cache memory cost. The baseline against which everything else is measured.
Grouped Query Attention (GQA)
How it works: Used in LLaMA 2/3, Mistral, and many 2023-2024 frontier models. Instead of one K and V per head, multiple query heads share a single K-V pair. Reduces the KV cache by the grouping factor (e.g., if 8 query heads share 1 K-V pair, KV cache is 8x smaller).
Tradeoff: Good quality with significantly smaller KV cache. Reduction comes from sharing K-V across heads, which can reduce expressiveness at very high group ratios. Widely adopted as the practical balance between MHA and extreme compression.
GQA was the dominant approach for KV cache reduction entering 2025. It works by reducing how many K-V pairs exist (sharing heads). MLA takes a different approach: it reduces the size of what gets stored per pair.
How MLA Works: Latent Compression
MLA was introduced by DeepSeek in the DeepSeek-V2 technical report (May 2024) and became a cornerstone of DeepSeek-V3. The key innovation is a low-rank joint compression mechanism: instead of caching the full K and V tensors, MLA caches a single compressed latent vector. At inference time, the needed K and V representations are reconstructed from this latent vector using a learned projection.
Training phase: learn the compression
During training, MLA learns two things simultaneously: the attention behavior of the model, and the mapping between a low-dimensional latent space and the full K and V tensors. The compression ratio is a hyperparameter set at training time. DeepSeek V2/V3 use a compression dimension significantly smaller than the full head dimension.
Inference (prefill): compress rather than cache
When processing input tokens, instead of storing a full K tensor and a full V tensor for each token at each layer, the model computes and stores a single compressed latent vector. This is the information that goes into the KV cache. The latent vector is much smaller: DeepSeek reports a reduction to roughly 5-7% of the standard MHA cache size, a compression ratio that enables up to 93% memory reduction.
Inference (decode): reconstruct at lookup time
When a new token needs to compute attention against prior tokens, MLA reconstructs the K and V tensors from the stored latent vectors using the learned projection matrices. This reconstruction happens on-chip during the attention computation and adds a small compute overhead (the projection multiply) in exchange for a dramatic reduction in memory bandwidth.
The quality preservation insight
DeepSeek's key finding is that the latent representation preserves the information needed for accurate attention. Unlike GQA (which reduces expressiveness by forcing heads to share K-V pairs), MLA maintains independent attention per head at the logical level. The compression is in the storage format, not in the attention behavior. This is why DeepSeek reports that MLA matches or exceeds MHA quality on benchmarks despite the dramatic cache reduction.
Translate Technical Depth Into Product Decisions
The AI PM Masterclass teaches you to read architectural changes like MLA and reason about what they mean for your product: cost structure, deployment options, and competitive dynamics. Taught live by a Salesforce Sr. Director PM.
Product Implications: Cost, Context, and Deployment
Why should a product manager care about the internal attention architecture of models they did not build? Because attention architecture drives the cost structure of inference, and cost structure determines what products are economically viable to build.
Long-context inference becomes cheaper
With a 93% KV cache reduction, a model using MLA can serve longer contexts on the same GPU memory. This means providers can offer 128K or 1M token contexts at lower cost than an equivalent MHA model. If you are building products where context length determines feature capability (document analysis, long conversation history, code review), MLA-based models may offer better cost-to-context-length ratios.
Higher concurrency on the same hardware
KV cache memory is one of the primary limits on how many simultaneous users a deployed model can serve. A smaller KV cache per request means more concurrent sessions fit on the same GPU fleet. For products with bursty or high-concurrency usage patterns, this directly translates to lower infrastructure cost at scale.
On-device and constrained-hardware deployment
The memory reduction from MLA makes it more feasible to run capable models in memory-constrained environments: on-device inference, edge servers, smaller cloud instances. If your product roadmap includes on-device AI features (mobile, embedded, IoT), the presence of MLA in a model's architecture is a meaningful signal for feasibility.
Inference cost is not only about parameter count
Before MLA, engineers evaluating model cost-per-token would look at parameter count and context length. MLA adds a third factor: attention architecture. A model with MLA may be cheaper to serve at 128K context than a smaller model using MHA at the same context length. When evaluating model APIs for your product, the attention architecture is now a relevant factor for cost modeling.
Why DeepSeek's pricing was disruptively low
DeepSeek-V3's launch pricing in early 2025 was 20 to 50 times cheaper than equivalent quality models from major providers. MLA is one of the architectural reasons. Reduced KV cache memory means more requests served per GPU, which directly lowers the marginal cost per token. Architecture-level efficiency improvements can create pricing advantages that are not easily matched by providers using older attention mechanisms.
Model provider selection for long-context products
When selecting a model API for a product with significant long-context requirements, ask providers which attention mechanism their deployed model uses. MLA or an equivalent low-rank KV compression approach is a positive signal for cost stability at high context lengths. Standard MHA at long context should command a cost premium; if pricing does not reflect this, investigate why.
MLA vs. GQA: The Key Differences
GQA and MLA are both KV cache reduction techniques, but they work differently and produce different tradeoffs. This comparison matters when you are evaluating models or when engineering teams are explaining architectural choices for a custom model deployment.
What gets compressed
GQA
The number of KV pairs. Multiple query heads share one K-V pair. Cache reduction comes from having fewer unique K-V pairs per layer.
MLA
The size of each stored unit. A compressed latent vector replaces the full K and V tensors per token. Cache reduction comes from smaller representations, not from sharing.
Effect on model expressiveness
GQA
Sharing K-V pairs across query heads reduces the independence of each head's attention pattern. At high compression ratios (many queries per K-V group), this can reduce the model's ability to attend to diverse patterns simultaneously.
MLA
Maintains independent attention per head at the logical level. The latent representation is shared as a storage format, but distinct K and V tensors are reconstructed per head at lookup time. Less expressiveness reduction for the same compression ratio.
Inference overhead
GQA
Minimal: no reconstruction step needed. At lookup time, the shared K-V pair is read directly. GQA simplicity is why it was adopted so widely before MLA.
MLA
Small added compute: a projection multiply to reconstruct K and V from the latent vector. This is fast (on-chip matrix multiply) and is offset in wall-clock time by the reduced memory bandwidth needed to read the smaller cache.
Compatibility with standard serving infrastructure
GQA
Directly compatible with standard attention kernels and KV cache management tools. No modifications needed.
MLA
Requires serving infrastructure that supports the latent-to-KV reconstruction step. Hugging Face Transformers and major inference frameworks have added MLA support following the DeepSeek releases, but older or proprietary serving stacks may require updates.
Which Models Use MLA and What Is Next
MLA debuted in DeepSeek-V2 and became the core attention mechanism in DeepSeek-V3 and DeepSeek-R1. As of mid-2026, it remains most prominently deployed in the DeepSeek model family, which is available via the DeepSeek API and through Azure AI, AWS Bedrock, and other inference providers.
What to watch for in frontier model announcements
KV cache compression ratios: When a new model announcement mentions a specific KV cache memory reduction, that is a signal the model uses MLA or an equivalent technique. A 5-10x reduction suggests a compression approach similar to MLA; smaller reductions suggest GQA or mixed methods.
Long-context pricing relative to short-context: Models with strong KV compression (MLA or equivalent) should show a smaller price premium for long contexts compared to standard MHA models. If a provider does not charge significantly more for 128K vs. 4K contexts, they likely have an efficient attention architecture.
Inference benchmark results: Tokens per second benchmarks at long context lengths reflect KV cache efficiency. An MLA model serving 128K context should show higher throughput per GPU than a comparable GQA or MHA model at the same context length.
Related techniques are emerging alongside MLA: cross-layer KV sharing (where adjacent layers share a single KV cache entry), MLA with decoupled rotary position embeddings (the variant DeepSeek uses to enable efficient key compression while preserving positional awareness), and MXFP4 quantization for KV cache data types. The direction is consistent: serving infrastructure that uses dramatically less memory per token at long contexts. For AI PMs, this translates into a multi-year tailwind toward cheaper, longer-context AI features.
Make Architecture Knowledge Work for Your Products
Understanding MLA is useful. Knowing how to use that understanding to make better model selection, cost modeling, and vendor evaluation decisions is what the AI PM Masterclass teaches. Four weeks, live instruction, hands-on projects.
Related Articles
Before you go: get the AI PM Minute
One tactic to make you a sharper AI PM, twice a week. 60 seconds to read. Free.
No fluff. Unsubscribe anytime.