How Mixture of Experts (MoE) Works: Router, Expert Collapse, and the VRAM Tax

Mixture of Experts (MoE) is a neural network architecture that routes each input token to a small subset of specialized sub-networks, called experts, rather than running it through the entire model. The result: a model can carry the knowledge of hundreds of billions of parameters while only paying the compute cost of a fraction of them per token.

DeepSeek V4 Pro makes that concrete. 1.6 trillion parameters but only 49 billion active per token. Usually, parameter count and compute cost move in lockstep. You want more knowledge, you pay more in FLOPS. MoE breaks that rule.

The first time I tried loading Mixtral 8x7B on my home server, I’d done the math on active compute and felt good about the setup. The model needs roughly 90GB VRAM to keep all eight expert blocks resident simultaneously. I had 48GB. It loaded in chunks, swapping expert blocks in and out of system RAM. Inference went from sub-100ms to multi-second latency. I’d planned for active parameters, not total parameters. That’s the most common mistake in MoE deployment.

Having shipped these architectures to production, I’ve seen exactly where the marketing slides fail. Most engineers trip up because they buy into the “frontier quality at a fraction of the cost” pitch. It’s misleading. You aren’t getting a free lunch, you’re just trading compute efficiency for a brutal VRAM tax.

Key takeaways

MoE routes each token to a small fraction of the model’s parameters, keeping inference cost low while total model capacity stays high. The router is a learned linear layer that specializes through training. You don’t assign topics to experts manually; the model figures that out through backpropagation. Loading an MoE requires VRAM for all experts, not just the active ones, and this is the most commonly underestimated deployment cost. Expert collapse is the primary training failure mode: without load-balancing loss, the router converges to a few dominant experts and wastes most of the model’s capacity. For API workloads, MoE is the reason frontier-quality inference now runs below $0.50 per million tokens.

The Mechanics of Sparse Activation: Router and Experts

MoE is just a sparse transformer. Dense models waste compute by forcing every token through every single parameter in the feed-forward network (FFN) layers. MoE fixes this by swapping that monolithic FFN for a group of smaller, specialized sub-networks called experts.

A Gating Network, or router, handles the routing. It acts as a traffic controller. When a token hits the router, it calculates weights to figure out which experts should handle the data. Most implementations use Top-K routing. It’s simple: the router picks the top-k experts with the highest scores and ignores the rest.

The router applies a single linear layer to the input token: G(x) = softmax(x · Wg), where Wg is the learned gate weight matrix. You pick the top-k values from G(x), zero out the rest, and renormalize. The final output is a weighted sum of those k experts’ results, which lets the model blend multiple specialized responses rather than committing to just one.

import torch
import torch.nn as nn
import torch.nn.functional as F

class TopKRouter(nn.Module):
    def __init__(self, d_model: int, n_experts: int, top_k: int = 2):
        super().__init__()
        self.top_k = top_k
        self.gate = nn.Linear(d_model, n_experts, bias=False)

    def forward(self, x: torch.Tensor):
        # x: (batch, seq_len, d_model)
        logits = self.gate(x)                          # (batch, seq_len, n_experts)
        scores = F.softmax(logits, dim=-1)
        topk_scores, topk_indices = scores.topk(self.top_k, dim=-1)
        topk_scores = topk_scores / topk_scores.sum(dim=-1, keepdim=True)
        return topk_scores, topk_indices

This is where the distinction between total and active parameters becomes critical. Total parameters cover everything, including the router. Active parameters are only the ones doing work for a specific token. Take DeepSeek V4 Pro: 1.6 trillion total parameters in the “knowledge base,” but only 49 billion active per token during inference. Gemma 4 27B runs the same principle at a scale local hardware can actually host: 26 billion total parameters with just 3.8 billion active per token.

On my home server running Ollama, a quantized Mixtral session that fit within my VRAM ceiling still showed unexpected latency spikes every few hundred tokens. The compute profile looked fine. The active parameter count was well within budget. The router was jumping between cold expert blocks, and each miss caused a stall. The benchmark that looked clean in isolation fell apart under realistic multi-turn sessions.

Don’t mistake this for manual assignment. You don’t tell Expert 1 to do Python and Expert 2 to do French. Specialization is emergent. Through backpropagation, the router figures out which experts are good at specific patterns, and the experts adapt accordingly.

Solving “Expert Collapse” and Load Balancing

If you’ve messed with custom MoE configs, you’ve probably hit expert collapse. It’s a “rich get richer” loop. Early in training, the router might find Expert A is slightly better than Expert B. Since the router updates via gradient descent, it starts dumping more tokens into Expert A. Expert A gets more data, gets better, and the router doubles down. It’s a feedback loop that kills diversity.

Eventually, the model ignores the vast majority of its experts. You’re left with a massive model that has the effective capacity of a tiny dense model. It’s a complete waste of VRAM and compute time.

I’ve seen this happen on internal experiment runs. A small MoE for on-device inference looked fine through the first thousand training steps. Then validation loss plateaued hard. Digging into the expert utilization logs, three of eight experts were receiving under 1% of tokens. The other five had collapsed into two preferred routes. The model’s effective capacity had dropped to something like a two-expert dense model. Fixing it meant adding auxiliary loss and restarting from an earlier checkpoint. The rerun cost two days.

To stop this, use Auxiliary Balancing Loss. Just add a penalty term to the loss function. If the router starts favoring one expert too heavily, the loss spikes. This forces the model to spread tokens across the whole pool. It basically tells the model that it can’t just take the easy path with Expert A.

Then there’s Router Z-loss. In large-scale MoE training, router logits can blow up, which leads to instability and crashes. Router Z-loss penalizes the absolute magnitude of those logits without messing with the softmax probabilities. It keeps the numbers in a range the hardware can actually handle so your gradients don’t explode.

MoE vs. Dense: Inference Speed and Compute Efficiency

When I first benchmarked Mixtral 8x7B against Llama 2 70B on my home server, the throughput numbers felt like a bug. The 8x7B was faster, sometimes by a factor of two, despite containing more total parameters. The hardware doesn’t care about total parameter count. It only processes the active ones.

Pricing as of May 2026: DeepSeek API Docs · TokenMix

Industry’s moving to MoE for one simple reason: efficiency. Dense models are expensive. If you jump from a 13B model to a 70B model, your compute cost per token spikes.

MoE breaks that link. Take Mixtral 8x7B. On paper, it looks like a monster, but it runs at the speed of a 13B dense model. Your hardware only handles matrix multiplications for the active experts. Everything else just sits in memory.

The first time I ran Mixtral 8x7B on hardware with a real VRAM ceiling, the throughput story checked out. Token generation was close to a 13B dense model. But I was running on a machine with 80GB available. On systems with less headroom, the performance curve inverts. The moment expert swap overhead kicks in, you’re paying latency costs that no routing efficiency can recover.

Pretraining gets the same boost. You can train an MoE faster than a dense model with the same total capacity. Because you only update a fraction of weights per token, you can pump up the model’s knowledge without blowing your FLOP budget.

It’s a trade-off between knowledge and spend. MoE lets us cram in massive amounts of world knowledge while keeping the cost per token low. Think of it as hiring a firm of 50 specialists instead of one generalist. You only pay for the two people actually working on your ticket.

The Hardware Tax: VRAM and Memory Requirements

Marketing claims end here. Engineering headaches start. MoE models might be computationally efficient, but they’re memory hogs.

Here’s the VRAM paradox. A dense 13B model only needs enough VRAM for 13B parameters. But an MoE model with 13B active parameters and 47B total parameters isn’t that simple. You can’t just load the active ones. The router has to send tokens to any expert instantly, so every single expert has to sit in VRAM.

I ran into this setting up an inference test bench for an MoE variant we were evaluating for on-device work. The active VRAM footprint passed all our pre-flight checks. The model crashed on first forward pass. The issue was that our memory estimator only accounted for active experts. All expert weights had to be mapped into device memory before the router could begin selection. The fix was trivial: update the estimator to account for total parameters. We’d already wasted most of an afternoon on it.

Expert Parallelism splits experts across different GPUs. Instead of duplicating the whole model on every card, GPU 1 takes experts 1 and 2, and GPU 2 takes 3 and 4. If the router on GPU 1 decides a token needs Expert 3, it ships that token across the NVLink or PCIe bus to GPU 2.

Now you’ve got communication costs. The “Capacity Factor” controls how many tokens a single expert can handle before the model drops them or reroutes them to avoid a bottleneck. Tighten it too much and quality drops. Loosen it, and you flood your interconnects with data, which kills your throughput.

If you’re hitting these walls, just use a high-VRAM GPU cloud provider or specialized AI infrastructure. It’s the only way to stop the latency spikes that happen when you try to swap experts in and out of memory.

Economic Analysis: API Pricing and Cost-to-Quality

MoE is the main reason token prices collapsed. We’ve crossed the sub-$1/M threshold. Frontier-level intelligence is officially a commodity.

Just look at the numbers. DeepSeek V4 runs at $0.30 per million input tokens. DeepSeek V4 Flash drops to $0.14. Dense frontier models like GPT-5.5 and Claude Opus 4.7 sit at $2.50 to $15.00 per million tokens. The provider’s compute cost depends on active parameters, not the total parameter count.

DeepSeek-V3 vs. GPT-4 proves the point. DeepSeek-V3 cost roughly $5.6 million in GPU rental costs for its final training run, while GPT-4 reportedly cost between $50 million and $100 million. That gap exists because DeepSeek bet on MoE from the start and only paid for 37 billion active parameters per token instead of activating everything.

Of course, there’s a quality trade-off. MoEs aren’t perfect. They usually sit 4 to 7 points behind the absolute frontier dense models on benchmarks. I built a documentation-search assistant that was running on a fine-tuned 7B dense model. When we swapped to Mixtral 8x7B, the benchmark delta was exactly the 4-5 point gap the literature predicts. The actual user-facing output quality was indistinguishable in review. The task had a low ceiling. Any model past a certain capability threshold produced the same answers. The cost dropped by half. Nobody noticed the switch. For 95% of production use cases, a 5-point benchmark drop is a rounding error if it means a 10x cost reduction.

Practical Implementation: Choosing MoE vs. Dense

Every MoE deployment I’ve been involved in started with the same question: what’s my actual bottleneck? If you’ve got plenty of memory but need to slash latency and cost, MoE is the obvious move.

Use MoE for large-scale RAG pipelines. In RAG, you’re usually shoving massive amounts of context into the prompt. A dense model with the capacity to handle that would be too slow and expensive to run at scale. MoE gives you the knowledge to handle complex queries without killing the user experience.

I moved a document Q&A pipeline from Llama 2 70B to Mixtral 8x7B for exactly this reason. The 70B model was choking on 8K context windows, and throughput dropped off badly as context length grew. With Mixtral, the active parameter count during inference stayed fixed regardless of context size, so throughput stayed predictable. End-to-end latency on a representative query dropped from 4.2 seconds to 1.8 seconds.

But don’t try to fine-tune these things lightly. Fine-tuning MoE models is a nightmare compared to dense ones. The router is temperamental. A small amount of specialized data can cause the router to collapse or over-specialize, which effectively kills the model’s general reasoning. If you need a model deeply specialized in a narrow domain through heavy fine-tuning, stick with a dense model. It’s just more stable.

For the current state of play: Gemma 4 27B is the strongest option for local deployment, with 26 billion total parameters and only 3.8 billion active per token, fitting on a single consumer GPU. DeepSeek V4 and V4 Pro are the go-to options for API-based workflows at frontier quality. Llama 4 Maverick and Qwen 3.6 35B-A3B round out the field for teams already in those ecosystems. Mixtral 8x7B was the model that proved MoE was viable at scale. Most teams have moved on, but it’s still referenced everywhere in the literature.

Don’t lock yourself into one architecture. Using an API aggregator like Together AI lets you pivot between these models without rewriting your entire infrastructure.

Frequently asked questions

Reed Hall

Staff engineer · ML accelerator hardware · AI tools practitioner

I'm a staff engineer at a large semiconductor company. I've been using AI tools in real engineering workflows since the early days, and I write about what actually works under production constraints.