Mixture of Experts Explained: Total vs Active Parameters

Mixture of Experts Explained

A mixture-of-experts model contains multiple expert networks and a router that selects a subset for each token or representation.

The design can provide a large total parameter capacity without activating every parameter for every token. It does not mean the inactive parameters disappear from memory, networking, training, or operations.

The router chooses a path

In a dense transformer layer, the same feed-forward network processes each token.

In a mixture-of-experts layer, a router scores available experts and sends the token to a selected subset. The selected experts process the token, and their outputs are combined.

Different tokens can take different routes. The model learns expert behavior and routing during training rather than relying on a human to label every token's specialty.

flowchart LR
    A["Token representation"] --> B["Router"]
    B --> C["Expert 1"]
    B --> D["Expert 2"]
    B -. "Not selected" .-> E["Expert 3"]
    C --> F["Combined output"]
    D --> F

This sparse path is the source of the arithmetic advantage. The whole system still needs to store and coordinate the experts.

Total and active parameters answer different questions

DeepSeek's V3 technical report describes 671 billion total parameters and 37 billion parameters activated for each token.

Total parameters describe the full learned parameter set. Active parameters describe the subset participating in computation for a token under the routing design.

"Only 37 billion parameters" is therefore wrong for the full V3 model. The model still has 671 billion total parameters.

No single count provides the hardware bill.

Sparse activation changes arithmetic

If a model activated every one of its total parameters for every token, its arithmetic would resemble a dense model at that full size.

By selecting fewer experts, a mixture-of-experts model can apply less compute per token while retaining a larger overall parameter pool. That can improve the relationship among capacity, quality, and computation under the authors' design.

The advantage is not free. The router adds work. Tokens must reach their experts. Expert outputs must return. The system must place experts across memory and devices.

The DeepSeek-V3 repository describes hardware-aware choices for precision, load balancing, parallelism, communication, and multi-token prediction. Architecture and implementation need to be read together.

Routing creates a load problem

If too many tokens choose the same expert, that expert becomes hot while other capacity sits idle. A system may need capacity limits, balancing methods, token dropping or rerouting, redundant placement, or scheduling.

Balancing pressure can also affect model quality if the system routes tokens for even usage rather than the best learned path.

DeepSeek reports an auxiliary-loss-free balancing strategy for V3. That is an author-reported design choice, not a general solution for every mixture-of-experts system.

Evaluation should include routing distribution, expert utilization, overflow, imbalance, failure, and quality under representative workloads.

Memory still holds the wider model

Active compute and memory footprint are different.

The weights for inactive experts still need to exist somewhere in the serving system. Training also needs activations, gradients, optimizer states, checkpoints, and other memory beyond a simple parameter count.

Quantization can reduce weight memory. Offloading can move data between memory tiers. Parallelism can spread the model across devices. Each choice changes latency, throughput, network traffic, complexity, quality, and reliability.

Do not infer that an active-parameter count tells you how many devices are required.

Communication can become the bottleneck

Experts may be distributed across accelerators or nodes. Tokens routed to remote experts must travel through the interconnect.

The relevant system therefore includes bandwidth, latency, topology, collective communication, congestion, placement, batch shape, sequence length, and overlap between computation and communication.

A design that reduces arithmetic but increases waiting can fail to deliver the expected speed.

Hardware and software co-design matters. Kernels, compilers, runtime, scheduler, networking, precision, batching, and workload determine delivered performance.

Training and serving have different pressures

Training optimizes parameters across large datasets and needs forward computation, backward computation, optimizer state, synchronization, checkpoints, and recovery.

Serving produces outputs under latency, throughput, concurrency, memory, availability, and cost constraints. Long reasoning traces can increase output tokens even when each token uses sparse activation.

A model can be efficient to train under one cluster design and difficult to serve under another workload. An architecture claim should name the stage.

Mixture of experts did not cause every DeepSeek result

DeepSeek V3 also reports Multi-head Latent Attention, FP8 mixed-precision training, load-balancing methods, multi-token prediction, optimized communication, data choices, objectives, and training methods.

The DeepSeek R1 paper adds post-training choices around reinforcement learning, cold-start data, supervised fine-tuning, and distillation.

The mixture-of-experts architecture is one part of the system. It should not receive credit for every quality, cost, or efficiency claim.

Evaluate the delivered workload

For a real system, record the exact model, runtime, hardware, precision, batch, sequence length, token distribution, quantization, expert placement, network, concurrency, latency distribution, throughput, memory, energy, failures, cost, and output quality.

Compare against a dense or alternative system at the same task and service requirement, not merely at the same total parameter count.

The practical lesson is not that mixture of experts makes compute cheap. It is that sparse activation moves the engineering question from one parameter number to routing and the full delivered system.

This page was developed with AI assistance from the E055 transcript and linked primary sources, then structured for human machine-learning, systems, infrastructure, and editorial review. It is a technical explainer, not a hardware sizing, cost, or deployment recommendation.