Research Note
Mixture of Experts Architecture Record
A mixture-of-experts transformer contains multiple expert sub-networks and a router that selects a subset for each token or representation. Sparse activation can provide
Mixture of Experts Architecture Record
Definition
A mixture-of-experts transformer contains multiple expert sub-networks and a router that selects a subset for each token or representation. Sparse activation can provide more total model capacity without activating every parameter for every token.
DeepSeek-V3 is the episode's concrete example:
https://github.com/deepseek-ai/DeepSeek-V3
https://arxiv.org/abs/2412.19437
The authors describe 671 billion total parameters and 37 billion activated per token. These figures belong to the specified architecture and do not mean that the full model stores only 37 billion parameters.
Parameter terms
Total parameters describe the full learned parameter set. Active parameters describe the subset participating in computation for a token under the routing design.
Neither number alone states memory footprint, bandwidth, latency, throughput, energy, quality, or cost.
Weights for inactive experts may still need to reside in memory across the serving system. Routing can distribute tokens unevenly. Expert placement can require communication across devices and nodes.
System map
| Layer | Question |
|---|---|
| Router | How are experts selected, and what happens when routing is wrong or unstable? |
| Capacity | How many experts are available and active? |
| Balance | How are hot experts and underused experts handled? |
| Memory | Where do all weights, optimizer states, caches, and activations live? |
| Communication | How do tokens move across devices or nodes to their experts? |
| Parallelism | How are data, tensor, pipeline, and expert parallelism combined? |
| Training | What losses, precision, scheduling, and failure controls are used? |
| Serving | What batch, sequence, quantization, caching, and routing conditions apply? |
| Operations | What monitoring, failure, scaling, and capacity controls are required? |
Tradeoff boundary
Sparse activation can reduce arithmetic relative to a dense model with the same total parameter count. It does not make routing, memory, network, synchronization, load balancing, serving, or operations free.
Delivered performance depends on hardware and software co-design, precision, batch, sequence length, network topology, expert placement, kernels, compiler, runtime, and workload.
Architecture alone cannot explain every reported efficiency or quality result. Data, objectives, training process, implementation, hardware, and evaluation also matter.
Publication language
Avoid "only 37 billion parameters." Say that the cited architecture has 671 billion total parameters and activates 37 billion per token under the authors' design.
Avoid presenting active-parameter count as direct proof of a dollar cost or hardware requirement.
Sources
Follow the evidence.
- github.com: LICENSEgithub.com
- bis.gov: commerce strengthens restrictions advanced computing semiconductors enhance foundry due diligence preventbis.gov
- arxiv.org: 2501arxiv.org
- NIST AI Risk Management Frameworknist.gov
- daltonanderson.ghost.io: deepseek vs nvidia the future of ai chip economicsdaltonanderson.ghost.io
- investor.nvidia.com: defaultinvestor.nvidia.com
- api-docs.deepseek.comapi-docs.deepseek.com
- bis.gov: 740bis.gov
- daltonanderson.net: deepseek vs nvidia the future of ai chip economicsdaltonanderson.net
- github.com: DeepSeek R1github.com
- open.spotify.com: 6jLI1bNwyoxI449vXJXzBVopen.spotify.com
- youtu.be: Qp24TkfT9XEyoutu.be
- bis.gov: 742bis.gov
- bis.gov: department commerce revises license review policy semiconductors exported chinabis.gov
- arxiv.org: 2412arxiv.org
- docs.nvidia.com: cudadocs.nvidia.com
- github.com: DeepSeek V3github.com