(24) Mixture-of-Experts Models Fit the Shape of the Enterprise

A large model that does not use all of itself

OpenAI's gpt-oss-120b contains 117 billion parameters, yet activates 5.1 billion for each token. Its smaller sibling has 21 billion total parameters and activates 3.6 billion. OpenAI says the quantised versions fit in 80 GB and 16 GB of memory respectively. IBM's Granite 4.0 H-Tiny takes the same idea further down the hardware stack: seven billion total parameters, one billion active, and an explicit design target of local applications. These numbers describe the attraction of mixture-of-experts models more clearly than another benchmark table.

A dense model applies the same feed-forward parameters to every token. A sparse MoE replaces some of those blocks with multiple feed-forward networks and a router that selects a small subset for each token. The model can therefore carry more parameters than it exercises on each forward pass. Shazeer and colleagues demonstrated the modern sparsely gated layer in 2017; GShard, Switch Transformer, GLaM and ST-MoE subsequently showed that conditional computation could scale across languages and tasks without scaling arithmetic at the same rate as total capacity.

That mechanism fits a specific corporate problem. Large organisations do not have one language or one task distribution. They contain legal clauses, risk policies, maintenance records, source code, product taxonomies, customer conversations and regional vocabularies. My thesis is conditional and testable: for a company with several recurring domain workloads, enough memory bandwidth and disciplined evaluation, a locally hosted open-weight MoE can provide greater usable task breadth per unit of inference compute than a dense model with comparable breadth. It loses that advantage when memory is scarce, demand is sporadic or the organisation cannot operate the surrounding system.

Corporate variety is the relevant form of scale

The case for MoE inside a company starts with heterogeneity rather than raw size. A bank may need one system to extract covenants, explain credit policy, draft code, classify complaints and answer questions in several languages. A manufacturer adds engineering standards, safety procedures, parts catalogues and service logs. Training or hosting a separate full model for every domain fragments controls and duplicates shared linguistic capacity. Sending every token through a single large dense network pays the full computational cost even when a request needs a narrow part of what the model has learned.

Sparse routing offers a middle position. Shared attention and other common parameters process every request, while selected expert blocks supply additional capacity. DeepSeekMoE made this division explicit by separating always-active shared experts from finer routed experts. DeepSeek-V2 and V3 extended that architecture at much larger scale. Qwen3 combines dense and MoE variants across a multilingual family. OLMoE provides an unusually open research record, including weights, data, code and logs. The common result is not that every MoE wins every comparison. It is that total capacity and active computation can be separated enough to make a broader model affordable to run repeatedly.

The benefit should be assessed on the organisation's own distribution. A useful test set would mix the work that shares syntax but differs in substance: an Italian policy question, a Python repository query, a credit memorandum and a maintenance instruction. The comparison should hold latency, hardware, context length and answer-quality thresholds constant. If the MoE does not cover more of that workload at the same active compute, the thesis fails for that company. Vendor benchmarks cannot settle the question because corporate value sits in the joint distribution of tasks, not the average public benchmark.

Experts are not departments

The word expert invites a tidy but misleading organisational chart. It is tempting to imagine one internal network learning law, another finance and a third German. Learned routing does not usually produce such legible boundaries. Tokens are routed at each MoE layer from their hidden representations; the selected path can change from token to token within the same sentence. Load-balancing objectives also discourage the router from sending everything to a small set of popular experts.

The evidence on specialisation is mixed for good reason. The original sparsely gated work observed syntactic and semantic patterns. OLMoE reported substantial specialisation, while recent multilingual analysis found routing aligned partly with linguistic families and differed by depth. DeepSeek's fine-grained and shared-expert design seeks less redundant routed capacity. Yet a 2026 study by Xi Wang, Soufiane Hayou and Eric Nalisnick found that routing similarity across five pretrained models could be explained by hidden-state geometry and that human-readable domain labels remained unreliable. Other work finds that load-balancing losses can promote overlap, and that a small group of experts may receive much of the traffic.

Corporate customisation should therefore avoid assigning policy meaning to expert identity. A routing trace is useful telemetry, not proof that a request stayed inside a notional legal or risk specialist. An enterprise cannot satisfy access control by naming expert 17 "Legal", nor explain a credit decision by reporting which feed-forward block fired. The non-obvious advantage of MoE is capacity sharing without organisational mirroring. The model can absorb varied patterns while the application retains the human-readable boundaries: permissions, document collections, tools, policies and evaluation suites.

Customisation belongs around and across the model

A company has four distinct things to customise: knowledge, behaviour, workflow and model capacity. Treating all four as a fine-tuning problem makes the system harder to update and audit.

Retrieval should carry knowledge that changes or has an owner. The original retrieval-augmented generation paper separated parametric memory from an external index, improving provenance and updateability. In a corporate deployment, that index can also enforce document-level permissions and expose the source used for an answer. A regulation, price list or procedure should be changed in the controlled repository rather than baked repeatedly into model weights. Retrieval quality still needs measurement; local hosting does not repair poor chunking, stale documents or an irrelevant search result.

Small adapters are better suited to stable behaviour such as output formats, terminology, classification boundaries or tool-calling conventions. LoRA freezes base weights and learns low-rank updates; QLoRA reduces the memory needed during adaptation by training adapters through a quantised base. Newer MoE-specific research complicates the picture in a productive way. Routed adaptation modules, MiLoRA and MoE-Sieve report that the routing pattern can guide where limited adaptation capacity is spent. The 2026 MoE-Sieve experiments found heavily skewed expert use and competitive results after adapting only a subset of frequently routed experts, although that result covers selected models and tasks rather than a general rule.

Full continued training has a higher bar. Branch-Train-MiX shows one route for training domains separately before combining them and learning token routing; sparse upcycling starts from a dense checkpoint. Mistral's 2026 Forge announcement applies this logic commercially to proprietary documentation, code and operational records. These approaches are credible when a company possesses a large, clean and durable corpus plus strong evaluation. For most deployments, the safer order is retrieval first, adapters second and broader post-training only after measured failure. The MoE supplies shared capacity; the corporate system supplies explicit specialisation.

Local control is an operating model

Running weights on company-controlled infrastructure changes the boundary around prompts, retrieved documents, outputs and logs. It can support data-residency requirements, reduce transmission to an external inference provider and allow the company to set retention, patching and access policies. Open-weight releases from Mistral, Qwen, IBM and OpenAI now work with common local stacks, including vLLM, llama.cpp, Ollama and TensorRT-LLM. Local deployment is no longer restricted to a research cluster, although production support remains a cost rather than a free feature.

Control of location does not establish control of use. The EU AI Act assigns obligations according to roles and risk, not the distance between a model and the server room. The European Data Protection Board's opinion on AI models addresses lawful use of personal data and the conditions under which a model might be considered anonymous. The UK Information Commissioner's Office requires purpose-specific data minimisation and security assessment. NIST's AI Risk Management Framework and its generative-AI profile call for governance, measurement and management across the lifecycle. NCSC and ENISA guidance similarly covers supply chains, deployment, monitoring, data poisoning, exfiltration and incident response.

A local MoE may reduce one exposure while adding others. The company now owns model provenance, runtime vulnerabilities, adapter lineage, access to weights, monitoring and rollback. IBM's signing of Granite checkpoints is a useful example of a control that becomes relevant when organisations download their own artefacts. OpenAI's adversarial fine-tuning work on gpt-oss illustrates the other side: accessible weights can be modified in ways a hosted provider cannot reverse. Locality creates a larger control surface, but it also makes that surface observable and configurable by the company.

The memory bill is the strongest counterargument

Sparse activation reduces arithmetic, not the total number of weights that must be stored or made available. Mixtral 8x7B activates roughly 12.9 billion of its 46.7 billion parameters per token, but the complete model still needs memory. The same distinction applies to newer models. Quantisation compresses the weights, while expert offloading can move less-used parameters between system memory and an accelerator. Neither technique removes the traffic.

This matters because autoregressive generation often waits on memory movement. Offloading research such as HOBBIT and Fast Inference of Mixture-of-Experts Language Models with Offloading reports large gains against earlier offloading systems, but those gains arise because the baseline problem is severe. Work on Apple Silicon NPUs identifies dynamic routing, irregular operators and many small kernel launches as additional costs. Across multiple accelerators, expert parallelism introduces communication and load-balancing work. Current vLLM documentation includes an expert load balancer precisely because real request distributions can route tokens unevenly.

A dense small model can therefore be the better local choice for a narrow, predictable task. A hosted API can be cheaper for low utilisation, occasional frontier reasoning or a team without inference engineers. OpenAI's own open-model guidance says self-hosting costs depend on infrastructure and operating approach. MoE economics improve when diverse demand is steady enough to amortise memory, deployment and evaluation. They deteriorate when a large checkpoint sits idle or when expert traffic crosses slow links.

The operational comparison must use completed work rather than tokens per second alone. Measure accepted outputs, tail latency, concurrency, retrieval failures, escalation rates, energy and staff time on the same hardware and corpus. Include a dense local baseline and a hosted baseline. MoE is a systems choice whose routing, quantisation, batching and memory topology determine whether its theoretical sparsity becomes useful capacity.

Build a governed model portfolio

The practical design is a shared local MoE beneath explicit corporate controls. Retrieval collections preserve ownership and freshness. Small adapters encode stable task behaviour. Application routing selects tools, policies and escalation paths. Evaluations are segmented by business domain, language, jurisdiction and risk. Sensitive, repetitive work stays on controlled infrastructure; unusual tasks can be sent to a stronger service when policy permits.

This architecture avoids forcing one model to become the visible organisation chart. It also avoids maintaining a separate foundation model for every team. The same base weights can support several bounded applications while access rules and evidence remain outside opaque expert routing. Model updates become portfolio decisions: replace the base after regression testing, revise a document immediately, or swap an adapter for one workflow.

The next procurement question should therefore be concrete. Given the company's real workload mix and hardware, does a candidate MoE meet more domain-specific quality thresholds at an acceptable cost and latency than the best dense local model and the permitted hosted alternative? Organisations that can answer that question with replayable evaluations will be able to customise local AI without confusing ownership of the infrastructure with evidence of performance.

Sources

Architecture and routing

1. Noam Shazeer et al., "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" https://arxiv.org/abs/1701.06538

2. Dmitry Lepikhin et al., "GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding" https://arxiv.org/abs/2006.16668

3. William Fedus, Barret Zoph and Noam Shazeer, "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity" https://arxiv.org/abs/2101.03961

4. Nan Du et al., "GLaM: Efficient Scaling of Language Models with Mixture-of-Experts" https://arxiv.org/abs/2112.06905

5. Barret Zoph et al., "ST-MoE: Designing Stable and Transferable Sparse Expert Models" https://arxiv.org/abs/2202.08906

6. Yanqi Zhou et al., "Mixture-of-Experts with Expert Choice Routing" https://arxiv.org/abs/2202.09368

7. Aran Komatsuzaki et al., "Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints" https://arxiv.org/abs/2212.05055

8. Damai Dai et al., "DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models" https://arxiv.org/abs/2401.06066

9. Niklas Muennighoff et al., "OLMoE: Open Mixture-of-Experts Language Models" https://arxiv.org/abs/2409.02060

10. Xi Wang, Soufiane Hayou and Eric Nalisnick, "The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise" https://arxiv.org/abs/2604.09780

11. Yuxin Chen et al., "Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steering" https://arxiv.org/abs/2601.14050

12. Hongcan Guo et al., "Advancing Expert Specialization for Better MoE" https://arxiv.org/abs/2505.22323

Open-weight models and enterprise releases

13. Mistral AI, "Mixtral of Experts" https://mistral.ai/news/mixtral-of-experts/

14. Mistral AI, "Cheaper, Better, Faster, Stronger" https://mistral.ai/news/mixtral-8x22b/

15. Qwen Team, "Qwen3 Technical Report" https://arxiv.org/abs/2505.09388

16. Qwen Team, "Qwen3" https://github.com/QwenLM/Qwen3

17. DeepSeek-AI, "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model" https://arxiv.org/abs/2405.04434

18. DeepSeek-AI, "DeepSeek-V3 Technical Report" https://arxiv.org/abs/2412.19437

19. IBM, "Granite 4.0" https://www.ibm.com/granite/docs/models/granite

20. Kate Soule and Dave Bergmann, "IBM Granite 4.0: Hyper-efficient, High Performance Hybrid Models for Enterprise" https://www.ibm.com/new/announcements/ibm-granite-4-0-hyper-efficient-high-performance-hybrid-models

21. OpenAI, "Introducing gpt-oss" https://openai.com/index/introducing-gpt-oss/

22. OpenAI, "gpt-oss-120b & gpt-oss-20b Model Card" https://openai.com/index/gpt-oss-model-card/

23. Databricks, "Introducing DBRX: A New State-of-the-Art Open LLM" https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm

24. Mistral AI, "Introducing Forge" https://mistral.ai/news/forge/

Customisation and retrieval

25. Patrick Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" https://arxiv.org/abs/2005.11401

26. Edward J. Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models" https://arxiv.org/abs/2106.09685

27. Tim Dettmers et al., "QLoRA: Efficient Finetuning of Quantized LLMs" https://arxiv.org/abs/2305.14314

28. Sainbayar Sukhbaatar et al., "Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM" https://arxiv.org/abs/2403.07816

29. Yilun Liu et al., "Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules" https://arxiv.org/abs/2508.02587

30. Andrea Manzoni, "MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning" https://arxiv.org/abs/2603.24044

31. Jingfan Zhang et al., "MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning" https://arxiv.org/abs/2410.18035

32. Ali Janati et al., "Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models" https://arxiv.org/abs/2608.07890

33. Ted Zadouri et al., "Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning" https://proceedings.iclr.cc/paper_files/paper/2024/file/6d00071564ec447466fc4577743cf1b3-Paper-Conference.pdf

Local inference systems

34. Hugging Face, "Mixture of Experts Explained" https://huggingface.co/blog/moe

35. Hugging Face, "Mixture of Experts (MoEs) in Transformers" https://huggingface.co/blog/moe-transformers

36. Artyom Eliseev and Denis Mazur, "Fast Inference of Mixture-of-Experts Language Models with Offloading" https://arxiv.org/abs/2312.17238

37. Peng Tang et al., "HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference" https://arxiv.org/abs/2411.01433

38. Afsara Benazir and Felix Xiaozhu Lin, "Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUs" https://arxiv.org/abs/2604.18788

39. Mu-Chi Chen et al., "Towards Building Private LLMs: Exploring Multi-Node Expert Parallelism on Apple Silicon for Mixture-of-Experts Large Language Model" https://arxiv.org/abs/2506.23635

40. vLLM, "Expert Parallel Deployment" https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment/

41. NVIDIA, "TensorRT-LLM Backend" https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/tensorrtllm_backend/README.html

42. NVIDIA, "MoE (Mixture of Experts)" https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/transformers-moe.html

43. NVIDIA, "Quantization Schemes" https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/quantized-types-schemes.html

Governance, privacy and security

44. OpenAI, "OpenAI Open-Weight Models (gpt-oss)" https://help.openai.com/en/articles/11870455-openai-open-weight-models-gpt-oss

45. National Institute of Standards and Technology, "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile" https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence

46. National Institute of Standards and Technology, "AI Risk Management Framework" https://www.nist.gov/itl/ai-risk-management-framework

47. European Union, "Regulation (EU) 2024/1689 Laying Down Harmonised Rules on Artificial Intelligence" https://eur-lex.europa.eu/eli/reg/2024/1689/oj?locale=en

48. European Data Protection Board, "EDPB Opinion on AI Models: GDPR Principles Support Responsible AI" https://www.edpb.europa.eu/news/edpb-opinion-on-ai-models-gdpr-principles-support-responsible-ai_ga

49. Information Commissioner's Office, "How Should We Assess Security and Data Minimisation in AI?" https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/guidance-on-ai-and-data-protection/how-should-we-assess-security-and-data-minimisation-in-ai/

50. UK National Cyber Security Centre, "Guidelines for Secure AI System Development" https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development/guidelines

51. European Union Agency for Cybersecurity, "Artificial Intelligence Cybersecurity Challenges" https://www.enisa.europa.eu/publications/artificial-intelligence-cybersecurity-challenges

52. European Union Agency for Cybersecurity, "Securing Machine Learning Algorithms" https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms