Mixtral 8x7B
Mixtral 8x7B is a large language model developed by Mistral AI using a Mixture of Experts (MoE) architecture, consisting of eight expert sub-networks each containing 7 billion parameters. During inference, only a subset of experts is activated per token, achieving high model capacity while maintaining low computational overhead. The model supports multilingual text understanding and generation, and demonstrates strong performance on benchmarks including code and reasoning tasks, representing one of the notable sparse MoE language models in the open-source community.
Core Facts
Google's Switch Transformer and Mistral's Mixtral 8x7B are typical representatives of the MoE architecture.
The large-capacity, small-activation design of MoE allows a model's knowledge capacity and inference cost to be decoupled.
The Mixture of Experts (MoE) architecture activates only approximately 10%–30% of total parameters during inference, significantly reducing FLOPs and GPU memory usage per inference
Timeline (last 90 days)
A typical MoE configuration such as Mixtral 8x7B has 8 experts per layer, and each token is routed to only 2 of them, with the parameter count activated in a single pass being about one third of the total (plus shared weights).
The MoE architecture was later widely adopted by models such as Mixtral 8x7B, GPT-4 (reportedly), and DeepSeek.
Open-source models such as Mixtral 8×7B and DeepSeek-V2 both adopt MoE architecture.
Mixtral 8x7B is a mixture-of-experts architecture model released by Mistral in late 2023.
In Mixture-of-Experts (MoE) models, each input token is routed to only 2–4 experts for processing, while the remaining experts remain dormant.
K=2 in MoE is currently the most common choice in practice, and models including DeepSeek-MoE and Mixtral 8x7B both adopt this setting.
In the MoE architecture, the Mixtral 8x7B model has a total of 46.7B parameters, but only approximately 12.9B parameters (2 experts) are activated per token, making inference costs close to those of a 13B dense model while achieving performance comparable to the 70B level.
Mixtral 8x7B has a total of approximately 46.7 billion parameters, but only 2 experts are activated per inference, making the actual computation equivalent to a model of approximately 13 billion parameters.
Mixtral 8x7B was the first open-source MoE model to attract widespread industry attention, using 8 7B experts (46.7B total parameters, 12.9B activated) to achieve performance close to Llama 2 70B.
Mixtral 8x7B has approximately 46.7 billion total parameters, but only activates approximately 13 billion parameters per forward pass; its inference compute is close to that of a 13B model, yet it still requires approximately 90 GB of VRAM (FP16).
7 more timeline events
All Facts (20)
Google's Switch Transformer and Mistral's Mixtral 8x7B are typical representatives of the MoE architecture.
90%VerifiedThe large-capacity, small-activation design of MoE allows a model's knowledge capacity and inference cost to be decoupled.
85%VerifiedThe Mixture of Experts (MoE) architecture activates only approximately 10%–30% of total parameters during inference, significantly reducing FLOPs and GPU memory usage per inference
85%VerifiedMistral AI's Mixtral 8x7B brought the MoE architecture into mainstream open-source visibility at the end of 2023.
80%VerifiedMistral AI's Mixtral 8x7B adopts a MoE architecture with a total parameter count of approximately 47B, but only approximately 13B parameters are actually activated during each inference.
65%VerifiedGPT-4 is reportedly built on an MoE design, and Mixtral 8x7B is a representative open-source implementation.
65%VerifiedMixtral 8x7B has a total of approximately 46.7B parameters, activates only approximately 12.9B parameters per forward pass, and reduces inference computation by approximately 60% compared to dense models of equivalent performance.
65%VerifiedThe MoE architecture dynamically selects a small number of experts to participate in computation through a router, achieving sparse activation; mainstream frontier models including GPT-4, Mistral, and DeepSeek all adopt similar designs
65%VerifiedMistral's Mixtral 8x7B made the MoE architecture widely popular in the open-source community.
65%VerifiedQwen3.5's parameter activation rate is 4.3%, while DeepSeek V3's 671B model has an activation rate of 5.5%.
65%Partially VerifiedMixtral 8x7B采用了混合专家（MoE）架构，在推理时只激活部分参数
75%UnverifiedMixtral 8x7B has approximately 46.7 billion total parameters, but only activates approximately 13 billion parameters per forward pass; its inference compute is close to that of a 13B model, yet it still requires approximately 90 GB of VRAM (FP16).
60%UnverifiedMixtral 8x7B uses a MoE configuration of 8 experts with top-2 selection.
60%UnverifiedThe mixture-of-experts architecture can be traced back to research by Jacobs et al. in 1991; Google's Switch Transformer (2021) and Mistral's Mixtral 8x7B (2023) brought it back into the mainstream.
60%UnverifiedMistral AI's 7B parameter version performs on par with early GPT-3.5
60%UnverifiedTypical representatives of the MoE architecture include Mixtral 8x7B and DeepSeek-V2.
60%UnverifiedA typical MoE configuration such as Mixtral 8x7B has 8 experts per layer, and each token is routed to only 2 of them, with the parameter count activated in a single pass being about one third of the total (plus shared weights).
50%UnverifiedThe MoE architecture was later widely adopted by models such as Mixtral 8x7B, GPT-4 (reportedly), and DeepSeek.
50%UnverifiedOpen-source models such as Mixtral 8×7B and DeepSeek-V2 both adopt MoE architecture.
50%UnverifiedMixtral 8x7B is a mixture-of-experts architecture model released by Mistral in late 2023.
50%