Dense Models Too Slow to Run Locally? How MoE Architecture Breaks Through the Performance Bottleneck

The Performance Dilemma of Local LLMs: Why Dense Models Can't Run Fast

As large language models (LLMs) advance at breakneck speed, more and more developers and enthusiasts want to run these powerful AI models on local hardware.

Background on LLM Development: Large Language Models (LLMs) are natural language processing models built on deep learning techniques, pre-trained on massive text corpora to learn linguistic patterns, knowledge, and reasoning capabilities. From BERT's debut in 2018, to GPT-3's stunning capabilities in 2020, to ChatGPT's worldwide explosion in 2022, LLM parameter counts have skyrocketed from hundreds of millions to hundreds of billions or even trillions. These models can not only handle traditional NLP tasks like text generation, Q&A, and translation, but also demonstrate complex abilities such as code writing, mathematical reasoning, and multi-step planning. However, the price of improved capabilities is an explosive growth in parameter scale, which directly creates performance challenges for local deployment.

Local deployment means better privacy protection, freedom from cloud service dependency, and lower long-term costs. But while the vision is compelling, the real-world performance bottlenecks are painfully obvious.

A viewpoint recently gaining traction in the community captures this tension perfectly: "Dense models run slowly on local hardware, but current progress is stunning — signaling that a technological shift is on the horizon." This brief statement precisely summarizes the core contradiction and trajectory of local AI deployment.

What Are Dense Models? Why Do They Run Slowly on Local Hardware?

Architectural Characteristics of Dense Models

A Dense Model refers to an architecture where all parameters participate in computation during inference. Regardless of the input, every layer and every neuron in the model must be activated and computed. This is the typical design of traditional Transformer architectures.

Computational Characteristics of the Transformer Architecture: The Transformer is the dominant architecture behind today's large language models, proposed by Google in 2017. Its core innovation is the self-attention mechanism, which processes all positions in a sequence in parallel and captures long-range dependencies. However, this fully-connected attention computation introduces quadratic computational complexity. In a dense Transformer model, generating each token requires passing through multiple layers of attention computation and feed-forward neural network (FFN) processing, with all parameter matrices participating in the computation. Take a 70B parameter model as an example: even using half-precision floating point (FP16), simply loading the model weights requires approximately 140GB of memory, and each forward pass involves trillions of floating-point operations — posing an enormous challenge to local hardware.

Consider a 70-billion parameter (70B) dense model: to generate a single token, the model theoretically needs to traverse all 70 billion parameters for matrix operations, placing extreme demands on memory bandwidth and compute power.

The Inherent Limitations of Local Hardware

Local hardware — whether consumer-grade GPUs or Apple's unified memory architecture — falls far short of data center-grade professional hardware in both memory bandwidth and VRAM capacity. The fundamental reasons dense models run slowly on local machines are:

Memory bandwidth bottleneck: Every inference pass requires reading all model weights; bandwidth directly determines token generation speed
VRAM capacity constraints: Large-parameter models consume massive amounts of VRAM, often exceeding what consumer devices can accommodate
Insufficient compute power: Full parameter activation creates enormous floating-point operation demands

The Decisive Impact of Memory Bandwidth: In LLM inference, the performance bottleneck often lies not in computational speed but in memory bandwidth — the rate at which data transfers from memory to compute units. This is because the growth in modern GPU compute capability (FLOPS) far outpaces the growth in memory bandwidth (GB/s). Take the consumer-grade RTX 4090 as an example: its FP16 compute reaches 82.6 TFLOPS, but memory bandwidth is only 1TB/s; meanwhile, the data center H100 offers 3.35TB/s bandwidth. During autoregressive decoding, the model generates tokens one at a time, re-reading all model weights for each token — a memory bandwidth-intensive process. A simple calculation: a 70B parameter model (140GB weights) at 1TB/s bandwidth has a theoretical maximum generation speed of about 7 tokens/second, before accounting for real-world system overhead.

This is why many users running large dense models locally encounter the frustrating experience of generating only a few tokens per second.

MoE Architecture: The Key to Breaking the Local Inference Performance Bottleneck

From Dense to Sparse Activation

In contrast to dense models, the Mixture of Experts (MoE) architecture has been making waves in recent years. The core idea behind MoE is "sparse activation" — while the model's total parameter count may be enormous, only a small subset of "expert" networks is activated during each inference pass.

How MoE Works: The Mixture of Experts concept was originally proposed by Jacobs et al. in 1991 and has recently experienced a renaissance in the large model domain. The core of the MoE architecture replaces traditional feed-forward neural network layers with multiple "expert" networks and introduces a gating network to determine which experts are activated during each inference. For example, the DeepSeek-V2 model has 236B total parameters but activates only 21B per inference; Mixtral-8x7B has 47B total parameters but actually activates only 13B. The advantage of this design is maintaining the model capacity and knowledge breadth that come with large parameter counts while dramatically reducing per-inference computation and memory access. The gating mechanism also allows different experts to learn knowledge from different domains, achieving more efficient parameter utilization. The tradeoff, however, is that the total model size is larger, requiring more storage space, and challenges like load balancing and training stability add engineering complexity.

For instance, an MoE model with hundreds of billions of total parameters might activate only a tenth or less during each inference. This means: while preserving the model's knowledge capacity and capabilities, the actual computation per inference is dramatically reduced.

The recent wave of MoE models from the open-source community is gradually bringing capabilities that once required top-tier hardware down to ordinary consumer devices.

Why This Signals "The Future Is Already Here"

The belief that local AI is on the verge of transformation is supported by deep technical logic:

Continuous architectural innovation: Technologies like MoE, quantization, and model distillation are maturing, steadily lowering the barrier to running large models locally

Model Quantization Explained: Model quantization converts model weights and activation values from high precision (such as FP32 or FP16) to low precision (such as INT8, INT4, or even lower). The core principle leverages neural network robustness — research shows that model parameters can tolerate a certain degree of precision loss without significantly impacting performance. Common quantization methods include: Post-Training Quantization (PTQ), which directly converts already-trained models; and Quantization-Aware Training (QAT), which simulates quantization effects during training. Advanced algorithms like GPTQ and AWQ use layer-wise optimization and weight importance analysis to maintain over 95% of original performance at 4-bit quantization. The benefits are significant: going from FP16 to INT4 reduces memory footprint and bandwidth requirements by 75%, making it possible to run 70B models on consumer-grade 24GB GPUs. Inference frameworks like llama.cpp deeply integrate quantization support, further driving the adoption of local deployment.
Rapid hardware ecosystem advancement: Apple's M-series chips with unified memory, and the proliferation of various AI-dedicated chips, provide a stronger hardware foundation for local inference

Advantages of Apple's Unified Memory Architecture: Apple's M-series chips use a Unified Memory Architecture (UMA), where the CPU, GPU, and Neural Engine share the same pool of high-bandwidth memory, eliminating the data copy overhead between CPU memory and GPU VRAM found in traditional PC architectures. The M2 Ultra offers up to 192GB of unified memory with 800GB/s bandwidth; the upcoming M4 Max is rumored to reach 546GB/s. This design is particularly well-suited for LLM inference: no need to transfer data between CPU and GPU, large models can be fully loaded into memory, avoiding the VRAM limitations of consumer GPUs. Additionally, Apple's Metal Performance Shaders and Core ML frameworks provide native optimizations for technologies like MoE and quantization. Real-world testing shows that an M2 Max with 96GB of memory can smoothly run quantized 70B models with performance approaching high-end gaming GPUs, at far lower power consumption and noise levels than traditional setups — offering Mac users a unique local AI experience.
The rise of open source: High-quality open-source models continue to emerge, and community optimization tools (such as llama.cpp, Ollama, etc.) make deployment easier than ever

The Ecosystem Value of Open-Source Inference Frameworks: Open-source inference frameworks like llama.cpp, Ollama, and vLLM are critical infrastructure for the local AI ecosystem. llama.cpp, developed by Georgi Gerganov, is implemented in pure C/C++, supports CPU inference and multiple quantization formats, and dramatically improves efficiency through SIMD instruction optimization and memory mapping — enabling even ordinary laptops to run large models. Ollama builds on this with a Docker-like model management experience, allowing users to download and run various open-source models with a single command. vLLM focuses on high-performance inference, optimizing VRAM usage and throughput through innovations like PagedAttention. The value of these tools lies not only in technical optimization but in lowering the barrier to entry: developers can quickly build local AI applications without needing deep understanding of low-level details. Active communities continuously contribute new model support, performance optimizations, and bug fixes, creating a virtuous cycle. The flourishing open-source ecosystem is transforming local AI from a geek toy into a practical tool.

The Real-World Value and Challenges of Local AI Deployment

Privacy Protection and Data Sovereignty

The most direct value of running large models locally is data privacy. All inference happens on the user's own device, with no need to upload sensitive data to the cloud. For privacy-sensitive scenarios in healthcare, legal, and financial sectors, this offers irreplaceable value.

Long-Term Cost Advantages

Local deployment avoids ongoing API call fees. For developers and enterprises that use AI heavily and frequently, a one-time hardware investment may be more cost-effective than ongoing cloud service subscriptions. Local models are also unaffected by provider rate limits, price increases, or policy changes, offering greater autonomy and control.

Challenges That Remain

Objectively speaking, locally-run models still lag behind top-tier cloud models in capability. The speed issues of dense models persist; and while MoE models are faster at inference, they actually have higher requirements for VRAM and memory capacity (since all expert parameters need to be loaded). This "speed vs. capacity tradeoff" is a practical consideration that local deployers must carefully weigh.

The Transformation Is Just Around the Corner

The core narrative is now clear: Dense models run slowly on local hardware due to hardware constraints, but architectural innovation — especially MoE — is rapidly changing this landscape.

We are at a critical inflection point. With the triple advancement of model architectures, quantization techniques, and hardware ecosystems, the vision of "smoothly running near-top-tier AI models on a personal computer" is moving from a distant dream to an imminent reality. For anyone following local AI deployment, the future is well worth looking forward to.

Key Takeaways

Dense models are slow for local inference due to full parameter activation, with memory bandwidth as the critical bottleneck
MoE architecture significantly reduces inference computation through sparse activation, representing a major breakthrough for local AI
Quantization techniques, unified memory architectures, and open-source tools are working together to drive the adoption of local deployment
Local AI offers unique advantages in privacy protection and cost control
A technological transformation is underway — the era of running high-performance AI models on personal devices is coming soon