The Complete Guide to Running GLM-5.2 Locally with 4 RTX 3090s for $4,000

A Milestone Moment for Open-Source Models

On June 17, 2025, Zhipu AI released the GLM-5.2 model. Benchmarks show its performance has reached GPT-4.8 levels, putting it on par with top closed-source models like GPT-5.5 and Claude OPUS. Even more exciting, GLM-5.2 is released under the MIT open-source license, meaning anyone with the right hardware can download and run it locally.

The MIT license is one of the most permissive open-source licenses available, allowing anyone to freely use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the software — the only requirement being that the copyright and license notices are included in all copies. By contrast, many open-source models use more restrictive licenses (such as Meta's Llama series, which previously used a community license with user count limits for commercial use). The MIT license means GLM-5.2 can be used directly in commercial products by companies of any size without additional licensing negotiations. This choice marks a bold step by a Chinese AI company in building the open-source ecosystem.

Bilibili creator Tech-Practice successfully built a local inference machine capable of running GLM-5.2 at home using 4 used RTX 3090 SXM4 GPUs, with a total budget of approximately $4,000. This article breaks down the key technical details and real-world performance results of the entire process.

Hardware and Software Configuration

Hardware Setup: 4×RTX 3090 SXM4

The core reason for choosing the RTX 3090 SXM4 is value for money. As a "sweet spot" GPU on the used market, four 3090s provide a total of 96GB of VRAM — critical for running large-scale language models. The total hardware build cost was kept to around $4,000.

It's worth explaining that the RTX 3090 SXM4 is NVIDIA's modular GPU version designed for data centers, using the SXM4 interface rather than the common PCIe slot. The SXM4 interface provides higher inter-GPU communication bandwidth via NVLink (up to 600GB/s), which is crucial for multi-GPU collaborative inference — when a model is sharded across multiple GPUs, they need to frequently exchange intermediate computation results, and high-bandwidth interconnects significantly reduce communication bottlenecks. On the used market, these GPUs typically come from decommissioned data center servers and are priced far below new cards of equivalent performance. Each RTX 3090 has 24GB of GDDR6X VRAM and 10,496 CUDA cores; four cards combined provide 96GB of VRAM, offering ample memory for large model inference. Note that the SXM4 version requires a dedicated motherboard and cooling solution and cannot be installed directly in a standard consumer PC, which is why building the complete system requires a certain level of technical expertise.

Software Stack: Linux + Llama.cpp

The software choices are straightforward:

• Operating System: Linux
• Inference Engine: Llama.cpp (newly compiled version)
• Quantization Method: Unsloth IQ quantization (dynamic 2-bit quantization)
• Model Size: ~23.9GB after quantization

Llama.cpp is an open-source C/C++ inference framework developed by Georgi Gerganov. Originally designed for Meta's LLaMA models, it has evolved into a universal inference engine supporting virtually all mainstream open-source large models. Its core advantages include: pure C/C++ implementation with minimal dependencies, eliminating the need for bulky Python deep learning frameworks; support for multiple quantization formats (GGUF) to accommodate different hardware configurations; flexible execution on CPU, GPU, or hybrid modes; efficient memory management with support for cross-device model layer allocation. Llama.cpp also includes a built-in HTTP server that can expose OpenAI API-compatible endpoints, enabling seamless integration with various frontend applications.

Since GLM-5.2's original model is enormous, fitting it directly into 96GB of VRAM still isn't enough, making a quantized version necessary. The Unsloth IQ quantization method was chosen here, compressing the model to dynamic 2-bit precision, with a final model size of approximately 23.9GB that loads comfortably into local VRAM.

Quantization is a technique that compresses model weights from high-precision floating-point numbers (such as 16-bit FP16) to lower-precision representations. Unsloth's IQ (Importance-based Quantization) method is a form of non-uniform quantization that dynamically allocates bits based on weight importance — weights with greater impact on model output retain more precision, while less impactful ones are compressed more aggressively. Dynamic 2-bit quantization means each parameter occupies only about 2 bits of storage on average, achieving roughly 8x compression compared to FP16 format. This explains how a model that might originally exceed 200GB can be compressed to 23.9GB. Of course, such aggressive compression inevitably leads to some precision loss, particularly in mathematical reasoning and complex logic tasks, but for everyday conversation and code generation, the quality degradation is typically within acceptable bounds.

Real-World Performance Testing

Basic Inference Speed

Multiple rounds of testing were conducted in command-line mode, with the following performance results:

• Token generation speed: ~6.3-6.8 tokens/second
• Prompt evaluation speed: ~10.6 tokens/second

To put this speed in perspective: in English, one token corresponds to roughly 4 characters or 0.75 words; in Chinese, one token corresponds to roughly 1-2 characters. So 6.5 tokens/second translates to approximately 5 English words or 6-13 Chinese characters per second. This is slightly slower than normal human reading speed (~4-5 English words per second), meaning users can essentially read the model's output in real time without waiting. For comparison, OpenAI's GPT-4o API typically achieves 50-100 tokens/second, and Claude's streaming output ranges from 30-80 tokens/second. While the local setup is slower, it's perfectly adequate for non-real-time batch tasks like code generation and document writing.

From a streaming perspective, over 6 tokens per second is entirely acceptable for interactive use. While not as fast as cloud APIs, considering this is a completely local, completely offline environment, the speed is already quite practical.

An Interesting Finding on GPU Utilization

An noteworthy phenomenon emerged during testing: only 1 of the 4 GPUs reached about 10%-13% utilization, while the others sat nearly idle, yet CPU usage was around 50%.

This is likely related to GLM-5.2's Mixture of Experts (MoE) architecture. MoE is a conditional computation paradigm where the model contains multiple parallel "expert" sub-networks and a router (gate) network. During inference, the router selects only a few experts to activate (typically 2-4) based on the input token's characteristics, rather than engaging all parameters in computation. This allows MoE models to have an extremely large total parameter count (providing greater knowledge capacity) while maintaining relatively low actual computational cost (since only a small fraction of parameters are used per inference).

GLM-5.2 likely employs a sparse MoE design similar to Mixtral or DeepSeek^Ad-V3, which explains why GPU compute utilization is low while CPU load is high — routing decisions and token scheduling are primarily handled on the CPU, while the actual matrix operations execute on the GPU but at limited scale. Additionally, with 2-bit quantization, the model's computational density (floating-point operations needed per byte of data) is low, and the GPU may spend most of its time waiting for data transfers rather than executing computations, which is another reason for low utilization. This means that if CPU-side processing efficiency or memory bandwidth utilization can be optimized in the future, there's significant room for overall inference speed improvement.

Impact of Thinking Mode on Inference Speed

GLM-5.2 offers an optional "Thinking Mode" (Thinking Part), similar to Chain of Thought (CoT) functionality. When maximum reasoning thinking mode is enabled, the AI displays detailed reasoning steps, but at the cost of significantly increased generation time.

Chain of Thought (CoT) is one of the most important breakthroughs in the large model field in recent years. Research has found that when models are guided to show their reasoning process step by step, their accuracy on math, logic, and complex problem-solving improves significantly. OpenAI's o1 series and DeepSeek-R1 are representative examples of this approach. GLM-5.2's thinking mode essentially has the model generate an internal reasoning process before giving its final answer. This reasoning process consumes additional token generation time but improves answer accuracy and reliability.

In the classic "How many R's are in Strawberry" test, the model with thinking mode enabled analyzes each letter step by step and ultimately gives the correct answer. However, for practical use, the creator recommends turning off the thinking feature for faster response times, unless you specifically need to see the model's reasoning process.

Code Generation in Practice: OpenWebUI Integration

API Server Mode

Through Llama.cpp's server mode, you can expose an API endpoint and then connect frontend tools like OpenWebUI to it, providing a web interface experience similar to using ChatGPT.

OpenWebUI (formerly Ollama WebUI) is an open-source web frontend interface designed specifically for local large model inference. It provides a ChatGPT-like conversational interface with support for multi-model switching, conversation history management, file uploads, RAG (Retrieval-Augmented Generation), and more. OpenWebUI can connect to multiple backends, including Ollama, Llama.cpp servers, and any OpenAI API-compatible endpoint. This allows users to get a professional-grade AI conversation experience without writing code, significantly lowering the barrier to using locally deployed models.

In testing, the creator asked the model to create an HTML file with a "LOVE" button. With thinking mode disabled, the model immediately began generating code with a streaming output speed of approximately 5.9 tokens/second.

Impressive Results

The final generated HTML file was not only fully functional but also featured:

• Polished UI design with animation effects
• Complete interactive functionality — clicking the button changes the text content
• Even sound effects

Prediction speed reached 6.5 tokens per second, with prompt evaluation at 3.3 tokens per second. For a fully locally-run open-source model, this code generation quality is already very close to commercial-grade AI assistants. This also validates that while 2-bit quantization is aggressive, the model still maintains a remarkably high capability level for structured output tasks like code generation.

Cost and Value Analysis

Let's do the math:

Item	Cost
4×RTX 3090 SXM4 (used)	~$3,000-3,500
Motherboard + CPU + RAM, etc.	~$500-1,000
Total	~$4,000

By comparison, using GPT-5.5 or Claude OPUS APIs in a high-frequency usage scenario could cost anywhere from tens to hundreds of dollars per month. Take a moderately active developer as an example: assuming roughly 1 million input tokens and 500,000 output tokens per month, at GPT-4-level pricing (input ~$30/million tokens, output ~$60/million tokens), the monthly cost would be about $60. At that rate, the $4,000 hardware investment would take about 5-6 years to recoup through saved API costs. However, for team use or high-frequency scenarios (such as hundreds of long conversations per day), the payback period can shrink to 1-2 years.

More importantly, local deployment provides a completely offline, completely private usage environment — for businesses and individual developers handling sensitive data, this value is immeasurable. In industries like healthcare, legal, and finance, data compliance requirements often prohibit sending sensitive information to third-party cloud services, and local deployment perfectly addresses this pain point. Additionally, local deployment means zero network latency dependency, unlimited usage, and complete control over model behavior.

Summary and Outlook

This experiment demonstrates several important trends:

1. Open-source models have approached or even reached the level of top closed-source models, and GLM-5.2's MIT license further lowers the barrier to entry
2. Running top-tier AI models on consumer-grade hardware is now a reality — four used RTX 3090s can build a functional local AI inference system
3. Mature quantization technology has made local deployment of large models feasible, with Unsloth's IQ quantization achieving a solid balance between compression ratio and quality
4. MoE architecture characteristics mean GPU utilization isn't the bottleneck — CPU optimization may be the key direction for improving local inference performance in the future

Looking ahead, as NVIDIA's next-generation consumer GPUs (such as the RTX 50 series) bring larger VRAM capacities and higher memory bandwidth, and as quantization algorithms continue to advance (such as 4-bit quantization achieving reasonable compression while maintaining higher precision), the local AI inference experience will continue to close the gap with cloud services. Meanwhile, AMD and Intel are also actively advancing their AI acceleration hardware ecosystems, promising more diverse hardware options and further cost reductions in the future.

Of course, 6 tokens per second still lags behind cloud services, and 2-bit quantization inevitably introduces some precision loss. But for users who prioritize data privacy and want to break free from API dependency, this setup already offers a very compelling option.