Real-World Comparison of 6 Open-Source LLMs for On-Premise Deployment: A Selection Guide to Avoid Pitfalls

In the wave of enterprises deploying large language models on-premise, a wrong choice often means tens of thousands of dollars in hardware costs going down the drain. Stacking up GPUs doesn't equal efficiency, and picking the wrong model can drag an entire intranet project into a maintenance quagmire. Based on a Bilibili creator's hands-on tests of on-premise deployment for 6 popular open-source LLMs, this article provides a comprehensive comparison across hardware cost, inference efficiency, ecosystem compatibility, and deployment difficulty—helping you avoid detours in real business scenarios.

What is on-premise deployment? On-Premise Deployment means running the entire inference service of a large language model on the enterprise's own servers or intranet environment, without data passing through external clouds—thereby meeting requirements for data compliance, information security, and low latency. Compared to calling cloud APIs, on-premise deployment requires enterprises to bear the costs of GPU hardware procurement, model quantization, inference framework setup, and long-term operations, but it achieves fully autonomous and controllable data. This is especially critical for heavily regulated industries such as finance, government, and healthcare.

It's worth noting that on-premise deployment is not as simple as downloading model weights to a local server—it also involves systems-engineering challenges such as inference framework selection, multi-GPU parallelism strategies, and cluster scheduling.

At the inference framework level, vLLM adopts PagedAttention technology to manage KV Cache in a fragmented manner. The KV Cache is the mechanism by which Transformer models cache the Key and Value matrices of the attention mechanism at each layer during autoregressive generation; its memory usage grows linearly with sequence length. PagedAttention eliminates fragmentation through a paging management approach similar to operating system virtual memory, boosting GPU memory utilization by over 30%. Ollama, on the other hand, excels with an extremely low configuration barrier, featuring built-in model format conversion and quantization workflows—suitable for quick single-machine validation, though its high-concurrency performance is limited. LMDeploy, developed by the Shanghai AI Laboratory, offers more targeted advantages in operator optimization for domestic models like InternLM and Qwen.

On the multi-GPU parallelism front, Tensor Parallelism splits a single layer's weight matrices by column or row across multiple GPUs, with each card responsible for only part of the matrix multiplication, and results aggregated via AllReduce communication—this requires extremely high inter-card interconnect bandwidth. NVLink's inter-card bandwidth (around 900GB/s) is far superior to PCIe (around 64GB/s), and cross-node communication relies on InfiniBand networks. Pipeline Parallelism splits by Transformer layers, placing the first few layers on the first card and the last few layers on the last card, using micro-batch pipelining to reduce bubble time. The former offers low latency but high communication overhead, while the latter offers high throughput but longer first-token latency. Choosing the wrong parallelism strategy can turn inter-card communication into a bottleneck, making actual inference speed even slower than a single card.

Cost-Effective Choices: DeepSeek and Qwen3

For enterprises with limited budgets seeking stable, reliable deployment, DeepSeek and the Qwen series are two options you simply can't ignore.

DeepSeek: The Practical Choice with the Lowest Hardware Cost

Among the 6 models, DeepSeek has the lowest hardware cost, saving enterprises a considerable amount on GPU expenses. Its core strengths lie in fast inference speed, high GPU utilization, and a simple fine-tuning process, making it ideal for budget-constrained teams.

The low-cost advantage of the DeepSeek series is no accident—the core behind it is the Mixture of Experts (MoE) architecture. The MoE architecture distributes model parameters across multiple "expert" sub-networks, and during inference each token activates only a few of these experts (e.g., 671B total parameters but only about 37B activated per pass). This maintains the capability ceiling brought by high parameter counts while drastically reducing the actual compute consumed per inference. The direct engineering benefit of this sparse activation mechanism is that the model's "effective compute" (measured in FLOPs) is far smaller than that of a fully dense model, while the Router dynamically selects the best-matching expert sub-network for each token—giving MoE models natural adaptive flexibility for task diversity. Compared to a dense model of equivalent capability, MoE puts less pressure on memory bandwidth during inference and achieves higher tokens/s, allowing DeepSeek to achieve higher throughput under the same hardware conditions and significantly improving the enterprise's deployment cost-effectiveness.

The GPU Utilization and Inference Throughput mentioned here are core metrics for measuring the cost-effectiveness of LLM deployment. GPU utilization reflects the proportion of GPU compute actually being used—low utilization means wasted compute and higher cost per token. Inference throughput is typically measured in tokens/s (the number of tokens generated per second), directly affecting user experience and concurrent processing capacity. Note that high GPU utilization does not always equal high inference efficiency—if a large amount of GPU time is spent on inefficient memory movement or communication waiting, high utilization actually indicates waste. Therefore, the two metrics must be considered together.

However, its weaknesses are equally clear—its agent orchestration capability is relatively weak, and its logic tends to break down when facing complex office automation workflows. So-called AI Agent orchestration refers to connecting the LLM with multiple tools, APIs, or sub-task workflows to form an automated multi-step decision chain—for example, automatically reading emails → calling the calendar API → generating meeting minutes → sending notifications. This places high demands on the model's instruction-following ability, Function Calling accuracy, and long-term context memory. Tool-calling accuracy is a key threshold for agent scenarios: the model must accurately identify when it should call an external tool, which tool to call, and in what parameter format—an error at any step breaks the entire automation workflow. DeepSeek's shortcomings here mean it's better suited for enterprises focused on data analysis, code assistance, and simple Q&A scenarios. Overall, it's a pragmatic and reliable practical choice.

Qwen3: The All-Rounder with the Most Complete Ecosystem

If there's one model that lets operations novices get started stress-free, Qwen3 is undoubtedly it. Its ecosystem compatibility is exceptional—so-called ecosystem compatibility directly determines the engineering complexity of enterprise deployment. Mainstream deployment frameworks such as vLLM, Ollama, and LMDeploy, as well as fine-tuning frameworks like LLaMA-Factory and Axolotl, offer varying degrees of support for different open-source models. Models with good ecosystems typically have well-developed quantized versions (GGUF, AWQ, GPTQ formats). Among these, GGUF is the mainstream format in the llama.cpp ecosystem, suited for CPU/consumer-grade GPU hybrid inference; AWQ (Activation-aware Weight Quantization) protects critical weight channels by being aware of activation value distributions, retaining more model capability at INT4 precision; GPTQ is a post-training quantization scheme based on approximate second-order optimization. All three have different strengths in speed, accuracy, and compatibility. Qwen3 excels in this regard—nearly all mainstream deployment tools and fine-tuning frameworks are compatible with it, and on-premise deployment cases across various industries are abundant, meaning fewer pitfalls and stronger community support.

Qwen3's excellent performance in knowledge-base scenarios is closely tied to its complete RAG toolchain ecosystem. The core demand for most enterprise on-premise deployments is building a Retrieval-Augmented Generation (RAG) knowledge Q&A system: enterprise documents are chunked, vectorized via an Embedding model, and stored in a vector database (such as Milvus or Chroma). When a user asks a question, the most relevant text chunks are retrieved first, then concatenated into the prompt and handed to the LLM to generate an answer. There are several easily overlooked engineering details in the RAG process: the document chunking strategy (chunk size and overlap ratio) directly affects retrieval granularity, the semantic spaces of the Embedding model and generation model need to be aligned as much as possible, and a Reranker model is usually needed for fine-grained reranking after vector retrieval to improve accuracy. Qwen3's advantage lies in the officially provided, specially optimized text-embedding series of Embedding models, which form a complete loop with the main model, ensuring both retrieval recall and generation quality. This is one of the core reasons for its best overall performance in multi-modal (text and image) knowledge-base scenarios.

The hardware barrier is equally friendly: a 70B-scale large model can run on two H20s or four 5090s, making long-term maintenance worry-free. Among these, the H20 is a data-center-grade GPU designed by NVIDIA specifically for the Chinese market, featuring 96GB of HBM3 memory. HBM (High Bandwidth Memory) offers 3–5x the memory bandwidth of GDDR, giving it a clear advantage in memory-bandwidth-sensitive workloads like LLM inference, and it's primarily aimed at inference deployment scenarios. The RTX 5090 is a consumer-grade flagship card released in 2025, with 32GB of GDDR7 memory and relatively high cost-effectiveness, suitable for budget-constrained teams. However, note that consumer-grade cards fall short of data-center cards in ECC (Error Correction Code) memory protection, thermal stability during long continuous operation, and NVLink interconnect support. The two represent two different hardware routes—"professional data-center card" versus "high-end consumer card"—and the choice requires weighing initial procurement cost, long-term maintenance risk, and business continuity requirements.

The only regret is that its ultra-long document parsing capability is not as prominent as specialized models. Overall, Qwen3 is suitable for nearly all general-purpose on-premise scenarios, especially for novice operations teams and multi-modal knowledge-base scenarios, making it the best all-around performer in this evaluation.

Top-Tier Capability: Zhipu GLM, the Intranet King

If you have higher demands for model capability, Zhipu GLM is currently recognized as one of the strongest open-source models available.

According to hands-on test data, GLM ranks in the top tier across all three dimensions of coding, agent collaboration, and Chinese comprehension. At the same time, it offers excellent compatibility with government and classified intranets, making it a great fit for high-security scenarios.

In terms of hardware requirements, two H20s are enough for simple business tasks, while complex automation workflows require at least four cards. Notably, four-card deployment scenarios often also involve fine-tuning of the quantization strategy. Quantization is the technique of compressing model weights from high-precision floating-point numbers (such as FP16) to low-precision integers (such as INT8 or INT4), which can drastically reduce memory usage and inference latency with minimal precision loss. Taking a 70B-parameter model as an example, FP16 precision requires about 140GB of memory, while INT4 quantization can compress this to about 40GB—a reduction of roughly 72%. The choice of quantization precision requires balancing model capability, memory limits, and inference speed. INT8 quantization affects accuracy on most tasks by less than 1%–2%, but INT4 quantization may cause a 3%–8% capability drop on complex reasoning and code generation tasks with strong logic. For models like GLM that focus on coding and agent capabilities, the quantization depth must be evaluated with particular caution, as excessive quantization can noticeably degrade output quality.

Four-card deployment also brings concurrent processing capacity considerations. LLM inference services typically use Continuous Batching technology to merge multiple user requests into a single forward pass, improving GPU utilization. Unlike static batching, continuous batching allows new requests to be inserted at any position in an ongoing batch, greatly reducing GPU idle waiting time—it is the core mechanism by which current inference service frameworks improve throughput. The higher the concurrency, the higher the Time to First Token (TTFT) rises, because a larger batch requires a longer Prefill stage to process all input sequences. GLM has high demands on memory and compute in complex agent workflows, so operations teams need to carefully set the maximum concurrency limit based on the latency tolerance of actual business—another concrete manifestation of its higher demands on operations team expertise.

GLM's main weakness lies not in capability itself but in ecosystem—online tutorials and third-party plugins are noticeably fewer than Qwen's, community resources are relatively scarce, and troubleshooting costs are higher when problems arise. Therefore, it is best suited for intranet teams with R&D capabilities, government classified units, and enterprises that require complex multi-step automation.

Specialized Tools and Lightweight Contenders

Some models don't aim to be all-rounders but instead excel in niche scenarios. Choose right and you get a killer tool; choose wrong and the cost-effectiveness is abysmal.

Kimi K2: In a League of Its Own for Long Document Parsing

Kimi K2 is nearly unbeatable in the arena of million-character long documents and batch PDF parsing, making it a killer tool for law firms and investment banks processing contracts.

Long Context Processing refers to a model's ability to maintain information-extraction accuracy when handling ultra-large context windows, typically measured by context window size, with mainstream models ranging from 32K to a million tokens. The technical challenges in this arena manifest in two main areas: first, memory usage grows near-quadratically with context length (the KV Cache mechanism, because every token must cache the Key/Value pairs of all historical layers). A million-token-level context under standard attention places nearly unacceptable memory demands, requiring alternative mechanisms such as Sliding Window Attention, Sparse Attention, or state space models (like Mamba) to reduce complexity from O(n²) to an acceptable range. Second is the "Lost in the Middle" phenomenon—research shows that standard Transformers accurately remember information at the beginning and end of a document but tend to miss key information in middle sections. Models optimized for long documents mitigate this through improved positional encoding (such as YaRN and RoPE extension) and specialized training data. This is the core of Kimi K2's differentiated competitiveness in this arena.

But its problem is that the cost is extremely high for daily enterprise OA automation and multi-user concurrency, making it severely unbalanced in cost-effectiveness for ordinary intranet businesses. It's only suitable for enterprises specializing in massive document parsing that don't involve daily general-purpose business—beyond this scope, the returns don't justify the investment.

MiniMax M3: A Customer Service Solution for Small Businesses

MiniMax M3 focuses on being lightweight with smooth conversation, running on a single card with low latency—more than enough for customer service and simple Q&A. But it falls short across the board on hard requirements like long document processing, complex reasoning, and code generation. Once the business scale expands, compute costs skyrocket, and the room for deep customization is also very limited.

Behind the lightweight advantage of single-card deployment lies a hidden concurrency bottleneck: the single card's memory ceiling directly limits the batch size of requests that can be processed simultaneously (i.e., the maximum Batch Size). Once daily active users exceed a certain scale, the request queue starts to back up, waiting times lengthen significantly, and user experience deteriorates rapidly. In addition, single-card solutions pose high-availability risks—a single point of failure causes complete service interruption, whereas multi-card or multi-node deployment can achieve failover through load balancing. This characteristic of being "sufficient at small scale but collapsing upon expansion" means MiniMax M3 is only suitable for customer service scenarios at small businesses or short-term, small-scale pilot projects, and should not serve as the infrastructure for long-term core business.

Promising Newcomer: Tencent Hunyuan 3

Tencent Hunyuan 3 is the most recently open-sourced model in this evaluation. Its capabilities are decent, and its integration with Tencent's full product lineup is a major highlight, with online knowledge-base performance that holds up.

But as a freshly open-sourced new model, enterprise deployment cases are extremely rare, first-time deployment is highly prone to pitfalls, optimization for mainstream frameworks is still incomplete, and it's hard to find ready-made solutions when problems arise. This also reflects a general pattern in the open-source LLM ecosystem: from open-source release to the community forming complete deployment tutorials, quantized versions, and industry fine-tuning solutions, a model typically requires a maturation period of 6 months to over a year.

This maturation period is especially evident at the engineering level: quantized versions of new models such as AWQ and GPTQ often lag the original release by weeks, because quantization requires model-structure-specific Calibration, and the quality of the calibration dataset directly affects post-quantization precision retention. Specialized inference framework optimizations (such as vLLM's dedicated CUDA scheduling kernels and Flash Attention adaptation) may take months to pass code review and merge into the main branch. Industry fine-tuning datasets and LoRA adapters based on the model (LoRA is a parameter-efficient fine-tuning technique that injects low-rank matrices alongside the original weights to achieve task adaptation with minimal parameters) depend even more on the long-term accumulation and open-source contributions of community researchers and industry practitioners. Before these engineering infrastructure pieces are in place, the labor cost of enterprises adapting on their own often far exceeds expectations. Therefore, Hunyuan 3 is better suited for large enterprises deeply tied to the Tencent ecosystem and backed by a professional operations team. For ordinary enterprises, the risk of entering now is relatively high, and a wait-and-see approach is recommended.

Selection Advice: Finding the Best-Fit Solution

From this hands-on ranking, we can see that on-premise deployment has no absolute optimal solution—only the best-fit solution:

• General first choice: Qwen3—complete ecosystem, low barrier, worry-free maintenance;
• Capability ceiling: Zhipu GLM—suitable for government classified intranets with R&D capabilities;
• Cost-effective choice: DeepSeek—for budget-constrained data analysis and code assistance scenarios;
• Specialized killer tool: Kimi K2—specializes in long documents, not general-purpose business;
• Lightweight solution: MiniMax M3—for small-business customer service pilots;
• Enter with caution: Tencent Hunyuan 3—suitable for large enterprises deeply tied to the Tencent ecosystem.

Building local offline models involves numerous details such as GPU configuration, quantization tuning, and cluster scheduling—model selection is only the first step. What truly determines the success or failure of a project is often the precise match between hardware configuration and business needs. Before formal deployment, it's recommended to conduct small-scale hands-on testing based on your own concurrency, document scale, and operations capabilities—for example, run stress tests at 1/10 of the production environment's concurrency load, focusing on Time to First Token (TTFT, the time from sending a request to receiving the first output token, which directly determines the user's perceived response speed) and throughput (tokens/s, which determines the maximum user scale the system can support on given hardware), and align these with business SLA requirements—to avoid paying for capabilities you don't need.

Key Takeaways

Key Takeaways

Key Takeaways