Deploying Open-Source LLMs in Production: What's Actually the Hardest Part?

A Soul-Searching Question from a Frontline Engineer

In a Reddit tech community, an engineer conducting industry research posed a question that hit a nerve: When running open-source large language models (LLMs) in production, what's truly the hardest part today?

This question resonated widely because it touches on a rapidly evolving yet still pitfall-laden domain. Open-source LLMs refer to language models whose weights, architecture, and training code are publicly accessible. Since 2023, open-source models like Meta's Llama series, Alibaba's Qwen series, and Mistral AI's Mistral series have developed rapidly. These models use Transformer architectures with parameters ranging from 7 billion to 70 billion, excelling at tasks like code generation, multilingual understanding, and reasoning. As open-source models like Llama, Qwen, and Mistral approach or even surpass closed-source models in certain scenarios, more teams are attempting to self-host inference services.

Self-hosting means enterprises deploy and run model inference services on their own infrastructure rather than calling third-party services via APIs. Inference refers to using a trained model to make predictions on new inputs, as opposed to training. In production, inference services must handle high-concurrency requests, manage GPU memory, and implement request queuing and batching. But between "getting a demo to work" and "reliably supporting production traffic," there's a chasm.

reddit original post discussion

The original poster listed a series of highly targeted questions covering model selection, infrastructure, cost, latency, observability, security compliance, and more. These questions themselves constitute an excellent "production-grade LLM deployment checklist."

The Full Technology Stack for Deploying Open-Source LLMs

Mainstream Inference Runtimes and Infrastructure Choices

The options mentioned in the original post reveal a rich technology stack for running open-source models in production:

Hosted APIs: Third-party inference providers like Together, Fireworks, and Groq, offering out-of-the-box solutions with per-token pricing.
GPU Cloud Platforms like RunPod: Providing elastic GPU resources, sitting between fully self-built and hosted solutions.
Kubernetes: For large-scale scenarios requiring fine-grained orchestration. Kubernetes is Google's open-source container orchestration system that automates deployment, scaling, and management of containerized applications. In LLM inference scenarios, it manages multiple inference service instances, implements load balancing, automatic failover, and resource scheduling.
vLLM / SGLang: Two major mainstream open-source inference engines. vLLM is a high-performance LLM inference engine developed by UC Berkeley, with its core innovation being PagedAttention technology—borrowing the virtual memory management concept from operating systems to store KV cache in blocks, reducing memory fragmentation and bringing GPU memory utilization close to 100%. SGLang (Structured Generation Language), developed by LMSYS, focuses on structured outputs and complex workflows, introducing RadixAttention mechanism that automatically detects and reuses common prefixes in prompts, particularly suitable for multi-turn conversations and Agent scenarios.
Dedicated GPUs: Teams pursuing ultimate performance and data isolation choose dedicated hardware.

Behind each choice lies a tradeoff between latency, throughput, cost, and controllability. There's no silver bullet, only tradeoffs.

Core Decision Variables for Selection

The original post distilled "what matters most" into six key metrics: latency, throughput, reliability, cost, scaling, and observability.

In real scenarios, these metrics often conflict. For example, increasing batch size to improve throughput typically sacrifices per-request latency; pursuing low cost with serverless may harm reliability due to cold starts. Serverless inference means paying only for actual compute used, with providers dynamically allocating GPU resources without user infrastructure management. However, cold start problems exist—the first request needs to load the model onto GPU (potentially taking 10-60 seconds), significantly impacting user experience. Engineers must determine priorities based on business scenarios—whether real-time conversation, batch processing, or Agent orchestration.

The Hardest Parts of Deploying LLMs in Production: Far Beyond Just Running the Model

Hidden Costs of Deployment and Integration

The original post specifically asks: "What's hardest about deployment and integration? What went wrong or took longer than expected?" This is precisely the most valuable part of community discussions.

According to widespread feedback from frontline engineers, the real challenges often aren't "making the model produce output," but rather:

GPU Resource Scheduling and Utilization: GPUs are expensive, and idle capacity is a massive cost sink. Balancing reserved resources with on-demand scaling between traffic peaks and valleys is an ongoing challenge.
Inference Engine Parameter Tuning: Parameters for vLLM and SGLang (max concurrency, KV cache size, quantization strategy) require repeated experimentation for specific models and hardware. KV cache refers to caching the Key and Value matrices of the attention mechanism in Transformer models to avoid redundant computation—a key technique for accelerating autoregressive generation. Quantization compresses model weights from FP16 to INT8 or INT4, reducing memory usage by 50-75% but potentially causing slight precision loss.
Missing Observability Systems: Unlike traditional web services, LLM inference monitoring involves unique metrics like token-level latency, Time To First Token (TTFT), and Time Per Output Token (TPOT). TTFT (Time To First Token) is the latency from request submission to generating the first token, determining user-perceived responsiveness—must be kept under 200ms for real-time conversations. TPOT (Time Per Output Token) is the average time to generate each subsequent token, determining streaming output smoothness. Off-the-shelf toolchains are still immature.

The Dilemma of Migration and Vendor Lock-in

The original post also asks "Have you switched providers/runtimes? What triggered the switch? What's preventing you from switching today?" This reveals a key reality: vendor lock-in exists in the LLM space too.

Once you've built a complete prompt engineering, caching, and monitoring system around a hosted API, migration costs skyrocket. What prevents switching is often uncertainty about new providers' reliability and performance. This includes concerns about API compatibility, data migration complexity, and new platform stability—in production, any insufficiently validated switch risks service disruption.

Cost, Security, and Trust: Three Barriers to Enterprise LLM Adoption

Demand for Transparent Inference Cost Structure

The original post bluntly asks: "Roughly how much do you spend on inference, including idle capacity?" and "When do you prefer serverless vs. dedicated GPUs?"

Behind this is enterprises' strong need for inference cost predictability. Cost calculation example: A service processes 1 million tokens during 3 peak hours and 100K tokens during 21 off-peak hours daily. With serverless ($0.50/million tokens), daily cost is $5.50; with dedicated A10 GPU ($1.50/hour), daily cost is $36, but peak latency is more stable. Serverless suits uncertain traffic with acceptable cold start latency; dedicated GPUs suit high-frequency, stable, latency-sensitive loads. True cost accounting must include idle GPU depreciation, or "cheap self-hosting" may actually cost more.

Hard Constraints of Security and Compliance

"What security/privacy requirements affect your choices?"—For regulated industries like finance and healthcare, data cannot leave borders or pass through third parties, directly determining they can only choose self-hosting or BYOC (Bring Your Own Cloud) solutions.

BYOC (Bring Your Own Cloud) is a deployment model where providers deploy software in customers' own cloud accounts, keeping data within customer environments while customers retain full control. For example: financial institutions are bound by GDPR and PCI DSS, healthcare by HIPAA, Chinese enterprises by the Data Security Law—these regulations all require sensitive data not leave specific jurisdictions or be transmitted to third parties without authorization. By running in customers' AWS, Azure, or Alibaba Cloud accounts, BYOC balances "software as a service" with "data stays local." This is why many enterprises accept higher operational costs to self-build inference platforms.

Key Elements for Building Trust

The original post's final questions are highly insightful: "What makes you trust a new provider or tool—benchmarks, free credits, SLAs, case studies, or BYOC?" and "Would you pay more for lower latency, higher reliability, or stronger control?"

SLA (Service Level Agreement) is a provider's commitment to availability, performance, and support response times, typically contractually bound with compensation clauses. Typical SLA metrics include: availability (e.g., 99.9% means monthly downtime under 43 minutes), API latency (e.g., P95 under 500ms), throughput guarantees, etc. Enterprise SLAs also include incident response times, data backup recovery time objectives, etc. SLA violations trigger service fee discounts or refunds, converting "trust" from subjective judgment to quantifiable metrics and economic incentives.

The answer is nearly certain: In production, the value of reliability and controllability far exceeds simply low prices. Business losses from a single outage may far exceed saved inference costs. Compared to consumer services' "best effort" promises, enterprise services require 99.95%+ availability SLAs, backed by redundant architecture, multi-region deployment, real-time monitoring, 24/7 operations teams, and massive investments.

Conclusion: A Product Requirements Document from the Community

This Reddit post appears to be research but actually outlines a complete pain point map for open-source LLM productionization. It reminds us: Competition in LLM deployment is shifting from "whose model is stronger" to "who can run models stably, economically, and securely."

For engineering teams, rather than chasing the latest models, first establish your evaluation framework: clarify business priorities for latency, cost, and compliance; rationally weigh self-hosting versus hosted services; and treat observability as a first-class citizen. For tool and platform vendors, this checklist is practically a ready-made product requirements document—whoever first solves GPU utilization, observability, and migration cost challenges will win the next wave of enterprise markets.