Build or Buy LLM Infrastructure? A 7-Month Hands-On Retrospective Reveals the Hidden Costs

A Real Lesson Learned Over 7 Months

In the wave of large language model (LLM) application deployment, almost every engineering team faces an unavoidable decision: should we build our LLM infrastructure from scratch, or buy an existing platform?

An engineer shared his 7-month journey on Reddit. He built LLM infrastructure from the ground up, and is now reflecting on whether he should have simply purchased a mature platform from the very beginning. Behind this seemingly simple either/or question lies a level of complexity that most teams don't realize until they're already deep in the weeds.

This article draws on his hands-on retrospective to outline when building vs. buying makes sense, break down the grossly underestimated engineering costs, and provide a side-by-side overview of the major tools on the market today.

When Does It Make Sense to Build LLM Infrastructure?

Building isn't without merit — in certain scenarios, it's the only right choice. Based on the original poster's summary, the following situations lean toward building:

Hard Compliance and Data Security Requirements

When data cannot leave your VPC (Virtual Private Cloud) and strict compliance requirements are in play, handing data to a third-party platform is simply not an option. Highly regulated industries like finance, healthcare, and government often fall into this category — data sovereignty trumps everything.

VPC (Virtual Private Cloud) is a core concept in cloud computing, referring to a logically isolated private network space carved out for users on public cloud infrastructure. Users can customize IP address ranges, subnets, routing tables, and network gateways, and data traffic is never mixed with that of other tenants. In highly regulated industries — such as SOC 2 and PCI DSS compliance in finance, HIPAA regulations in healthcare, and GDPR in the EU — there are strict requirements governing where data is stored, how it's transmitted, and who can access it. Once data is sent to a third-party LLM platform's API endpoint, it may traverse external networks, be stored on external servers, or even be used for model training — all of which could constitute compliance violations. Therefore, "data not leaving the VPC" means the entire inference process — from the input prompt to the model's output — must be completed within the enterprise's own controlled network boundary.

Token Volume Is Large Enough

When token consumption reaches a certain scale, the cost of calling commercial APIs directly becomes unsustainable. At that point, building your own inference infrastructure and self-hosting models can actually drive down per-unit costs over the long term.

Understanding this requires grasping the economics of tokens. A token is the basic unit of measurement for LLM text processing. Large models don't process text directly — they first use a tokenizer to split text into tokens. In English, one token corresponds to roughly 4 characters or 0.75 words; in Chinese, a single character typically maps to 1–2 tokens. Commercial APIs (such as OpenAI's GPT-4 or Anthropic's Claude) charge based on the number of input and output tokens, with output tokens typically priced 2–4x higher than input tokens. When a company's daily call volume reaches millions or even billions of tokens, API costs skyrocket. For example, GPT-4 Turbo's input pricing is approximately $10 per million tokens — an application processing 1 billion tokens per day could face API costs exceeding $10,000 daily. At that scale, purchasing GPU servers to self-host open-source models (like Llama 3, Mistral, etc.) involves significant upfront investment, but the per-unit inference cost can drop by an order of magnitude, offering substantial long-term economic advantages.

Need to Fine-Tune on Proprietary Data

If your core business depends on fine-tuning on proprietary data and you don't want that data exposed to any external platform, building is practically the only option. The model's capabilities themselves become your moat.

Fine-tuning refers to continuing the training of a pre-trained large model using a domain-specific or task-specific dataset, significantly improving the model's performance in that domain. Related technical approaches include full fine-tuning, LoRA (Low-Rank Adaptation), and QLoRA — all parameter-efficient fine-tuning methods. Full fine-tuning requires updating all model weights and demands extremely high GPU memory (fine-tuning a 70B parameter model may require hundreds of GB of VRAM), while LoRA freezes the original weights and trains only low-rank decomposition matrices, drastically reducing resource requirements. The core value of fine-tuning is this: while general-purpose large models are powerful, in specific professional scenarios (such as legal text analysis, medical imaging report generation, or enterprise internal knowledge Q&A), models fine-tuned on domain data often achieve far superior accuracy and consistency compared to general models. This is why the author says "the model's capabilities themselves become your moat" — fine-tuning data and training methodologies are competitive barriers that are difficult to replicate.

When Should You Just Buy an LLM Platform?

In contrast to building, the advantages of buying a mature platform lie in speed and focus. The author believes buying is the better deal in these situations:

Your team doesn't have MLOps engineers: Without dedicated machine learning operations capability, forcing your way through infrastructure will only slow everything down. MLOps (Machine Learning Operations) applies DevOps principles to machine learning systems. MLOps engineers are responsible for the entire model lifecycle: building training data pipelines, model training and version management, model packaging and deployment, scaling inference services up and down, and monitoring and alerting in production environments. This role requires competence across three domains simultaneously — software engineering (containerization, CI/CD, microservices architecture), infrastructure operations (Kubernetes, GPU cluster management), and machine learning (model evaluation, data drift detection) — making it a scarce talent in the market. Teams without MLOps engineers who attempt to build LLM infrastructure typically have backend engineers fill in, but these engineers usually lack experience in model serving, GPU resource scheduling, and inference performance optimization, leading to frequent issues at scale such as latency jitter, OOM (Out of Memory) errors, and model version chaos.
You need to ship fast: The business priority is delivering a product ASAP, not polishing underlying infrastructure.
Your use case isn't a competitive differentiator: If you're building relatively standardized scenarios like RAG, text summarization, or chatbots, and owning the infrastructure itself won't give you any competitive edge, then building is just reinventing the wheel. RAG (Retrieval-Augmented Generation) is one of the most popular architectural patterns in LLM applications today. The core idea is to retrieve relevant content fragments from external knowledge bases (such as company documents, databases, or web pages) before sending the user's question to the LLM, then append those fragments as context to the prompt so the LLM generates answers grounded in that context. RAG addresses two key LLM pain points — knowledge cutoff limitations and hallucination. A typical RAG pipeline includes document chunking, embedding (vectorization), storage in a vector database (such as Pinecone, Weaviate, or Milvus), semantic retrieval, reranking, and final generation. Although RAG has become a relatively standardized pattern, there are still numerous engineering details to tune in practice.

The bottom line: When infrastructure isn't your source of differentiation, you shouldn't be investing precious engineering resources in it.

The Grossly Underestimated Hidden Engineering Costs

This is the most valuable part of the entire post — and the root cause of most teams falling into the trap.

The author makes a sharp observation: many people think "building LLM infrastructure" is just connecting to a model API and writing some calling logic. But once you're actually in it, you discover that a complete production-grade infrastructure encompasses a multitude of independent subsystems:

Routing Logic: How to intelligently distribute requests across multiple models/providers. LLM routing goes far beyond simple load balancing. In real production environments with multiple models and providers, the routing system needs to make real-time decisions across multiple dimensions: request complexity (use smaller models for simple questions, larger models for complex reasoning to optimize cost), real-time latency and availability of each provider (automatically switch when an API endpoint slows down), capability differences of each model on specific tasks (Claude for code generation, GPT-4 for data analysis, etc.), token cost budget constraints, and rate limit management. More sophisticated routing strategies also involve semantic routing — automatically selecting the most appropriate model or processing pipeline based on the semantic content of user input. A robust routing system also needs to handle retry logic, request queuing, priority sorting, and cross-region scheduling — its engineering complexity rivals that of an independent microservice gateway project.
Fallback Handling: Backup strategies for when the primary model is unavailable or times out.
Prompt Version Management: Iteration, rollback, and tracking of prompts.
Cost Tracking: Precisely accounting for spending per call, per user, and per feature.
Evals Pipeline: An automated system for continuously measuring model output quality. The evaluation pipeline is one of the most easily overlooked yet most critical components of an LLM production system. Unlike unit tests in traditional software, LLM outputs are non-deterministic — the same input can produce different outputs at different times, and the standard for "correct" is often ambiguous. The evaluation pipeline must address several core issues: how to define evaluation metrics (accuracy, relevance, safety, format compliance, etc.), how to build high-quality evaluation datasets (golden datasets), how to implement automated evaluation (commonly using the LLM-as-Judge approach, where another powerful model judges the target model's output quality), and how to perform regression testing when models are swapped or prompts are changed. Without a reliable evaluation system, teams cannot quantitatively answer the fundamental question "Did switching models make things better or worse?" — and all optimization becomes a shot in the dark.

The author emphasizes: "Each of these is not a weekend side project — each one is an independent engineering project in its own right." Worse still, most teams only realize these costs after they've already committed to building, at which point they're in too deep to turn back.

This is exactly what the title means by "too late" — it's already too late. During the decision-making phase, they only saw the tip of the iceberg. The real workload was hiding beneath the surface.

Side-by-Side Comparison of Major LLM Infrastructure Tools

To help those who come after, the author provided practical commentary on the major tools in the LLM infrastructure space. These assessments come from frontline practice and are highly valuable as reference.

orq.ai

Integrates routing, prompt management, observability, and evaluation into a relatively comprehensive all-in-one solution. The downside is that it's newer, and its third-party integration ecosystem is still catching up.

LangSmith

Tracing and observability are well-executed, but prompt management features are underdeveloped. The author considers it "designed more for engineers than for cross-functional teams" — meaning the barrier to entry is higher for non-technical roles like product managers and operations staff.

Helicone

Easy to get started with, great visualization, and low onboarding cost. However, its weakness lies in limited functionality — it essentially stays at the observability layer with limited ability to extend beyond that.

Portkey

Focuses on routing and reliability, excelling in this area. But governance and evaluation depth are only average.

LiteLLM

Open-source and flexible, offering a high degree of freedom. But the author cautions: the workload of self-hosting is much larger than it appears, and enterprise-level support is limited. Open-source doesn't mean free — operational costs need to be factored into the total bill.

How to Make the Right Build vs. Buy Decision

Synthesizing this retrospective, we can distill a pragmatic decision framework:

First, ask about differentiation. Does infrastructure constitute your competitive advantage? If the answer is no, lean toward buying.

Second, assess compliance and scale. Must data stay on your internal network? Is token volume large enough that API costs are unacceptable? Only when these hard constraints are met does building have a solid justification.

Third, honestly evaluate your team's capabilities. Choosing to build without MLOps engineers is like pushing your team into a bottomless pit that continuously drains resources.

Fourth, don't underestimate the "supporting engineering." Routing, fallback handling, evaluation, cost tracking — factor all of these into your TCO (Total Cost of Ownership), then compare against platform subscription fees. The conclusion will often flip.

TCO (Total Cost of Ownership) is a classic analytical framework in enterprise IT decision-making, and it's especially important in the build vs. buy decision for LLM infrastructure. The TCO of a build approach includes not only explicit costs — such as GPU servers or cloud GPU instance fees and engineer salaries — but also a large number of hidden costs: the opportunity cost of engineers being distracted from core product development, business interruption losses caused by system failures, ongoing investment in patching security vulnerabilities, long-term accumulation of technical debt, and the risk of knowledge loss due to staff turnover. A common misjudgment is that teams only account for the initial build hours when estimating self-build costs, while ignoring the ongoing maintenance, upgrade, and scaling costs after the system goes live — which often account for 60%–80% of total engineering investment. By contrast, platform subscription fees, while more visible on the balance sheet, are predictable and fixed, and they shift a large portion of operational responsibility to the platform provider.

More often than not, "buying" isn't a compromise born of insufficient capability — it's a rational choice to focus on core value. True engineering wisdom lies in knowing what to do yourself and what to hand off to others.