AI's Second Half: The Decisive Battle Shifts from Models to Infrastructure

From the Model Arms Race to the Infrastructure Race

Over the past few years, competition in the AI field revolved almost entirely around models themselves. Who had more parameters, who scored higher on benchmarks, who could be first to release the next flagship model—these became the core yardsticks of competitiveness. Every iteration of models like GPT, Claude, Gemini, and Llama sent shockwaves through the industry. However, as the capabilities of large models gradually converge and open-source models rapidly catch up with closed-source ones, the era of building a moat solely on model performance is coming to an end.

The real watershed appears in the face of a simple reality: possessing a powerful model is only the starting point. Whether you can deploy models into real business scenarios efficiently, reliably, and cost-effectively is what determines success or failure. The next era of AI is no longer just about "whose model is smarter," but about "whose infrastructure is stronger."

Why AI Infrastructure Has Become the New Battleground

Model Capabilities Are Rapidly Being Commoditized

When the capability gap between top-tier models shrinks to a point imperceptible to ordinary users, the scarcity of the models themselves diminishes. The deeper reason behind this trend lies in the high degree of standardization of the Transformer architecture.

The 2017 paper "Attention Is All You Need" by Google Brain fundamentally changed the deep learning landscape: the Transformer replaced the sequential dependency structure of RNN/LSTM with the Self-Attention mechanism, enabling models to process sequence data in parallel and dramatically improving training efficiency. The core innovation of self-attention is that it allows each position in a sequence to interact directly with all other positions, with a computational complexity of O(n²), but through parallelized GPU computation, the overall training throughput far exceeds traditional RNN architectures. Multi-Head Attention further allows the model to capture multiple semantic relationships simultaneously across different subspaces, while Positional Encoding compensates for the attention mechanism's inherent lack of positional awareness using sinusoidal functions or learnable vectors. It is precisely this highly modular and parallelizable architectural design that enables different institutions to rapidly reproduce and iterate on similar technical foundations. Notably, the Transformer was not originally designed specifically for language models but was proposed for machine translation tasks—this "accidental generality" precisely demonstrates the profundity of its architectural abstraction. It unified the sequence modeling problem into the fundamental operation of "attention allocation," allowing it to seamlessly transfer to heterogeneous domains such as text, images, audio, and even protein structure prediction, thereby establishing an almost unshakable architectural dominance.

Subsequently, OpenAI systematically demonstrated Scaling Laws in its 2020 paper "Scaling Laws for Neural Language Models"—that under the premise of jointly scaling data, compute, and parameter count, model performance exhibits a predictable power-law growth pattern. This discovery transformed AI research from an "art of trial and error" into an "engineering science," and also explains why major institutions competed to expand training scale. The Chinchilla law (DeepMind, 2022) further revised the optimal compute allocation ratio, pointing out that for a given compute budget, data volume and parameter count should scale proportionally, rather than overemphasizing parameter scale as in early practice, driving research toward more efficient training ratios. However, the diminishing marginal returns of Scaling Laws are becoming increasingly apparent—from GPT-3 to GPT-4, the increases in parameter count and compute investment have far outpaced the gains in performance. This is also the deeper driver behind the industry's shift toward alternative paths such as inference efficiency optimization, synthetic data generation, and reinforcement learning post-training (RLHF/RLAIF). RLHF (Reinforcement Learning from Human Feedback) trains a reward model to simulate human preference scoring, then fine-tunes the language model using reinforcement learning algorithms like PPO to make its outputs better align with human value judgments; RLAIF (Reinforcement Learning from AI Feedback) replaces human annotators with another large model, greatly reducing the data cost of alignment training. The rise of these two paths marks that "Post-Training" has become a capability-building stage as important as pre-training, further diluting the competitive advantage of simply piling on parameters.

Since 2017, the vast majority of mainstream large models have been built upon similar architectural paradigms, and the broad validation of Scaling Laws has made the path to capability improvement even more predictable. The flourishing of the open-source community has further accelerated the commoditization process: open-source models such as Meta's LLaMA series, Mistral, and Falcon continue to narrow the performance gap with closed-source models, and the Hugging Face platform now hosts over 500,000 models. LoRA (Low-Rank Adaptation) decomposes full-parameter updates into the product of two low-rank matrices, training only about 0.1%-1% of the parameters during fine-tuning; QLoRA builds on LoRA by introducing 4-bit NormalFloat quantization and double quantization techniques, compressing the memory requirement for fine-tuning a 65B-parameter model from over 780GB to about 48GB, enabling it to be completed on a single consumer-grade GPU. Such parameter-efficient fine-tuning techniques allow small teams to obtain domain-specific models at extremely low cost, further breaking down the technical barriers of top players. Today, a small or medium-sized team can fine-tune a model that meets specific needs based on open-source weights. When models gradually become "replaceable components," the center of gravity of the value chain inevitably shifts upstream and downstream, and AI infrastructure is precisely the core link that captures this portion of the value.

Inference Cost Determines Commercial Viability

Training a large model is certainly expensive, but for the vast majority of enterprises, what truly consumes resources continuously is inference. Every user request means real, tangible compute expenditure. When an AI product reaches millions or even tens of millions of daily active users, tiny differences in inference efficiency are sharply amplified into enormous cost gaps.

The performance bottleneck of large model inference is essentially the tension between memory bandwidth and compute density. Take the A100 GPU as an example: its peak FP16 compute is 312 TFLOPS, but its HBM2e memory bandwidth is only 2TB/s, meaning that loading the weights of a 175B-parameter model from memory alone requires about 175 seconds of pure bandwidth time—this "Bandwidth Wall" determines that the core direction of inference optimization is to reduce data movement rather than simply boosting compute. Understanding this bottleneck requires distinguishing the characteristics of two types of computational tasks: Compute-Bound operations (such as matrix multiplication) are limited by the FLOPS ceiling, while Memory-Bound operations (such as element-wise activation functions and LayerNorm) are limited by bandwidth—the latter dominates in large model inference. This is precisely why simply piling on compute often improves inference latency less than expected, while the FlashAttention algorithm, which focuses on reducing HBM accesses (by keeping intermediate results of the attention matrix in SRAM through tiled computation rather than repeatedly writing them back to HBM), can bring a 2-4x real-world speedup. A complete technology stack has formed around large model inference optimization: model quantization compresses weights from FP32/FP16 to INT8 or even INT4 formats. Among these, GPTQ minimizes quantization error layer by layer based on approximate second-order optimization (the Hessian matrix), while AWQ (Activation-aware Weight Quantization) maintains precision by protecting critical weight channels with significant activation values. Both can keep the performance loss within 1-2% at INT4 precision and reduce memory usage by about 75%. KV cache management targets the autoregressive generation characteristics of the Transformer attention mechanism, avoiding redundant computation by caching the Key-Value matrices of historical tokens—without caching, each newly generated token would require recomputing attention over all historical tokens, producing O(n²) redundant computation, which is especially fatal for scenarios handling 32K or even 128K long contexts. The PagedAttention proposed by vLLM borrows the paging concept of operating system virtual memory, managing the KV cache in fixed-size non-contiguous memory blocks and mapping logical addresses to physical addresses via a Block Table, reducing memory fragmentation from about 60% to less than 4% while supporting cross-request KV cache sharing (Prefix Caching), so that requests with the same system prompt need not be recomputed. Continuous Batching dynamically inserts new requests into the batch being generated at the iteration level, solving the GPU idle problem caused by varying sequence lengths in traditional static batching, boosting GPU utilization and throughput by several to over ten times compared to traditional static batching. Inference-specific chips represented by the Groq LPU adopt a deterministic compilation execution model, eliminating the scheduling overhead of GPU general-purpose architectures, while Nvidia H100's Transformer Engine is specifically optimized for Transformer operations through FP8 precision and hardware-level sparse matrix acceleration, representing the trend of specialization at the hardware level. The combined effect of these optimization methods directly determines the actual cost per thousand requests and together forms the lifeline that decides whether a product can be profitable.

Reliability and Scalability Are the Thresholds for Productization

Getting a demo running in the lab and supporting massive concurrent requests in a production environment are two completely different orders of magnitude. Latency, throughput, availability, fault recovery, elastic scaling—these capabilities repeatedly honed in the traditional cloud computing field are subject to even higher requirements in AI scenarios. Concepts such as SLO (Service Level Objectives) and error budgets defined by Google's SRE (Site Reliability Engineering) system are being introduced into AI service management, but the non-deterministic output of AI systems, the long-tail latency distribution (P99 latency is often dozens of times P50), and the high cost of elastic GPU resource scaling mean that traditional SRE methodologies need targeted adaptation. The P99 long-tail latency problem of AI inference services is particularly prominent: because the number of tokens generated by autoregressive generation is unpredictable (the same prompt may generate 50 to 5000 tokens under different temperature parameters), traditional capacity planning methods based on modeling response time distributions fail, requiring the joint control of tail latency through mechanisms such as output token budget-based request truncation (Max New Tokens limits), asynchronous streaming returns (Streaming), and speculative Early Stopping. Whoever can lower unit cost while guaranteeing service quality holds the initiative for scaled deployment.

The Four Core Layers of AI Infrastructure

The Compute and Hardware Layer

The bottom layer is chips and compute supply. The scarcity and high cost of GPUs have driven the entire industry into fierce competition for compute resources, while also giving rise to specialized hardware optimized for inference scenarios. The current high concentration of the GPU supply chain (TSMC handles the vast majority of advanced-process chip fabrication, and CoWoS advanced packaging capacity has become the core bottleneck for H100/H200 shipments) makes compute acquisition itself a strategic resource. CoWoS (Chip-on-Wafer-on-Substrate) is TSMC's advanced packaging technology that stacks and packages GPU dies and HBM memory on the same substrate via a Silicon Interposer, making the data bandwidth between HBM and the GPU dozens of times that of the traditional PCIe interface—the H100's HBM3 bandwidth reaching as high as 3.35TB/s is thanks to this. The scarcity of CoWoS capacity became a hard bottleneck for Nvidia's shipments in 2023, reflecting the extreme fragility of the AI compute race at the supply chain level. Meanwhile, the ASIC route represented by the TPU (Tensor Processing Unit), the wafer-scale integration route represented by the Cerebras WSE, and the Dataflow Architecture route represented by SambaNova are challenging GPU dominance from different dimensions. A vast technical and commercial ecosystem is forming around the scheduling, virtualization, and utilization optimization of compute.

The Model Serving and Orchestration Layer

Above compute lies the service layer responsible for efficiently running models. It needs to solve a series of engineering problems such as model loading, request routing, dynamic batching, and multi-model coexistence. Speculative Decoding is an important inference acceleration technique that has emerged in recent years: a small "Draft Model" rapidly generates a candidate token sequence, which is then verified in parallel by the large model. Leveraging the fact that verification by the large model is faster than generation, it boosts throughput by 2-3x without loss of output quality. The theoretical basis of speculative decoding is that the computational bottleneck of large model inference comes from memory access rather than arithmetic: verifying N candidate tokens requires almost the same memory access as verifying 1 token (weights need to be loaded only once), yet it can confirm outputs at multiple positions in parallel. Thus, the effective throughput gain is positively correlated with the Acceptance Rate of the draft model—typically a 70%-85% acceptance rate brings about a 2x speedup. A variant, Self-Speculative Decoding, further generates drafts by skipping some Transformer layers, avoiding the engineering complexity of maintaining a separate draft model. In multi-model coexistence scenarios, Time-Multiplexing and Space-Multiplexing strategies need to be dynamically weighed according to model size, request priority, and SLA requirements. The maturity of this layer directly determines how much actual capacity can be squeezed out of the same hardware, and is key to model deployment efficiency.

The Data and Retrieval Layer

With the popularization of technologies such as RAG (Retrieval-Augmented Generation), vector databases, and context management, infrastructure around data is becoming increasingly important. RAG was proposed by Meta AI in 2020, and its core insight comes from distinguishing between two ways large models store knowledge: Parametric Knowledge (solidified in model weights, subject to timeliness problems and costly to update) and Non-Parametric Knowledge (stored in external knowledge bases and updatable in real time). RAG combines the advantages of both by dynamically retrieving relevant documents and injecting them into the context at inference time, effectively solving the knowledge timeliness and hallucination problems of large models—the latter stemming from the model confusing statistical co-occurrence relationships with semantic facts during training, while the verifiable source text provided by external retrieval can significantly constrain the model's generation space.

In terms of technical implementation, documents are first transformed into high-dimensional vectors (typically 512-3072 dimensions) via an Embedding model. This process maps semantically similar text to adjacent positions in the vector space, stores them in a vector database, and builds ANN (Approximate Nearest Neighbor) index structures such as HNSW (Hierarchical Navigable Small World graph, achieving logarithmic retrieval complexity through a multi-layer skip-list structure) or IVF (Inverted File Index, partitioning the vector space through clustering to accelerate retrieval), enabling the recall of semantically relevant documents from billions of vectors within millisecond-level latency. Specialized solutions such as Pinecone, Weaviate, Milvus, and Chroma, along with PostgreSQL's pgvector extension, each occupy a place, with differences mainly reflected in dimensions such as indexing algorithms, distributed architecture, filtering performance, and operational complexity. Notably, vector databases face the dual challenges of the "Curse of Dimensionality" and precise filtering efficiency in Metadata Filtering scenarios: when users need to perform compound queries based on both vector semantic similarity and structured attributes (such as date ranges and document categories), Pre-filtering may drastically reduce the candidate set of the ANN index leading to lower recall, while Post-filtering may waste significant retrieval compute. How to balance between the two is a core competitive point of architectural differentiation among vendors. Current frontier evolution directions of RAG include hybrid retrieval (dense vectors + BM25 sparse retrieval, fusing the two results via RRF or linear weighting), multi-hop reasoning retrieval (iteratively decomposing complex questions and retrieving multiple times), and Microsoft's proposed GraphRAG (a knowledge-graph-based RAG variant that parses documents into entity-relationship graphs, supporting structured reasoning across documents), further improving accuracy in complex Q&A scenarios. The quality of a model's output largely depends on the context fed to it, and how to efficiently store, retrieve, and update this data is itself a complex system requiring dedicated infrastructure support.

The Orchestration and Toolchain Layer

The layer closest to applications is the orchestration framework that links together models, data, and external tools, along with supporting tools for monitoring, evaluation, and observability. The emergence of LangChain in late 2022 defined three core abstractions: Chain (a directed graph that sequentially calls LLMs and tools), Agent (dynamically deciding the sequence of tool calls based on paradigms such as ReAct/Plan-and-Execute), and Memory (short-term context window management and long-term vector storage state persistence). LlamaIndex, meanwhile, focuses on the refined management of data indexing and retrieval, introducing more granular abstraction levels such as Node Parsing, index type selection, and query routing. Together, the two laid the foundation for the "Chain" and "Tool Use" paradigms. The ReAct (Reasoning + Acting) paradigm alternately generates "Thought" and "Action" steps, explicitly decoupling the large model's reasoning process from tool calls, making the Agent's decision-making process traceable and debuggable; while the Function Calling mechanism, formally introduced into the API by OpenAI in 2023, guides the model to output program-parsable JSON-format call instructions by injecting a structured tool description Schema into the prompt, narrowing the gap between natural language intent and code execution down to the structured parsing level, becoming a key infrastructure for Agent engineering. Multi-Agent frameworks such as AutoGen and CrewAI further introduce inter-agent collaboration and role division mechanisms (such as the Planner-Executor-Critic triangular architecture), enabling tasks to be decomposed in parallel and coordinating the information flow among multiple agents via a Message Bus or shared state (Blackboard).

As AI moves from single calls to multi-step Agent workflows, entirely new engineering challenges emerge: state persistence requires reliably saving intermediate results across long-running tasks spanning minutes or even hours, and coping with process crashes and network interruptions; idempotency design requires that the same step produces no side effects (such as sending duplicate emails or writing to a database repeatedly) when retriggered due to timeouts or errors; determinism guarantees require Agents to have auditable and rollback-capable abilities at critical decision points (such as code execution and file writing), which is especially difficult given the inherently random nature of large model outputs; multi-agent scenarios must also address Race Conditions and distributed consistency issues, where the CAP theorem constraints of traditional distributed systems still apply. Observability tools such as LangSmith, Langfuse, and Helicone make originally black-box Agent behavior debuggable and quantifiable through OpenTelemetry-standard trace linkage tracking, span-level latency analysis (precise to each LLM call, tool call, and retrieval operation), prompt version management and A/B testing, as well as token consumption and cost attribution analysis—this is the necessary infrastructure for AI systems to move from experiment to production, and a direct response to this demand.

Implications for Entrepreneurs, Technical Decision-Makers, and Engineers

This structural transformation means vastly different opportunities and challenges for different roles.

For entrepreneurs, rather than confronting head-on in the model field already heavily guarded by giants, it is better to deeply cultivate a specific niche of AI infrastructure. Directions such as inference optimization, cost management, Agent orchestration, and evaluation tools still hold vast unmet needs. These "selling shovels" businesses often have clearer business models and more solid moats. It is worth noting that competition in the infrastructure field is equally fierce: open-source alternatives (such as vLLM to commercial inference services) will continue to compress commercialization space, and differentiated moats often come from enterprise-grade integration capabilities, compliance certifications, and proprietary data flywheels, rather than pure technical leadership.

For enterprise technical decision-makers, the focus of selection is shifting from "which model to use" to "how to build a sustainably evolvable AI infrastructure." Models may keep being replaced, but the underlying data pipelines, service architectures, and operational systems should possess sufficient abstraction capability and flexibility to adapt to the rapidly changing model ecosystem. Building a Model-Agnostic abstraction layer and avoiding deep binding to the proprietary API features of a single model provider are key architectural decisions for reducing migration costs. Specifically, separating and encapsulating Embedding generation, prompt templates, and model invocation logic means that switching the underlying model only requires modifying the adaptation layer rather than rewriting business logic—this is how this principle manifests in engineering practice; whereas for systems that have deeply customized proprietary features such as OpenAI Function Calling, the actual cost of migrating to another provider often far exceeds expectations. This lesson is prompting more enterprises to list "vendor lock-in risk" as an evaluation dimension during the architecture review stage.

For individual engineers, mastering infrastructure-related skills such as model deployment, performance optimization, and distributed systems is rapidly rising in market value. Being able to call an API is no longer scarce; the engineering ability to truly run AI systems stably, fast, and economically—including CUDA programming and GPU performance analysis, distributed training and inference framework tuning, and building observability for AI systems—is the long-term core competitiveness.

Conclusion

The first stage of AI was about exploring "what models can do," while the next stage is about the engineering practice of "how to use models reliably and economically." As the spotlight gradually shifts from dazzling model launch events to the less sexy but crucial infrastructure, the maturity of the industry is quietly improving. Every technological wave in history has matured along a similar trajectory: the internet era shifted from the browser wars to the competition over CDN, databases, and cloud infrastructure; the mobile internet era shifted from the operating system battle to the contest over app distribution and payment infrastructure. This transformation of the AI era is both the natural result of technological maturity and the inevitable choice as commercialization pressure forces the return of engineering rationality. The true winners of the future may not be the companies with the strongest models, but the players who build the most efficient AI infrastructure.

Key Takeaways