Inside the LLMD Project: How Open Source Inference Achieves Production-Grade Scale

As open source AI models grow increasingly powerful, deploying them efficiently and reliably in production has become a real challenge for many teams. Carlos Costa, an IBM Research Distinguished Engineer and co-founder of the LLMD project, shared the project's latest progress ahead of the upcoming PyTorch Conference North America in San Jose—an open source project aimed at bringing open source inference to production-grade scale.

From Experiment to Production: LLMD's Path to Maturity

Just over a year after its founding, the LLMD project has moved from early exploration into a new stage of maturity. According to Carlos Costa, the project can now deliver state-of-the-art performance for agentic serving while supporting the key capabilities required for day-to-day operations, including multi-tenant quality of service (QoS) and zero-downtime operations.

These capabilities are critical for enterprise-grade AI deployments. Traditionally, open source inference solutions have struggled to deliver both performance and functionality: either sacrificing manageability for low latency, or sacrificing throughput for stability. LLMD's goal is to break this trade-off, enabling open source models to carry large-scale, multi-tenant production workloads without compromising performance or productivity.

LLMD stands for "LLM-D," short for Large Language Model Deployment, an open source project jointly initiated by IBM Research, Red Hat, and other teams. It is not an inference engine itself, but a distributed inference orchestration framework for Kubernetes. Its design approach uses high-performance inference engines such as vLLM as execution backends, organizing GPU resources across multiple machines into a unified service plane through scheduling, routing, and caching layers. LLMD draws heavily on Kubernetes' mature patterns from the microservices world—declarative configuration, controller loops, autoscaling—and adapts them to LLM inference, a scenario extremely sensitive to latency and memory. The industry typically compares it with projects like NVIDIA's Dynamo and KServe, all focused on how to simultaneously optimize GPU utilization and end-to-end latency under multi-node, multi-model, multi-tenant conditions. Understanding this positioning helps clarify why LLMD emphasizes "production-grade" rather than simply chasing single-inference speed.

Performance Breakthroughs in Agentic Serving

With the rise of AI agent applications, inference services face new challenges. Agents typically need to call models frequently, handle multi-turn conversations, and execute complex tasks, placing higher demands on inference engines in terms of concurrency, latency control, and resource scheduling.

LLMD achieves state-of-the-art performance in agentic serving, meaning it can better support the efficient operation of agent workloads. The project is said to have helped many adopters in Greece's open source AI community make better use of their AI infrastructure.

Agentic serving refers to the inference service model provided for AI agent workloads, which is fundamentally different from traditional single-turn question-answering inference. An agent may initiate dozens or even hundreds of model calls in a single task, alternating between planning, tool calling, observing results, and re-reasoning, forming what is known as the "think-act-observe" loop. Such workloads have three notable characteristics: heavy interdependence between requests, unpredictable output length per request, and the need for KV Cache to be reused or quickly switched across multiple calls. As a result, inference engines must support prefix cache sharing, fine-grained continuous batching, and priority-based preemptive scheduling—otherwise either memory is wasted badly or long-tail latency spirals out of control. The metrics for measuring agentic serving performance are no longer simply tokens/s, but task completion time, cost per task, and the number of concurrent agents. When LLMD claims state-of-the-art, it means leading on these new metrics.

Multi-Tenant QoS and Zero-Downtime Operations

Beyond performance, LLMD also tackles operational challenges in production environments. Multi-tenant quality of service guarantees allow different users or teams to share the same inference infrastructure without interfering with each other; zero-downtime operations ensure the system continues serving during upgrades, scaling, or failure recovery.

These features are key to whether enterprises hoping to make AI inference a core business support can truly deploy it. Carlos Costa emphasized that all these capabilities are achieved without sacrificing performance or productivity, reflecting LLMD's deep architectural considerations.

Multi-tenant QoS (Quality of Service) in inference scenarios means: when multiple users, teams, or applications share the same GPU pool, the system can provide differentiated service level guarantees for different tenants. This typically requires three layers of mechanisms: first, admission control and quota management to prevent one tenant from occupying all concurrency; second, priority-based request scheduling so that high-priority tenants' requests get lower queueing latency; third, resource isolation through memory quotas, time slices, or dedicated instances to avoid interference. Zero-downtime operations requires completing model version switches, node scaling, and failure recovery without interrupting in-flight requests, with common techniques including traffic canarying, connection draining, and rolling replica replacement. Together, these two capabilities determine whether an LLM inference platform can move from "usable" to "trusted for core business," and they were also the most obvious weaknesses of open source solutions compared with commercial products in the past.

Community Collaboration and Future Outlook

LLMD's progress is not isolated. Carlos Costa revealed that in the PyTorch Conference keynote, IBM will join forces with Red Hat and Google to showcase the next generation of LLMD capabilities. This cross-company open source collaboration model is a microcosm of the continued prosperity of the open source AI ecosystem.

For developers interested in open source inference, agent deployment, and AI infrastructure, LLMD's evolution is worth following. It seeks to answer a core question: can open source models meet the demanding requirements of performance, reliability, and operability at production scale? Based on current progress, the answer is becoming increasingly affirmative.

PyTorch Conference North America is an annual technical conference hosted by the PyTorch Foundation and supported by the Linux Foundation, usually held in San Jose, focusing on the latest advances in the PyTorch ecosystem in training, inference, compilation, and distributed systems. Keynotes are generally presented jointly by core contributing companies and foundation members, and are therefore often used as a stage for joint announcements of major open source projects. This arrangement of IBM, Red Hat, and Google sharing the stage carries symbolic significance in the open source AI field: it shows that standardization of LLM inference is no longer dominated by a single vendor, but tends toward forming cross-cloud, cross-hardware public infrastructure under open governance. For developers, this kind of collaboration often means better framework compatibility, less vendor lock-in, and faster upstream feature contributions.

One-line Summary

LLMD brings open source inference to production-grade scale, combining agent performance, multi-tenant QoS, and zero-downtime operations.

Paragraph Summary

Carlos Costa, IBM Research Distinguished Engineer and co-founder of the LLMD project, shared the latest progress of this open source inference project ahead of PyTorch Conference North America. Just over a year old, LLMD has moved from early exploration to maturity, delivering state-of-the-art performance for agentic serving while supporting multi-tenant quality of service and zero-downtime operations—key production capabilities. While traditional open source inference solutions often force a trade-off between performance and manageability, LLMD aims to let open source models carry large-scale, multi-tenant production workloads without sacrificing performance or productivity. The project has also helped adopters in Greece's open source AI community make better use of their AI infrastructure. IBM will join Red Hat and Google at the conference to showcase next-generation LLMD capabilities, highlighting the value of cross-company open source collaboration.