China Merchants Bank in Practice: K8s + KEDA Powers a Unified Training-Inference Tidal Scheduling Architecture

The Core Tension in Unified Training-Inference: Fundamentally Different Resource Needs

Deploying large models in production reveals a natural conflict between training and inference compute demands. In their KubeCon presentation, the IT department of China Merchants Bank (CMB) cut straight to the pain point: training jobs require stable, sustained compute resources — any interruption is costly — while inference services need flexible, metric-driven elastic scaling to handle fluctuating online traffic.

Maintaining separate resource pools for each workload type inevitably leads to idle capacity and wasted cost. CMB's goal was clear: run both training and inference workloads simultaneously on a single shared compute resource pool, allowing compute to flow freely between the two.

The platform runs on a large-scale heterogeneous compute pool made up primarily of three types of domestic AI accelerators, hosting models across different parameter scales including Qwen, DeepSeek^Ad, GLM, and MiniMax. This heterogeneous, multi-model environment places elevated demands on the scheduling architecture.

Architecture Design Based on CNCF Open-Source Projects

CMB ultimately chose to build its unified training-inference foundation entirely on the CNCF open-source ecosystem, rather than adopting an integrated appliance solution. The overall architecture can be understood through two pipelines: training and inference.

Training Pipeline: From Quota Admission to Data Warm-Up

When a user submits a job from the training platform, it first reaches Kueue for admission control and quota allocation. Once the quota check passes, the Kubernetes native Scheduler works with HAMi to allocate compute resources and spin up Pods.

In parallel, Fluid pre-warms the datasets and model weights needed for training, loading them into DRAM close to the compute nodes to minimize I/O wait times. Once everything is ready, the training framework inside the Pod — such as TorchElastic or MS-SWIFT — begins training.

Kueue is a Kubernetes-native batch job queue manager designed for resource-intensive batch workloads like AI/ML training. It implements multi-tenant quota management through a two-level abstraction: ClusterQueue defines cluster-level resource quotas (e.g., total GPU count), while LocalQueue distributes quotas to individual namespaces (teams or business lines). When resources are available, Kueue admits jobs and triggers actual Pod creation; when resources are constrained, jobs are suspended in the queue — avoiding over-scheduling, resource preemption, and fragmentation. HAMi (Heterogeneous AI Computing Virtualization Middleware) is a device plugin designed for heterogeneous GPUs and NPUs. It supports slicing a single accelerator card by memory or compute ratio across multiple Pods, addressing fine-grained allocation of domestic heterogeneous accelerators in Kubernetes environments. Together, these tools enable resource management that goes beyond "do we have a card?" to "exactly how much compute do we have?"

Fluid is a CNCF sandbox project — a data orchestration and acceleration framework for cloud-native environments. Through Dataset and Runtime CRD abstractions, it accelerates data from remote storage (object storage, HDFS, etc.) into distributed caches on memory or NVMe close to compute nodes. In large model training scenarios, model weight files often exceed tens of gigabytes; pulling them from remote storage on every job start causes significant I/O wait and drags down actual GPU utilization. Fluid's warm-up mechanism pre-loads data into local cache before the training job is scheduled, so the training framework enters the compute phase almost immediately after startup. Fluid also supports data-affinity scheduling — preferentially placing Pods on nodes that already have cached data — further reducing cross-node data transfer overhead.

Inference Pipeline: Prefill/Decode Disaggregation and Elastic Scaling

On the inference side, the internal MaaS platform handles all user-facing traffic. The platform deploys Prefill/Decode (PD) disaggregated instances in an RBG-like fashion, or directly launches single-node or multi-replica small-model instances using native K8s scheduling policies. Specific inference instances are served by vLLM or SGLang.

The monitoring layer collects instance metrics via Prometheus, which are then consumed by KEDA to configure autoscaling — supporting both cold-start scale-from-zero and horizontal scale-out from 1 to N.

PD disaggregation (Prefill/Decode separation) is an important architectural pattern for optimizing LLM inference. LLM inference consists of two phases: the Prefill phase processes the full input prompt and computes the KV Cache in one pass — compute-intensive but short; the Decode phase auto-regressively generates output token by token — memory-bandwidth-intensive and longer in duration. Deploying them on separate instances allows independent scaling and optimization tailored to each bottleneck, preventing mutual interference that would otherwise degrade GPU utilization. KEDA (Kubernetes Event-Driven Autoscaler) is a CNCF graduated project that extends Kubernetes native HPA to support autoscaling triggered by external metrics (e.g., Prometheus metrics, message queue depth) and natively supports scale-to-zero with on-demand cold starts — making it an ideal fit for the tidal traffic patterns of inference services. TPOT (Time Per Output Token) is a core metric for measuring LLM inference throughput quality, directly reflecting the smoothness of streaming output as perceived by the user. Using it as the autoscaling trigger threshold is far more representative of actual inference service load than CPU or memory utilization.

Tidal Scheduling: Compute Flows Freely Between Training and Inference

The essence of the architecture lies in the word "tidal." In their demo, CMB illustrated this with a two-card resource pool: in the initial state, one card runs training and the other runs inference.

Inference instances are configured with autoscaling metrics based on TPOT (Time Per Output Token). When load is applied to inference Pods and request volume rises, KEDA detects the metric threshold breach and triggers a scale-out event — the system reclaims the compute resource from the training job and transfers it to inference, spinning up an additional inference Pod to serve traffic.

This scheduling logic aligns naturally with real business rhythms: during peak hours (typically daytime), training resources are yielded to inference to serve more online users; during off-peak hours (e.g., overnight), inference resources are released to run training jobs. Compute flows back and forth between the two modes like tides, significantly improving overall utilization.

Why They Chose Open-Source Over Integrated Appliances

CMB evaluated integrated solutions early in their selection process. While such solutions can solve problems quickly in the short term, they have a critical weakness: when the upstream training or inference tech stack evolves, a closed, integrated solution is difficult to adapt accordingly.

Large model technology iterates at a rapid pace — inference engines like vLLM and SGLang, along with training frameworks, are all updating at high frequency. The modular combination of CNCF open-source projects (Kueue, HAMi, Fluid, KEDA, Prometheus, vLLM/SGLang) allows each component to be independently upgraded or replaced, ensuring the architecture remains evolvable over the long term. For a financial institution making long-term technology investments, this is the rational choice.

Quantified Results After One Year in Production

After approximately one year of building and operating the system, CMB delivered a concrete set of results:

• Core utilization increased from 35% to 60% — nearly doubling;
• Inference costs dropped by more than 60% year-over-year;
• Multi-tenant foundation reuse on the training side saved approximately 80% of training resources;
• Data acceleration technologies like Fluid significantly reduced model startup times for both training and inference jobs.

Critically, these gains were achieved without compromising service quality — throughout the entire process, inference services maintained performance benchmarks of TTFT under 5 seconds and TPOT under 50 milliseconds. This demonstrates that tidal scheduling does not trade user experience for cost savings; it delivers wins on both fronts.

Implications for the Industry

CMB's implementation offers a replicable reference model for AI infrastructure in financial services and large enterprises. Three core lessons stand out: first, acknowledge the fundamental resource differences between training and inference, and harmonize them with a unified foundation rather than isolated pools; second, use metric-driven autoscaling (e.g., based on TPOT) to align resource allocation more closely with actual load; third, commit to an open-source, composable technology path to preserve room for evolution in a rapidly iterating large model ecosystem.

For organizations with heterogeneous compute and utilization challenges, the combination of Kubernetes + KEDA + HAMi + Fluid is well worth a deep look.