The Five Unsolved Challenges of Production MLOps: A Deep Dive into GPU Scheduling, Fault-Tolerant Training, and Model Observability

The "Last Mile" Dilemma of MLOps

MLOps (Machine Learning Operations) is an engineering practice framework that extends DevOps principles to managing the lifecycle of machine learning systems. It spans the entire workflow of data management, model training, evaluation, deployment, monitoring, and iteration. Gartner listed it as one of the top ten strategic technology trends of 2023. Unlike the CI/CD pipelines of traditional software, the output quality of ML systems is jointly determined by three factors: data, code, and models. Among these, data and model states are difficult to version-control directly with Git the way code is—and this is precisely the core source of complexity that distinguishes MLOps from traditional DevOps, as well as the fundamental reason why so many "textbook answers" repeatedly fail in production environments.

When building production-grade machine learning infrastructure on Kubernetes, you quickly realize one thing: the "basic functions" like pipeline orchestration, experiment tracking, and model deployment all have mature tools to choose from. What truly keeps engineers up at night are the deeper problems that have no standard answers and no out-of-the-box solutions.

An engineer building production-grade ML infrastructure once posed a question on Reddit: setting aside the "which tool should I use" selection debate, which problems in production MLOps are genuinely unsolved? The question quickly struck a chord, because it touched on the most authentic pain points in the current field of ML engineering.

This article systematically examines these five hardcore challenges and analyzes why they remain so difficult to fully resolve.

Challenge One: Fault-Tolerant Training on Spot Instances

The Fundamental Conflict Between Cost and Stability

Spot (preemptible) instances are a billing model in which cloud providers sell their idle compute capacity at a discounted price—AWS calls them Spot Instances, Google Cloud calls them Preemptible/Spot VMs, and Alibaba Cloud calls them Preemptible Instances. The core mechanism is this: when the cloud provider needs to reclaim resources, it sends an interruption notice to the user about 2 minutes in advance, then forcibly terminates the instance. It is precisely this mechanism that makes their price typically only one-third or even less than that of on-demand instances. For large model training tasks that may require hundreds of GPUs running continuously for days, this cost gap is extremely tempting. But Spot instances can be reclaimed by the cloud provider at any time, which raises the core conflict: how do you avoid losing existing training progress when an instance is randomly preempted?

It is worth noting that there are subtle but critical differences in how each cloud provider delivers interruption signals. AWS Spot Instances deliver the 2-minute warning to user programs via the EC2 Instance Metadata Service (IMDS) and the EventBridge event bus; Google Cloud's Preemptible VMs are forcibly reclaimed after running for a maximum of 24 hours; while its Spot VMs use demand-based dynamic interruption similar to AWS. In large model training scenarios, engineering teams typically deploy an "Interruption Handler" daemon that listens for the cloud provider's interruption signal and, within the 2-minute window, attempts to trigger one final checkpoint save—this has become the de facto engineering convention, but its reliability heavily depends on whether I/O bandwidth is sufficient at that moment.

On the surface, the answer seems to be "save checkpoints frequently," but reality is far more complex:

Checkpoint I/O Overhead: Checkpoints for large models can easily reach hundreds of GB, and saving too frequently severely slows down the training pace, while object storage bandwidth may become a new bottleneck. Frameworks like DeepSpeed have introduced sharded checkpoint saving, which splits the model state into multiple smaller files written in parallel—alleviating this problem to some extent, but still not fundamentally solving it.
Distributed State Consistency: In scenarios that mix data parallelism, model parallelism, and pipeline parallelism, when a node is preempted, how do you consistently roll back and recover the state of the entire training cluster?
Elastic Training: The ideal scenario is that training automatically adapts to a new topology when the number of nodes changes. TorchElastic (now integrated into PyTorch's official distributed module torch.distributed.elastic) is exactly the elastic training framework Meta designed for this purpose. Its core abstraction decouples the "membership" of training processes from the "training logic," dynamically negotiating the set of participating nodes via a Rendezvous mechanism, allowing training jobs to automatically re-form the network and recover from the latest checkpoint when nodes are dynamically added or removed. In practical deployments, engineering teams typically pair it with etcd or Zookeeper as the Rendezvous backend. However, TorchElastic's elasticity is almost unusable in Pipeline Parallelism scenarios—because the number of nodes in each pipeline stage must be strictly fixed, and any node dropping out means the entire pipeline needs to re-plan its partitioning strategy. This remains quite fragile in real-world engineering, especially under mixed parallelism strategies, where the logic for handling topology changes is extremely complex, and the industry currently has no universal solution.

Although frameworks like TorchElastic and DeepSpeed provide partial capabilities, achieving truly "seamless fault tolerance with optimal cost" remains an open industry problem.

Challenge Two: Fair GPU Scheduling Across Teams

The "Politics" Behind Scarce Resources

GPUs are the most expensive and scarce computing resource within an enterprise. When multiple teams share the same GPU cluster, scheduling problems quickly evolve into challenges deeply intertwined with both technology and organization:

Fairness vs. Utilization: Strictly partitioning GPUs by quota leads to large amounts of idle resources; allowing over-borrowing makes it difficult to guarantee that high-priority tasks can promptly reclaim resources.
Gang Scheduling: Distributed training requires "either grab all N GPUs at once, or take none at all"—this strategy originated in parallel computing research in the 1980s. Its core insight is: for tightly coordinated parallel tasks, if the various tasks cannot obtain computing resources simultaneously, the communication waits between tasks will cause overall throughput to collapse. In the deep learning era, collective communication operations like All-Reduce require all participating nodes to synchronously reach a specific execution point, making this need even more pressing. Kubernetes' default scheduler kube-scheduler uses a per-Pod greedy strategy and inherently does not support this semantic: when resources are tight, it may successfully schedule some training Pods while the remaining Pods stay Pending due to insufficient resources, and the Pods already holding GPUs fall into idle waiting, forming a "partial occupancy deadlock" that wastes large amounts of resources.
The Cost of Preemption: Forcibly preempting a running training task means directly losing its training progress—which pulls the problem back to the fault-tolerance challenge itself.

To address these problems, Volcano (a CNCF sandbox project) introduces abstractions like Queue and PodGroup to natively support Gang Scheduling and batch semantics; Kueue is the Kubernetes community's next-generation job queue project; projects like Koordinator are also exploring finer-grained co-scheduling strategies; and Run:ai is a commercial solution providing enterprise-grade GPU scheduling and quota management. Nevertheless, compared with mature HPC job scheduling systems (such as SLURM), the scheduling maturity of cloud-native environments still lags noticeably behind, and the triangular constraint of "high utilization, fair allocation, and low scheduling latency" remains difficult to satisfy simultaneously.

Challenge Three: Reproducibility of Data and Models

The Invisible "Drift"

Many teams believe that introducing MLflow or DVC solves the reproducibility problem, but the most insidious challenges in production environments lie precisely in:

Multi-Dimensional Version Coupling: A single training result is jointly determined by the code version, data version, hyperparameters, random seed, and even the underlying dependency library versions. The engineering cost of fully locking down all variables is extremely high.
Training-Serving Skew: This is the number one culprit behind models that are "excellent during training but fall apart after deployment." Google listed it as a core source of ML technical debt in its classic paper Machine Learning: The High-Interest Credit Card of Technical Debt. Its technical root cause is that the training phase typically uses batch frameworks like Spark and Pandas to offline-process historical data to generate features, while online inference requires millisecond-level real-time computation—the two sets of code are often maintained separately by different teams. Subtle inconsistencies such as different normalization denominators, differences in categorical feature encoding order, and divergent null-handling logic can all cause silent degradation in online performance. Here is a typical example: offline training uses Python's Pandas to compute a user's average spending over the past 30 days, while online it uses Java to query Redis for real-time aggregation—subtle differences between the two pieces of code in timezone handling, null filling, and numerical precision all cause systematic shifts in feature values.

One of the core design goals of a Feature Store is precisely to eliminate this skew through unified feature logic that is "defined once, shared offline and online." A Feature Store is essentially a feature computation and serving middleware with dual storage backends: offline storage (typically columnar storage like Parquet/BigQuery) serves batch training data backfills, while online storage (typically low-latency KV stores like Redis/DynamoDB) serves real-time inference queries. Feast uses Python to define feature views and materializes them separately into offline and online storage, while Tecton further supports real-time computation of streaming features. However, introducing a Feature Store is not "free": it requires teams to abstract feature definitions from business code scattered across various places into a unified feature engineering layer, a migration effort that often takes months in legacy systems, while Tecton's commercial licensing costs also deter many small and medium-sized teams.
Data Drift Detection: After a model goes live, the distribution of input data slowly changes over time. How to automatically detect distribution shifts and trigger retraining still lacks a universal engineering paradigm.

Challenge Four: Model Observability and Debugging

The Failure Boundary of Traditional Monitoring Systems

The software engineering field has mature APM and logging systems, but "failures" in ML systems are often silent—the service reports no errors, the basic metrics look normal, yet the model's prediction quality is quietly degrading.

The Delayed Labels Problem: This is the core difficulty that distinguishes ML monitoring from traditional software monitoring. The true labels for online predictions are often delayed by days or even weeks—a credit scoring model needs to wait months for the actual default to occur, while a recommendation system needs to wait for the user's click or purchase behavior. During this time window, engineers can usually only rely on "Proxy Metrics" to indirectly infer the model's health: such as changes in the PSI (Population Stability Index) of the input feature distribution, or statistical distribution shifts in prediction scores.

PSI originally came from the field of financial credit scoring. Its essence is an approximation of the symmetric KL divergence between two probability distributions, and it typically uses the empirical thresholds of PSI<0.1 for a stable distribution, 0.1–0.2 for slight drift, and >0.2 for significant drift. However, PSI is sensitive to the choice of binning boundaries and is not sensitive enough to changes in the tails of the distribution. Academia has proposed stricter alternatives based on the Wasserstein distance (Earth Mover's Distance) and Maximum Mean Discrepancy (MMD)—the Wasserstein distance, rooted in optimal transport theory, better captures changes in the geometric structure of distributions; MMD maps distributions to a high-dimensional space via kernel functions and then compares mean differences, offering superior theoretical properties in high-dimensional feature scenarios. But the computational cost and scalability of these methods in high-dimensional feature spaces remain challenges, and how to set reasonable alert thresholds and avoid a flood of false positives still heavily depends on business domain knowledge.
Interpretability and Failure Attribution: When a prediction goes wrong, does the problem lie at the data, feature, or model level? The path to localization is long, often requiring troubleshooting across multiple systems one by one.
End-to-End Data Lineage Tracking: From raw data to the final prediction, spanning multiple systems and team boundaries, it is nearly impossible to fully reconstruct the complete chain when problems occur. Tools like Evidently AI, WhyLabs, and Arize AI focus on this area, and among them, Evidently AI also supports "unsupervised drift detection" based on changes in input feature distributions in business scenarios that lack timely feedback signals (such as loan default prediction). But there is currently no mature solution that is universal across all scenarios.

Challenge Five: Balancing Inference Cost and Latency

The New Battlefield After Large Models Go Live

As LLMs enter production environments at scale, engineering challenges on the inference side are becoming increasingly prominent:

GPU Utilization: A single inference request typically cannot fully saturate GPU compute power, requiring Dynamic Batching. Unlike static batching, which waits to gather a full batch of requests, Continuous Batching allows new requests to be inserted into existing batches in flight. Implemented by inference engines like vLLM and TensorRT-LLM, it can boost GPU throughput several-fold—but batching itself still introduces additional queuing latency, posing a challenge for latency-sensitive businesses.
KV Cache Pressure: The KV Cache (Key-Value Cache) is the core mechanism for inference optimization in the Transformer architecture. Its principle is to cache the Key and Value matrices of already-generated tokens in GPU memory, avoiding the need to recompute the attention weights of all preceding tokens each time a new token is generated, thereby reducing the time complexity of inference from O(n²) to O(n). However, taking a 70B-parameter LLM as an example, under a 32K context window, a single request's KV Cache may occupy several GB of memory, severely compressing the number of concurrent requests that can be served.

The PagedAttention technique proposed by vLLM is one of the most influential breakthroughs in the field of LLM inference optimization in 2023. Its core innovation borrows the paging management concept of operating system virtual memory, dividing the KV Cache into fixed-size "pages" and allocating memory on demand through a mapping table from logical blocks to physical blocks, eliminating the fragmentation waste caused by traditional contiguous memory allocation, and enabling the number of concurrent requests that can be served under the same memory to increase 2-4x. It is worth noting that PagedAttention's memory savings primarily come from eliminating fragmentation and pre-allocation waste, rather than compressing the data volume of the KV Cache itself—the latter requires independent techniques such as KV Cache quantization (e.g., FP8 quantization). Prefix Caching, on the other hand, targets scenarios where a System Prompt is shared, allowing multiple requests to reuse the KV Cache of the same prefix, further reducing redundant computation overhead.
Multi-Model Coexistence Scheduling: How do you efficiently host dozens or even hundreds of models on limited GPU resources, achieving on-demand loading and unloading? Layered offloading of model weights is a common strategy, but the Cold Start Latency of GPU instances is far higher than that of CPU services, which comes at a great cost in latency-sensitive scenarios.
Cost Predictability: LLM inference costs fluctuate dramatically with context length, making budgets difficult to estimate accurately and introducing uncertainty into business decisions.

Conclusion: Why These Challenges Remain Unsolved for So Long

Overall, the unsolved challenges of production MLOps share one common characteristic: they all sit at the intersection of technology, cost, and organization.

Spot fault tolerance and GPU scheduling are an ongoing tug-of-war between technology and cost; reproducibility and observability are deep friction between technology and engineering processes; and cross-team resource allocation points directly to the structural conflict between technology and organizational collaboration. These problems cannot be solved by introducing a single tool—they require systematic evolution in engineering practices, platform architecture, and even team collaboration models.

For ML engineers feeling their way along this path, recognizing which problems are the truly tough nuts to crack is itself the first step toward maturity.