Self-Healing Releases: Using Agentic AI to Automatically Fix Production Failures

Starting with the CrowdStrike Outage

Remember the CrowdStrike incident that paralyzed the world? Daniel Oh and Kevin Dubois from Red Hat/IBM opened their KubeCon talk with a personal story: they were attending a conference in Berlin when the outage struck — five minutes before their talk. Kevin's flight was cancelled outright, stranding him in Berlin for two or three days. Daniel's flight was delayed six hours. Even Starbucks was unable to take orders because the Microsoft systems it relied on had gone down.

The irony is that this global catastrophe was triggered by a tiny content update — a small, seemingly inconsequential change. Yet that minor tweak brought down entire systems, and recovery took one to two weeks. This is the core question the two speakers posed: a CI/CD pipeline alone isn't enough. You need to ensure everything is healthy before deploying to production, and you need the ability to automatically fix things when they go wrong.

Progressive Delivery: Validate Before You Scale

The first layer of the solution is Progressive Delivery. Rather than pushing new features to all production environments in one "big bang," a canary approach gradually rolls out traffic — starting with 1–2% of users, then scaling to 3–4% once things look stable, and eventually reaching full rollout. The moment something goes wrong, the system immediately rolls back to the previous stable version (N-1).

Kevin emphasized that the key to this entire process is automation. Traffic shifting, health checks, continuing the rollout or triggering a rollback — all of it is decided automatically by the system based on metrics and telemetry data, rather than waiting for users to call in complaints before someone manually rolls back.

Underpinning all of this is GitOps. Git serves as the single source of truth, managing not only application code (Java, .NET, Python) but also all configuration files — Kubernetes CRDs, Services, ConfigMaps, Secrets, and other manifests. These configurations need to be externalized from the application itself, enabling loose coupling and independent management. Combined with Argo CD or Flux, whenever the desired state changes, the production environment automatically aligns to it.

That said, Argo CD itself updates the desired state as a full replacement in one shot. To achieve progressive delivery, you need tools like Argo Rollouts working alongside it.

---

GitOps is an operational paradigm that uses a Git repository as the Single Source of Truth, first proposed by Weaveworks in 2017. The core idea: all desired states for infrastructure and applications are stored as declarative configuration files in Git, every change must be triggered via a Git commit or PR, and an automated system continuously reconciles the actual running state to match the desired state defined in Git. Argo CD and Flux are the two most widely adopted GitOps tools today — both run inside a Kubernetes cluster, continuously watching the Git repository and auto-syncing. Unlike traditional "push-based" CI/CD, GitOps uses a pull-based architecture where an in-cluster controller actively pulls the target state, offering advantages in security and audit traceability — every production change maps to a traceable Git commit, and rollbacks are as simple as git revert.

---

The Limits of Static Analysis Templates

Argo Rollouts uses AnalysisTemplates to define health check criteria. You can integrate metric sources like Prometheus and set conditions — for instance, monitoring for 5xx errors like 500s and 503s.

But what if the failure isn't a 500 error? Kevin pointed out that static YAML configurations can only cover scenarios you've anticipated in advance. To catch all possible anomalies, teams end up writing a growing collection of different Prometheus queries, which quickly becomes unwieldy. On top of that, hand-written YAML is prone to human error and unexpected issues.

This is exactly where agentic AI comes in — instead of continuously writing custom templates and rules, let AI handle new, unforeseen scenarios in a dynamic and autonomous way.

---

Argo Rollouts is an advanced release controller for Kubernetes that extends the capabilities of native Deployments, with built-in support for canary releases, blue-green deployments, and other strategies. AnalysisTemplate is a Kubernetes custom resource (CRD) used to define health evaluation logic during a rollout: you specify the data source (Prometheus, Datadog, Web Hook, etc.), query expressions, thresholds, and evaluation frequency, and Argo Rollouts periodically runs these analyses, automatically deciding whether to proceed or roll back. This mechanism decouples the definition of "what counts as healthy" from the release process and enables version-controlled health policies. However, it remains fundamentally static — all judgment logic is baked into YAML before deployment and cannot dynamically adapt to new anomaly patterns at runtime. This is the ceiling of purely rule-driven approaches.

---

Building an AI Analysis Agent with Java and Quarkus

The two speakers made an interesting technology choice: building the AI agent system in Java. Many would ask why not Python — Kevin's answer: the modern Java stack based on Quarkus starts fast, has extremely low memory footprint, and delivers excellent throughput. Paired with LangChain4j (a project inspired by Python's LangChain but independently developed), it makes it easy to integrate LLM capabilities into Java applications, while being more "enterprise-ready" in terms of observability, metrics, and security.

The entire system is called the "Kubernetes Agent." They wrote a lightweight plugin for Argo Rollouts (implemented in Go) that calls into this agent system. In the AnalysisTemplate, instead of a Prometheus provider, they use a plugin provider pointing to the team-built argo project metrics AI plugin.

Configuration only requires a few parameters: which Pods are the stable version (used as a baseline), which are the canary, and optionally some extra prompt context for the specific application. The AI uses this to determine whether the canary version is performing on par with or better than the stable version.

---

LangChain4j is an independent Java ecosystem implementation of the LangChain concept (not an official Java port of LangChain), providing a unified abstraction layer for interacting with various LLMs — supporting OpenAI, Anthropic, local Ollama, and more — with built-in support for tool calling, RAG (Retrieval-Augmented Generation), memory management, and agent orchestration. Quarkus is a cloud-native Java framework led by Red Hat; its core advantage is moving much of the framework initialization work to compile time, enabling millisecond startup times and very low memory usage when combined with GraalVM native images — particularly valuable for agent systems that need to respond quickly. Compared to the Python ecosystem, Java solutions integrate more seamlessly into existing enterprise monitoring (Micrometer/Prometheus), security (Vault integration), logging standards, and dependency audit workflows, reducing the friction of introducing a new technology stack.

---

The Agent Workflow: Analyze, Score, and Auto-Remediate

The overall agent workflow is elegantly designed:

• Parallel collection: Multiple parallel agents simultaneously gather diagnostic information and metrics (Pod logs, telemetry from both stable and canary versions)
• Analysis agent: Sends the aggregated data to the LLM to determine whether a problem exists
• Scoring agent: Because LLMs can sometimes "hallucinate," a dedicated scoring agent evaluates the quality of the analysis, forming a feedback loop
• Decision relay: Once a pass/fail conclusion is reached, the signal is sent back to Argo Rollouts synchronously to trigger an immediate rollback or continue the rollout

Kevin noted that this type of analysis task doesn't require a massive frontier model — a smaller model (the demo used a locally deployed Qwen 3.8) handles it well and runs fast.

The team went one step further by incorporating automated remediation as an asynchronous step. Since the LLM has already analyzed the root cause, why not let it attempt a fix? Because production has already been rolled back and the incident is resolved, remediation can proceed asynchronously without urgency: if it's a simple problem the LLM can handle, it automatically creates a remediation Pull Request; if it's more complex, it generates a GitHub Issue with a detailed description for the on-call team.

---

LLM hallucination refers to the phenomenon where a large language model generates content that sounds plausible but is factually incorrect or fabricated — a significant risk in engineering scenarios that demand precise judgment. The independent scoring agent introduced here is a practical engineering countermeasure against hallucination — essentially a form of multi-agent verification, similar to the "four-eyes principle" in code reviews. This design pattern is also known as a Reflection Loop in agent systems: a generator agent produces an initial output, an evaluator agent scores its quality, and if the score falls short, the system re-prompts and iterates until the output meets the standard. Choosing a small local model like Qwen 3.8B over cloud-based large models like GPT-4 not only reduces latency and cost, but also avoids sending production logs and sensitive metrics to third-party services — a critical consideration for enterprise compliance and data privacy.

---

Live Demo: From Fault Injection to Automated Fix PR

Despite shaky conference Wi-Fi, the demo ran end-to-end successfully. Kevin intentionally pushed a new image containing a NullPointerException directly to the main branch.

After the push, Argo CD and Argo Rollouts detected the new version and began the canary rollout: 90% of traffic went to the stable version, 10% to the canary. After waiting 10 seconds, the analysis phase began — agents started pulling metrics and logs from both versions. The analysis results came back quickly: while the canary's metrics were within thresholds, the logs showed repeated, severe NullPointerExceptions. The decision: roll back. Traffic instantly switched back to the stable version.

Meanwhile, the remediation workflow triggered asynchronously in the background. After LLM generation and scoring agent validation (with re-prompting and retry if quality was insufficient), a Pull Request was opened on GitHub — the fix being null checks added to prevent the NPE.

It's worth emphasizing that the current approach preserves human in the loop — the fix is submitted as a PR and requires human review before merging. But Kevin also noted that if you're feeling bold, it's entirely possible to make this a fully automated loop where the system continuously attempts fixes and re-deploys until it produces a stable version.

Engineers Are Still the Captain

The two speakers closed with a pragmatic perspective: AI can automate more and more work, but you're still "the captain of the ship, the pilot of the plane." AIOps and agentic AI are assistants — you're the one who needs to orchestrate and maintain overall control.

They've open-sourced everything: the lightweight Argo Rollouts plugin (Go), the demo application, the Java agent system, and a progressive delivery configuration (targeting OpenShift) for one-click deployment of the entire stack (Argo CD, Rollouts, app, and agents). For developers looking to build agent applications in Java, they've also prepared a dedicated workshop.

The value of this "self-healing release" approach lies in combining the safety net of progressive delivery with the dynamic analysis and auto-remediation capabilities of agentic AI — so the next small change in the spirit of CrowdStrike doesn't escalate into a global disaster.