Pinterest Medic: Engineering an AI Agent for Automated Spark Failure Diagnosis

In large data platform teams, providing high-quality support to partner teams who depend on the underlying infrastructure is a never-ending battle. Spark job failures, distributed system debugging, and priority trade-offs — these daily challenges consume enormous amounts of engineering time and energy. Pinterest Staff Engineer Dražko Profirović shared their solution at the AI Engineer conference: Medic for Apache Spark, an agent-based automated diagnostic tool.

This article traces Medic's evolution from prototype to a multi-agent architecture, along with the key engineering lessons the team learned along the way.

Why Build Medic

The support on-call rotation for a data platform team has two unavoidable pain points. First, a high barrier to troubleshooting: debugging Spark — or any distributed system — is inherently difficult, especially for engineers new to the framework. Second, unclear prioritization: should you first help one team fix a failing job, or unblock another team facing an imminent deadline? Human engineers are forced to make difficult trade-offs with limited time.

LLMs happen to break this constraint — they can scale knowledge and capability horizontally on demand. Pinterest's vision for the diagnostic tool was straightforward: a user simply asks, "Why did this job fail?" and receives a deep research report containing a chain of evidence for the root cause, along with fix recommendations tailored to the job's context. Furthermore, the agent needed to be deployed wherever users actually work — in Slack and the Airflow UI.

From Prototype to Bottleneck: The Limits of a Single ReAct Agent

The team started by exposing data resources via the Model Context Protocol (MCP) and connecting them to an LLM. MCP is an open protocol standard introduced by Anthropic in late 2024, designed to standardize how LLMs connect to external data sources and tools — its core idea is similar to the USB-C standard: different data systems only need to implement a single MCP Server interface to be callable by any LLM application that supports an MCP Client. In Pinterest's case, MCP allowed Spark job metadata, logs, metrics, and other distributed data resources to be exposed to the LLM through a unified interface, eliminating redundant integration work for each data source. This made it possible to have the model reason directly about Spark jobs within an MCP-enabled conversation. The approach worked in practice, but was highly dependent on carefully crafted prompts from a human operator.

The team then expanded the prototype into a single Reasoning and Acting (ReAct) Agent. ReAct is an LLM agent paradigm proposed by a Google research team in 2022, where the model alternates between producing a reasoning chain (Thought) and invoking external tools (Action), then continues reasoning based on the tool's output until it reaches an answer. This paradigm works well for open-ended problems, but when a single agent must handle complex tasks, the reasoning chain grows unwieldy, context window pressure surges, and it becomes difficult to precisely control execution paths through prompt engineering. This agent was given a single comprehensive prompt defining a problem-solving methodology, a structured output format for reports, and examples of common failure patterns. With these capabilities, Medic was opened to beta users.

But early trials exposed numerous flaws:

• Prompt tuning became unsustainable: a single prompt shouldering all responsibilities meant that adding detail in one area often degraded behavior in another.
• Inconsistent response quality: analysis was sometimes superficial, sometimes excessively verbose.
• Lack of control: it was difficult to keep the agent on track.
• Frequent context window exhaustion: the massive log output of production jobs rapidly consumed tokens, causing reasoning to break down.
• Weak testing strategy: end-to-end testing relied on manual validation in production, production data got cleaned up, test results were anecdotal, and there was no reliable way to confirm whether new changes had broken previous improvements.

Building a Solid Engineering Foundation with Observability and Testability

Facing these problems, the team first invested in observability and testability. They used OpenTelemetry to push trace data to LangFuse, examining the agent's execution steps as waterfall charts to identify the causes of low-quality responses. OpenTelemetry is an open-source observability framework incubated by the CNCF that provides a unified standard for collecting traces, metrics, and logs — it has become the de facto standard for cloud-native observability. LangFuse is an open-source observability platform designed specifically for LLM applications, supporting fine-grained tracing of every LLM call and tool invocation by an agent, displayed as a waterfall chart. Combining the two allowed the team to inspect the agent's execution trajectory the way they would debug a traditional microservice, pinpointing exactly which reasoning step or tool call caused quality to degrade — a critical piece of infrastructure for engineering AI systems.

Equally important was building an end-to-end testing framework. The approach is elegantly simple:

• Record mode: the agent calls real downstream systems, and tool responses are captured as fixtures, saved to the filesystem, and committed as code.
• Playback mode: the agent runs against fixtures instead of production data, performs its analysis, and generates a report.

The test suite then scores the report according to pre-written offline evals. For example, one eval checks whether the recommended fixes exceed three items — if the agent generates too many recommendations, the score drops, keeping the final report concise. This mechanism allows the team to measure quality with quantitative metrics rather than intuition, and to expand test coverage while ensuring improvements don't introduce regressions.

Diving Deep into Logs and Metrics: Improving Signal-to-Noise Ratio

With testing in place, the team began optimizing log processing in depth. Log noise is enormous, and many exceptions are actually benign, so simply focusing on the last exception is unreliable. The initial regex-based heuristic filtering approach didn't scale well, so the team built an exception classification pipeline.

The core idea: learn which exceptions commonly appear in successful jobs, treat them as potential "red herrings," and filter them out. The agent fingerprints and clusters exceptions, then ranks them by content relevance and recency relative to the job's termination time. The agent no longer consumes logs directly; instead it interacts through two MCP tools: one to fetch the top-k truncated exceptions, and one to fetch the full logs for a specific exception. This significantly improved the signal-to-noise ratio and reduced the risk of the LLM being "anchored" by misleading exceptions.

For metrics, the challenge was similarly the context window. Raw time-series data is very unfriendly for long-running production jobs and is token-inefficient.

Pinterest's approach here is clever: metrics analysis is handled inside an isolated sub-agent. They convert raw time-series data into charts, stitching them into a single overview image (resembling a Grafana dashboard but annotated with useful information like min/max values), then attach this image to the LLM conversation and let the model reason about patterns in the data.

The biggest advantage of using an image: regardless of how long a Spark job runs, the input token count stays fixed. The sub-agent can identify anomalous signals such as executor counts dropping to zero, prolonged stalls, or bottlenecks, then summarize its findings and return them to the parent agent — keeping the context window healthy at all times.

Multi-Agent Architecture: A Qualitative Leap in Control

The final step in Medic's evolution was refactoring the single ReAct agent into a multi-agent architecture, built on top of the LangGraph deep agent library. LangGraph is a stateful multi-agent orchestration framework developed by the LangChain team, whose core abstraction is modeling agent workflows as directed graphs (DAGs or graphs with cycles), where nodes represent agents or processing steps and edges represent control or data flow. Compared to linear Chain patterns, LangGraph natively supports cyclic reasoning, conditional branching, and parallel execution — making it especially well-suited for complex tasks requiring multi-agent collaboration. The deep agent library Pinterest uses is a further abstraction on top of LangGraph, with built-in "scaffolding" tools like a task todo list and virtual filesystem that help agents maintain coherent state during long-running tasks, preventing them from "losing their way." Each agent has its own dedicated prompt and a subset of MCP tools — a design philosophy in line with the experience of coding tools like Claude Code and Codex.

After the refactor, the team was finally able to decompose the bloated single prompt into specialized roles, with each agent having a clear responsibility, making it easy to maintain and test independently. An unexpected benefit: expanding the project scope became as simple as adding a new prompt — it's precisely how the team extended Medic to also support optimization of Spark SQL jobs.

The Complete Diagnostic Workflow

The full workflow operates as follows:

1. A user request enters the system, and the intent is classified as either a "simple Q&A" or "deep diagnosis."
2. If it's a deep diagnosis, the Triage Agent determines the lifecycle state of the Spark job; if failure is confirmed, it invokes a tool to generate a set of failure hypotheses.
3. Each hypothesis is investigated in parallel by a Research Agent, which collects evidence to validate it and returns a confidence score and root cause.
4. The Supervisor selects the highest-confidence root cause and invokes the Healer Agent, which uses a vector database ingested with operational runbooks to generate fix recommendations. The vector database supports fast semantic similarity retrieval by encoding text semantically as high-dimensional vectors — it's a core component of Retrieval-Augmented Generation (RAG) architectures. The Healer Agent uses the root cause description as a query vector to retrieve the most relevant runbook fragments, then the LLM generates specific recommendations in the context of the job — avoiding the token waste of stuffing the entire runbook into the context window while ensuring fix recommendations have a clear knowledge source.
5. Finally, the Supervisor assembles a properly formatted report.

Results and Engineering Takeaways

The multi-agent architecture proved highly effective, providing the strongest level of control over system behavior, while the log processing improvements dramatically reduced incorrect root causes. Worth noting: the team experimented with using LangGraph's workflows to make agents more deterministic, but found this approach far more brittle compared to the ReAct paradigm — a counterintuitive but important engineering judgment.

Looking ahead, Pinterest is experimenting with automatically improving agents by absorbing user feedback from historical sessions, and sees broad potential for extending this pattern to other distributed systems like Flink and Trino.

The Medic for Apache Spark case provides a solid engineering blueprint for "landing AI agents in production infrastructure operations": the real value lies not in flashy model capabilities, but in the continuous refinement of engineering fundamentals — observability, testability, and architectural decoupling.

Key Takeaways