Nike Builds Self-Healing Telemetry System with MCP and AI Agents

When Telemetry Pipelines Learn to Self-Regulate: Nike's Closed-Loop Control Practice

In the observability space, a long-standing contradiction persists: the more complex the system, the more easily configuration management becomes a bottleneck. Panos Tsilopoulos, former Director of Observability Platform Engineering at Nike, recently shared how they built a closed-loop telemetry system using OpenTelemetry, OpAMP, and the MCP protocol, allowing AI agents to participate in operational decisions under strict guardrails. According to him, during major sales events like Black Friday, Nike needs to manage approximately 100,000 OpenTelemetry Collectors, while the team is small enough to be described as "a bunch of people held together with tape."

Why Build a Custom Control Plane

A year and a half ago, there were virtually no mature commercial solutions on the market that could handle Nike's scale. Bindplane was one of the few options, but its cost was prohibitive for Nike. So the team built a custom control plane based on the OpAMP protocol, grouping Collectors by topology and infrastructure to control the blast radius of changes. Panos emphasized that OpAMP is essentially a network protocol, and a single push can affect the entire fleet, so grouped management is a safety baseline.

OpenTelemetry Collector is a vendor-neutral telemetry data collector under the CNCF umbrella that supports receiving, processing, and exporting metrics, logs, and traces. Its configuration is defined via YAML, including sections such as receivers, processors, exporters, and service. In large-scale deployments, Collectors typically run as a fleet, with each instance handling specific data pipeline processing tasks. Nike manages approximately 100,000 Collectors, meaning configuration changes require precise control over impact scope—otherwise a single erroneous push could cause a global outage.

OpAMP (Open Agent Management Protocol) is an open protocol in the OpenTelemetry ecosystem for remotely managing collector fleets. It defines the communication mechanism between servers and agents, supporting configuration delivery, status reporting, and health checks. OpAMP itself does not specify configuration content, but provides secure transport and grouping capabilities. Nike leverages OpAMP's group management features to group Collectors by topology and infrastructure, thereby limiting the blast radius of changes to a controllable scope.

Division of Labor and Closed Loop Among Three Protocols

OTLP standardizes the data side, OpAMP standardizes the Collector control side, and MCP standardizes structured access for AI agents to both. Together they form a closed loop: agents observe the telemetry backend and fleet status through MCP read-only tools, draft configuration change proposals, which after human approval are delivered to Collectors by the OpAMP server, and then based on validation results, either a full rollout or rollback is decided. According to the live demo, humans only need to perform two actions throughout the entire process: reviewing the proposal and approving execution.

MCP (Model Context Protocol) is an open protocol proposed by Anthropic to standardize structured interaction between AI agents and external tools and data sources. MCP servers expose tools, and agents execute queries or operations by calling these tools. In Nike's closed loop, MCP serves as the unified interface for agents to access the telemetry backend and fleet status. Tools are divided into read-only tools (such as querying metrics, logs, fleet overview) and write tools (only for creating proposals), and CI blocks any tool with apply capability from being merged. This design ensures that the agent's scope of action is strictly limited to observation and proposal stages, while actual changes still require human approval and OpAMP delivery.

Typical Scenarios: Cardinality Explosion and Sampling Recovery

In retail scenarios, product labels like SKUs landing on metrics can cause exponential growth in active time series. Nike's custom observability platform is based on Grafana and ClickHouse, processing about 200TB of data daily, and cardinality explosions can jeopardize Mimir stability—even without downtime, it's unnecessary compute waste. The agent's task is to draft filtering or transformation rules for the problematic service. Another scenario is reducing trace sampling rates for affected services during incidents to restore full-fidelity context. Panos shared a production case: during a large-scale chaos experiment in China before Double 11, a change caused multiple metrics' cardinality to increase by 30%, directly overwhelming the Mimir ring in that region—and on Double 11 day, Nike China's revenue accounts for about 50% of the annual total, roughly $5 billion.

Cardinality in observability refers to the unique number of metric label combinations. Each unique label combination generates an independent time series stored in time-series databases (such as Mimir, Prometheus). When label values have high-cardinality characteristics (such as SKU, user ID, request ID), the number of time series can grow exponentially, causing memory and storage pressure to surge, and even overwhelming the time-series database. In Nike's retail scenarios, product labels landing on metrics easily trigger cardinality explosions. The agent's task is to draft filtering or transformation rules to remove or aggregate high-cardinality labels at the Collector's processor layer, thereby protecting backend stability.

Guardrails as Code: Architectural Boundaries, Not Prompts

Nike's guardrail system is fully codified and version-controlled. Policies are a YAML file, and any change entering production must go through code review. The most notable design is the coexistence of allow lists and deny lists. The allow list defines what the agent can currently do, growing from an initial 5 entries to about 20; the deny list represents red lines that must never be crossed—for example, "never allow the agent to redirect telemetry data to another destination," meaning exporter configurations are always on the deny list. Panos explained that keeping both prevents the allow list from being accidentally expanded by broad entries (such as processors.*)—the deny list takes precedence, and its changes are easily identifiable in code review.

Other quality gates include: evidence receipts—instead of requiring agents to output text summaries ("they're habitual bullshitters"), they are required to attach the actual queries executed and data returned; protected labels—Tier 0 critical service configurations are untouchable by agents, and even if the issue affects these services, the agent can only modify the rest; and canary releases (first pushed to no more than 5% of the fleet), validation gates, and automatic rollback. There are also hard constraints at the MCP tool level: there is only one write tool for creating proposals, no apply tool exists, and if anyone tries to add an MCP tool with apply capability in CI, the build will fail directly.

Canary Release is a deployment strategy to reduce change risk by first pushing changes to a small subset of target instances (e.g., no more than 5% of the fleet), observing whether validation metrics are normal, and then gradually expanding. In Nike's closed loop, canary releases are combined with validation gates and automatic rollback: after agent proposals are approved, the OpAMP server first delivers the configuration to a small number of Collectors, and after telemetry data is returned, the system automatically verifies whether cardinality has recovered and whether metrics are lost. If validation passes, a full rollout occurs; otherwise, automatic rollback. This mechanism minimizes the impact of change failures, especially in scenarios managing a fleet of 100,000 Collectors.

Rejection and Correction in the Demo

The live demo simulated a cardinality explosion in the checkout service. Claude, through MCP tools, sequentially called fleet overview, metric queries, and log queries, identified that a newly added label line caused the problem, and drafted a transformation rule with supporting evidence. The first proposal used a service=checkout selector, which was rejected by policy—because that label is protected and would affect payment-critical Collectors. After receiving purely natural language feedback, the agent narrowed the selector to tier=standard and resubmitted, which was approved. After human review and approval, the change was first delivered to 2-5% of the 40 simulated Collectors, telemetry returned showing cardinality restored to normal without metric loss, and then rolled out to the remaining 38.

Failure Modes and Implementation Advice

Panos admitted that the test suite for this system was gradually built by "attacking the guardrails themselves," and was very fragile initially. Regarding proposal format, the demo only used local text; in production systems, one would naturally think of GitHub-style Pull Requests, but OpAMP's simplest implementation is stateless and requires additional database support. His three closing recommendations are: put boundaries in the architecture, not in prompts; guardrails are part of production integration; and open standard nodes make the system portable. For teams hoping to replicate this path, this means the value of AI agents is not in replacing human decision-making, but in freeing humans from repetitive observation and drafting, while using engineering guardrails to hold the impassable bottom line.