BLACKBOX: Injecting Memory into Incident Response with a Knowledge Graph

Production incidents rarely occur in isolation. Behind a single alert, there may be a recent deployment, a downstream dependency, an overloaded database, an unfamiliar service owner, and a similar event from months ago that feels eerily familiar. For on-call engineers, the real challenge is not just finding the answer, but quickly connecting all the context and explaining why that answer is trustworthy.

BLACKBOX is an AI incident investigation agent built to solve exactly this problem. It uses FalkorDB as a persistent operational knowledge graph, connecting services, dependencies, alerts, deployments, runbooks, people, and historical retrospectives into a single network.

From Alert to Explanation: How Graph Traversal Locates Root Causes

Traditional alert investigation often relies on engineers switching between multiple systems to search. BLACKBOX takes a different approach: when an incident occurs, it does not settle for retrieving isolated records, but traverses the graph structure to understand how the alert relates to the rest of the system.

Suppose a high-latency alert appears on notifications-api. BLACKBOX simultaneously examines several potentially relevant clues: whether the connection pool is exhausted, whether a recent PR changed the database pool configuration, which deployment introduced the issue, which services depend on notifications, who owns the related code, and whether a similar failure pattern has occurred historically.

The agent follows these relationship paths to find the likely root cause, determine the impact scope, identify relevant experts, and generate an actionable explanation. Unlike simply returning a list of search results, this answer comes with a reasoning path through the system, allowing engineers to see how the conclusion was reached step by step.

Multi-hop Graph Traversal is a core query pattern in graph databases, referring to starting from a node and following edge relationships through multiple consecutive hops to discover indirect connections. In the BLACKBOX scenario, a typical multi-hop query might look like: alert node → associated service → recent deployment of that service → PR introduced by the deployment → PR author. This traversal capability enables the agent to cross multiple system boundaries and automatically connect clues that were originally scattered across different tools. In contrast, traditional relational databases require multiple JOIN operations to express similar logic, and performance degrades sharply as relationship depth increases. Graph databases store relationships as first-class citizens, and multi-hop query latency is approximately linear with path length, making them better suited for root cause tracing with uncertain depth in operations scenarios.

FalkorDB as the Operational Memory Layer

The core design of BLACKBOX is that investigations do not disappear when the incident ends.

After each analysis, BLACKBOX writes structured findings back to FalkorDB, including root causes, affected services, solutions, owners, and retrospective knowledge. Subsequent investigations can directly use this information as context, forming a feedback loop: alerts trigger investigations, the agent traverses associated data, identifies findings and recommends solutions, investigation results are written back to the graph, and future incidents can recall this knowledge.

The first investigation may require exploring a large number of relationships; subsequent investigations with similar characteristics can start from stored context and reach effective answers faster.

Operational Knowledge Graph is a knowledge base that explicitly models operational domain entities and their relationships as a graph structure. Unlike general knowledge graphs, it focuses on operational semantics such as service topology, deployment pipelines, alert rules, personnel ownership, and incident retrospectives. Its value lies in turning relationships that were originally implicit in engineers' heads or scattered across documents—such as "who knows what," "what depends on what," and "how was it fixed last time"—into queryable structured assets. BLACKBOX writes the structured findings of each investigation back to the graph, essentially continuously expanding the nodes and edges of this knowledge graph, giving the system cumulative memory. This design differs from RAG solutions that rely solely on vector retrieval: RAG recalls text fragments, while the operational knowledge graph recalls deterministic relationship paths between entities, and the latter offers stronger interpretability and precision in root cause localization.

Why Choose a Graph Database Over Documents or Tables

Operational systems are inherently interconnected. Services depend on services, deployments change services, PRs belong to engineers, incidents affect multiple customer-facing systems, runbooks describe how to recover components, and retrospectives explain what happened in the past. These relationships are difficult to fully preserve in flat documents or isolated tables.

FalkorDB provides BLACKBOX with a natural representation that enables it to ask multi-hop questions: Which services depend on the alerting service? Which recent deployment is related to this failure? Who owns the code involved in the suspected root cause? Which customer systems are within the impact radius? Have we seen a similar incident before? What solution was used to recover last time?

Because the results come from connected graph paths, BLACKBOX can explain not only what it believes but also how it arrived at that judgment. This interpretability is particularly critical in incident response scenarios.

The Difference Between Cold and Hot Investigations

BLACKBOX intuitively demonstrates the difference between investigation modes with and without prior memory.

A cold investigation starts from the alert and traverses the graph to discover relevant context, potentially requiring examination of dependencies, deployments, code changes, personnel, and runbooks. A hot investigation leverages retrospectives and findings written during previous runs, can recall learned knowledge, reduces unnecessary tool calls, and focuses on confirming known patterns.

The interface presents this process in real time: users can see graph traversal, the agent's tool calls, findings being written, and the final explanation.

An Interface Built for Real-Time Collaboration

BLACKBOX includes a real-time investigation interface for incident response. As the agent works, the frontend streams progress and updates the graph visualization. The UI displays active alerts, causal paths through affected services, related deployments and PRs, customer-facing impact radius, relevant owners and responders, agent reasoning events, findings and retrospective write-backs, and cold vs. hot investigation comparison results.

This visibility matters during incidents. Engineers need to understand what the system is doing, rather than waiting for a black-box answer without supporting evidence.

The Role of Language Models and the Deterministic Fallback

In BLACKBOX, the language model is responsible for selecting tools, interpreting graph query results, summarizing evidence, and generating clear incident explanations. But the graph remains the source of operational truth.

BLACKBOX also includes a built-in deterministic planner as a fallback. When the model is unavailable or a request exceeds the configured tool budget, the system can still run the core investigation workflow based on the graph. This ensures the resilience of demos and incident workflows while maintaining the interpretability of findings.

Deterministic Planner refers to a component that executes investigation steps according to predefined rules or a state machine, without relying on the randomness of large language models. In BLACKBOX, it serves as a degradation path for the LLM: when the model API is unavailable, responses time out, or tool call counts exceed the budget, the system automatically switches to the deterministic planner, executing core graph queries in a fixed order (such as checking dependencies, retrieving recent deployments, and finding historically similar events). This "LLM-first, deterministic fallback" architecture pattern is particularly important in operational agents, because incident response scenarios demand extremely high availability and predictability and cannot allow investigations to be completely interrupted by model-side failures. Although the deterministic planner is less flexible than the LLM, it ensures the investigation process completes within a controlled scope, and its output is fully reproducible for post-hoc auditing.

Key Insights from the Build Process

The development of BLACKBOX reinforced a principle: agent memory is most valuable when it is structured and connected to the world it operates in. Simple conversation records cannot tell us which service caused the incident, who owns it, or which systems depend on it. Graph structures explicitly preserve these relationships and let future investigations use them. FalkorDB enables persistent memory, multi-hop reasoning, and real-time operational workflows to combine in a single system.

Next Steps

The next step for BLACKBOX is to integrate real-time observability, ticketing, deployment, and chat systems so the graph stays up to date as services change and incidents evolve. Productionization also requires stronger access control, tenant isolation, rate limiting, and richer query observability.

The long-term vision is to build an incident response system that becomes more useful with every investigation, helping engineers move from alerts to understanding, and from understanding to action. BLACKBOX is built around a simple idea: every incident should make the next one easier to resolve.

One-line Summary

BLACKBOX uses the FalkorDB graph database as persistent memory to build an interpretable AI incident investigation agent.

Paragraph Summary

BLACKBOX is an AI agent for production incident investigation. Its core design uses the FalkorDB graph database as a persistent operational knowledge graph, connecting services, dependencies, alerts, deployments, runbooks, personnel, and historical retrospectives into a traversable network. When an alert fires, the agent locates the root cause through multi-hop graph traversal and outputs an explanation with a reasoning path rather than simple search results. Structured findings from each investigation are written back to the graph, forming a feedback loop from cold to hot investigations that lets similar incidents reuse historical knowledge to accelerate localization. The language model handles tool selection and evidence summarization, but the graph is the source of operational truth, with a deterministic planner as a fallback to ensure resilience. Future plans include integrating real-time observability, ticketing, and chat systems, and adding production capabilities such as access control and tenant isolation.