Rashomon: The Open-Source "Black Box" Monitoring Tool for AI Coding Agents

When AI Agents Start Writing Code, Who's Watching Them?

Coding agents like Claude Code and Cursor Agent are evolving from "autocomplete assistants" into "autonomous executors" — they proactively run commands, modify files, invoke sub-agents, and execute tests. While this brings a huge leap in productivity, it also raises a widely overlooked question: when an agent is operating on your behalf, do you really know what it's doing?

An open-source project called Rashomon on Reddit is trying to answer that question. Its author pitches it as "Sentry for AI coding agents" — not tracking application crash logs, but tracking every real action an agent takes during execution. The analogy cuts right to a genuine pain point in modern AI engineering: the absence of Observability.

What Problem Is Rashomon Trying to Solve?

In traditional software engineering, tools like Sentry help developers capture runtime exceptions and reconstruct error stack traces. In agentic workflows, however, "going wrong" takes on far more complex and subtle forms. According to the author, Rashomon focuses on recording the following categories of execution details:

• Command execution: Which shell commands the agent actually ran
• Test failures: Which tests failed during execution
• File changes: Which files were modified and what exactly changed
• Sub-agent activity: The behavior chain when a primary agent spawns subtasks
• Say-do gaps: The discrepancy between what the agent actually did and what it reported to you

That last point deserves special attention. LLM-powered agents carry a structural risk: their natural language summaries don't necessarily reflect what truly happened. An agent might claim it "fixed the bug and all tests passed," while in reality it skipped certain steps or the tests never actually ran. Rashomon's core value is making this kind of "say-do inconsistency" visible — a concept neatly reflected in its name (Rashomon, referencing the classic film where the same event is recounted in conflicting versions).

---

Observability originates from control theory and in software engineering refers to the ability to infer a system's internal state from its external outputs. It is typically built on three pillars: Logs, Metrics, and Traces. Sentry is an iconic product in this space, focused on capturing runtime exceptions and reconstructing the surrounding context to help developers quickly identify root causes. In traditional software, code behavior is driven entirely by deterministic logic, so the observability challenge is mainly about finding errors that have already occurred. AI agents introduce probabilistic decision-making — the same input may produce different execution paths, and agents autonomously choose tools and decompose subtasks, forming dynamic behavior chains. This makes traditional logs and stack traces difficult to apply directly, requiring observability solutions designed specifically around "agent behavior" rather than "code execution."

---

Why the "Say-Do Gap" Is the Central Problem

The most insightful aspect of this project is its skepticism toward an agent's self-reported reliability.

In human-AI collaboration, developers often make decisions based on an agent's text summaries — "it says it's fixed, so I'll merge it." But agent outputs are fundamentally probabilistic, prone to hallucination and overconfidence. Once its reports diverge from its actual actions, errors get silently introduced into the codebase while human reviewers remain completely unaware.

Rashomon treats the execution record as "objective evidence" and compares it against the agent's "subjective narrative." This design philosophy is essentially building an audit and accountability mechanism into agentic workflows. As more teams let AI write production code, this kind of traceability could shift from "nice to have" to "compliance requirement."

---

LLM "hallucination" takes a distinctive form in the context of coding agents. When a model generates an execution summary, it isn't simply "replaying" a log of completed operations — it's generating natural language probabilistically based on context. This means the agent's report is fundamentally an inference, not a playback. When the execution chain is long and intermediate steps are compressed or lost within the context window, the model may unintentionally generate a summary that doesn't match what actually happened, presenting it with an air of confident certainty. This is fundamentally different from a human lying — it's closer to memory distortion. For this reason, relying solely on an agent's text output for code review introduces a systemic blind spot, not just an occasional low-probability risk.

---

The Open Questions the Author Is Asking

Notably, the Reddit post itself is a classic open-source "demand validation" exercise. The author openly admits they're still figuring out the product direction and poses several questions to the community:

• Would you use a tool like this?
• How would you improve it?
• What capabilities would it need for you to install it and keep using it?
• What do existing tools already do well? What's still missing?

This kind of public inquiry reflects the true state of the AI toolchain today: the space is too new for established answers. No de facto standard has emerged for agent observability. Tools like LangSmith, Langfuse, and Helicone primarily focus on tracking at the LLM call layer, while dedicated monitoring at the "coding agent execution layer" remains largely unexplored. Rashomon is aiming squarely at that gap.

---

LangSmith, Langfuse, and Helicone are the leading LLM observability platforms today. Their core capabilities center on tracing each API call's inputs and outputs, token consumption, latency, and cost — with some supporting visualization of multi-step chain call sequences. These tools are extremely valuable for debugging prompt effectiveness and optimizing model call costs. However, their perspective stops at the "model call layer" — they record what was sent to the model and what the model returned, not what the agent actually did on the operating system. When an agent runs git commit, modifies a config file, or calls an external API, those side effects occur entirely outside the LLM call layer and are invisible to existing tools. Rashomon targets exactly this blind spot: going beneath model call logs down to the real system execution level.

---

Real Value and Real Challenges

From a product perspective, Rashomon's direction is sound — but for developers to actually "install it and stick with it," it still needs to clear a few hurdles:

Integration cost. Developers won't tolerate a steep onboarding tax for an additional monitoring layer. It needs to hook into the execution flow of mainstream agents like Claude Code with minimal friction — ideally plug-and-play.

Signal-to-noise ratio. Recording "everything" is easy, but what's truly useful is surfacing anomalies within a mountain of execution logs — automatically highlighting moments of say-do inconsistency, rather than making users sift through raw logs themselves.

Open-source moat. As an open-source tool, its strengths lie in transparency and self-hostability, which is attractive to teams concerned about data privacy who don't want their code execution records uploaded to a third party. But open source also means the project needs community co-building to sustain ongoing iteration.

Final Thoughts

Rashomon is still in a very early exploratory stage — at the moment it looks more like a "proof of concept + needs assessment" than a finished tool. But the question it's asking — how do you trust an AI that writes code on your behalf — will only grow sharper as agent capabilities expand.

When an agent shifts from "advisor" to "executor," observability stops being optional. Whether or not Rashomon ultimately becomes the Sentry of this space, the question it raises is already one that every developer using AI for coding should take seriously. The project repository is open on GitHub, and developers who are interested can follow its progress.