Open-Source Agent Analytics Tool: Separating Execution Success from Task Success

The Core Problem Being Overlooked: Did the Agent Actually Accomplish Anything?

As AI Agents become increasingly widespread, more and more developers are confronting an uncomfortable reality: every step the agent takes appears to execute successfully, yet the agent ultimately fails to solve the user's actual problem. A developer on Reddit shared an open-source analytics tool they built, and its central insight is to rigorously separate two distinct dimensions: execution success and task success.

This distinction may sound simple, but it cuts to the heart of how we evaluate agents. Traditional monitoring typically focuses on whether API calls return 200s, whether tools are triggered correctly, and whether the pipeline completes — all of which are measures of execution-level success. But an agent can succeed on every individual step while completely failing at the overall goal.

Why These Two Types of "Success" Must Be Measured Separately

Execution Success ≠ Task Success

Consider a concrete example: an agent tasked with booking a flight for a user might successfully call the search API, successfully parse the returned results, and successfully fill out the booking form — every sub-step executed perfectly. But it booked the wrong date, or selected a flight that doesn't match the user's needs at all. From an execution monitoring perspective, this run is a green light. From the user's perspective, it's a complete failure.

If a development team only watches execution-level metrics, they'll conclude the system is running fine — while being completely blind to the real loss of user value. This is precisely the blind spot this open-source project aims to address.

The Diagnostic Value of Separating the Metrics

Splitting these two dimensions apart lets teams pinpoint the root cause of problems with much greater precision:

• If execution success rate is high but task success rate is low, the problem lies in reasoning, planning, or understanding user intent — not in the stability of the toolchain.
• If execution success rate is also low, the issue points to hard failures in infrastructure, tool integrations, or error handling that need to be fixed at the engineering level first.

This layered attribution makes the direction for optimization clear and avoids the frustration of having all metrics jumbled together with no obvious place to start.

Why an Open-Source Analytics Tool Matters

Filling the Gap in Agent Observability

Most LLM monitoring and observability tools on the market today focus heavily on traditional metrics: token consumption, latency, call chain tracing, and so on. Evaluation at the business level — whether the agent actually completed its task — remains a relatively underdeveloped area of the tooling ecosystem. By releasing this analytics approach as open source, the author lowers the barrier for teams to build their own evaluation infrastructure.

The value of open source here goes beyond free access. It also means the evaluation logic is transparent and auditable. Developers can clearly see how "task success" is defined and determined, and customize it for their own business context — because what counts as "task success" varies enormously across applications, and a generic black-box standard rarely fits.

A note on observability: The concept of observability originates from control theory and in software engineering generally refers to the ability to infer a system's internal state from its external outputs — logs, metrics, and traces. For traditional microservices, the three pillars of observability are relatively mature. But LLM Agents are special: their "internal state" includes semantic-level reasoning processes. Why did the model make a particular decision? Did it correctly understand the user's intent? None of this can be measured with latency or error codes. This forces agent observability to extend upward into the "business semantics layer" — and existing tooling is largely unprepared for this. Tools like LangSmith, Langfuse, and Arize Phoenix have begun offering tracing and evaluation features, but structured measurement of "task-level success" remains a widespread gap in the industry.

Real-World Value for Production Environments

For teams that have already deployed agents to production, this kind of analytics tool is especially valuable. As agents move from demos to real business workflows, the gap between "seems to work" and "actually reliable" gets magnified. Continuously tracking task success rates helps teams catch signals of quality degradation before a wave of user complaints hits.

Practical Takeaways Worth Noting

This project is an individual developer's open-source effort, but it reflects an increasingly important trend in agent engineering: evaluation frameworks are evolving from "was the process correct" to "was the outcome correct."

For teams building agent applications, there are a few key lessons to take away:

• Build layered evaluation metrics — don't let green lights at the execution layer hide failures at the task layer.
• Clearly define what "task success" means in your specific business context. This is often harder than the technical implementation.
• Treat evaluation as a continuous process, not a one-time test — especially when models and prompts are being iterated on frequently.

It's worth noting that due to limited information in the original source, specific implementation details about this tool — including which frameworks it supports and the technical mechanisms used to determine task success — remain unclear. Interested developers are encouraged to visit the open-source repository directly to understand its actual capabilities and integration options.

On determining "task success" technically: This is genuinely hard. Common approaches include rule-based heuristics (e.g., checking whether the output contains specific fields), using another LLM as a judge (LLM-as-a-judge) to score results, and comparison against a human-annotated golden set. Each approach has a cost: rules don't generalize well, LLM judges can themselves be wrong and add extra overhead, and human annotation doesn't scale easily. This is precisely why "clearly defining task success" is often harder than building the technology — it requires product, business, and engineering teams to reach consensus on "what counts as getting it right," which in many organizations is itself an unsolved problem.

Conclusion

Separating "execution success" from "task success" is a key signal that agent evaluation is maturing. It's a reminder to every agent developer: a completed pipeline is not the same as a solved problem. True success should be measured by whether the user's goal was actually achieved. The emergence of open-source analytics tools like this one is filling an important gap in agent observability and quality assurance.