AI Agent Operation Recovery Benchmark: When Task Completion Scores Zero

When a business system accepts an AI Agent's operation but the confirmation receipt is lost in transit — what should the Agent do? Retrieve the original result, reconnect to the running task, hold the outcome as uncertain, or simply submit a new write operation? This seemingly simple question is precisely the key scenario for measuring Agent reliability and safety.

A developer designed a benchmark called "Agent Operation Recovery" around this problem and attempted to submit it to the Kaggle Benchmark Challenge. This article is both an honest account of that submission attempt and a revealing look at how Agents actually perform in failure-recovery scenarios.

Test Design: The Rigorous Thinking Behind 24 Synthetic Cases

The benchmark centers on a core tension: how can an Agent avoid re-executing irreversible operations when a confirmation receipt is lost, while still advancing legitimate business that genuinely needs to be completed?

The developer designed 24 synthetic cases covering a range of scenarios: in-flight operations, lost confirmation receipts, unknown execution outcomes, legitimate new intents, authorization changes, and goal-binding changes. These scenarios span four business domains — tickets, inspections, appointments, and inventory deductions — with five tools running on an in-memory simulator, involving no real customer systems or production transactions.

The evaluation framework deliberately distinguishes four levels: tool call attempts, accepted actions, actual effects produced, and the final report. Of all cases, 17 require advancing business progress, while 7 require returning unknown, blocked, or pending results. This means simply refusing all operations cannot pass the test. More critically, new write operations are intentionally designed as non-idempotent, so the simulator does not hide unsafe duplicate dispatches through automatic deduplication. This design ensures the test genuinely exposes dangerous Agent behavior.

Idempotency is a core concept in distributed systems design: an operation is idempotent if executing it once produces the same effect as executing it multiple times. For example, "set inventory to 100" is idempotent, while "reduce inventory by 10" is not — executing the latter twice yields an incorrect result. In Agent recovery scenarios, idempotency is critical: if the original operation succeeded but the confirmation was lost, idempotent operations can be safely retried, while non-idempotent operations (such as duplicate charges or duplicate dispatches) can cause serious business damage. By deliberately making write operations non-idempotent, this benchmark strips away that "technical safety net," forcing the Agent to rely on its own reasoning to decide whether to retry rather than depending on system-level deduplication as a fallback.

Models Tested and the Reality of Restricted Access

The frozen research batch, labeled 20261009-full96-v1, compared deepseek-v4.1-flash and deepseek-v4-pro via Tencent Cloud TokenHub using the Kaggle Benchmarks SDK. Each model ran twice across all 24 cases — 48 episodes per model, 96 total — each time with a fresh conversation and simulator state, with caching disabled.

The developer was unusually candid about the limitations of this data: these are locally run results from an external vendor, and vendor-side identifiers cannot pin down immutable model versions; the two repetitions constitute limited reproducibility evidence, not independent random sampling or significance testing.

The real complications arose with native Kaggle platform access. When the developer attempted native access with their original account, early calls to google/gemini-3.7-flash and qwen/qwen3-next-80b failed due to CN region restrictions. A subsequent fresh, uncached private access probe — attempting deepseek-ai/deepseek-r1-0528, openai/gpt-oss-120b, and zai/glm-5 — also failed, returning HTTP 403/PermissionDenied with no successful model responses.

The developer particularly emphasized an important principle: missing native observations do not equal a score of zero. No cached external outputs were presented as native Kaggle results, and no external models were relabeled as Kaggle models. This strict separation of data provenance boundaries is especially rare in today's benchmark environment, which is so often filled with inflated claims.

Core Finding: The Paradox of Completing Tasks but Scoring Zero

The external research fully observed all 96 planned episodes and retained all failure records. The results are striking:

Frozen Research Metric	Flash	Pro
Strict safe/correct resolution	15/48	0/48
Final JSON parse failures	27/48	48/48
Correct business progress / episodes requiring advancement	34/34	34/34
Duplicate write attempts / eligible episodes	0/40	0/40
Duplicate effects / observed episodes	0/48	0/48

The data reveals a thought-provoking paradox: both models achieved perfect scores on the diagnostic "business progress" metric (34/34), and neither produced any duplicate write attempts or duplicate effects — indicating strong performance on the core safety dimension. Yet on "strict safe/correct resolution," Flash scored only 15/48 and Pro scored 0/48.

The key issue is final JSON parse failures: the Pro model encountered final_not_json parse failures in all 48 episodes. The strict contract requires the final output to be valid JSON, with identity and claims backed by evidence, a complete effect ledger, and compliant tool behavior. Once JSON parsing fails, subsequent semantic checks are skipped entirely — this is the technical root cause of the "task completed but scored zero" phenomenon.

The developer carefully notes that parse failures or a strict score of zero do not prove the original write operations failed. Automatically resubmitting based on this alone could introduce a side effect the experiment itself never observed. Moreover, successful execution does not erase prohibited tool call attempts or payload-binding errors — the runtime may have rejected an unauthorized request, but the trace still exposes the model's flawed decision.

Effect Ledger is the record structure used in this test framework to track the actual side effects an Agent produces. Unlike trace logs that only record tool call intent, the effect ledger specifically registers real state changes that occurred in external systems — such as inventory deductions, ticket creation, or appointment modifications. Distinguishing the three levels of "tool call attempt," "accepted action," and "actual effect produced" is designed to identify a dangerous pattern: a model may submit an unauthorized request that is rejected by the system (a tool call attempt exists), yet no effect occurs at the system level (the effect ledger is empty) — but strict scoring still records this as an error decision, because the trace exposes the model's overreaching intent, even if that intent was never fulfilled.

Honest Disclosure of Eligibility Gaps: External Results ≠ Platform Scores

What is most commendable about this submission is its unflinching disclosure of its own eligibility gaps. The developer explicitly states: there is no publicly accessible Kaggle benchmark URL, and no successful native Kaggle results exist for this submission. Private task creation, notebook saves, and an empty results table cannot substitute for that.

The challenge organizers confirmed that external APIs and cached results do not meet the "platform-run" requirements of this challenge. The developer chose to honestly disclose this eligibility gap rather than packaging the attempt as "accepted."

That said, the independently published research remains fully auditable — including a fixed reproduction source, a public full96-v1 release (containing cases, scorer, manifest, and all 96 episode records), and an earlier published research article. The experiment used the Kaggle Benchmarks SDK, with recovery and binding-related questions referencing the ACC (Agent Capability) specification and BailingHub.

These findings are built on a clearly scoped, small synthetic contract involving only two models from the same family and a single vendor, and do not demonstrate production safety or general equivalence across models. The developer's stated next steps are to vary recovery tool phrasing, evidence noise, and interpretation checks without replacing the frozen observations.

ACC (Agent Capability) Specification is a framework for describing and standardizing Agent capability boundaries, covering expected Agent behavior within a given toolset, authorization scope, and goal constraints. BailingHub is the associated coordination and registration mechanism. In Agent recovery scenarios, the ACC specification matters because it clearly defines which recovery behaviors fall within the Agent's authorized scope and which constitute overreach beyond the original task delegation. For example, re-querying task status is typically within scope, while unilaterally deciding to resubmit a payment without new authorization may violate the goal binding of the original delegation. The specification provides an actionable reference for evaluating Agent decision boundaries in ambiguous states.

Implications for Agent Reliability Evaluation

This case carries value for the broader Agent evaluation field that goes well beyond the submission results themselves. It clearly demonstrates that an Agent's "task completion" and "strict safety contract compliance" are two dimensions that must be measured separately. A model that scores perfectly on business progress and shows zero risk of duplicate operations can still fail completely under strict scoring due to output format issues.

More importantly, the methodological example it sets — strict separation of data provenance boundaries, full disclosure of limitations, complete retention of failure records — represents the attitude that serious benchmarking should embody. At a time when Agent safety is receiving increasing attention, this approach of "exposing gaps rather than gilding results" is far more instructive than a polished leaderboard number.