How to Prevent Duplicate Refunds After an AI Agent Crashes: CellaFlow's Durable Execution Approach

Multi-agent systems are evolving beyond the simple "prompt → model → tool → response" pattern into complex execution flows deeply embedded in production environments. When an agent starts triggering refunds, placing orders, or calling payment APIs — operations that cannot be safely reversed — a seemingly simple question becomes extremely thorny: what happens if the process crashes midway through?

A developer building CellaFlow shared their thinking on Reddit around this "crash benchmark." The core argument is straightforward: Agents are ephemeral. Execution shouldn't be.

The Classic Problem Hidden in a Refund Scenario

Imagine this flow: a text agent decides to issue a refund → the system refunds $100 → at that exact moment, the pod running the refund logic crashes → the result never gets written to a checkpoint → the agent restarts → does it issue the refund again?

The most obvious issue is "duplicate side effects" — the customer gets refunded twice. But the author points out an equally important, often-overlooked problem: what if the mechanism you introduced to prevent duplicates ends up permanently deadlocking the entire workflow?

This shifts the discussion from "safety" to "liveness." It's not enough to guarantee that "the operation wasn't executed twice" — you also need to guarantee that "the operation will eventually be completed."

Benchmark Comparison of Four Approaches

The author built an open-source, reproducible benchmark comparing four approaches across multiple failure modes: No guard, Postgres advisory lock, Durable claim, and CellaFlow.

Failure modes tested include: concurrent retries, crash after claiming work, crash after external side effect occurs, slow worker, worker permanently disappearing, and ownership transfer.

The simplified results are illuminating:

Failure Mode	No Guard	Advisory Lock	Durable Claim	CellaFlow
Concurrent retries	5 side effects	5 side effects	1	1
Crash after claim	2	2	1	1
Crash after side effect	2, recoverable	2, recoverable	1, deadlock	2, recoverable
Slow holder	5	5	1, waiting	1, waiting
Holder disappears	-	-	1, deadlock	2, recoverable

The key insight: "happens only once" does not equal "succeeds." Durable claim performs well at preventing duplicates, but deadlocks in two scenarios — it gets "no duplicates" right while sacrificing "recoverability."

Why Durable Claim Isn't Enough

The author illustrates Durable claim's fatal flaw with a simple flow: worker A claims an operation → A crashes → the claim record still exists → worker B arrives, sees "already claimed" → deadlock.

The claim did its job — it prevented a duplicate. But it also prevented anyone else from completing the work.

To make claims recoverable, you inevitably end up adding: claim + lease + heartbeat + ownership transfer + fencing. At that point, you're no longer "adding an idempotency check" — you're building an entire execution system. That's exactly what CellaFlow aims to provide.

Lease and Heartbeat are standard mechanisms in distributed systems for managing resource ownership. A lease is an ownership claim with an expiration time — a worker acquires a time-limited lease when claiming a task and must renew it via heartbeat before it expires, otherwise the system assumes the worker has gone offline and allows another worker to take over. A heartbeat is essentially the worker periodically signaling "I'm still alive" to the coordinator.

Ownership Transfer refers to the process of handing task control to a new worker after the original worker's lease expires, requiring an atomic update to the ownership record to prevent concurrent contention. Fencing is the complementary safety mechanism: each time ownership transfers, the system increments an "epoch number," and validates that the current worker's epoch is still the latest on every write operation. This way, even if a zombie worker revives, its stale epoch token is rejected and it cannot write outdated state. Only by combining all four mechanisms can you achieve both "no duplicates" and "recoverability."

The Hardest Problem: The Side Effect Already Happened

The truly unsettling scenario is: claim the operation → call the payment API → the payment API accepts the request → process crashes → the result is never recorded.

If the external system doesn't participate in your protocol, the runtime has no way of knowing whether the payment actually succeeded. You're left with two bad options: don't retry (might get permanently stuck), or retry (might double-charge).

The author honestly admits: no distributed systems trick can provide "exactly-once" semantics for an arbitrary external API that doesn't participate in the transaction. If the downstream API supports idempotency keys, use them. CellaFlow's goal is something else — making the execution process surrounding the side effect itself durable and recoverable: once ownership is lost, another worker can eventually take over rather than letting the workflow deadlock permanently. That distinction is the core of what they want to measure.

Idempotency Key is the industry-standard approach to handling duplicate calls to external APIs. The caller includes a unique identifier with each request, and the external service is responsible for deduplication: if a request with the same idempotency key has already been processed, it returns the original result without executing the operation again. Major payment services like Stripe and Braintree natively support idempotency keys.

Exactly-once semantics is the strongest delivery guarantee in distributed systems, meaning an operation is executed exactly once — neither lost nor duplicated. By contrast, at-least-once (may duplicate) and at-most-once (may be lost) are weaker, easier-to-implement guarantees. When uncontrollable external systems are involved, exactly-once is typically only achievable through "the external system actively participating in two-phase commit (2PC)" or "the external system providing an idempotent interface" — otherwise, no matter how clever the caller is, it cannot unilaterally guarantee exactly-once. This is the fundamental reason the author candidly states that "no distributed trick can provide exactly-once for an arbitrary external API."

The Coordination Challenge of Multi-Agent Systems

When there's more than one agent in the system, things get more interesting. Imagine a support agent, billing agent, and fraud agent all pointing at ticket 12345. They may have completely different sessions, thread IDs, execution histories, models, and prompts, but they might all independently conclude: "this customer needs a refund."

If you use "each agent's own idempotency key," the system sees three different operations (support/thread-A, billing/thread-B, fraud/thread-C), but from a business perspective this should be the same thing.

The author makes a valuable point: the unit of coordination shouldn't be the caller — it should be the "work" itself. CellaFlow's approach is to bind idempotency scope to the business identifier:

@tool(
    tool_name="issue_refund",
    scope=IdempotencyScope.SCOPE_SHARED,
    shared_on=["ticket_id"],
)

This way, no matter which agent initiates it, as long as ticket_id = 12345, they converge to the same execution. The agent's identity becomes secondary; the work's identity comes first.

Idempotency Scope determines which dimensions the system uses to judge "whether two calls are the same thing." The most common design uses caller identity (e.g., session ID, thread ID) as the scope — sufficient for single-agent scenarios, but in multi-agent scenarios it causes the same business operation to be treated as multiple distinct requests.

Binding scope to business identifiers (such as ticket_id, order_id) is a "work-centric" design philosophy, in contrast to the "caller-centric" thinking of traditional RPC frameworks. Similar patterns appear in event-driven architectures and the Saga pattern: the subject of coordination is the business transaction itself, not the microservice instance that triggered it. For multi-agent systems, this means maintaining a registry of "work in progress" at the framework level rather than the business code level, using business keys as the global deduplication basis.

Zombie Workers and Fencing

There's also a "zombie" problem: worker A holds an operation → gets stuck → lease expires → worker B takes over → B completes the work → A suddenly wakes up.

A isn't necessarily malicious or broken — it simply doesn't know it's already lost ownership. Without fencing, it might come back after B has taken over and write stale state, corrupting data consistency.

So the system needs to know not just "has this work been claimed?" but "who owns it right now, and is this worker still allowed to act?" That's exactly what ownership epochs and fencing are designed to solve.

The Problem Space CellaFlow Wants to Solve

The author concludes: the model can make the right decision — the hard part is ensuring that decision remains correct when processes crash, networks time out, workers disappear, retries kick in, two workers compete, ownership changes, an old worker wakes up, and another agent independently attempts the same business operation.

The required primitives ultimately come down to: durability + shared work identity + idempotency + leases + recovery + fencing.

This discussion is worth reading for anyone building production-grade agents, workflow engines, distributed systems, or payment/order infrastructure. The author also candidly says that more than whether CellaFlow wins the benchmark, what they really want to hear is: if you've ever had a worker crash after executing an irreversible side effect but before recording the result, how did you handle it? Both the benchmark and the project are open-sourced on GitHub.