Durable Execution for AI Agents: Checkpoints, Retries, and Recovery

AI agents are moving from demo toys to production systems, but one fundamental question remains unresolved: when a process crashes, a service is deployed, or multi-day human approval is required, how does the agent's execution state survive? Durable Execution exists precisely for this. Drawing on LangGraph's crash tests and type-checked code, this article breaks down how checkpoints, retries, and recovery let agents "stay alive" through failures.

Why Agents Need Durable Execution

Traditional script-style agents keep state in memory. Once the process terminates, all intermediate reasoning, tool call results, and pending steps are lost. Failures in production come from far more sources than you might imagine: containers evicted by the scheduler, rolling deployments, downstream API timeouts—or simply waiting for a human approval that takes three days to complete.

The core idea of durable execution is to write each step's execution result to recoverable storage. The agent is no longer a "run-once" function but a workflow that can be paused, replayed, and branched. According to the original article, this lets agents survive crashes, deployments, and multi-day approval cycles instead of losing all progress at the first exception.

Durable Execution first matured in the distributed workflow space, with systems such as Temporal, Cadence, and AWS Step Functions built around it as a core abstraction. Its fundamental contract is to persist the execution state of a long-running process—local variables, call stack, pending steps—to external storage, so the process can pause or exit at any point and resume precisely in another process later. Unlike traditional database transactions, durable execution targets orchestration flows that may last hours or even days, rather than millisecond-level operations. For agents, this means LLM calls, tool executions, and human approvals can all be treated as recoverable activity units rather than one-shot function calls.

It's important to distinguish this: "storing state in a database" is not the same as durable execution. The latter requires the framework to rebuild the exact execution position on recovery, including branch paths and the returned results of completed steps, thereby avoiding duplicate calls to external services. This is the fundamental difference from simple logging or periodic snapshots.

Checkpoints: Turning Every Step into Recoverable State

A checkpoint is the cornerstone of durable execution. Whenever the agent completes a node—an LLM call, a tool execution, a routing decision—the system serializes and persists the current state. On recovery, the execution engine continues from the most recent valid checkpoint rather than rerunning from scratch.

LangGraph's crash tests demonstrate this vividly: forcibly killing the process mid-flow, then restarting, the agent resumes from the last checkpoint, completed steps are not re-executed, and unfinished steps continue to advance. The original article emphasizes that a checkpoint is not merely "saving progress"—it also defines the semantic boundary of recovery: which side effects have already occurred and which must be retried.

Consistency Between Checkpoints and Side Effects

An easily overlooked detail is side effects. If a tool call has already written data to an external system but the checkpoint has not yet been persisted, recovery may result in duplicate writes. When discussing retry mechanisms, the original article points out that durable execution must be paired with idempotent design: either the tool itself is idempotent, or completion markers for side effects are recorded in the checkpoint.

Idempotency means that executing the same operation once and multiple times produces the same external effect. In the context of durable execution, idempotency is the key property that makes recovery safe. Common implementations include: generating a unique request ID for each tool call and deduplicating in the external system; replacing blind writes with "check-then-write" upsert semantics; or explicitly recording the completion status of each side effect in the checkpoint and skipping completed calls on recovery.

Notably, idempotent design often requires cooperation from external systems—for example, payment gateways supporting idempotency keys, or message queues supporting deduplication. If the downstream system itself provides no idempotency guarantee, checkpoint markers on the agent side can only achieve "at-least-once" semantics, and conflict detection is still needed at the business layer. Therefore, the idempotent semantics of tool nodes should be considered together with checkpoint granularity at the design stage, rather than left for recovery logic to patch up after the fact.

Retries: Distinguishing Retryable from Non-Retryable Failures

Retrying is not simply running again. A durable execution framework needs to determine the failure category: network timeouts, rate limiting, and temporary service unavailability are usually retryable; whereas parameter errors, permission denials, or business logic failures should terminate or be routed to human handling.

The practice given in the original article is to declare retry policies at the node level, including maximum retry count, backoff strategy, and retryable exception types. Combined with type-checked code, retry configurations can be verified at compile time, avoiding discovering missing policies only at runtime. This "type-safe durability" approach turns the agent's fault-tolerance behavior from an implicit convention into an explicit contract.

Backoff Strategy determines the waiting time between retries. The simplest is a fixed interval, but production systems more commonly use exponential backoff (e.g., 1s, 2s, 4s, 8s) combined with random jitter to avoid the "thundering herd" effect where a large number of failed requests retry simultaneously. For agent nodes calling external APIs, waiting time should also be dynamically adjusted based on server rate-limit responses (such as 429 status codes and the Retry-After header).

When retries are exhausted, durable execution frameworks typically do not discard the task outright but route the flow into a failure-handling branch or a human-intervention queue, which naturally connects to the human approval scenarios mentioned in the original article. Reasonable backoff and cap settings, plus fast failure on non-retryable errors, can prevent agents from wasting large amounts of call quota during failures while ensuring eventual consistency.

Recovery: The Key to Cross-Deployment and Multi-Day Approvals

The scenario that truly separates production-grade agents from prototypes is long-running flows requiring human intervention. Consider a procurement approval agent that waits three days for human sign-off after submitting a request. A process cannot stay suspended waiting indefinitely; durable execution lets the agent release resources while waiting, then resume from the checkpoint and continue subsequent steps once approval arrives.

The same applies to deployment scenarios. Rolling updates kill old processes; if agent state exists only in memory, in-flight tasks are lost. Durable execution externalizes state to the storage layer, so a new-version process can take over from the same checkpoint, achieving seamless migration. The original article's crash tests cover such scenarios, verifying that the post-recovery execution path matches the uninterrupted one.

Recovery Is Not Replay

It's important to distinguish recovery from replay. Replay re-executes historical steps to rebuild state, which is costly and may trigger side effects; recovery directly continues from persisted state. The goal of durable execution is the latter—checkpoints store state snapshots rather than operation logs, making recovery faster and safer.

Practical Recommendations

For teams building production-grade agents, the practices from the original article can be summarized as follows: choose a low-latency, highly available backend for checkpoint storage; define explicit idempotent semantics for each tool node; use type checking to constrain retry and recovery configurations; and proactively inject crashes and deployment interruptions in tests to verify recovery paths.

Durable execution is not a nice-to-have feature but the infrastructure that takes agents from demos to production. When agents need to run for hours, days, or across multiple deployments, checkpoints, retries, and recovery together form their "survival mechanism."

One-line Summary

Durable execution uses checkpoints, retries, and recovery to keep agents alive across crashes, deployments, and multi-day approvals.

Paragraph Summary

The article points out that traditional script-style agents keep state in memory, so process termination loses all intermediate results, while production failures include container eviction, rolling updates, downstream timeouts, and multi-day human approvals. Durable execution writes each step's result to recoverable storage, turning the agent into a workflow that can be paused, replayed, and branched. Checkpoints are the cornerstone, storing state snapshots rather than operation logs, so recovery resumes from the latest valid checkpoint instead of replaying from scratch. Retries must distinguish retryable from non-retryable failures and declare maximum counts, backoff strategies, and exception types at the node level, verified at compile time via type checking. Recovery supports cross-deployment takeover and long-running human intervention; implementation also requires attention to checkpoint storage selection, tool idempotent semantics, and proactive fault-injection testing.