Agent Retry Logic Hides a Race Condition: Query Lies and Duplicate Dispatch

Your AI Agent dispatched an action, then immediately called the provider's query endpoint to confirm success. The query returns "not found," so you mark it as failed and retry — but the action actually already succeeded. You just created a duplicate dispatch.

This isn't a bug in the retry logic. It's a flaw in the trust model for query results.

The Root Problem: Successful Writes Don't Mean Visible Reads

Many providers use an eventually consistent model. A write returns HTTP 200 with no errors, but the artifact doesn't immediately appear on read replicas. Depending on the provider and load, replication lag can be 200 milliseconds, 2 seconds, or even 30 seconds.

A query executed immediately after dispatch reads a clock that hasn't synchronized yet.

There are two failure modes that look completely identical here:

Query result: "not found"
Case A: The action genuinely failed — no write occurred
Case B: The action succeeded — the read replica hasn't synchronized yet

Both return the same result, but only Case A should trigger a retry.

This is a textbook race condition: the Agent's verification model assumes "read-your-writes" visibility is immediate, but read replicas in distributed systems have inherent synchronization delay. When retry logic depends directly on query results for decision-making, the replication window becomes a breeding ground for duplicate dispatches.

Eventual Consistency is a consistency model widely adopted in distributed systems, as opposed to strong consistency. Under strong consistency, once a write returns success, all subsequent reads immediately see the new value; eventual consistency only guarantees that "if no new writes occur, all read replicas will eventually converge to the same state," without making strict promises about how long convergence takes. Typical distributed storage (such as Amazon S3, DynamoDB, Cassandra) and many SaaS provider APIs are built on this model because they need to balance availability and latency — synchronous replication to all replicas would significantly increase write latency.

Read-Your-Writes is a type of consistency session guarantee, meaning that after a client writes, that same client's subsequent reads will definitely see the write. Single-node databases typically satisfy this guarantee naturally, but cross-region, multi-replica distributed systems often require additional mechanisms (such as session tokens or primary replica routing) to achieve it. AI Agent verification models typically implicitly assume "read-your-writes is immediately visible," while the actual provider API being called may not offer this guarantee. This cognitive gap is the root cause of the race condition.

The Fatal Flaw of the Naive State Machine

A typical retry confirmation function might look like this:

suspend fun confirmDispatch(record: ActionRecord, provider: ActionProvider): ActionRecord {
    val artifact = provider.lookup(record.dispatchId)
    return if (artifact != null) {
        record.copy(delivery = DeliveryState.CONFIRMED)
    } else {
        // BUG: assumes "not found" equals "failed"
        // but it might just be "not yet propagated"
        record.copy(delivery = DeliveryState.FAILED)
    }
}

If this code runs immediately after dispatch, the "not found" response is ambiguous — it can't distinguish genuine failure from replication lag. Once marked as FAILED, the retry queue may initiate a duplicate dispatch.

According to StareBrain's technical analysis, this pattern is especially dangerous for on-device actions: when the accessibility layer confirms a UI operation has been triggered, whether the downstream provider has recorded the operation belongs to a different clock. Two clocks, one verification model — the gap between them is the source of duplicate dispatches.

Race Condition refers to when a system's output depends on the uncontrolled ordering or timing of events. In this case, there's a race between the Agent's query operation and the read replica's replication operation: the query may arrive before replication completes or after, and the two timings return different results, but the caller cannot determine from the result itself which timing occurred.

Race conditions are dangerous in distributed systems because they're often non-reproducible — under low load or in local test environments, replication lag might be only a few milliseconds, queries almost always read the latest data, and the bug hides. Once in production, cross-region deployments, or high-load periods, replication lag amplifies to seconds and duplicate dispatches suddenly erupt. Therefore, fixes for this class of problems cannot rely on "making queries faster" or "increasing retry counts" — they must introduce explicit constraints on the time dimension at the model level, turning uncertain timing into decidable states.

The Fix: Minimum Age Gating

The core idea: PENDING records must not enter the reconciliation process before the consistency window closes.

data class ActionRecord(
    val actionId: String,
    val dispatchId: String,
    val providerObjectId: String?,
    val delivery: DeliveryState,
    val acceptance: AcceptanceState,
    val dispatchedAt: Instant,
    val confirmedAt: Instant?,
    val eligibleForReconciliationAt: Instant  // dispatched_at + consistency_window
)

fun ActionRecord.isEligibleForReconciliation(): Boolean {
    return Instant.now().isAfter(eligibleForReconciliationAt)
}

The reconciliation loop checks eligibility before querying:

suspend fun reconcileIfEligible(record: ActionRecord, provider: ActionProvider): ActionRecord {
    if (!record.isEligibleForReconciliation()) {
        // Too early — query results are untrustworthy
        return record
    }
    val artifact = provider.lookup(record.dispatchId)
    return if (artifact != null) {
        record.copy(
            providerObjectId = artifact.id,
            delivery = DeliveryState.CONFIRMED,
            confirmedAt = Instant.now()
        )
    } else {
        // Now "not found" genuinely means failure; the consistency window has closed
        record.copy(delivery = DeliveryState.FAILED)
    }
}

This effectively adds a "freshness period" to read operations in distributed systems: query results within the window constitute no evidence whatsoever.

Minimum Age Gating is philosophically aligned with Lease and Fencing Token concepts in distributed systems: by introducing a time-based barrier, the system refuses to make decisions based on data during a known-untrustworthy time window, rather than trying to eliminate the latency itself.

Unlike common exponential backoff retries, minimum age gating isn't "wait a bit and try again" — it's "until the window closes, no query result is accepted as evidence." The key difference: backoff retries still consume query results, just with delayed consumption; gating completely ignores results within the window, semantically severing the reasoning chain of "query returns not found → determine failure."

Implementation requires attention to clock source selection — monotonic clocks should be used rather than wall clocks for computing eligibility time, to avoid errors from NTP adjustments or timezone changes. In cross-process, cross-machine scenarios, you may also need to rely on timestamps or version numbers provided by the server rather than local time.

Where Consistency Window Values Come From

Window durations aren't arbitrary. Code should configure specific values per provider:

object ConsistencyWindows {
    // From provider documentation
    val STRIPE_EVENTS = Duration.ofSeconds(5)
    // From empirical observation (no public SLA)
    val GOOGLE_CALENDAR = Duration.ofSeconds(10)
    // Conservative default when no documentation exists
    val DEFAULT = Duration.ofSeconds(30)
}

Key principle: if the provider doesn't publish replication lag, measure it yourself. Dispatch a test write, poll the read endpoint, record when it becomes visible. Always test under load, not ideal conditions.

Late Confirmations and Replacement Races

Another related problem: what happens when a confirmation signal arrives after the record has already been marked FAILED and a replacement has been queued?

suspend fun handleLateConfirmation(record: ActionRecord, artifact: ProviderArtifact): ActionRecord {
    return when (record.delivery) {
        DeliveryState.FAILED -> {
            // Late confirmation — the action succeeded despite the earlier failure verdict
            // Cancel any queued replacement before updating state
            cancelQueuedReplacement(record.actionId)
            record.copy(
                providerObjectId = artifact.id,
                delivery = DeliveryState.CONFIRMED,
                confirmedAt = Instant.now()
            )
        }
        DeliveryState.PENDING -> {
            // Normal confirmation path
            record.copy(
                providerObjectId = artifact.id,
                delivery = DeliveryState.CONFIRMED,
                confirmedAt = Instant.now()
            )
        }
        DeliveryState.CONFIRMED -> {
            // Already confirmed — idempotent, no change
            record
        }
    }
}

Replacement dispatch must read delivery state immediately before firing, not at scheduling time. Scheduling a replacement and actually dispatching are two separate moments; the state check belongs to the dispatch moment.

Idempotency means the same operation produces the same effect whether executed once or multiple times. In distributed systems, due to network partitions, timeout retries, and message replays, any cross-service call must assume "messages may arrive repeatedly." In this case, handleLateConfirmation returning the original record directly for CONFIRMED state is an expression of idempotent design — receiving the same confirmation signal repeatedly doesn't change state or trigger side effects.

More insidious is the replacement race: scheduling a replacement and executing it are two separate moments. If the record is FAILED at scheduling time but a late confirmation arrives before execution and transitions it to CONFIRMED, then executing the replacement based on the scheduling snapshot would cause a duplicate dispatch. Therefore replacement logic must re-read state at the last moment before actual dispatch, not reuse the state copy from scheduling time. This is why distributed transactions often say "there must be no gap between check and use" — all state-based decisions should occur near the point where side effects happen.

Behavior in the Complete State Machine

enum class DeliveryState {
    PENDING,      // Dispatched, awaiting confirmation
    CONFIRMED,    // Query returned the artifact
    FAILED        // Query still returned no result after consistency window closed
}

fun ActionRecord.resolvedStatus(): ActionStatus = when {
    delivery == DeliveryState.PENDING && !isEligibleForReconciliation() -> 
        ActionStatus.DENIED_UNRESOLVED  // Window still open
    delivery == DeliveryState.PENDING && isEligibleForReconciliation() -> 
        ActionStatus.DENIED_UNRESOLVED   // Query needed
    delivery == DeliveryState.FAILED -> ActionStatus.FAILED
    delivery == DeliveryState.CONFIRMED && acceptance == AcceptanceState.REJECTED -> ActionStatus.FAILED
    delivery == DeliveryState.CONFIRMED && acceptance == AcceptanceState.ACCEPTED -> ActionStatus.CONFIRMED
    else -> ActionStatus.DENIED_UNRESOLVED
}

DENIED_UNRESOLVED won't decay to FAILED from an early query — it requires a query outside the window to transition. Queries within the window constitute no evidence.

Practical Implications

The consistency window is part of your verification model. If it doesn't appear in code as a named constant (rather than a magic number), it's an assumption. And assumptions are where duplicate dispatches live.

For developers, this means that when designing Agent retry logic, the provider's consistency window must be treated as a first-class citizen. Don't assume query results reflect the current true state — explicitly model "when queries can be trusted." This is a paradigm shift from "query equals truth" to "queries must wait for truth to become visible."

Two clocks, one verification model — the gap between them is your source of duplicate dispatches. Put that gap in code, not in luck.

One-line Summary

AI Agents must explicitly model provider consistency windows to avoid duplicate dispatches caused by read replica lag.

Paragraph Summary

The article points out that when an AI Agent queries a provider for confirmation immediately after dispatching an action, if the provider uses an eventually consistent model, a successful write doesn't mean immediate readability — a "not found" query result may just be replication lag rather than genuine failure. A naive state machine treating "not found" directly as FAILED triggers unnecessary retries, causing duplicate dispatches. The fix is to introduce minimum age gating: setting an eligibleForReconciliationAt for each record (dispatch time plus consistency window), where query results within the window constitute no evidence. Consistency windows should be named constants determined from documentation or measurement. The article also discusses that late confirmations require idempotent handling, that replacement dispatch must immediately read the latest state, and emphasizes that queries within the window won't let DENIED_UNRESOLVED decay to FAILED.