An Elixir Polling Loop Burned $63 Overnight: Lessons from a Code Review

Starting from the Anomalous Card on the Board

One morning I opened the board and found a card in an unusual state — not crashed, but unusually active. Behind the card had accumulated a large volume of activity records starting from the previous night, and I hadn't touched it since lunch.

On instinct, I started digging into what had happened that night. The result was shocking: 243 agent runs in 8 hours, roughly 31 million input tokens, $64.25 spent — of which $63.10 was pure waste, with zero valid commits.

What made it even harder to accept: this code wasn't written by me. It was auto-generated by Claude Code on the Camelot platform based on a task I had created. I read the diff, approved the merge. At the time, it looked completely normal.

The loop didn't stop because I caught it or blocked it — it stopped because the runner eventually crashed on malformed output, pushing the task into an error state. In hindsight, even that crash seemed like the most responsible member of the team.

A Two-Minute GitHub Poller

In Camelot AI, tasks flow through a board. When an agent opens a Pull Request, the task enters the pr stage and the poller takes over: check GitHub every two minutes and react. Merged → done; CI failed → have the agent re-engage; reviewer requested changes → have the agent re-engage.

The poller itself is a Camelot task. I described the requirement in natural language, and the agent auto-generated the PR. Using Elixir's Ash and Oban frameworks, the trigger required almost no code to write:

triggers do
  trigger :check_pr_status do
    action(:check_pr_status)
    scheduler_cron("*/2 * * * *")
    queue(:github)
    max_attempts(3)

    where(
      expr(
        not is_nil(pr_number) and
          stage == :pr and
          project.status == :active
      )
    )
  end
end

That's the whole thing. AshOban looks up matching rows every two minutes based on the where expression and runs the action for each. I've hand-written dozens of triggers like this — boring but reliable, and that familiarity is exactly what made me skim this diff.

The problem is that behind this boring, reliable trigger is an action that spins up a container and burns LLM tokens.

The Cost of a Cascading Trigger

The core issue is that the code tests a state ("someone requested changes") rather than an event ("someone requested changes and I haven't handled it yet").

GitHub's reviews endpoint returns all historical review records. Once a CHANGES_REQUESTED review is submitted, it exists in the list forever. Even if the reviewer later approves, the old CHANGES_REQUESTED remains.

So has_review_state?(reviews, "CHANGES_REQUESTED") is really asking "has this PR ever been requested for changes?" rather than "does this PR currently need changes?" Once true, always true.

Every two minutes, the trigger matched; every two minutes, the condition was true; every two minutes, Camelot dispatched an agent to fix feedback that had already been addressed hours earlier. The agent dutifully looked, found nothing to do, exited cleanly — 243 times.

This is what's called a level-triggered check. In an ordinary Elixir application, a redundant level trigger just means one extra database query every two minutes, and you'd never notice. But here the executor spins up a container, clones the repo, and sends roughly 130K tokens of context to a language model.

Redundant polling is a rounding error in a normal app. Here, it's the bill.

There was also a second-order effect that made things worse: the prompt includes the PR conversation history. Each useless run adds its own noise to the conversation, causing the per-run cost to climb steadily from about $0.12 early on to $0.45 later. The loop wasn't just wasting — it was accelerating the waste.

Level-triggered vs. edge-triggered are fundamental concepts in electronic engineering and digital system design, with literal meanings from signal level (a sustained high/low potential) versus signal edge (the instantaneous transition from high to low or low to high). Level-triggered means "respond continuously as long as the signal is in a certain state"; edge-triggered means "respond once, only at the moment the signal changes." This distinction applies equally to distributed systems and event-driven architectures: treating "there exists some unhandled feedback" as the trigger condition (level-triggered) causes repeated responses; treating "a new piece of feedback was just submitted" as the trigger condition (edge-triggered or event-based) means each piece of feedback is consumed exactly once. In practice, doing level-triggering safely usually requires a persistent "watermark" marker for what's been processed — for example, recording a last_seen_at timestamp or message ID — to converge "state exists" into "state exists and has not yet been processed." A level trigger without a watermark degenerates into an infinite loop once the state stays true for a long time, which is exactly the root cause of this incident.

Why All Three Lines of Defense Failed

This system wasn't without limits. Three mechanisms were supposed to stop it, but the loop bypassed them all.

The freshness check was dead code on this path. The system has a pr_comments_seen_at timestamp specifically to avoid reprocessing feedback. But it was only queried in the comments branch, and the comments branch sits below the reviews branch within the same cond. The reviews branch never touches it. The guard existed, was correct, but was unreachable.

The branch reset the counter that was supposed to cap it. Look at this call:

request_changes(task, comments, 0)

The third argument is the auto-fix attempt counter. The system caps consecutive re-dispatches (default 2), but that cap only counts merge-conflict and CI-failure fixes, and this branch passes 0, resetting the counter on every pass. The guard was actively disarmed by the code, twice an hour, all night long.

Branch order made recovery impossible. CHANGES_REQUESTED was checked before APPROVED. Even if a reviewer later approved the PR, the loop kept triggering — the earlier branch matched first, and the approval branch was never reached.

None of these are exotic bugs. Each one reads as normal in a code review — I know this firsthand, because every line appeared on my screen and I read it all "smoothly."

A key design feature of the GitHub PR reviews API is often misread: GET /repos/{owner}/{repo}/pulls/{pull_number}/reviews returns the list of all submitted review records on that PR, each record containing state (APPROVED, CHANGES_REQUESTED, COMMENTED, DISMISSED, etc.), a submitted_at time, and the reviewer's identity. In other words, it's an append-only log, not a "current verdict snapshot." The same reviewer can submit multiple reviews, and old and new records coexist in the list. To determine "whether this PR is currently requested for changes," you must group the records by reviewer, take each person's latest stateful review, and combine it with the last commit time to determine whether that verdict has been superseded by a later push. Doing Enum.any?(state == "CHANGES_REQUESTED") on the list directly is equivalent to asking "has anyone ever requested changes historically," and once true it stays true forever — this class of error is extremely common in homegrown GitHub tooling.

The Fix: Compare Reviews Against the Last Commit

Two changes, and the important one isn't the obvious one.

First, collapse the review list into each reviewer's current verdict before asking any question:

@review_states ~w(APPROVED CHANGES_REQUESTED DISMISSED)

def latest_reviews(reviews) do
  reviews
  |> Enum.filter(&(&1["state"] in @review_states))
  |> Enum.group_by(&reviewer_login/1)
  |> Enum.map(fn {_login, submitted} ->
    Enum.max_by(submitted, &review_date/1)
  end)
end

Group by reviewer, take each reviewer's latest stateful review. COMMENTED and PENDING carry no verdict and never override a previous verdict.

Second — and this is what actually stops the loop — apply the same staleness guard to reviews that the comments path already had:

def changes_requested?(reviews, last_commit_date, seen_at) do
  reviews
  |> latest_reviews()
  |> Enum.any?(fn review ->
    review["state"] == "CHANGES_REQUESTED" and
      unaddressed?(review_date(review), last_commit_date, seen_at)
  end)
end

The verdict must be newer than the last commit (pushing a fix counts as addressed) and not yet processed. Two independent conditions are what mark a review as handled.

In the actual incident, this alone would have stopped the loop on the first pass. The review was submitted at 11:46:46Z, while the head commit was 11:52:34Z. The review had already gone stale before the loop even began. One timestamp comparison — that was the whole tragedy.

The Second Bug: An Optional Call Inside a with Chain

There was a second failure in the same PR, same smell.

The poller chained five GitHub calls in a single with:

with {:ok, pr_data} <- Client.get_pull_request(...),
     {:ok, reviews} <- Client.list_pull_request_reviews(...),
     {:ok, comments} <- Client.list_pull_request_comments(...),
     {:ok, commits} <- Client.list_pull_request_commits(...),
     {:ok, check_runs} <- fetch_check_runs(...) do
  # ...
else
  {:error, reason} ->
    Logger.warning("Failed to check PR for task #{task.id}: #{inspect(reason)}")
end

This looks like good Elixir code. The problem is that the last call requires the GitHub App's Checks: Read permission, and without it GitHub returns 403.

One 403 and the whole with collapses into a single log line. Merge-conflict handling, review handling, comment handling, merged detection — all stop, silently failing for every task in the pr stage, all because of one optional enhancement call.

The fix is to explicitly allow that call to fail:

@spec best_effort_check_runs({:ok, [map()]} | {:error, term()}) :: {:ok, [map()]}
def best_effort_check_runs({:ok, runs}), do: {:ok, runs}

def best_effort_check_runs({:error, reason}) do
  Logger.warning("check-runs unavailable (#{inspect(reason)}); treating as no checks")
  {:ok, []}
end

Degrade to "no checks" and keep going. CI failures won't be detected until the permission is granted — annoying but survivable. Everything else keeps working, which is the point.

with provides one error channel for the whole chain — that's a feature. When one link in the chain isn't required, it becomes a bug, and it fails silently.

Elixir's with expression is used to chain multiple operations that can fail: it matches each clause in order, and as soon as one clause matches the pattern in the else branch, the entire chain short-circuits to else. The value of this design is making "the first error" a unified failure exit, avoiding layers of nested error-handling code. But it also carries a hidden risk: with implicitly treats every step in the chain as a "required success condition," so once a step is optional (say, an API call purely to enhance information), a non-fatal failure silently destroys all downstream logic in the chain. The idiomatic remedy is to wrap optional calls in an adapter function that always returns {:ok, ...} (like best_effort_check_runs/1 in the article), downgrading it from "success/failure" semantics to "success/degrade," and expressing that contract explicitly in the @spec. This lesson applies to any chained construct with a unified error channel, including Rust's ? operator compositions and catch boundaries in Promise chains.

How I've Changed the Way I Work

I've been reviewing agent diffs the wrong way. I review naming, structure, fit with the codebase, whether I'd write it the same way. All of that was fine here, which is exactly why I merged. Now I ask three questions of any diff involving a loop: What makes it trigger? What makes it stop? What does one pass cost?

Level-triggered checks need a termination condition, not just a trigger condition. "Is this state true?" isn't enough. "Is this state true and have I not yet processed it?" is the real question, and the second half must be persistent.

Never reset a guard inside a branch the guard protects. If I have to point to the one line that cost me $63, it's that 0. One character I didn't write and didn't question.

Best-effort calls don't belong in a with chain. If a step is optional, write a function that always succeeds and say so in the type signature.

Cost is a first-class signal. I had logs, I had session records. What I didn't have was anything that could tell me "this task ran 243 times today" or "this task cost $60." A counter and a threshold would have caught this in 20 minutes instead of 8 hours.

AI agent-generated code review is developing into a new discipline distinct from traditional code review. Traditional CR focuses on naming, structure, test coverage, and conformity to team conventions; those dimensions are still necessary for agent-generated code, but nowhere near sufficient, because the failure modes of agent code stem from the hallucination of "it looks familiar" — it copies common patterns but is insensitive to the context in which those patterns live. For this class of code, the industry is gradually recommending several additional checkpoints: the trigger condition of loops/scheduling logic (what starts it), the termination condition (what stops or dedupes it), the real cost of a single execution (does it spin up a container, call an LLM, produce a billing event), whether error paths are silently swallowed, and whether guards have unreachable or bypassable branches. Writing these questions explicitly into the review checklist narrows the incident surface of agent code far more than raising attention to code style.

Conclusion

Three lines for the trigger, four lines for the loop branch. Nothing complicated in this incident would have mattered in an ordinary CRUD app — redundant polling would cost a few hundred cheap queries, and I'd never have known.

What changed is the executor at the other end of the loop. When the executor spends money, "harmless redundant work" stops being a category. Every pass has to prove its worth.

And the harder part to write. Camelot exists because I believe agent work should pass a human gate before becoming reality. The gate was there, and it worked exactly as designed. The gate was me, on a Tuesday evening, looking at a diff that looked like something I'd write, clicking approve. The human approval loop isn't a guarantee — it's a place that requires an actual human to be present — and that's a much harder commitment than what I thought I was making.

My practice says: notice the recurring thought, stop feeding it. Apparently my poller needed that teaching, and so does my reading.

One-line Summary

A level-triggered PR poller that checked historical review state instead of current status fired 243 agent runs overnight, wasting $63.

Paragraph Summary

One morning the author found a board card unusually active, and tracing it back revealed 243 agent runs in 8 hours the previous night, consuming roughly 31 million input tokens and $64.25 — of which $63.10 was pure waste, with zero valid commits. The root cause was a GitHub poller generated from natural language that looked boring and reliable: it checked the PR every two minutes, but its logic asked "has it ever been requested for changes" rather than "does it currently need changes," and since the GitHub reviews API returns a historical list, once CHANGES_REQUESTED appears it stays true forever. All three defenses that should have stopped the loop failed: the freshness check was unreachable on the reviews branch, the auto-fix counter was reset by a 0 argument, and branch order made the approval branch unreachable. The fix is to compare reviews against the last commit time, add a staleness guard, and degrade gracefully on optional API calls. The author reflects on how AI-agent code should be reviewed.