Coinbase Engineer: How to Determine Whether an AI Agent Can Operate Unattended

Whether an AI agent can truly take over a business process isn't a decision made on gut feeling—there's a verifiable set of criteria for judging it. In a talk at AI Engineer, Coinbase engineer Evan Kormos laid out a complete methodology spanning flow diagrams, an evaluation system, and a phased handover of control.

Starting from Policy: Writing "Intent" as Rules

Kormos's starting point is interesting: not the model, not the tools, but policy. He calls policy "the organization's original intent," or "the original system prompt"—it defines how work should be done, especially around risk and compliance. When writing it, use explicit normative verbs like must, may, and should.

The key move is separating build state from run state. Policy is defined or modified at a point in time, while the system keeps running and keeps changing. Once the two are separated, policy changes have something like a "compilation" moment and don't get tangled up with the rapid evolution at runtime.

In the classic triad of people, process, and technology, Kormos sees the agent as an operating layer that cuts across all three: it collaborates with people, executes part of the process itself, and is itself made of technology. The focus of this talk lands on "the intersection of agents and process"—what work to give them, what boundaries to draw, and how to build trust in their output.

DAG and DCG: Deterministic on the Outside, Adaptive on the Inside

The basic shape of a process is a graph. The business expects work to flow in a particular direction through a series of gates—for example, changes must be approved before deployment, and customers must be verified before an account can be opened. This kind of loop-free process can be expressed with a directed acyclic graph (DAG), which captures steps and dependencies. In the open-source world, Apache Airflow is a typical example of expressing processes and their dependencies as DAGs in Python code.

Kormos gave several scenarios where policy-driven processes are expressed as DAGs: commercial lending, listing assets in specific regions, and regulated software delivery. Software delivery is currently the area with the highest agent adoption, and it's the example he expands on most.

Interestingly, inside every node of the DAG there's actually a loop. Take the code review node: execute several steps, judge whether the work is "good enough," release it to the next step if so, and if not, absorb the feedback and iterate. This forms a directed cyclic graph (DCG), and open-source tools like LangChain and LangGraph can help manage DCG state.

This is the core recipe of the whole approach—deterministic on the outside, adaptive on the inside. The outer process, gates, and policy boundaries are stable; the inner "generate—check—iterate" loop lets the agent try repeatedly.

A DAG (directed acyclic graph) is a classic data structure for describing process steps and dependencies. Its "acyclic" property guarantees that work can only move forward and never develop circular dependencies, which makes it naturally suited to expressing business processes with approval gates. A DCG (directed cyclic graph), by contrast, allows cycles between nodes, capturing stages like "generate—check—iterate" that require repeated attempts. The key to combining the two is nesting: each node of the DAG encapsulates a DCG loop, with the outer layer guaranteeing process determinism and auditability and the inner layer accommodating the agent's adaptive behavior. Tools like LangGraph manage the loop state of a DCG through state machines and checkpoint mechanisms, making the iteration process pausable, resumable, and traceable.

Exception Flows and Bottlenecks: The Two Things Most Easily Overlooked When Adding Agents

Processes don't always go as planned, so you must design exception flows. There are two principles: don't let work get stuck, and don't pre-design for every edge case. Exception flows should be monitored—if delivered code is canceled or fails security standards, those are exactly the events you most need to know about.

Another, more subtle problem is traffic migration. Once an agent joins and one stage gets faster, pressure downstream shows up immediately: when code review gets faster, builds have to keep up; when builds get faster, security scanning becomes the new bottleneck. Kormos's judgment is that as processes are parallelized and accelerated, the bottleneck keeps shifting, so observability and measurement of the overall process matter more than writing the workflow tooling itself.

The Trustworthiness of the Loop Depends on the "Check"

Why is this hard? Kormos points to a fundamental challenge of AI engineering: the high variability of inputs and outputs. He believes higher-quality input can buy better output at lower compute cost, which comes back to building "trusted context."

He offered a concrete practice: inject the DAG's state into the agent so it knows where it sits in the global picture, giving it a longer time horizon—which in turn requires optimizing the cost and efficiency of this loop.

But the real key is the "check" step. Defining what counts as "good" is the hardest part, and you also need reliable evaluation methods; otherwise, more passes don't equal better results. In his words: the trustworthiness of the loop equals only the trustworthiness of the check.

Replacing "a Feel for the Model" with an Evaluation System

Traditional software builds quality confidence through systematic mechanisms like unit tests and test cases, so that when code changes you can detect whether something else broke. But agents are probabilistic: the same request can produce different answers, and multiple answers may all be acceptable, so comparing outputs word for word is far from enough.

Kormos's alternative: build representative datasets, clearly define the criteria for "good," and run evaluations repeatedly in series, treating outputs as statistical objects rather than doing single comparisons. Layer evals and traces (runtime logs) on top, and you get a confidence mechanism that complements traditional software testing. This means judging agent quality shifts from "feel" to "research and statistical data."

In the context of AI engineering, evals (evaluations) refer to systematically measuring the quality of model output using a representative set of input samples. Unlike the deterministic assertions of traditional unit tests, evals focus on pass rates and distributions in a statistical sense. Traces, meanwhile, record every call, input/output, and intermediate state in a complete agent run, and are the raw material for post-hoc attribution and discovering failure modes. The two are complementary: evals answer "whether overall quality meets the bar," while traces answer "exactly which step a given failure occurred at." Feeding typical failure cases exposed by traces back into the evals dataset creates a loop of continuous improvement—precisely the core mechanism for using engineering to replace subjective feeling in building quality confidence.

Which Nodes Can Be Handed to Agents?

Back to the lending example: KYC and financial review can run in parallel, while the credit memo, risk review, documentation, and decision proceed in sequence. For each node in the DAG, you need to ask a set of questions:

• Can it be easily checked against policy?
• Can it run within a bounded number of loops?
• Can it be traced?
• Can it fall back to a human when something goes wrong?

If the answer to all of these is "yes," the node is relatively suitable for automation. Kormos stresses that the point of discussion isn't "whether to use AI" but "whether we expect AI to take over a decision at a given point." Stages like the final decision should still be driven by humans.

The requirement that a node "can fall back to a human" is essentially the practical demand of human-in-the-loop design. In an agent system, fallback capability means a node must retain its complete intermediate state and decision rationale so that when a human takes over, they can continue from the breakpoint rather than starting over. This usually relies on checkpoint mechanisms and persistable run state. In addition, fallback implies a design for permission boundaries: an agent's actions should be revocable or overridable, and especially for high-risk operations such as production environment changes and fund transfers, an interface for human intervention must be reserved. Automated nodes without fallback design often pay a high price once they fail, so this is often treated as a disqualifying condition for whether a node can be automated.

Phased Handover: From Full Supervision to Handling Exceptions

The methodology ultimately lands on a phased path of handing over control, which Kormos calls "hill climbing":

1. Full supervision: The agent works alongside humans, who verify outputs one by one, refine datasets together with domain experts, define the criteria for "good," and stand up the confidence mechanism.
2. Sampled checks: Once measured confidence reaches the threshold set by policy, humans review only a portion of outputs while running more systematic evaluations through traces, ensuring the parts humans don't see still meet the bar.
3. Exception handling: After crossing the confidence threshold, humans mainly handle exceptions in the process while continuing to participate in refining datasets and safeguarding output quality.

The significance of this path is that unattended operation isn't a switch—it's a gradual curve with metrics, thresholds, and fallback.

A Few Conclusions

Kormos's closing is concise: deterministic on the outside, adaptive on the inside; build quality confidence through evaluation and statistics rather than a feel for the model; quality is a team sport, and AI engineers are responsible for turning product signals from all sides into well-built applications backed by MLOps; and agents will bring acceleration across the entire operating model.

For teams considering letting agents take over real business processes, the value of this framework isn't in any particular tool choice, but in how it breaks "can we trust it" down into engineering problems that can be designed, measured, and progressively verified.

One-line Summary

A Coinbase engineer proposes policy-driven design, DAGs wrapping DCGs, and statistical evaluation to gradually hand over business processes to agents.

Paragraph Summary

The article presents a methodology from Coinbase engineer Evan Kormos for letting agents take over business processes. It starts with writing organizational policy as explicit rules and separating build state from run state. Process modeling follows "deterministic outside, adaptive inside": DAGs express gates and dependencies, while nodes use DCGs to carry generate—check—iterate loops. After agents are introduced, teams must address exception flow design and bottleneck shifts caused by traffic migration; process observability matters more than the tooling itself. Because agent outputs are probabilistic, representative datasets and series-based statistical evaluation should replace word-for-word comparison to build confidence. Deciding whether a node can be automated requires answering four questions, and the handover path runs from full supervision to sampled checks to exception handling.