Coinbase's 9B Anti-Fraud Model Beats Opus 4.5: A Reproducible Methodology

Coinbase fine-tuned an anti-fraud model from open-source Qwen3.5-9B, and it beat Claude Opus 4.5 on their own risk-control benchmark. But the real story isn't the model itself — it's the entire system built around it.

What Coinbase Built

Onramp routes transactions to a risk service. Existing ML models and rule engines do the first-pass screening, and an LLM agent reviews downstream only the selected transactions, using recent transaction history and written review guidelines.

There are three key design decisions:

• Output the level only. The model returns a risk level (e.g., RISK_LEVEL: 5) with no explanation. Under a roughly 7,000-token prompt, Opus's P50 latency is about 1.5 seconds
• Decision-making lives in external code. The application layer builds context, parses the level, and executes the decision policy. The model itself never performs payment operations
• Ban-only authority. The agent can ban transactions that upstream approved, but it cannot override upstream ban decisions

In an A/B test at 50% traffic, the experimental group with the agent enabled saw a 30% reduction in fraudulent transactions and a 22% reduction in fraudulent amount. Coinbase estimates the fraud losses saved are roughly 3x the inference cost.

Why the Reward Function Is "Verifiable"

Most LLM reinforcement learning needs a judge model to assess answer quality. Fraud doesn't. Historical transactions eventually have outcomes: chargeback or confirmed fraud, or nothing. So the reward is simply a deterministic function of the model output and the known label.

Part 3 adds two details worth reproducing:

• Format check. Outputs that cannot be parsed are penalized
• Class balancing. When fraud samples are rare, "always output low risk" looks highly accurate. The reward function weights fraud and normal samples separately so a constant output can't game the score

GRPO samples a group of responses for the same prompt and pushes the model toward samples that score above the group mean. No value network, no judge model required.

RLVR (Reinforcement Learning with Verifiable Rewards) has been an important line of work in recent LLM post-training, with representative efforts including DeepSeek^Ad-R1 and the math/code training in Tulu 3. The core idea: when the correct answer to a task can be programmatically verified (e.g., a math problem has a unique numeric answer, code passes unit tests, a transaction is ultimately confirmed as fraud or legitimate), you no longer need to train a judge model to score it — a deterministic verification function can generate the reward signal directly. This has three benefits: rewards are no longer constrained by a judge model's biases and capability ceiling; rewards are fully reproducible during training, which makes debugging easier; and the cost of training and maintaining a reward model is eliminated. The trade-off is limited applicability — only tasks whose "outcomes are verifiable" can take advantage of this, while open-ended writing, creative dialogue, and similar tasks still rely on human preferences or model judges.

GRPO (Group Relative Policy Optimization) is a policy optimization algorithm proposed by DeepSeek as a simplified alternative to PPO. It samples a group (e.g., 8 or 16) of responses for the same prompt, uses the group's average reward as a baseline, and shifts probability mass toward samples scoring above the group mean while suppressing those below. Compared with PPO, GRPO removes the value network (critic), significantly reducing memory footprint and engineering complexity, while naturally fitting scenarios with "multiple answers to the same question and comparable rewards." In the fraud setting, each prompt corresponds to one transaction, multiple risk-level outputs are sampled, and the final chargeback outcome verifies which level was correct — precisely matching GRPO's assumptions.

Why Use Amount-Weighted Recall Instead of Just F1

F1 counts events. Missing a $5,000 withdrawal and missing a $12 card test count the same. Amount-weighted recall asks: what proportion of fraudulent amount did you intercept?

The core point of Part 2 is that a model can miss very few transactions yet miss a huge amount of money. Coinbase's benchmark contains 16,140 transactions, 7,293 users, and 813 confirmed fraud cases, drawn from the nine weeks before the agent went live to ensure the agent's own interceptions didn't affect the observations. The result: newer versions of Opus, Sonnet, and GPT all underperformed the previous version on recall, F1, and amount-weighted recall.

F1 score is the harmonic mean of precision and recall, measuring classification quality at the "event count" level. It treats every transaction as an equally weighted sample, so recall on the minority class (fraud) matters as much as precision on the majority class (legitimate). F1's limitation is that it assumes all errors cost the same — which almost never holds in risk control, medical screening, anomaly detection, and similar settings. Amount-weighted recall weights each fraud sample by the amount involved and asks: what share of total fraudulent amount did you intercept? When the amount distribution of fraud samples is extremely long-tailed (most fraud is a few dollars of card testing, a few are large withdrawals), the two metrics can point to completely different models. In practice, you usually also need to track amount-weighted precision or the equivalent "legitimate amount wrongly banned," because optimizing recall alone encourages the model to ban indiscriminately. This category of metrics is also often called "business-weighted metrics" or "cost-sensitive metrics," with mature usage in Kaggle's IEEE fraud detection competition and in production payment risk monitoring.

Notebook-Scale Pattern Validation

The author wrote a synthetic-data version in about 200 lines of Python (seed 18): 600 transactions, 47 labeled fraud, total amount $68,484.50. Upstream rules screen first, and routing selects 579/600 for review. The reviewer returns only strings like RISK_LEVEL: 4, external code parses with a strict regex, and bans when level ≥ 4.

The small-scale test surfaced several notable phenomena:

F1 and amount-weighted recall picked different winners. The speed reviewer caught 24/29 card tests but blocked zero withdrawals that upstream had approved. The context reviewer caught 12/12 withdrawals (amount recall 0.979) but wrongly banned $49,411.22 in legitimate transactions. Neither metric alone is enough — you must track both precision and wrongly-banned legitimate amount.

A general-purpose small model added no value here. LiquidAI's d1-omni-600M ran on CPU, described each transaction in natural language, and added 18 new bans — all false positives. Recall stayed at upstream's 0.319.

Class-balanced rewards caught the lazy strategy. "Always output RISK_LEVEL: 1" had a plain accuracy of 0.922 but an average reward of −0.000.

Ban-only authority limits direction, not damage. A deliberately buggy reviewer emitted RELEASE, RISK_LEVEL: 9, and prose, causing 344 parse errors. All upstream bans stayed intact, but it still randomly added 111 bans, 104 of them legitimate transactions. You need alerting on the ban rate.

When Not to Copy This Approach

No trustworthy labels. RLVR depends on the quality of the outcome labels. If fraud confirmation is slow, noisy, or missing, the reward becomes guesswork.

Low transaction volume. Coinbase's benchmark alone has 813 confirmed fraud cases. If you only have dozens, a prompt-based approach with a fixed evaluation set is cheaper and easier to reason about.

Policy changes weekly. Natural-language guidelines can be adjusted without changing the model. Policy baked into weights cannot. If rules change faster than you can retrain and evaluate, keep policy in prompts and external code.

No lifecycle owner. You need evaluation on domain outcomes, unified versioning of weights, prompts, and policy, and a tested rollback plan.

The prompt-engineering approach means writing the review policy, transaction history, and decision criteria into the system prompt and letting a general-purpose LLM output judgments directly, without touching any weights. Its advantage is fast policy iteration — modify the prompt and ship, with no data labeling, training, or evaluation cycle. Its downsides are heavier dependence on model inference cost and latency, and the behavioral boundary is entirely up to the model. When you have only dozens of fraud samples, policy changes weekly, or the team lacks the capability to maintain an RL training pipeline, the prompt approach is usually more economical. By contrast, the weight fine-tuning approach bakes policy into model parameters: prompts are shorter and behavior is more stable at inference, but every policy update requires a fresh round of data labeling, training, and regression evaluation. The article's proposed criterion is: can the policy change more slowly than you can retrain and evaluate? If the answer is "no," keep policy in prompts and external code, and let the model output only a narrow label.

Four Steps You Can Start This Week

1. Freeze a labeled evaluation set from before the model intervenes in decisions
2. Have the model return only one parseable token; keep policy in code
3. Score every candidate on precision, recall, F1, amount-weighted recall, and wrongly-banned legitimate amount
4. Only then consider reinforcement learning. The reward function you need is already in your evaluation framework

The core insight of this methodology isn't "replace a large model with a small one" — it's "take decision authority away from the model." The model handles one narrow task: outputting a risk level. Everything else is controlled by deterministic code. This architecture makes verifiable rewards possible and makes system behavior auditable and rollback-friendly.