Breadth Is the Cheap Axis: Why AI Search Pays Off More Than Thinking Harder

On seven out of eight circuit design tasks, an LLM-driven search method beat the previous best-known solution. The paper is about quantum circuits, but what keeps nagging at me is where it spent its compute.

It didn't ask the model to "think harder." It had the model generate a large number of candidates, then spent the real budget on filtering them. The model is the cheapest part of the loop; search is the expensive part, and that's exactly where the payoff hides.

Look at how most agent architectures burn tokens, and you'll see the opposite orientation: one chain of thought stretched ever longer, repeated exhortations to "reason carefully," bigger context windows. All depth, no breadth. Depth is what you reach for when you can't generate alternatives—when your only move is to polish that one answer.

The Number I Actually Track

Hit rate at N. Of N candidates, how often does at least one meet the bar?

If this curve flattens as N grows, your sampler is broken. You're paying 8x and getting eight copies of the same answer. If it climbs, breadth is doing real work, and you should keep scaling N until the curve hits an inflection point. That inflection point is your budget line, and it's different for every task. I've never guessed it right—I've only ever measured it.

Diversity is the whole game. Temperature 0.2, eight samples from the same prompt—that's just one sample with extra steps. In my experience, reframing the question beats tuning temperature. Ask the same problem as a constraint problem, as a search problem, as "what would break this." Different decompositions produce genuinely different candidates. Tiny perturbations of the same prompt just give you a batch of sibling samples.

Temperature is a parameter that controls randomness when an LLM samples tokens, typically ranging from 0 to 2. The lower the temperature (e.g., 0.2), the sharper the probability distribution, and the model tends to pick the highest-probability token, producing more deterministic and conservative output. The higher the temperature, the flatter the distribution, and low-probability tokens get a chance to be selected, producing more divergent output. But temperature has a fundamental limitation: it only shifts the sampling location on the same probability distribution—it doesn't change the distribution itself. That's why sampling eight times at temperature 0.2 tends to yield tiny variants of the same answer rather than genuinely different ideas. This is exactly why the article stresses that "reframing the question beats tuning temperature": changing the prompt fundamentally reshapes the conditional probability distribution, whereas tuning temperature just picks a different sampling point on the same distribution.

Hit rate at N (pass@N) is a core metric for evaluating the sampling quality of generative models, derived from the pass@k measure in code generation. It answers the question: if you independently sample N candidates, what's the probability that at least one meets the target criterion? Computationally, it's usually estimated as 1 minus the probability that all N candidates fail. This metric matters because it decouples "model capability" from "sampling strategy": if pass@N approaches 1 quickly as N grows, the model actually "knows" the answer and just missed on a single sample—so increasing N is efficient. If pass@N grows slowly or plateaus as N increases, the model simply doesn't cover the correct answer's distribution, and increasing N just wastes compute. The "inflection point" mentioned in the article is where the marginal return of the pass@N curve starts to decline, and it directly gives you the optimal sampling budget for that task.

Why Nobody Does This

Mostly latency. Fan-out feels slower even when it isn't—if your provider allows concurrency, eight parallel calls finish at roughly the same time as one. Then there's cost accounting: per-token billing makes 8x look like 8x, and nobody wants to explain that line item on a bill. Then there's the mental model of the application: "one request, one response." Fan-out breaks that model—your logs go from a single line to a tree.

These are all real friction points. But none of them is a reason to say "this technique doesn't work."

Fan-out is a classic pattern in distributed systems: one request is duplicated into multiple parallel copies, each processed separately, then results are aggregated. In LLM agents, fan-out means concurrently launching N samples for the same task, then having a filter (verifier, scoring model, heuristic rule) pick the best candidate. Fan-out contrasts with serial "extended chain of thought": the latter keeps self-correcting along a single trajectory, while the former spreads the exploration space horizontally. The engineering obstacles to fan-out lie mainly in three places: first, latency perception—even if concurrent calls take roughly the same total time as a single call, users still psychologically feel that "sending eight requests" is slower; second, per-token billing makes cost scale linearly with no amortization mechanism; third, observability—logs go from a single linear trajectory to a tree structure, making debugging and attribution more complex.

Where It Breaks Down

Tool calls.

If each candidate is a full agent trajectory, then N candidates means N sets of side effects. N writes, N emails, N charges. For text, sampling is nearly free; for actions, sampling is extremely expensive. Once candidates start touching the real world, breadth is no longer the cheap axis.

This points toward a shape I've been converging on: keep the execution phase single-threaded and put the breadth upstream—in the part that only produces text. Propose broadly, execute once. Expensive and irreversible steps get one chance; cheap and reversible steps get as many as possible.

Side effects in the agent context refer to irreversible changes that candidate trajectories make to the external world. Pure text generation is idempotent: generate a hundred candidates, and the worst case is wasted tokens—the world state is unchanged. But once candidate trajectories include tool calls—writing to a database, sending an email, calling a payment API, executing a shell command—each candidate genuinely alters external state. This structurally breaks the "sample broadly, then filter" pattern: you can't eliminate candidates after they've executed, because elimination can't undo side effects that have already occurred. Workable mitigations include: executing candidates in a sandbox or transaction and committing only after confirmation; or abstracting side-effecting actions into two-phase operations that can be rolled back. But these all require system-level support—they're not something you can solve by simply swapping a prompt.

What I'm Taking Away From This Paper

Not "the model now understands physics better." The interesting conclusion is narrower and more useful: when producing candidates is cheap and filtering candidates is also cheap, the optimal strategy isn't to work harder at getting one answer right—it's to rapidly obtain a hundred answers.

That's not as clever as it sounds. But it's also the direction I keep adjusting my own loops toward, and it works more often than the cleverer version.

Paper link: https://arxiv.org/abs/2610.22367v1