Testing Opus 5.5's Six Effort Levels: Spent $500 to Run 18 Apps

A Brutal Test Costing Nearly $500

Which effort level should you use for AI coding tools? This question puzzles most developers. A YouTube creator did something many have thought about but never done: having Opus 5.5 build three completely different apps at six different effort levels, running a total of 18 builds.

The experiment was inspired by a similar test by Nate Hurk, but this creator wanted to verify a more fundamental question — whether the type of app affects the optimal choice of effort level. After all, a 3D scene, a chess engine, and a music production tool face completely different types of problems.

The entire experiment consumed nearly $500 in token credits, close to 800 million tokens, and over 20 hours. Each run used a fresh session, the same prompt, no follow-up corrections allowed, and the model was deliberately not told to test its own code — whether it self-tested was itself one of the measured metrics.

Three Apps, Three Challenges

Night Shift: 3D Studio Scene

The first app was an interactive 3D recreation of the creator's YouTube studio, with the core test being visual fidelity and modeling detail. From low to UltraCode, the visual quality did improve: low was crude but fully functional; medium added an entrance animation but had a strange black occlusion; high introduced a transition effect entering the studio, with noticeably improved lighting and detail; extra and max converged visually, but max was actually worse than extra in some interactions — for example, the sign-switching function stopped working.

At UltraCode, nearly every object was interactive — the play button could be toggled, the microphone swayed, and scrolling code even appeared on screen. But the creator's own verdict was blunt: for the night shift project, he actually preferred extra's result.

Checkmate: Writing a Chess Engine from Scratch

The second app required Opus to implement complete chess rules from scratch and write an AI engine, without using any chess libraries. All levels successfully completed the task, but the real test came in the subsequent tournaments.

The creator himself is a 1700-1800 rated player, yet could not beat the AI at any level. But the tournament results were even more surprising: max won undefeated with 8 wins and 2 draws, extra took second with 8 points, and the most expensive UltraCode only placed third — the two versions still tuning their engines at the 3-hour limit ended up producing the strongest play.

low even beat medium in the rankings, showing that effort level and logical capability are not linearly related.

Why "Self-Testing" Becomes the Key Variable

High effort levels balloon in time cost, mainly not because code generation slows down, but because the model is allowed to repeatedly execute a "generate—verify—fix" loop. In tasks like the chess engine, the model can compile its own code, run games, tally wins and losses, and adjust search depth, evaluation function weights, and other parameters accordingly. This means the effort level is really buying the number of iterative tuning cycles, not simply the amount of code.

This also explains a counterintuitive phenomenon in the experiment: UltraCode was still tuning its engine at the 3-hour limit, and its final playing strength was actually worse than the earlier-converging max and extra. Tuning has diminishing marginal returns, and a longer search chain does not necessarily lead to a better strategy — under a limited time budget, a version that locks in earlier may actually win because it is more complete. Therefore, the number of self-tests only shows "how much checking was invested," not that "the result is necessarily better."

Beat Lab: Single-File Music Studio

The third app was a complete music production tool, including a piano roll, mixer, and visualizer, with all sound effects generated through code and no external audio files allowed. All levels successfully implemented the core requirement of code-generated sound effects.

An interesting reversal appeared with Beat Lab: although UltraCode was more expensive, it actually ran much faster than max and extra. The reason was that it launched 10 different sub-agents working in parallel — a behavior not observed in the other two apps.

What Is an "Effort Level"

In the APIs and product interfaces of some frontier models, "effort level" does not refer to a change in the number of model parameters, but rather a dial for the model's reasoning budget. It is usually expressed through two dimensions: one is how long the model is allowed to "think" before giving a final answer (i.e., the length of the extended reasoning chain), and the other is how many self-verifications, tool calls, or parallel sub-agent tasks the model is allowed to initiate. From low to the highest tier, cost and time increase approximately exponentially, but capability gains are not proportional — this is precisely the core issue this experiment set out to verify.

Several concepts that are easy to confuse need to be distinguished: effort level adjusts "how much compute to spend solving the same problem," while model version determines "what capability is used to solve it"; context caching (cache read) is a cost-reduction technique that reuses repeated prompts and prior context in a session, so its unit price is extremely low. In this experiment, 96% of tokens were cache reads, but what really determined the bill was output tokens, which accounted for only 1%.

The Cost Curve: Nearly Doubling at Each Level

The cost data for the three apps showed a clear pattern. low for all three apps combined was under $4; medium about $14; high about $27; extra about $50; max about $82; and UltraCode jumped directly to over $300.

The creator also broke down where the tokens went: 96% of tokens were cache reads — because in each session the model had to re-read all previous context. But this portion is extremely cheap, only about 20 cents per million tokens. What really burns money is output tokens: although they account for only 1% of the total, they are charged at $20 per million tokens and ultimately made up about 42% of total cost.

The change in self-test counts was equally striking: at low it was almost zero, rising to 77 to 125 times at the max and UltraCode levels. This is the main reason high effort levels balloon in time — not that the model writes more slowly, but that it spends a lot of time repeatedly checking its own output.

The Cost Structure of Cache Reads and Output Tokens

The inference cost of large models consists of three parts: input, output, and cache hits. The data set revealed by this experiment — 96% / 1% / 42% — is essentially caused by the difference in unit prices across the three tiers: cache reads usually cost only about $0.2 per million tokens, while output tokens can reach $20 per million, a difference of about a hundredfold. Therefore, even if output accounts for only a tiny fraction of total tokens, it can still swallow nearly half the bill.

This leads to a practical optimization principle: controlling output length saves more money than controlling input length. In actual use, reducing lengthy explanatory text and asking the model to return code or structured results directly often lowers costs more than trimming prompts. Conversely, once large blocks of reference material are placed into context and cached, their marginal cost is extremely low and they can be used without worry. Understanding this pricing structure helps developers judge whether "paying more to raise the effort level" is really worth it.

How to Choose an Effort Level

Based on the complete data from this experiment, the creator offered his own recommendations:

low for validating ideas. It can generate a complete 3D scene in a few minutes for less than $1. For rapid prototyping, its cost-effectiveness is unmatched.

high as the daily default. It usually finishes within an hour, costs $7 to $11 per app, performs self-testing, and is good enough for most needs.

max for logic-intensive tasks. In scenarios like the chess engine that require correctness guarantees, max's self-test frequency and tuning depth can make a real difference.

UltraCode is not worth it in most cases. It costs 2 to 6 times as much as max, takes significantly longer, and even timed out in multiple tests. For these three apps, UltraCode never produced the best result. The creator recommends reserving it only for genuinely large multi-module projects.

One notable conclusion is that effort level affects logical tasks far more than visual tasks. The chess engine's playing-strength ranking was almost positively correlated with effort level, but visual effects began to plateau after extra — spending more money did not make the image better.

This means that when choosing an effort level, you should judge based on the nature of the task: for visual projects, high may be enough, while only projects involving complex logic are worth spending more on.

Paragraph Summary

A creator spent nearly $500, close to 800 million tokens, and over 20 hours having Opus 5.5 build three apps — a 3D studio scene, a chess engine, and a music production tool — across six effort levels from low to UltraCode, for 18 builds total. The results showed costs nearly doubling at each level, with UltraCode exceeding $300 combined across the three apps yet placing only third in the chess engine tournament and never producing the best result; max won undefeated. In terms of cost structure, 96% of tokens were cheap cache reads, while output tokens made up only 1% but contributed 42% of the cost. Self-test counts soared from nearly zero at low to 77 to 125 at the highest tier, the main cause of the surge in time. The creator recommends: low for prototyping, high as the daily default, max for logic-intensive tasks, and UltraCode not worth it in most scenarios.