Has Claude Opus 5.5 Been Quietly Nerfed? LiveNerf Benchmark Tracks Model Capabilities in Real Time

A Question That Keeps Haunting Users: Are Models Being Secretly Nerfed?

A persistent suspicion has long circulated in the AI community: that large models get quietly weakened by their developers after release — a practice commonly called "nerfing." Users frequently complain that "this model has gotten dumber lately," but such complaints tend to stay in the realm of subjective impressions with little quantifiable evidence to back them up.

For Anthropic's Opus 5.5, one Reddit developer decided to let the data speak. He launched an open-source benchmarking project called LiveNerf, tracking the model's performance from day one of its release in an attempt to answer a simple yet thorny question: has Opus 5.5 actually been nerfed?

LiveNerf project discussion on Reddit

How LiveNerf Works

The core idea behind this project is daily re-testing and long-term tracking. The author re-runs a set of independent benchmarks every day — such as GPQA (graduate-level scientific Q&A) and SWE-bench (software engineering task benchmark) — and plots the results in a GitHub repository, building a capability curve that evolves over time.

How Test Questions Are Selected

The question selection process is worth noting. Rather than picking questions at random, the author uses Claude to filter for questions where "Opus 5.5 is most likely to make mistakes." This design is intended to improve sensitivity — if the model's capabilities genuinely decline, performance on its weak spots will break down first, surfacing problems earlier.

Control Group and the Decision Threshold

To establish a reference point, the author uses the previous generation Opus 5 as a control benchmark. The judgment criteria are also clear-cut: if Opus 5.5's score on LiveNerf deviates by more than 7.5 points, it's flagged as a potential signal of capability regression.

The value of this approach lies in converting a vague collective impression — "the model got dumber" — into a quantifiable metric with a defined threshold that can be validated over the long term.

GPQA (Graduate-Level Google-Proof Q&A) is a scientific question bank designed by domain experts at the graduate level, covering subjects like physics, chemistry, and biology. Questions are deliberately curated to ensure they can't be answered through simple searches, making it one of the leading benchmarks for measuring deep reasoning in large models. SWE-bench focuses on software engineering tasks, requiring models to generate code patches that pass tests based on GitHub Issue descriptions — considered a more rigorous standard for evaluating real-world coding ability (as opposed to code completion). Both are "high-floor, low-ceiling" hard benchmarks where subtle performance fluctuations are amplified, making them better suited for detecting marginal changes in model capability than general-purpose benchmarks.

Current Verdict: No Nerfing Detected So Far

As of the author's post, monitoring results show no signs of nerfing have been detected. Opus 5.5's daily performance has remained stable, with no crossing of the 7.5-point warning threshold.

That said, the author remains cautious about his own data. He explicitly states that a reliable baseline won't be established until day 10, and that no formal judgment will be made before day 20. This restraint is exactly what this kind of monitoring project needs — avoiding hasty conclusions drawn from a few days' worth of noisy data.

Collective Delusion or Real Phenomenon?

This project touches on a recurring flashpoint in the AI community: how much of the reported nerfing is actually a genuine technical adjustment, and how much is users' psychological expectations playing tricks on them?

The author is clear-eyed about this. He plans to release LiveNerf data again in the coming week or two if the community suddenly erupts with complaints about the model being nerfed — using it to test whether the outcry is "hysteria or data-backed."

This approach of placing subjective discourse alongside objective data is far more constructive than mere argument. It at least provides a falsifiable framework: if the data shows no movement while complaints spike, the issue likely lies in changes to users' workflows or expectations; conversely, if the data drops in tandem, the developer may have genuinely made some kind of adjustment.

The Value and Limitations of This Kind of Monitoring

LiveNerf represents a spontaneous form of community oversight over AI developers' transparency. In an era where model capabilities and service quality are largely opaque, third-party continuous tracking tools fill an important trust gap.

But its limitations should also be acknowledged. A project maintained by a single developer may have biases in question pool size, testing frequency, and methodology. Using Claude to select the "hardest questions" could itself introduce selection bias. Furthermore, factors like model output randomness, API load fluctuations, and temperature parameters can all affect a single day's scores — which is precisely why the author emphasizes the need for a long-term baseline.

For developers who care about Opus 5.5's long-term stability, this open-source repository offers a data window worth watching. Whether the model has been nerfed will ultimately be answered by time and the graphs.

The temperature parameter is the core hyperparameter controlling output randomness in large models — higher values produce more varied outputs, while lower values yield more deterministic results. In benchmark testing, if temperature isn't fixed across runs, the same question may receive different answers on different days, and the resulting score fluctuations can be misread as capability changes. Additionally, server-side load balancing may route API requests to different model replicas running on different hardware or software versions, introducing further noise. This is why rigorous continuous monitoring projects need to control the random seed, fix the temperature, and rely on sufficiently large sample sizes to average out the random error of individual test runs — until then, a single day's data points have very limited reference value.