Are AI Capabilities Underestimated? The Non-Convex Truth Hidden by Average Scores

The Illusion One Chart Creates

A thought-provoking idea has been circulating in the tech community lately: if you only look at certain comprehensive benchmark charts, you might conclude that today's large language models aren't much more capable than they were six months ago. Progress seems to have stalled, with diminishing marginal returns at every turn.

But does that conclusion actually hold up? The developer who put forward this observation made a sharp point: models have improved far more rapidly in mathematics and computer science than in virtually any other domain. In other words, the "plateau" we seem to be seeing may well be an illusion created by averaging.

This raises a critical methodological question: when we use a single composite score to measure a highly uneven capability system, what are we actually measuring?

The Averaging Trap: Why Comprehensive Benchmarks Mislead

Most people evaluating AI progress rely on some kind of "overall capability" metric — a single number that attempts to summarize a model's full range of abilities. This approach is intuitive and easy to communicate, but it's also deeply misleading.

Imagine a model that leaps from 60 to 90 in mathematical reasoning and from 55 to 88 in code generation, while showing only modest gains in open-ended writing, commonsense reasoning, multilingual understanding, and similar dimensions. When you average all these dimensions together, those two areas of dramatic improvement get "diluted" by the many areas that barely moved. The resulting overall curve looks unremarkable.

The original author's analogy is wonderfully apt: we are essentially trying to describe a model's "median capability" with a circle. A circle is neat, symmetrical, and easy to understand. But the actual distribution of capabilities is nothing like that.

Why Math and CS Are Moving the Fastest

There are structural reasons why mathematics and computer science are advancing so rapidly:

• Strong verifiability: Math problems have definitive answers, and whether code runs correctly or passes test cases can be determined automatically. This provides clear, cheap, and massively scalable reward signals for reinforcement learning.
• Synthetic data generation: Unlike subjective tasks that require human annotation, math and code problems can be generated programmatically at scale — models can even generate their own problems and self-verify the answers.
• Explicit reasoning chains: Problems in these domains tend to have clear logical steps, making them ideally suited to emerging paradigms like Chain-of-Thought reasoning and test-time compute scaling.

These properties are precisely what allowed recent technical breakthroughs — especially reasoning models — to deliver disproportionately large gains in these areas.

The Non-Convex Shoggoth: The True Shape of LLM Capabilities

The most imaginative framing in the original post is this: a model's true capability envelope looks less like a smooth shape and more like a highly non-convex shoggoth.

A shoggoth is a creature from Lovecraft's fiction — a shapeless, amorphous mass covered in tentacles and eyes, widely used in AI circles as a metaphor for the way large language models are simultaneously powerful, inscrutable, and internally chaotic. Here, the author uses it to describe the shape of capability distributions: not a smooth circle, but a bizarre form that protrudes enormously in some directions while caving inward in others.

"Non-convex" is the operative word. A convex shape is one where any line drawn between two points stays within the shape — it's regular and predictable. Non-convex means the distribution of capabilities across dimensions is fractured, discontinuous, even counterintuitive: a model might solve Olympiad-level math problems while making elementary mistakes in simple everyday commonsense reasoning.

This jagged intelligence is a defining characteristic of modern large language models, and it explains why user experiences are so polarized. When one person marvels at a model's brilliance while another scoffs at its stupidity, they're simply touching different surfaces of the same non-convex shape.

Implications for the Industry: How to Accurately Assess AI Capability

What might seem like a purely technical discussion carries significant implications for the broader industry.

Benchmark Systems Need to Be Rebuilt

If a single composite score systematically obscures real progress, then many existing leaderboards and benchmarks have a fundamental flaw. A more sensible approach would involve:

• Presenting capabilities by dimension rather than collapsing them into a single number;
• Focusing on capability boundaries and peaks, not just medians;
• Developing dynamic benchmarks that capture "non-convex" characteristics — identifying which directions a model is surging in and which remain stagnant.

A Better Basis for Deployment Decisions

For developers and organizations, this means the right question isn't "is this model good?" but rather "is it strong on the specific dimension I need?"

A model that is advancing rapidly in code and mathematics is revolutionary for use cases like programming tools, quantitative trading, and scientific computing — but may offer only marginal improvement for content creation, emotional companionship, or complex social judgment tasks. Understanding the non-convexity of the capability envelope is what enables sound technology selection.

Progress Is Far from Stalled

The most important takeaway is this: AI progress is far slower than the "average curve" suggests only if you're looking at it the wrong way. It's simply concentrated unevenly in specific domains. When we lament that "AI doesn't seem to be making breakthroughs anymore," we're probably just observing that the real breakthroughs are happening in dimensions we don't interact with in our day-to-day use.

Conclusion

Using a circle to measure a shoggoth will always produce a distorted picture. This observation from the developer community is a reminder that as AI systems grow increasingly complex, oversimplified metrics can create serious cognitive blind spots.

Truly understanding AI progress requires us to let go of our fixation on a "single score" and instead embrace the messier, non-convex, jagged reality — one that is charging forward at full speed in certain directions. Mathematics and computer science may be just the first tentacles this shoggoth has extended. The rest of it is still growing in the dark.