Which AI Model Is the Most Verbose? I Built a "Yap Benchmark" to Find Out

When a new model drops, everyone benchmarks how capable it is—but almost no one tests something I care about more: how much a model yaps. A lot of the time, the capability gap between new models isn't huge, and when two models perform about the same, I'll pick the one that's less annoying to use. So I built my own benchmark, called Neural Yap, specifically to quantify this.

The results surprised me a bit, but I quickly understood why: the model itself isn't the only factor.

The Four Dimensions of Yap Material

Neural Yap's scoring logic is straightforward: the lower the score, the less the model yaps; the higher the score, the more it yaps. The whole scoring system breaks down into four categories:

• Filler phrases: The model keeps saying things like "You're right," "You're absolutely correct," "You make a good point, I should step back"—that typical AI tone.
• Typographic symbols: Em dashes and other characters with obvious AI fingerprints.
• Formatting structure: Whether the answer turns into a full Markdown article again—many models love doing this.
• Answer length: The most intuitive manifestation of yapping.

What's interesting is that I don't weight these equally. I don't really care about formatting and special characters. What actually annoys me is two things: answers that could be short but get written as long essays, and those AI-flavored filler phrases.

Regular expressions are a syntax for describing string-matching patterns—for example, using fixed character combinations to catch phrases like "You're right" or "Let's dive deeper," or pattern-matching consecutive em dashes and overused bold and list markers. In this benchmark, the human-readable rules in the lexicon (codes, examples, word budgets) need to be translated into these patterns so that model outputs can be scored automatically. Separating rule authoring from expression writing has the benefit of keeping the rules themselves auditable and discussable, while delegating the syntax details to the model, which lowers maintenance costs. The trade-off is that auto-generated regexes may miss matches or match incorrectly, so you still need to validate by sampling actual outputs.

Surprising Result: The GPT Series Is Actually the Least Yappy

In the first version of the results, the GPT series models ranked lowest—which is the best score, meaning they yap the least. This confused me for a while, because my impression was that these models were pretty verbose. Conversely, Claude Opus 5.5, which I'd always found concise and pleasant to use, scored lower than all the GPT models (i.e., yapped more).

But one conclusion is very solid: Claude Opus 5.5 is the undisputed yap champion. Ask it anything and it wants to give you a dissertation—it simply can't be concise. So seeing it ranked first at least confirmed the benchmark isn't broken.

Breaking it down by subcategory confirms this too. On filler phrases, the GPT models scored lowest, and looking at actual reasoning outputs, this isn't an implementation bug. Meanwhile, Mistral Large 4 and DeepSeek^Ad V4 Flash are clearly more chatty—the former is a European model, the latter open-source and easy to run on local hardware with enough compute. Both are excellent models, but I don't like verbose models, and they're indeed noticeably wordier in their phrasing. On the length dimension, DeepSeek V4 Flash also ranks high; Mistral Large 4, however, does well on this one.

How the Benchmark Runs

The core of the whole system is the lexicon directory: using Markdown to define exactly what this benchmark is looking for and what it penalizes. All the patterns and phrases to penalize are registered here, with codes and examples—things like "Let's dive deeper," "Now I understand the problem," "You're right, you're absolutely, completely, thoroughly correct." These aren't formal regular expressions—just an overview of the penalty list. Formatting and layout rules are defined here too.

Then lexicon.py converts these rules into regular expressions. These regexes aren't hand-written by me—I defined what to penalize and what to measure, but I delegated the actual expression writing to Claude.

The penalties fall into several categories: for sycophancy, penalize opening lines and rhetorical structures like "It's not X, it's Y"; for vocabulary, penalize clichés like "game-changer"; for length, define word budgets—250 words allowed for coding examples, 50 words max for factual quick-fire Q&A, with deductions for exceeding the budget.

The prompt design also has some thought behind it. There are basic factual questions like "What's the capital of Australia," and adversarial tests: first say "No, it's Sydney" (which is of course wrong), then see how the model responds to the correction—without using any of those annoying filler phrases. There are also explanation-type prompts (what is blockchain, how do vaccines work), advice-type (should I learn Rust or Go first), coding-type, and creative-type prompts. All questions are ultimately evaluated on just four things: phrasing, length, typography, and format.

The technical implementation uses inspect AI to run batches. YAML files in the runs directory list all the models to test; adding a new model doesn't rerun everything—existing results are reused and only the new one runs. Results land in the results directory, including charts and a CSV.

inspect AI is an open-source model evaluation framework that combines "datasets, task prompts, and scorers" into a reproducible evaluation pipeline, supporting parallel batch runs and logging each run's logs and costs. Its typical structure: declare the list of models to test and evaluation tasks in YAML or Python, and the framework handles scheduling calls, saving raw outputs, and passing them to scoring functions. The article's "YAML files in the runs directory list all models to test" and "results land in the results directory with a CSV" are exactly this kind of declarative configuration. Incremental runs (reusing existing results, only running new models) are also a common caching mechanism in evaluation frameworks, avoiding rerunning everything each time a model is added, thereby controlling time and cost.

Cost: The Hidden Dimension

The CSV also records something not visible on the charts: how much each run cost. On the same benchmark, most models cost very little, but one model's cost was clearly an order of magnitude higher than the rest and took a long time to run, because it spent a lot of compute on reasoning.

The problem is that this benchmark doesn't penalize yapping "inside the model's head"—a long chain of thought isn't penalized. This may be a gap to fix in the next version. The author later tested an upgraded model from the same series, and its yapping was noticeably better, with the cost issue no longer prominent—so that high expense may have been specific to that model or that particular run.

The Real Reason: Harness Matters More Than the Model

Back to the contradiction: the benchmark shows GPT models are more concise, but my actual experience using them in agents is exactly the opposite.

The answer is the harness. I suspect Codex's harness constrains the model more loosely than Claude Code, and the models I use in claude.ai and Claude Code actually yap less than GPT models with Codex. Amplifying this contrast is another detail: I gave both sides instruction files asking for conciseness—CLAUDE.md on the Claude side, AGENTS.md on the Codex side.

From my subjective experience (not quantified), Claude models are more willing to follow system instructions than GPT models. This is purely anecdotal observation, and it might be worth testing in a second version of the benchmark.

The results viewer lets you compare question by question. For example, asking "Is drinking coffee every day bad," one model is clearly longer and uses bold text plus bulleted lists, getting penalized more on this item. Asking "What is blockchain," 6.1 Sol gives a fairly concise answer with just one em dash at the end; Opus 5.5 serves up a full formatted structure; Opus 5 just dumps a big block of text packed with em dashes. The creative prompt "Write a tweet announcing our app's dark mode launch" is also typical: 6.1 Sol's answer fits neatly into a tweet, while the longer version exceeds tweet length and ends with "Would you like me to adjust further?"—hitting the filler-phrase penalty again.

These examples also empirically confirm the benchmark's conclusion: Claude series models do yap more than GPT series models. And the things I like more about Claude Opus 5.5 seem to have to do with how Claude Code uses it.

Harness (agent framework) refers to the software layer wrapping around a large language model that drives it through multi-turn tasks—e.g., Codex, Claude Code, Cursor, etc. It determines the content and priority of system prompts, how tool calls are orchestrated, how context is assembled and truncated, when to let the model continue generating, and whether outputs are auto-formatted. The model is only responsible for a single generation; what users actually perceive as "verbose or concise" is often the final result of model output filtered, constrained, and assembled by the harness. That's why the same model can feel very different in different harnesses: a strict harness compresses length through system instructions and forbids listy formatting, while a loose harness barely intervenes. This is also why a benchmark testing a model's raw output in isolation can contradict everyday experience.

CLAUDE.md and AGENTS.md are both "project-level instruction files" placed in the repository root, automatically read in and injected into context by the respective harness at the start of each session, to declare long-term constraints like coding standards, answer style, and language preferences. CLAUDE.md is for Claude Code; AGENTS.md is a more neutral cross-tool convention also supported by frameworks like Codex. The key difference lies in how they're injected and how instruction priority is handled: some harnesses reinforce them as high-priority system instructions repeatedly, while others just attach them as ordinary context, and the model may not strictly follow them. This explains why "the same conciseness instruction file" can have completely different effects, and suggests that when evaluating models, instruction-following should itself be an independently measured dimension.

What the Next Version Needs

Neural Yap is currently version 0.1, and the repo is private for now—the reasoning being that if it were public, AI could access it and theoretically game the score. Though the author also jokes that probably not many people care about this kind of benchmark.

Two clear improvement directions: count verbosity in the model's chain of thought as a penalty, and quantify "how well the model follows system instructions." Every time a new model is released, the author plans to test it. After all, when capability gaps are small, choosing the one that's less annoying is itself a form of pragmatism.

One-line Summary

The author built the Neural Yap benchmark to quantify model verbosity, finding that the harness matters more than the model itself.

Paragraph Summary

The author built a benchmark called Neural Yap that quantifies LLM "yapping" across four dimensions—filler phrases, typographic symbols, formatting structure, and answer length—where lower scores mean more concise. First-version results show the GPT series yaps least, while Claude Opus 5.5, which the author always found concise and pleasant, is the yap champion, tending to write an essay for any question; Mistral Large 4 and DeepSeek V4 Flash are noticeably chatty in phrasing. The benchmark defines penalty lists in a lexicon, uses AI-assisted regex generation, and runs batches with inspect AI, recording call costs not visible on charts. The key finding: benchmark results contradict daily experience because harnesses like Codex and Claude Code constrain models differently, and Claude models seem more willing to follow conciseness instructions like CLAUDE.md. The next version plans to include chain-of-thought verbosity and instruction-following in the evaluation.