9,050 AI Scripts Later: Breaking Down AI Writing Slop and Evaluation Bias

You've probably read too many versions of this paragraph already, maybe even this week:

"In today's fast-paced world, AI isn't just a tool—it's a paradigm-shifting tapestry. Let's dive deep into this transformative landscape where the real magic happens."

The Towards AI team now uses AI heavily to draft videos, courses, books, and articles, in the thousands. The team uses Claude for almost everything, but the content that needs review often sounds like the paragraph above. Since June, the author started quantifying "which models can actually write like me." After 9,050 scripts, one unexpected finding was that these "slop phrases" are actually easy to fix.

Experimental Setup: Turning "My Voice" Into a Scoring Rubric

The author selected ten of his own video scripts, some published and some held back to prevent data contamination. Four people spent nearly a month, about 400 hours, polishing them into as clear and detailed a rubric as possible, and turning "why they work" into measurable criteria.

During evaluation, models could access the author's voice guide, storytelling guide, output rules, past examples, brief, and research materials, to see if they could regenerate decent content. Then four LLM judges each scored the output: one GPT, one Claude, one open-source model, and the recently added Jev.

As of now, 181 configurations have been measured, covering all thinking levels, with plans to cover all future new models. The author emphasizes that the most important thing isn't the leaderboard itself, but the conclusions learned from building this new benchmark, reviewing thousands of generations, and repeatedly iterating on the rubric.

Rubric and LLM-as-judge: A rubric here isn't just a simple scorecard—it breaks down "why this draft works" into reproducible, item-by-item verifiable dimensions, such as voice consistency, structural clarity, banned-phrase hit rate, argument density, etc., each with weights and judgment examples. LLM-as-judge means having a large model score the generated results according to the same rubric, which is far cheaper than human evaluation and can be repeated at scale, but it's known to have biases like position preference, length preference, and "favoring its own output." That's why the author needs a multi-judge panel and tries to fix the rubric and prompts before scoring, so that score changes mainly reflect model capability rather than noise in the judging process.

Low Slop Doesn't Mean Good Writing

The top 15 configurations all scored above 90 on the slop check, meaning that with sufficiently detailed prompts, sentences like the opening one can basically be controlled. Among the 181 configurations, the "cleanest" model was GPT-6 Astra (max effort), with a slop score of 94.9.

But be careful: slop is only one metric. Astra currently ranks 62nd on the overall leaderboard because it doesn't sound like the author: tone and voice only 82, while Opus is 90. The Claude series occupies the top 11 overall.

The author's core takeaway is: removing a few "Claude-isms" is just a quick edit, annoying but fixable in one pass; rebuilding a voice is different, requiring a lot of brainpower, restructuring text, and swapping examples, so for now the author still leans toward Claude for writing.

The difference between slop and tone and voice: Slop (clichés) usually refers to high-frequency, empty expressions that can be applied across contexts, such as "in today's fast-paced world," "paradigm-shifting tapestry," "let's dive deep and explore," etc. It's more like surface filler in the text, fixable by deleting it or replacing it with concrete information. Tone and voice points to a deeper writing persona: sentence rhythm, word preferences, example habits, argumentation style, and what topics the author naturally gets excited or restrained about. A model can have almost no slop yet still not sound like you, because it's just swapped in a clean but unfamiliar voice. That's also why a high slop score only means the text isn't "greasy," not that it's "like me."

The Marginal Return of Max Effort Is Very Small

Opus 5.5 at max effort is the best writer measured, scoring 90.8 out of 100. At xhigh, it scores 90.0, costing $0.69 per script instead of $2.75.

In other words, a 4x price increase yields less than a 1% score improvement. The author's choice is to default to xhigh, and only turn on max for the few drafts where every sentence matters.

Cheap Models Change the "Unit of Work"

The open-weight model GLM-5.3 Flash scores 88.2, at 1.5 cents per script. Opus 5.5 at max is nearly 190 times more expensive, for only about 3% higher score.

This isn't just "generating similar writing with GLM is nearly 200 times cheaper"—it's that with the right approach, you can generate better results than Claude using open-source models, while being cheaper. The team also measured variance across generations: if you generate 10 drafts with GLM and pick the best one, the result gets extremely close to Opus 5.5, while still being 20 times cheaper. If you add a strong judge, a good rubric, and can merge paragraphs from different drafts into a "best-of stitch," in theory it would clearly outperform Opus 5.5 at lower cost.

The author's conclusion is direct: generation is now cheap, selection is the hard part. Spend money on judging and selecting, not on generating.

pass@k and multi-draft selection: When a single generation is cheap enough, a more effective strategy isn't chasing "get it right in one shot" but generating k drafts and picking the best, corresponding to the pass@k idea commonly used in code generation evaluation. A single piece from an open-weight model like GLM-5.3 Flash costs very little, so quantity can make up for single-generation quality gaps: after generating 10 drafts, the best draft often matches a single output from a more expensive model. If you then bring in a strong judge to select by rubric, or even stitch the best paragraphs from multiple drafts into a "stitched draft," overall quality may even exceed that of a single expensive model. The core change is that generation is no longer the bottleneck—judging, selection, and merging are the new cost centers.

Every Judge Favors Its Own Lab

Compared to other judges, the GPT judge gives OpenAI drafts +2.75 points and deducts -2.14 points from Anthropic drafts. The DeepSeek^Ad judge gives about +1 point to DeepSeek and Kimi. The Claude judge has the smallest tilt, about half of the others, but it still favors its own and, surprisingly, likes Google even more. Jev is relatively neutral and slightly prefers Gemini models, but it can't give reasons for its scores, making it less stable on these tasks; the author guesses it may be because writing leans more toward "System 2"-style judgment.

No judge is neutral. Use a panel, and never let the writer's own lab be the only scorer.

The AI Writing Pipeline I'd Adopt

The criterion for choosing a model isn't "writes the most impressively" but "reaches the goal with minimal editing and without driving you crazy." Then:

1. Write long, very specific instructions: your voice, structure, rules.
2. Show what to do and what not to do. Name the phrases you hate, and check drafts in code rather than trusting the prompt.
3. Attach full text of your best past work, ideally explaining why they're good.
4. Generate multiple drafts with cheap open-source models.
5. Have a high-thinking judge from another lab select or merge the best parts by a clear rubric.
6. Run one or two rounds of review-and-edit with another judge. Beyond two rounds, the author hasn't seen much benefit.
7. Keep a human at the end. Even Opus 5.5 drafts need a person.

One caveat: this measures the author's own voice on his own scripts—just one data point, not the truth. Before switching anything, test with your own writing. Building this evaluation pipeline has been very useful; the team has already adapted it to many clients' custom tasks with good results.

The author will release the full benchmark ToneBench, including the methodology behind every number, in a talk at Turing room — Evals Track on Tuesday, October 13 at 10:40.

Why cross-lab judging panels are needed: Large-model judges' preferences are often tied to their training sources, alignment goals, and style tendencies, showing up as higher scores for their own models and lower scores for competing labs' models. This bias doesn't automatically disappear even if the judge model itself is strong, because it carries a systematic taste for "what counts as good writing." Letting the writer's own lab's model be the sole scorer is like letting the same set of preferences be both player and referee. The safer approach is to use multiple judges from different labs, averaging or voting when necessary, and requiring judges to give item-by-item reasons according to an explicit rubric. If a judge can't explain its scoring basis, its scores should be downweighted, because unexplainable preferences amplify noise.

One-line Summary

An evaluation of 9,050 AI scripts shows: slop is easy to remove, voice is hard to imitate, and multi-draft plus cross-lab judges is more cost-effective.

Paragraph Summary

Using 9,050 scripts and 181 configurations, the author built an evaluation system centered on "writing like me." Results show that slop can be effectively controlled with detailed prompts; GPT-6 Astra scores highest on slop but still doesn't sound like the author; the real challenge is tone and voice, which is why the Claude series occupies the top 11. On cost, Opus 5.5 from xhigh to max raises the price 4x for less than 1% score improvement, while cheap open-source model GLM-5.3 Flash generating multiple drafts and picking the best can approach Opus 5.5 at about 20x cheaper. All LLM judges have lab preferences, so a cross-lab panel is necessary. Based on this, the author gives a writing pipeline: long instructions, positive and negative examples, code-checking banned phrases, multi-draft generation, cross-lab judge selection and merging, and final human oversight.