Cleaning Up Claude's Token-Verbose Output with a Separate LLM: A Dual-Model Pipeline Architecture in Practice

When AI Coding Assistants Become Too "Verbose"

When using Large Language Models (LLMs) for programming assistance, an increasing number of developers have identified a common pain point: model outputs are becoming increasingly verbose. A recent discussion on Hacker News — "Clean up Claude 5's token vomit with a separate LLM" — precisely captured this phenomenon and sparked heated debate.

The term "token vomit" is the community's vivid description of AI assistants' excessive output. It refers to models producing, alongside the core answer, large amounts of redundant explanations, repetitive confirmations, excessive disclaimers, and unnecessary pleasantries. These not only dilute the density of useful information but also tangibly consume users' token budgets and reading time.

The Cost Dilemma Behind Verbose Output

For developers relying on API-based billing, redundant output translates directly into real monetary costs. In LLM API services, tokens are the fundamental billing unit — one token corresponds to roughly 3-4 English characters or 1-2 Chinese characters. Taking current mainstream model pricing as an example, both Claude 3.5 Sonnet and GPT-4o price output tokens at approximately $15 per million tokens. If a response that should be 200 tokens balloons to 600 tokens, the cost per call triples. For production environments handling tens of thousands of daily calls, this means thousands of dollars in additional monthly expenditure. When this redundancy accumulates across thousands of calls, the cost difference becomes quite substantial.

The deeper issue is that this verbosity is often not accidental but a systematic result of how models are trained. Specifically, current mainstream models universally employ RLHF (Reinforcement Learning from Human Feedback) for alignment training, during which human annotators rank model outputs by preference. Research shows that annotators tend to select longer, more detailed answers as "better" outputs — a phenomenon known as "length bias." In the process of optimizing reward signals, models learn the strategy of "saying more is always safer than saying less," causing outputs to become systematically verbose. This isn't an isolated problem with any particular model but a structural side effect of current mainstream alignment methods. To achieve higher scores on benchmarks and human preference evaluations, models are guided to generate responses that appear "thoughtful" and "comprehensive." However, in actual engineering scenarios, developers often need concise, directly executable code snippets rather than answers wrapped in layers of explanation.

The core idea discussed is quite inspiring: since we can't easily change the model's own output habits, why not introduce a separate, lighter-weight LLM as a post-processing layer specifically responsible for "cleaning up" the main model's output and extracting the truly valuable core content.

The Technical Logic of the "Dual-Model" Cleanup Approach

The essence of this approach is a pipeline architecture: let the powerful but verbose primary model (like Claude) handle generation, then let a cheaper, faster auxiliary model handle condensation.

Why Use a Second LLM Instead of Rule-Based Filtering

Some might question: why not simply use regex or simple rules to strip out redundancy? The reason is that model redundancy exists at the semantic level, not as fixed patterns. Disclaimers, repeated explanations, and polite filler phrases take infinitely varied forms, making traditional rule matching unable to achieve full coverage — it either accidentally removes useful information or misses redundant content. An LLM that understands semantics can more intelligently determine what's core and what's noise.

From a natural language processing perspective, this is essentially a composite task of extractive summarization and information filtering. Redundant content might appear in completely different sentence structures — "Please note this is only a suggestion," "It should be pointed out that in actual use you should consider...," "I hope this helps" — these expressions differ greatly in grammatical structure but serve the same semantic function. A rule-based system would need to enumerate all possible patterns, while a semantics-based model can directly judge at the intent level whether a passage carries information essential to solving the problem.

Weighing Costs and Benefits

On the surface, introducing a second model seems to add calling costs. But the key lies in differentiated model selection: the capabilities needed for cleanup are far less demanding than for generation. The generation stage requires deep reasoning, code comprehension, and creative abilities, while the compression and refinement stage is essentially a text summarization and information filtering task. Therefore, lightweight models like GPT-4o-mini, Claude 3 Haiku, or Gemini Flash can be selected for execution — these models typically cost 1/10 to 1/50 of flagship models. For example, Claude 3 Haiku's output price is only $1.25 per million tokens, dramatically reducing the marginal cost of post-processing compared to the primary model.

If the primary model's verbose output is compressed to one-third or less of its original length, downstream storage, transmission, and subsequent processing stages all continuously benefit. For scenarios where LLM output needs to be fed into other systems — such as automated testing after code generation, formatting after document generation, or passing context to the next call in multi-turn conversations — concise output significantly reduces the comprehension burden on downstream models and context window consumption.

Divergences and Reflections in Community Discussion

This approach also sparked different voices within the community. Some developers argue that using one LLM to patch another LLM's "flaws" is essentially a stopgap measure that treats symptoms rather than causes. The real solution should come through more precise system prompts, output format constraints (such as requiring models to return responses in JSON Schema or specific structured formats), or model providers optimizing for conciseness at the training level.

In fact, many models already support controlling output verbosity through prompts, such as explicitly requesting "return only code, no explanations." Some APIs also provide structured output features (like OpenAI's Function Calling and JSON Mode) that can constrain output format to some degree. However, practice has shown that model compliance with such instructions isn't always stable, especially in long conversations or complex tasks. As information accumulates in the context window and attention disperses, verbose habits tend to "creep back" — a phenomenon sometimes called "instruction drift." This is precisely the practical foundation for the post-processing approach's value — it provides a deterministic "safety net" ensuring that regardless of how the primary model's output fluctuates, the content ultimately delivered to users or downstream systems has undergone quality control.

Implications for AI Engineering Practice

This discussion, seemingly focused on "verbose output," actually reflects an important trend in current AI application engineering: Compound AI Systems are becoming the norm.

This concept was formally proposed by Berkeley AI Research in early 2024, with the core argument that future AI system performance improvements will come more from system architecture design than from purely relying on single model capability improvements. No single model can achieve optimality across all dimensions — it might have strong reasoning but verbose output, or be concise but lack depth, or be fast but insufficiently accurate. Combining multiple models according to their respective strengths into a pipeline, with each stage playing to its specialties, is becoming the mainstream paradigm for building reliable AI systems.

From "generate-review" to "generate-clean," from RAG (Retrieval-Augmented Generation, which combines external knowledge base retrieval systems with generative models to improve answer accuracy) to multi-agent collaboration (as implemented by frameworks like Microsoft's AutoGen and CrewAI, where multiple AI roles cooperate through division of labor), the common kernel of these patterns is using specialization to compensate for the limitations of any single model. In code generation, the "generate-test-fix" iterative pipeline has been proven to produce quality several times higher than single-pass generation.

For developers building AI products, this case conveys a pragmatic signal: don't obsess over finding one "perfect" model, but learn to use engineering approaches to organize existing model capabilities into systems that meet actual needs. Cleaning up "token vomit" is just one small entry point, but the compound thinking it represents is the lasting approach to addressing LLMs' various imperfections. This thinking requires developers to shift from the simple pattern of "calling one API" to the systems engineering perspective of "designing an intelligent pipeline" — and the latter is the critical leap from prototype to production-grade AI systems.

Key Takeaways