DoGBench Released: The First User-Facing Documentation Generation Benchmark — No Model Scores Above 50%

DoGBench: Filling the Gap in User Documentation Generation Evaluation

As large language models rapidly advance in code generation, question answering, and content creation, one long-overlooked use case is finally coming into focus: generating documentation for end users. DoGBench (Documentation Generation Benchmark) has been introduced as the first evaluation benchmark specifically designed for this task, measuring whether models can produce technical documentation that is genuinely valuable to users — readable, accurate, and complete.

The most striking finding from DoGBench is this: no model tested so far has scored above 50%. This result stands in sharp contrast to the everyday impression many people have of AI's writing abilities, and reveals just how difficult documentation generation truly is.

Why Documentation Generation Is Harder Than It Looks

Many assume that documentation generation is simply a matter of translating code or feature descriptions into natural language — something large models should excel at. But user-facing docs carry unique requirements, and these are precisely what DoGBench is designed to capture.

The Dual Challenge of Accuracy and Completeness

A quality user document must not only read smoothly — it must be technically precise. Models frequently generate text that sounds coherent but contains factual errors or omits critical steps. For installation guides, API references, or how-to tutorials, even a single incorrect parameter can completely block a user.

Structure and Usability Requirements

Good documentation requires a well-organized hierarchy, clear examples, and an understanding of the user's mental model. This demands that a model not only "know" the information, but "know how to organize" it to serve real readers. Scores consistently falling below 50% indicate that current models still have significant gaps in this kind of integrated capability.

What "No Model Scores Above 50%" Actually Means

This number needs to be understood in the context of the benchmark's scoring methodology. A rigorously designed benchmark often produces low scores that reflect the task's high bar — not that models are entirely useless. The value of DoGBench lies precisely here: it establishes a quantifiable, comparable standard for a chronically underestimated task.

Previously, evaluating the quality of AI-generated documentation relied mostly on subjective judgment or general-purpose text metrics like BLEU and ROUGE — neither of which can capture whether documentation actually helps real users accomplish their goals. As the first user-oriented documentation generation benchmark, DoGBench fills this gap and gives researchers and developers a way to compare model performance in terms that are much closer to real-world value.

Implications for Developers and the Tool Ecosystem

For teams building AI documentation tools, code assistants, or knowledge base products, DoGBench points to several important considerations.

First, don't assume large models will automatically produce high-quality documentation. Scores consistently below 50% are a reminder that human review and mechanisms like Retrieval-Augmented Generation (RAG) remain necessary in production environments.

Second, documentation quality evaluation should return to the core question: "Can users actually complete the task?" — not surface-level fluency. The emergence of DoGBench may push the industry toward more practical quality standards.

Third, this is a clear signal for improvement. Existing models still have enormous room to grow on this task — whether through better training data, task-specific fine-tuning, or tool-augmented generation, there are real opportunities to meaningfully raise the bar.

---

Retrieval-Augmented Generation (RAG) is currently one of the leading techniques for reducing hallucinations in large language models on knowledge-intensive tasks. The core idea is straightforward: before the model generates a response, it retrieves the most relevant passages from an external knowledge base — such as product documentation, a code repository, or an internal wiki — and feeds those passages as context alongside the query, giving the model's output a verifiable grounding. In documentation generation specifically, RAG can help anchor the model to the correct API parameters for a specific version, the right command-line syntax, or the latest operational steps — reducing factual errors caused by outdated training data or fuzzy knowledge boundaries. That said, RAG doesn't solve problems of structure and readability; it primarily addresses the accuracy gap.

Limitations and Observations

It's worth noting that public discussion of DoGBench remains quite limited — with only sparse attention and comments on platforms like Hacker News. The specific composition of its test set, its scoring dimensions, and the full list of evaluated models still await more detailed documentation. The credibility of any benchmark depends on whether its task design reflects real-world scenarios and whether its scoring is reproducible. The claim that "no model scores above 50%" is best treated as a warning about task difficulty, rather than a definitive verdict on any particular model's capabilities.

As more teams integrate AI into technical writing workflows, evaluation tools like DoGBench — focused on genuine user value — may become an important benchmark for measuring real progress.

Background: BLEU and ROUGE

BLEU (Bilingual Evaluation Understudy) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation) are long-established automated text evaluation metrics in natural language processing. BLEU measures quality by calculating n-gram overlap between generated text and reference text, and was originally designed for machine translation; ROUGE emphasizes recall and is commonly used for summarization evaluation. Both share a fundamental limitation: they only measure surface-level word matching and cannot assess logical coherence, actionability of steps, or whether a user can actually complete a task by following the documentation. In a technical documentation context, an installation guide that closely mirrors a reference document in phrasing but gets a key parameter wrong could score highly on BLEU/ROUGE while being completely useless in practice. This is precisely why a benchmark purpose-built for user-facing documentation matters.