The Dude System: How Dual-Detection Multi-Agent AI Catches Inconsistencies Between Papers and Code

When Submission Volume Outpaces Human Review Capacity

In recent years, the volume of paper submissions in AI has exploded, with many papers providing open-source code to support reproducibility. However, a long-overlooked problem is becoming increasingly prominent: discrepancies often exist between the methods described in papers and their actual code implementations.

The Reproducibility Crisis: Reproducibility is the cornerstone of scientific research — the ability of other researchers to replicate experimental results based on paper descriptions. Yet in recent years, a severe "reproducibility crisis" has emerged across multiple disciplines. A 2016 Nature survey revealed that over 70% of researchers failed to reproduce others' experiments, and more than 50% couldn't even reproduce their own. In AI and machine learning, this problem is particularly acute. Statistics from Papers With Code show that while the proportion of papers with open-source code has been rising year over year, inconsistencies between code and paper descriptions remain widespread. Common discrepancies include: hyperparameter settings that don't match paper descriptions, missing data preprocessing steps, oversimplified key details of model architectures, and incomplete documentation of experimental environments. These inconsistencies may stem from unintentional oversights, poor code version management, or the "embellishment" of methods during paper writing. Such discrepancies not only waste other researchers' time on reproduction efforts but can also propagate erroneous research conclusions, eroding the trust foundation of the entire academic community.

These inconsistencies may be unintentional implementation deviations or deliberate or inadvertent over-promotion, all of which ultimately undermine research credibility. As submission volumes far exceed human review capacity, the academic community urgently needs automated methods to detect discrepancies between papers and code.

The Rise of Large Language Models: LLM-based detection approaches have thus entered researchers' sights. Large Language Models are deep learning-based natural language processing models, typically containing billions to hundreds of billions of parameters. Representative LLMs include OpenAI's GPT series, Google's PaLM and Gemini, and Meta's LLaMA. These models learn statistical patterns and knowledge representations from massive text datasets through pre-training, enabling them to understand complex natural language instructions and generate coherent text outputs. In academic research scenarios, LLMs demonstrate powerful text comprehension and reasoning capabilities, able to parse technical descriptions in papers, understand implementation logic in code, and establish semantic mappings between the two.

However, existing single-agent LLM approaches suffer from two major limitations: limited context capacity (context windows typically range from 4K to 128K tokens — roughly 3,000 to 100,000 words — making it difficult to process complete long papers and large codebases in a single pass) and a unidirectional perspective on discrepancy detection, resulting in poor recall — many genuinely existing inconsistencies go undetected. Additionally, LLMs may produce "hallucinations" — generating content that seems plausible but is actually inaccurate — posing a significant challenge in discrepancy detection tasks that demand high precision.

Addressing this pain point, a recent arXiv paper (arXiv:2609.03416v1) introduces Dude, the first Dual-Detection Multi-Agent System designed for paper-code discrepancy detection.

The Hidden Pitfalls of Multi-Agent Design

Upgrading from a single-agent to a multi-agent detection system should intuitively improve detection capability. But the research team discovered that the problem is far more complex than it appears.

Architectural Principles of Multi-Agent Systems: Multi-Agent Systems (MAS) are a classic architecture in artificial intelligence, where multiple autonomous agents collaborate, negotiate, or compete to accomplish complex tasks together. Each agent has specialized responsibilities and domain knowledge, exchanging information through well-defined communication protocols. In LLM-based multi-agent systems, each agent is essentially an LLM instance endowed with a specific role and set of instructions. For example, in the Dude system, a "paper parsing agent" focuses on understanding the method descriptions in academic papers, a "code parsing agent" focuses on analyzing source code implementation logic, and a "coordination agent" is responsible for integrating findings from different perspectives. This division-of-labor design philosophy stems from the "divide and conquer" approach, enabling the system to transcend the cognitive boundaries of a single LLM and handle more complex, multidimensional tasks.

The Challenge of Granularity Asymmetry

The Dude authors identified a key insight: there is a granularity asymmetry between paper-language and code-language. Papers tend to describe methods using highly abstract, generalized natural language, while code consists of extremely specific, line-by-line implementation details. The "granularity" of these two languages operates at entirely different levels.

When multiple agents collaborate under this granularity asymmetry, two typical problems arise:

• Over-interpretation: Agents may draw excessive inferences from vague paper descriptions, imposing non-existent "implementation details" onto the code for comparison.
• Over-reporting: Agents tend to flag any minor phrasing difference as an "inconsistency," generating a large number of false positives.

In other words, while multi-agent systems theoretically have stronger recall capability, without proper constraints they severely sacrifice precision, producing excessive noise that actually reduces practical value. The challenge in multi-agent systems lies in designing effective collaboration mechanisms to avoid information conflicts, redundant work, or responsibility diffusion among agents.

Dude's Two Core Mechanisms

To boost recall while suppressing false positives, Dude incorporates two key mechanisms that represent the system's core innovations compared to previous approaches.

Granularity-Aligned Negotiation

The first mechanism is granularity-aligned negotiation. This mechanism enables the paper-parsing agent and the code-parsing agent to "converse" and compare at the same granularity level. By aligning the abstraction levels of both languages, the system avoids pseudo-discrepancies caused by level mismatches, reducing over-interpretation at its source.

Two-Stage Salience Filtering

The second mechanism is a two-stage salience-filtering mechanism. After agents produce candidate discrepancies, the system doesn't accept them wholesale. Instead, it runs two rounds of filtering to screen out insignificant or unimportant "pseudo-discrepancies." This mechanism effectively suppresses agents' over-reporting behavior, ensuring the final output focuses on genuinely meaningful inconsistencies.

These two mechanisms work in concert, enabling Dude to strike a balance between the high-recall advantage of multi-agent systems and precision, resolving the inherent contradiction in previous multi-agent designs.

Experimental Results: Simultaneous Gains in Recall and Precision

The research team evaluated Dude on real-world paper-code discrepancy datasets, with impressive results:

• Compared to baseline methods, Dude achieved up to 22.8% improvement in both recall and precision;
• F1 score improved by up to 18.7%.

Deeper Meaning of Evaluation Metrics: Recall and Precision are two core evaluation metrics in information retrieval and classification tasks, and a trade-off typically exists between them. Recall measures "how many of what should be found were actually found," calculated as: true positives / (true positives + false negatives); Precision measures "how many of what was found are correct," calculated as: true positives / (true positives + false positives). In the paper-code discrepancy detection scenario, high recall means finding as many real inconsistencies as possible (reducing missed detections), while high precision means that reported discrepancies are genuinely valid (reducing false alarms). Traditional detection systems often face the dilemma of "you can't have your cake and eat it too": relaxing detection criteria improves recall but increases false positives, while tightening criteria misses real issues. The F1 score is the harmonic mean of recall and precision, providing a comprehensive measure of overall system performance.

These results demonstrate that Dude doesn't sacrifice one metric to boost the other — it achieves simultaneous optimization of both recall and precision, improving recall without compromising precision. For discrepancy detection tasks, the significant F1 score improvement is particularly meaningful — it indicates the system can both "catch everything" (fewer missed detections) and "catch accurately" (fewer false alarms).

Significance and Outlook

Dude's value lies not only in being the first dual-detection multi-agent system for paper-code consistency checking, but also in revealing and addressing a universal problem in multi-agent system design — granularity asymmetry.

Paper-code comparison is just one example among many "cross-modal, cross-granularity" verification tasks. Similar granularity asymmetry problems exist in requirements documents vs. code, design specifications vs. implementations, natural language instructions vs. program behavior, and many other scenarios. Dude's approach of granularity-aligned negotiation and salience filtering offers valuable insights for multi-agent system design across these related domains.

From a broader perspective, as AI research enters an era of "submission floods," automated reproducibility review tools will become essential infrastructure for the academic ecosystem. If systems like Dude can mature further, they could assist reviewers, replicators, and even researchers themselves in promptly identifying and correcting discrepancies between papers and code before and after publication, thereby improving research quality and credibility across the field. The widespread adoption of automated detection tools would not only save the time costs of manual review but also establish a more transparent and trustworthy academic evaluation system, driving a paradigm shift in AI research from "rapid publication" to "high-quality reproducibility."

As a newly published piece of research, Dude is still in its early stages. Its generalization capability on larger and more diverse datasets, as well as its effectiveness in real-world review workflows, still requires further validation. However, the problem direction it identifies and the solutions it proposes undoubtedly lay an important foundation for the emerging topic of automated paper-code consistency detection.