DeepInstructor: An Agentic Framework for Research Idea Evaluation Driven by Structured Academic Experience

From Generation to Evaluation: The Bottleneck in Research Ideation Is Shifting

As automated scientific discovery advances, large language models (LLMs) can now generate research ideas at unprecedented scale. This creates a new problem: when ideas are no longer scarce, the real bottleneck shifts from "generating ideas" to "judging ideas." Who decides which of these massive volumes of ideas truly have novelty, significance, and feasibility?

Existing evaluation methods largely rely on two approaches: the model's own parametric knowledge, or unstructured retrieval-augmented generation. Both have clear shortcomings — neither produces the kind of "experience-grounded" reasoning that a human mentor brings to the table. When a seasoned reviewer evaluates a paper, they draw on years of domain experience and countless comparisons with similar work — not simple knowledge recall or text matching.

DeepInstructor's Core Idea: Turning Evaluation into Reasoning Over Experience

This arXiv paper proposes DeepInstructor, an agentic framework that reframes research idea evaluation as "a reasoning process over structured academic experience." Its key innovation is treating evaluation not as a one-shot scoring task, but as a traceable, interpretable reasoning process.

Building the Experience Graph

At the foundation of DeepInstructor is an experience graph constructed from 58,607 peer reviews. This scale is significant — it means the system doesn't rely on the model's implicit knowledge, but instead explicitly organizes the judgments that real reviewers have accumulated through practice into structured data. These review comments are decomposed and linked, forming a retrievable experience network that allows the AI to "reference how others have judged similar work" during evaluation.

ReAct-Based Agentic Retrieval

Built on top of this graph, DeepInstructor employs an agent based on the ReAct (Reasoning and Acting) paradigm. When evaluating a given idea, the agent retrieves dimension-specific evidence rather than offering a blanket conclusion. This "evidence-per-dimension" design ensures every judgment can be traced back to the experience supporting it, dramatically improving interpretability — which is precisely what pure parametric methods lack most.

ReAct (Reasoning and Acting) is a language model reasoning paradigm proposed by a Google Research team in 2022. Its core idea is to have the model alternate between generating reasoning chains (Chain-of-Thought) and executing external tool calls or information retrieval actions. Unlike pure chain-of-thought prompting, ReAct allows the model to "pause, query external information, and continue reasoning based on the results" mid-process, forming a think–act–observe loop. This design is particularly well-suited for tasks requiring multi-step verification: the model doesn't need to rely on one-shot memory recall but can dynamically supplement evidence during reasoning. In DeepInstructor's context, judging dimensions like "novelty" and "feasibility" often requires different reference points. The ReAct framework lets the agent retrieve relevant review experience for each sub-question separately, rather than using a single broad query to cover all dimensions — ensuring every conclusion is backed by concrete experiential evidence.

The DeepInstruct Dataset: Controlled Pairwise Comparisons

To train and validate this framework, the researchers also introduced the DeepInstruct dataset. Its distinguishing feature is controlled pairwise comparisons across three core dimensions:

• Novelty: Does the idea have original value?
• Significance: What is the idea's potential impact?
• Feasibility: Can the idea realistically be implemented?

Using pairwise comparisons rather than absolute scores is a pragmatic choice. Humans are generally more reliable when judging "which of A or B is better" than when assigning "a score to A." Pairwise comparisons also align better with human judgment and provide a more stable benchmark for measuring model performance.

Pairwise comparison as an evaluation paradigm has deep roots in machine learning, and has been widely adopted since reinforcement learning from human feedback (RLHF) gained popularity. Its advantage lies in avoiding the subjective ambiguity of absolute scoring: different reviewers may interpret what "7 out of 10" means very differently, but tend to reach higher consensus on "A is more novel than B." Statistically, pairwise comparison data can be converted into global rankings via methods like the Bradley-Terry model, preserving relative preference information while reducing annotation noise. This is also why mainstream LLM alignment training broadly favors preference data over scoring data. The DeepInstruct dataset applies this thinking to research idea evaluation, making the training signal more closely reflect how human reviewers actually make decisions — in practice, reviewers facing multiple submissions are fundamentally making relative comparisons, not isolated absolute judgments.

Experimental Results: Significant Improvement in Alignment with Human Judgment

The experimental results reported in the paper show that DeepInstructor substantially outperforms existing baseline methods. In terms of alignment with human judgment:

• Hit@1 improved by 24.4%
• Hit@2 improved by 29.7%

Hit@1 and Hit@2 measure how closely the model's rankings match human preferences. An improvement of nearly 30% demonstrates that grounding evaluation in explicit structured experiential reasoning is indeed more aligned with expert judgment than relying on the model's implicit knowledge.

Significance and Limitations

The most important takeaway from this work is that research idea evaluation can be grounded in explicit reasoning over structured academic experience, rather than black-box model intuition. As automated research pipelines grow increasingly sophisticated, a trustworthy and traceable evaluation component will become the critical gatekeeper of the entire system. DeepInstructor offers a path to turning "reviewer experience" into an asset that can be systematically reused.

That said, its limitations deserve honest acknowledgment. The experience graph is derived from historical reviews and may inherit existing reviewer biases. Peer review itself is subject to disagreement and noise when judging "novelty" and "significance." Furthermore, whether performance on controlled pairwise comparison datasets generalizes to real-world, open-ended idea evaluation scenarios remains to be validated. As a newly published arXiv preprint, its conclusions still await broader replication and scrutiny.

Even so, the direction DeepInstructor points toward — enabling AI evaluators to reason "with evidence and precedent," much like a human mentor — holds genuine reference value for the broader field of automated scientific discovery.