Re-Search: Using the 20 Questions Game to Dissect LLM Bottlenecks in Scientific Ideation

Why Can LLMs Judge Good Ideas but Not Generate Them?

Scientific ideation is never a one-shot process. Researchers refine ideas step by step based on feedback, yet existing evaluations typically look only at final outputs, completely obscuring the intermediate exploration process. It is hard to know: what detours does an LLM take when exploring research ideas? What limits it from raising truly valuable questions?

A recent paper titled "Re-Search: Dissecting LLM Ideation Through a Two-Player Questioning Game" answers this question with a clever gamified design. It reframes the scientific ideation process as a "20 Questions" game, letting the model rediscover a paper's core idea through pure Q&A interaction, thereby completely isolating "ideation generation" from "experiment execution."

Turning Scientific Ideation into a "20 Questions" Game

Re-Search's design is straightforward: given the research question of a published paper, the model can only obtain binary feedback by asking "yes/no" questions, with the goal of progressively approaching the paper's core idea. The entire process involves no code execution or experimental operations, purely testing the model's search ability within a hypothesis space.

The research team built this evaluation from 200 ICML 2026 papers. Each time the model asks a question, it receives a "yes" or "no" feedback, and its conjectures evolve gradually across Q&A rounds. This design decomposes ideation generation into a traceable trajectory, allowing researchers to precisely observe how the model adjusts hypotheses and where it gets stuck.

The "20 Questions" game is a classic guessing game: one party thinks of an object (such as an animal or item), and the other party progressively narrows down the possibilities by asking up to 20 questions that can only be answered "yes" or "no," ultimately guessing the answer. This game is often used as an abstract model of information retrieval and hypothesis testing, because it requires the questioner to efficiently search a vast hypothesis space under limited binary feedback. In artificial intelligence, the 20 Questions game is used to evaluate an agent's active exploration, information gain maximization, and reasoning abilities. Re-Search borrows this framework, replacing "guessing the object" with "guessing the paper's core idea," making the LLM's ideation process observable and quantifiable.

The First 30 Questions Decide Success or Failure; Most Models Plateau Afterward

Testing six recent LLMs revealed a pattern: almost all progress is concentrated within the first 30 questions (out of 100 total). In other words, the first 30% of Q&A rounds essentially determine the game's final outcome; afterward, most models enter a plateau and no longer produce substantive advances.

This finding itself has practical implications. If the model fails to find the right direction in early Q&A, additional rounds later are unlikely to compensate. The "search window" for ideation is narrower than imagined, and the quality of early questions matters far more than their quantity.

The Bottleneck Is Generation, Not Judgment

Through intervention experiments controlling candidate questions and hypothesis spaces, the study reaches a counterintuitive conclusion: the model's bottleneck is generation ability, not judgment ability. When provided with a good idea or a good question, the model can recognize its value; but when asked to produce these ideas or questions from scratch, it often cannot.

This distinction is crucial. It means that simply improving a model's generation ability may be less effective than introducing ideas and questions from external sources during the search process. What the model lacks is not "taste" but "productivity."

In machine learning and cognitive science, "generation" and "judgment" are often distinguished as two different abilities. Generation refers to producing novel and useful content from scratch, such as proposing hypotheses, designing questions, or creating text; judgment refers to evaluating, scoring, or selecting among existing candidates, such as identifying which idea is more valuable. The two are asymmetric in computational complexity: generation typically requires searching a vast combinatorial space, while judgment only requires comparing given options. LLMs perform excellently on judgment tasks, partly because pretraining objectives (such as next-token prediction) and reinforcement learning from human feedback (RLHF) emphasize evaluation and ranking abilities more than construction from scratch. Therefore, the "judgment stronger than generation" phenomenon discovered by Re-Search is consistent with current LLM training paradigms, and also explains why introducing an external candidate pool can effectively compensate for generation shortcomings.

Two Failure Modes: Over-Exploitation and Over-Exploration

Further analysis of ideation trajectories revealed two typical model failures: one type of model over-exploits a single direction, repeatedly digging deeper along one path without being able to break out; the other over-explores too many directions, scratching the surface of each and failing to converge.

This is exactly analogous to the exploration-exploitation dilemma in reinforcement learning. Good scientific ideation requires finding a balance between the two, and current models seem to lack this dynamic adjustment capability.

The exploration-exploitation dilemma is a core problem in reinforcement learning and decision theory: an agent must trade off between "exploiting" the currently known best option to obtain certain returns and "exploring" unknown options to discover potentially better solutions. Over-exploitation leads to getting stuck in local optima and missing better global solutions; over-exploration leads to dispersed resources and failure to converge. In the scientific ideation scenario, this dilemma manifests as: over-exploiting models repeatedly dig into the same direction and cannot break out of mental fixedness; over-exploring models constantly switch directions and lack deep accumulation. An ideal strategy requires dynamically adjusting the exploration rate based on feedback, for example using optimistic initial values, Thompson sampling, or entropy regularization. Current LLMs lack this dynamic adjustment mechanism and often exhibit extreme tendencies toward exploration or exploitation.

Design Implications for Automated Research Systems

Re-Search's conclusions point to a concrete design principle: since models can judge things they cannot generate, enriching the search space—introducing ideas and questions from external sources—may be more effective than improving a single model's generation ability. This has direct reference value for building automated research (autoresearch) systems: rather than pursuing a stronger generator, design a better candidate pool and filtering mechanism.

Decomposing the ideation process into traceable Q&A trajectories also shifts evaluation from "looking at results" to "looking at the process." This perspective may reveal the true shortcomings of LLMs in scientific ideation better than any single metric.

One-line Summary

Re-Search uses a 20 Questions game to find that LLM ideation bottlenecks lie in generation rather than judgment, with progress concentrated in the first 30 questions.

Paragraph Summary

Re-Search designs the scientific ideation process as a "20 Questions" game, letting LLMs rediscover a paper's core idea through yes/no Q&A alone, thereby isolating ideation generation from experiment execution. Evaluation on 200 ICML 2026 papers shows that six LLMs make almost all their progress within the first 30 questions, then plateau. Controlled experiments show that models can recognize good ideas and good questions (judgment ability) but cannot generate them on their own (generation ability)—the bottleneck is generation, not judgment. Failure modes fall into two categories: over-exploiting a single direction and over-exploring too many directions. The study suggests automated research systems should prioritize enriching the search space of external ideas and questions rather than merely improving model generation ability.