Evaluating Inquiry Learning in LLM Agents: Hypothesizing, Reasoning, and Exploration

When LLMs Become Scientists: An Exam on Inquiry Ability

Scientific inquiry is never a display of a single capability, but a cyclical process of "hypothesis—experiment—observation—revision." Large language models perform impressively on individual tasks such as inductive reasoning, hypothesis generation, and data analysis—but can they handle it when these abilities are strung together into a complete inquiry chain?

A study titled Evaluating Inquiry Learning in LLM-Based Agents: Hypothesizing, Reasoning, and Exploration takes aim at exactly this question. The researchers point out that prior work has mostly tested LLMs' scientific abilities in isolation, lacking a unified, interactive evaluation framework to observe how models integrate these capabilities across an entire inquiry cycle.

Experimental Design: Turning the Countdown Task into a "Math Laboratory"

From Static Problems to an Interactive Environment

The research team's approach is quite clever: they extended the classic Countdown task into a math environment where agents no longer passively answer questions, but actively manipulate variables, collect observations, and infer relationships between variables. This effectively transforms a static math problem into a miniature scientific system where "experiments" can be performed.

In this environment, agents need to complete several key actions:

• Exploration strategy: deciding which variable combinations to try first
• Causal discovery: identifying dependency relationships between variables from observed data
• Hypothesis revision: adjusting conjectures based on new evidence

This setup allows researchers to observe multiple dimensions of inquiry behavior simultaneously, rather than just checking whether the final answer is correct.

The Countdown task is a common benchmark in mathematical reasoning and planning: given several numbers and a target number, the goal is to combine the given numbers into the target using addition, subtraction, multiplication, and division. Traditional Countdown is a single-turn static Q&A where the model outputs an expression and that's the end—it tests search and arithmetic ability. This study makes a key modification: converting the problem-solving process into multi-turn interaction, where the model can proactively propose a variable combination and observe feedback, then infer hidden rules from it. In this way, what was originally a "one-shot solve" problem becomes a closed loop of "repeated experimentation—observation—induction." Agents no longer compete on single-step reasoning accuracy; instead, their behavioral characteristics are exposed in areas such as designing experiments, allocating trial-and-error budgets, and extracting patterns from noisy feedback. The benefit of this modification is that it decomposes inquiry ability into observable interaction trajectories, allowing exploration strategy, causal discovery, and hypothesis revision to be scored independently, providing a unified experimental foundation for the subsequent evaluation framework.

Cross-Comparison of Seven Models and Multiple Prompting Strategies

The experiment tested seven LLMs paired with multiple prompting strategies. The results show that the models' intrinsic inquiry ability is quite limited—the success rate for discovering the underlying expression (i.e., truly understanding the system's rules) was low. They tend to probe randomly rather than strategically design experiments; when faced with contradictory evidence, they also lack the ability to effectively revise hypotheses.

Structured Guidance: A Remedy from Education Science

The Unexpected Effect of Scaffolded Prompting

The most interesting finding of the study is that when researchers introduced structured guidance inspired by educational scaffolding, model performance improved significantly. In other words, inquiry learning support strategies designed for human learners—such as step-by-step hints, guiding questions, and staged checkpoints—can also help LLMs better complete inquiry tasks.

This result has two implications. On one hand, it shows that LLMs are not entirely devoid of inquiry potential—they simply lack the ability to self-organize and self-regulate. On the other hand, it suggests a practical direction for improvement: rather than expecting models to spontaneously develop a scientist's way of thinking, it is better to provide structured support at the interaction level.

Educational scaffolding originates from Vygotsky's "zone of proximal development" theory. It refers to the temporary hints, demonstrations, or step-by-step guidance a teacher provides when learners still struggle to complete a task independently, gradually withdrawing them as competence grows. Common scaffolding forms include: breaking complex tasks into sub-steps with sequential dependencies, posing guiding questions, setting checkpoints at key nodes, and providing imitable examples. The cleverness of this study lies in transferring this human education concept into prompt engineering: using structured prompt templates in place of a human teacher to define stages for the LLM's inquiry process and indicate what to do next. This is both a prompt engineering strategy and a kind of "cognitive scaffold" at the interaction level. Its effect is significant because it compensates for the model's shortcomings at the metacognitive level—namely, its lack of self-monitoring about which stage of inquiry it is currently in and what kind of information gain it should pursue next.

Why Simple Guidance Works

The researchers' explanation is that structured guidance helps models maintain clearer stage divisions during inquiry—what to do first, what to do next, and when to go back and verify. Without such guidance, models easily get "lost" in the exploration space, repeatedly trying similar hypotheses without realizing it.

Limitations and Implications: How Far Are LLMs from Being "Autonomous Scientists"?

Strategic Exploration Is a Clear Weak Point

The core shortcomings exposed by the experiment center on strategic exploration and iterative hypothesis revision. Excellent scientific inquirers do not blindly try things at random; they design the next experiment based on existing observations so that each step provides as much new information as possible. LLMs perform poorly in this regard, often lacking this awareness of "information gain."

The core of strategic exploration is an awareness of "information gain": an excellent inquirer will not waste experimental budget on hypotheses that have already been verified, but will prioritize experiments that can reduce uncertainty as much as possible regardless of the outcome. In cognitive science, this corresponds to the idea of "optimal experimental design," and in reinforcement learning it connects to active learning and Bayesian experimental design. The finding of this study is that LLMs perform close to random probing in such decisions: they tend to repeat similar hypotheses and repeatedly operate on variables already supported by evidence, while rarely proactively designing critical experiments that could falsify the current conjecture. The direct consequence of lacking information-gain awareness is low sample efficiency—with the same number of interactions, human inquirers can lock onto the underlying rules faster, while the model may remain for a long time in observation regions with low discriminative power. This is also one of the important reasons for its low success rate in discovering underlying expressions.

Practical Implications for AI Agent Development

For developers building AI agents, this study conveys a pragmatic signal: at the current stage, it is unrealistic to expect LLMs to autonomously complete open-ended scientific inquiry tasks. A more feasible path is to embed guidance mechanisms into system design—whether scaffolding at the prompt engineering level or stage checkpoints in multi-turn interaction, both can substantively improve inquiry quality.

The study also reminds us that when evaluating LLMs' scientific abilities, high scores on individual benchmarks can be misleading. The real challenge lies in the connection and integration between abilities, and this is precisely where current models are weakest.

Conclusion

This work constructs a unified, interactive evaluation framework for LLM inquiry ability, and by modifying the Countdown task it enables comprehensive assessment of exploration strategy, causal discovery, and hypothesis revision. The performance of seven models reveals the clear deficiencies of current LLMs in autonomous inquiry, while also demonstrating the effectiveness of structured guidance. For the AI research community, this is both an honest "capability checkup report" and a clue pointing toward improvement: to teach LLMs to think like scientists, perhaps we must first start by "teaching" them how to be guided.

One-line Summary

A study builds an interactive framework to evaluate the full LLM inquiry cycle, finding that autonomous inquiry ability is limited, but structured guidance can significantly improve performance.

Paragraph Summary

Aiming at the question of whether LLMs can handle a complete scientific inquiry chain, this study constructs a unified, interactive evaluation framework and transforms the classic Countdown task into a math environment where variables can be actively manipulated, thereby comprehensively assessing exploration strategy, causal discovery, and hypothesis revision. Experiments testing seven LLMs and multiple prompting strategies find that the models' intrinsic inquiry ability is quite limited: they tend to probe randomly rather than strategically design experiments, with a low success rate in discovering underlying rules. However, when structured guidance inspired by human educational scaffolding is introduced, model performance improves significantly, indicating that they are not without inquiry potential, but lack self-organization and self-regulation. The study points out that strategic exploration and iterative hypothesis revision are currently the most prominent weaknesses, and offers pragmatic advice for AI agent development: at this stage, guidance mechanisms should be embedded into systems rather than expecting models to spontaneously develop a scientist's mindset.