How Do Humans Build Forecasting Bots? Interviews with 15 Developers Reveal Three Major Challenges

Interviews with 15 forecasting bot builders reveal three major challenges: fact acquisition, coherent reasoning, and performance evaluation.
Based on qualitative interviews with 15 successful participants in an AI forecasting competition, this article explores the practical dilemmas human builders face when constructing forecasting bots. The study reveals three major challenges: introducing ineffective complexity when tuning bots to acquire factual research; testing in opportunistic and piecemeal ways when achieving coherent reasoning; and struggling to accurately grasp capability boundaries when understanding actual performance. The paper argues that future innovation in processes, tools, and human-AI collaboration should target these specific pain points rather than merely pursuing model reasoning capabilities. As the study comes from a single source, its conclusions should be viewed with caution, but it provides valuable empirical material for understanding the obstacles to deploying AI forecasting.
The Rise of Forecasting Bots and the Role of Human Builders
AI models' increasingly powerful complex reasoning capabilities make them attractive tools for predicting the future. But for AI forecasting systems to truly take off, human builders have a great deal of work to do—tuning bots to acquire facts, achieving coherent reasoning, and evaluating their actual performance. A qualitative interview study of 15 forecasting bot developers reveals the recurring dilemmas in this work.

Forecasting is the systematic inference of the future based on past and present data. As AI models demonstrate increasingly strong capabilities on complex reasoning tasks, researchers have begun exploring their use for forecasting work. However, the paper notes that if AI forecasting systems are to be adopted, human builders will bear the responsibility of constructing and evaluating these systems.
To understand what this work actually looks like, the research team conducted qualitative interviews with 15 forecasting bot builders. These interviewees were sampled from successful participants in a recent AI forecasting competition. They had both practical experience and measurable results, making their experiences highly informative.
The paper's core contribution is that it does not stop at abstract discussions of "whether AI can forecast," but instead delves into the details of builders' daily work, revealing three specific challenges.
Qualitative interview research is a social science research method that uses semi-structured or open-ended conversations to gain deep insight into interviewees' experiences, perceptions, and practical details, rather than relying on quantitative statistics. It typically employs purposive sampling, selecting representative or information-rich cases, and distills patterns through thematic analysis and similar methods. In this study, the 15 interviewees came from successful participants in an AI forecasting competition; this sampling ensured that the sample had both practical experience and measurable results, thereby revealing the real dilemmas in builders' work. The strength of qualitative research lies in capturing the details of "why" and "how," but its limitation is the small sample size and the difficulty of statistically generalizing conclusions, so the authors emphasize that its universality should be viewed with caution.
Challenge One: Tuning Bots to Acquire Factual Research
The first challenge centers on "tuning bots to acquire factual research." Forecasting requires grounding in reliable factual information, but getting AI systems to autonomously acquire and filter facts is not simple. Builders need to design mechanisms that guide bots to find relevant and accurate data sources, rather than relying on the model's internal knowledge, which may be outdated or incorrect.
This process involves extensive debugging and trial and error. The paper notes that builders often introduce complexity that does not pay off—that is, the additional modules or processes they add ultimately do not improve forecast quality but instead increase the system's burden.
Factual research in the forecasting context refers to systematically collecting, verifying, and integrating external information relevant to the forecasting question, rather than relying on static knowledge stored in model parameters. Large language models have knowledge cutoff dates and may contain hallucinations or outdated information, so builders need to design mechanisms such as retrieval-augmented generation (RAG), tool calling, or web search to enable bots to dynamically acquire authoritative data sources. However, this process involves complex decisions such as data source screening, timeliness assessment, and relevance ranking, and builders often need to repeatedly debug prompts and processes. The paper notes that builders tend to add unnecessary modules (such as extra verification steps or redundant data sources), and this complexity does not improve forecast quality but instead increases the system's burden and debugging difficulty.
Challenge Two: Achieving Coherent Reasoning
The second challenge is "achieving coherent reasoning." Forecasting is not simple fact retrieval; it requires connecting multiple pieces of information into a logical chain of inference. AI models have grown stronger at complex reasoning, but keeping their reasoning process coherent and interpretable remains difficult.
Builders need to design prompts, processes, or external structures to guide model reasoning, but the paper finds that their testing approaches are often "opportunistic and piecemeal." This means builders lack systematic evaluation methods, responding to problems as they arise rather than establishing a rigorous testing framework from the start.
Coherent reasoning requires AI models to connect multiple pieces of factual information according to logical relationships, forming a traceable inference chain from premises to conclusions. In forecasting tasks, this means the model needs to identify causal mechanisms, weigh different pieces of evidence, and maintain consistency, rather than simply piecing together fragments. Although large models have improved on complex reasoning benchmarks, their reasoning processes may still skip steps, contradict themselves, or be disturbed by irrelevant information. Builders often guide models through chain-of-thought prompting, external reasoning frameworks, or structured processes, but the paper finds that their testing approaches are "opportunistic and piecemeal"—that is, they lack preset evaluation protocols and instead patch things up as specific errors arise. Such piecemeal testing makes it difficult to systematically expose weak links in the reasoning chain, causing coherence problems to be masked or to recur.
Challenge Three: Understanding How Bots Actually Perform
The third challenge is "understanding how well your bot actually performs." Evaluating forecasts is inherently complex: the future has not yet happened, so how do you judge whether a forecast is good? Builders need to design evaluation mechanisms, but the paper notes that they often struggle to accurately grasp the true boundaries of a bot's capabilities.
This challenge is intertwined with the second: if testing is piecemeal, then understanding of bot performance will also be incomplete. Builders may overestimate or underestimate a bot's capabilities, thereby affecting the direction of subsequent tuning.

Forecast evaluation is inherently difficult: future events have not yet occurred, so ground truth is not immediately available, and builders must rely on proxy indicators such as historical backtesting, probabilistic scoring rules (such as the Brier score), or expert review. However, each of these methods has limitations—backtesting may overfit to history, scoring rules depend on parseable forecast formats, and expert review is subjective and expensive. Without a systematic evaluation framework, builders find it difficult to accurately define a bot's capability boundaries and are prone to overestimating or underestimating its performance. The paper notes that the piecemeal nature of evaluation and the piecemeal nature of reasoning testing are intertwined: incomplete testing leads to one-sided understanding of bot performance, which in turn affects tuning direction, forming a vicious cycle. This highlights the need to develop standardized, reusable evaluation tools.
Implications for Innovation in Tools and Processes
The paper argues that a detailed description of builders' challenges helps clarify the problems that future innovation in processes, tools, and human-AI team collaboration should target. In other words, rather than abstractly pursuing stronger forecasting models, it is better to first solve the specific pain points builders encounter in their actual work.
For example, if builders generally introduce ineffective complexity in fact acquisition, then tool design should help simplify this step; if testing is piecemeal and opportunistic, then more systematic evaluation frameworks need to be provided. These directions are closer to actual needs than simply improving model reasoning capabilities.
Consensus and Divergence Across Multiple Perspectives
It should be noted that the analysis in this article is mainly based on a single research source and has not yet been cross-validated by multiple independent sources. Therefore, the above conclusions should be regarded as an interpretation of that specific interview study rather than a broad consensus in the field. Nevertheless, the study's depiction of the details of builders' actual work provides valuable empirical material for understanding the obstacles to deploying AI forecasting systems.
For developers and product designers concerned with AI application deployment, this study reminds us that the bottleneck of AI forecasting lies not only in model capabilities but also in how humans construct, tune, and evaluate these systems. Ignoring the realities of builders' work makes it difficult to translate even the strongest reasoning capabilities into reliable forecasting tools.
One-line Summary
Interviews with 15 forecasting bot builders reveal three major challenges: fact acquisition, coherent reasoning, and performance evaluation.
Paragraph Summary
Based on qualitative interviews with 15 successful participants in an AI forecasting competition, this article explores the practical dilemmas human builders face when constructing forecasting bots. The study reveals three major challenges: introducing ineffective complexity when tuning bots to acquire factual research; testing in opportunistic and piecemeal ways when achieving coherent reasoning; and struggling to accurately grasp capability boundaries when understanding how bots actually perform. The paper argues that future innovation in processes, tools, and human-AI collaboration should target these specific pain points rather than merely pursuing model reasoning capabilities. As the study comes from a single source, its conclusions should be viewed with caution, but it provides valuable empirical material for understanding the obstacles to deploying AI forecasting.
Related articles

Gemini 4 Neon Review: Benchmark Scores Rival Carbon, API Pricing and Quota Breakdown
Google Gemini 4 Neon is now live, benchmark scores closely tracking Carbon, API pricing 3/15 at twice Hydrogen's rate. This article compiles Bilibili creator testing: Neon can replace 90% of Carbon scenarios, AntiGravity quota consumption is only one-third of Carbon, plus subscription channel restrictions and 2API ban risks.

Gemini 4 Helium Coming Soon? Signals of Google Loosening Subscription Restrictions
Google team member hints releases will continue; community speculates Gemini 4 Helium (Flash Lite) may be next. Subscription ban controversy gets a response as product lead says "Soon," suggesting restrictions may be lifted. This article covers Gemini developments, OpenAI Coding Agent and risk management, Doubao real-time audio-video API, and compute business dynamics.

AI-Assisted React Development in Practice: Optimizing a Video App and Handling Errors
A 14-year veteran developer demonstrates AI-assisted React video app optimization: compressing video from 80MB to 5MB (~94%) via ImageKit, plus reliable error handling and retry mechanisms. Includes planning, code understanding, and verification methods for working with AI.