Rejecting Hallucinated Citations: A More Honest Research Agent Practice

The most frustrating problem with research agents is often not a lack of capability, but overconfidence. Fabricating citations while still speaking with full authority, or being so abstract that when something goes wrong you have no idea which step failed. A developer on Reddit (u/spidervirus), fed up with these two extremes, built a smaller, more transparent research agent — ResearchAgent — with a simple core philosophy: only say what it actually did, and only cite sources it actually retrieved.

From "Black-Box Output" to "Explicit Stages"

The most direct design choice of this project is breaking the research process into clearly visible stages: plan → search/verify → synthesize. Each stage has explicit inputs and outputs, rather than cramming everything into a single model call. The developer admits this was done to make failures diagnosable — when the result is wrong, you can pinpoint whether the planning went off track, the retrieval missed key information, or the synthesis stage introduced bias.

Whether this "explicit staging" is worth it is one of the questions he proactively raises in the post. The extra steps do introduce some friction in the workflow, especially when the question is simple and staging feels heavy-handed. But for research tasks that demand credibility, explainability is itself part of the value.

Confidence Scoring: Giving "Uncertainty" a Voice

Another core design of the agent is calibrated confidence scoring with justifications, along with limitations listed by severity. In other words, it doesn't just give a number — it explains why that confidence level was chosen and which aspects have weak evidence. This directly addresses the pain point mentioned at the outset: a hallucinated citation stated with 95% confidence is far more dangerous than an answer with 60% confidence that flags uncertain sources.

According to the developer, the agent only cites real sources it has actually retrieved. This seems basic, yet it's a weak spot for many research agents — generating citations that look plausible but don't actually exist. As for how well-calibrated the confidence scores themselves are, the developer is also soliciting community feedback, indicating this remains an open question rather than a validated conclusion.

Calibration in probabilistic forecasting and machine learning refers to the consistency between a model's output confidence and its actual accuracy rate. A "well-calibrated" system should, over the long run, be correct about 80% of the time when it claims 80% confidence; if it frequently gives wrong answers with 95% confidence, it is in an "overconfident" miscalibrated state. Calibration and accuracy are two different dimensions: a model can be very accurate but poorly calibrated (e.g., always reporting 99% confidence), or have average accuracy but good calibration (willing to admit uncertainty). Common evaluation methods include reliability diagrams and Expected Calibration Error (ECE), which bucket predictions by confidence level and compare the actual hit rate in each bucket. For research agents, calibration quality directly determines whether users can rely on confidence scores to judge whether human review is needed — which is why the original article lists this as an open question.

Local Operation and Human Approval Gates

On the engineering side, the project stays lightweight: optional human approval gates, and a fully local operation mode — it runs with Ollama and DuckDuckGo, with no mandatory dependence on cloud services. For developers concerned about data privacy or wanting to experiment at low cost, this is a pragmatic starting point.

The project also includes progressive examples and tutorials, designed so users can see how each component works rather than treating it as a monolithic black box. This "disassemblable" approach is consistent with the staged design.

Ollama is an open-source local LLM runtime framework that provides a command line like ollama run and a local HTTP API, enabling one-click download and execution of open-weight models such as Llama, Mistral, and Qwen without building your own inference service or relying on cloud APIs. DuckDuckGo serves as the retrieval source here, offering search result interfaces that require no API key (such as the Instant Answer API or HTML endpoint), suitable for low-cost, privacy-focused prototyping; compared to official Google/Bing search APIs, it makes trade-offs in result richness, rate limits, and stability. The combination means the entire "search — reasoning — synthesis" pipeline can run entirely on a local machine, with data never leaving the device — but the cost is being limited by local compute and the reasoning capability of consumer-grade models, with long-context handling and quality ceilings typically lower than cloud-based large models.

Who This Kind of Agent Is For

If you're after an ultra-smooth one-click experience, the explicit staged workflow may feel verbose. But if you're working on research tasks that require traceability and credibility judgment, or you're building your own agent and want to understand what happens at each step, this approach of putting uncertainty on the table is worth considering. The two questions the developer raises at the end of the post — whether confidence scores are trustworthy and whether explicit stages are a help or a burden — are exactly the directions the community needs to validate together for this kind of tool going forward.

One-line Summary

A developer fed up with research agents' overconfidence built a transparent research agent with staged processing, traceable sources, and calibrated confidence.

Paragraph Summary

Reddit developer u/spidervirus, frustrated that research agents either fabricate citations or make failures impossible to locate, built ResearchAgent. It explicitly divides the research process into three stages — planning, search/verification, and synthesis — so errors can be traced to specific steps; it only cites real sources actually retrieved and provides calibrated confidence scores with justifications and severity-ranked limitation notes. Engineering-wise, it supports optional human approval gates and can run fully locally (Ollama + DuckDuckGo), with progressive examples and tutorials included. The developer proactively seeks community input on two open questions: whether confidence scores are trustworthy and whether explicit stages are a help or a burden. The article argues that for research tasks requiring traceability and credibility judgment, this approach of putting uncertainty on the table is worth considering.