Open-Source MCP Server: Automatically Verifying the Authenticity of AI-Generated Paper Citations

When AI Starts Fabricating Citations

Generative AI is increasingly being used in academic writing, but a subtle yet serious problem has emerged alongside it: AI can fabricate citations that appear real but do not actually exist. This phenomenon is known as "citation hallucination." Large language models may generate references with perfect formatting—complete with authors, journals, and even DOI numbers—yet these citations simply do not exist in the real world.

What is a DOI? A DOI (Digital Object Identifier) is a permanent, unique code assigned by the academic publishing industry to every formally published work, taking a form like 10.1038/s41586-023-06792-0. Managed by the International DOI Foundation (IDF), publishers apply for DOIs from registration agencies such as CrossRef when a paper goes online. The DOI was designed to solve the problem of broken web links—even if a paper migrates to a new server, the DOI always points to the same article. Precisely because of this authoritative status, AI-generated fake citations bearing DOIs are especially dangerous: users often treat the presence of a DOI as an endorsement of a reference's authenticity, when in reality the model has merely learned the string-format patterns of DOIs and can generate codes that "look legitimate" but cannot actually be resolved.

The root of this problem lies in the very training mechanism of large language models. LLMs learn linguistic patterns by predicting the next word across vast amounts of text. They have "learned" the formatting patterns of academic citations—the arrangement of author names, journal titles, years, and DOI numbers—but their training process does not embed a queryable, real-time "fact database." Models tend to generate content that "sounds plausible" rather than content that "actually exists." This "fluency trap" is particularly deadly within highly structured citation formats: a fabricated citation is often flawless in its formatting, and detecting it manually requires painstaking, one-by-one verification.

What merits deeper examination is that this hallucination is not a "random error" of the model, but an inevitable product of its statistical learning nature. During training, LLMs learn the statistical patterns of language from trillions of tokens through self-supervised learning—essentially learning "what kind of word sequences are reasonable in human writing." When asked to generate academic citations, the model makes a "plausible inference" based on the citation formatting patterns it has learned: it selects combinations of author names, journal titles, and research topics that frequently co-occur in the training corpus, stitching together complete-looking citation entries. This process is fundamentally different from "fact retrieval"—the model is not querying a database that stores real literature; it is performing statistical interpolation.

The Dual Influence of Self-Supervised Learning and RLHF Self-Supervised Learning is the core training paradigm of modern large language models: the model requires no human annotation and trains itself solely by predicting masked or truncated tokens in text, thereby extracting the statistical structure of language from internet-scale corpora. This approach is extremely efficient, yet fundamentally incapable of distinguishing between "linguistically plausible" and "factually true"—the model's objective function is to minimize prediction error, not to maximize factual accuracy. Building on this, Instruction Fine-tuning makes models better at following user instructions and generating well-formatted output; RLHF (Reinforcement Learning from Human Feedback) further reinforces the style of responses that "look useful and fluent" by having human evaluators score model answers. The problem is that human evaluators often find it difficult to verify citation authenticity item by item during scoring, which means RLHF may sometimes "reward" fabricated citations that are formatted more perfectly—the model learns "what a high-quality citation should look like" but not "a high-quality citation must actually exist." This mechanistic flaw is precisely the deeper reason why citation hallucination is difficult to eliminate through simple prompt optimization.

Even more concerning is that models trained through Instruction Fine-tuning and RLHF perform better at adhering to formatting conventions, and may as a result generate "more perfectly formatted" fake citations—further increasing the difficulty of manual detection.

For researchers, this is a fatal hidden danger. Once a fake citation slips into a formal paper, the consequences range from damaging academic reputation to facing the risk of retraction. Recently, a developer shared his solution on Reddit: an open-source MCP server capable of automatically fact-checking AI-generated citations before they ever reach a research paper.

What Is MCP, and Why Is It Well-Suited for Citation Verification

An Introduction to the MCP Protocol

MCP (Model Context Protocol) is an open standard open-sourced by Anthropic in 2024, designed to provide large language models with a unified interface for interacting with external tools and data sources. At its core, it adopts a client-server architecture: the AI assistant acts as the client and communicates with the MCP server via the standardized JSON-RPC protocol; the server, in turn, encapsulates specific external tool capabilities—such as querying databases, accessing APIs, or executing specific validation logic. This "decoupling of tools from models" design means developers do not need to modify the model itself; they simply deploy a new MCP server to extend external capabilities to any MCP-supporting AI client (such as Claude Desktop).

From a broader technical perspective, MCP solves a long-standing "M×N integration problem" in AI capability extension. Before MCP, whenever an AI application wanted to connect to N external tools, it often had to develop a separate adaptation layer for each tool, leading to enormous duplicated effort and inconsistent interfaces.

The Technical Essence of the M×N Integration Problem Imagine an AI platform that simultaneously supports M different models (GPT-4, Claude, Gemini, etc.) and wants all these models to be able to call N kinds of external tools (search engines, databases, code executors, etc.). Without a unified standard, each model requires a separate adaptation interface for each tool, resulting in a total workload of M×N—as the number of models and tools grows, maintenance costs balloon multiplicatively. MCP transforms this problem into M+N: each model only needs to implement the MCP client protocol once, and each tool only needs to implement the MCP server interface once, with the two communicating through a standardized JSON-RPC message format. The protocol defines three core interaction capabilities: Tools (allowing models to perform operations with side effects), Resources (allowing models to access read-only data such as files and databases), and Prompts (allowing servers to inject structured context into models). This aligns with the classic computer science idea of "interface abstraction"—just as an operating system's device driver model lets hardware vendors integrate into the ecosystem without understanding the details of upper-layer applications, MCP lets tool developers extend capabilities to models without understanding the internal implementation of any specific model.

By defining a unified communication specification—clearly specifying the interaction formats for three core capabilities: Tool Definition, Resource Exposure, and Prompt Templates—MCP enables any server that conforms to the MCP specification to be plug-and-play by any MCP client. This closely mirrors the design philosophy of the USB interface: a standardized interface decouples peripherals from hosts, greatly reducing the friction of ecosystem expansion. MCP servers can be implemented in any programming language and communicate with clients via standard input/output (stdio) or HTTP+SSE, making independent development of vertical scenarios like academic citation verification possible.

Through an MCP server, an AI assistant can call external capabilities in real time during a conversation. This architecture is naturally suited for citation verification scenarios: when the AI generates a citation during writing, the MCP server can immediately step in, compare that citation against real academic databases, and intercept false information "at the source of the problem"—rather than waiting until the paper is finished to manually screen for errors.

From Passive Checking to Active Interception

Traditional citation verification usually happens after writing—researchers must manually check each reference one by one, a massive workload that is prone to omissions. The core innovation of this MCP server lies in moving verification forward to the generation stage: every time the AI produces a citation, the system instantly verifies its authenticity, enabling a "verify-as-you-write" workflow.

Breaking Down How It Works

The typical workflow of such paper citation verification tools includes the following key steps:

• Citation Extraction: Identifying and parsing structured citation information from AI-generated text, including fields such as title, author, journal, year, and DOI.
• Database Comparison: Cross-querying the extracted citations against authoritative academic databases to confirm whether the literature actually exists. The current mainstream data sources that can be integrated include: CrossRef (a DOI registration agency operated by a consortium of publishers, indexing over 150 million journal articles with DOIs, offering the most authoritative coverage of published literature and being the top choice for verifying DOI authenticity), PubMed (maintained by the U.S. National Library of Medicine, focused on the life sciences and medicine, covering approximately 35 million records with coverage depth in these fields far exceeding general databases), Semantic Scholar (a multidisciplinary free database built by Allen AI, employing AI-assisted literature parsing technology capable of processing full-text PDFs and automatically extracting citation relationship graphs, offering a fully open free API—a common integration choice for open-source tools), and arXiv (an important source of preprints in science and engineering, indexing large numbers of the latest research results that have not yet undergone peer review, especially important for tracking cutting-edge research in fields like AI, physics, and mathematics, though its preprint nature requires users to keep in mind that the content has not been peer-reviewed).

Differences in the Indexing Logic of the Four Major Academic Databases These four databases differ fundamentally in coverage and technical architecture, which directly affects the practical effectiveness of citation verification tools. CrossRef's core value lies in its status as the "authoritative DOI registration agency"—publishers must register a DOI with CrossRef when publishing an article, making CrossRef the gold standard for verifying whether an article has been formally published, though the quality of metadata returned by its API varies, and abstracts and author information for older literature are sometimes missing. PubMed is maintained by the U.S. National Center for Biotechnology Information (NCBI); beyond indexing published articles, it also links biomedical data such as gene sequences and clinical trials, and its MeSH (Medical Subject Headings) vocabulary is an important tool for semantic retrieval in the life sciences—but literature outside the medical field is almost entirely absent from its coverage. Semantic Scholar's uniqueness lies in its AI-driven literature graph: it not only indexes literature metadata but also automatically parses citation relationships through machine learning, building data on "paper influence" and "citation lineage," which makes it especially useful for tracing the intellectual evolution of a concept; its fully free open API also makes it the top integration point for open-source citation verification tools. arXiv, meanwhile, is positioned as a preprint platform where articles can be published without peer review—highly timely but of variable quality. In the context of citation verification, the presence of arXiv means that "an article can be found" does not equal "an article has been academically validated"—a subtle distinction that users must pay special attention to when interpreting verification results.

The disciplinary emphases and indexing logic of different databases determine the applicable scope of verification tools, so users should select the appropriate combination of data sources based on their research field.

• Authenticity Judgment: Based on the comparison results, marking statuses such as "Verified," "Not Found," or "Information Mismatch" to help users quickly locate suspicious citations.
• Feedback Return: Returning verification results in real time to the AI assistant or user, so they can correct or delete false citations.

The value of this design lies in embedding the mechanism for safeguarding academic integrity directly into the AI-assisted writing workflow, rather than as a standalone, after-the-fact remedy tool.

Why Paper Citation Verification Is Becoming Increasingly Important

The First Line of Defense for Academic Integrity

As more and more researchers use tools like ChatGPT and Claude to assist with literature reviews and paper writing, the scale of the AI citation hallucination problem continues to grow. The academic community has already conducted systematic quantitative research on this: analyses published across multiple academic platforms in 2023 show that mainstream LLMs have error rates ranging from 30% to over 60% when generating academic citations, with the specific figures varying significantly depending on prompting methods, disciplinary fields, and model versions. Error types show clear patterns: completely fabricated "ghost citations" account for about half of the errors, while another common error type is "partially correct"—the article title genuinely exists, but the author, journal, or year information has been replaced or confused. This type of error is more deceptive and harder to detect through simple full-field matching.

The Cost of Academic Retraction Once citing false literature is discovered, the consequences often exceed researchers' expectations. Data from Retraction Watch, an international retraction monitoring organization, shows that the number of academic papers retracted annually has grown more than tenfold over the past decade, with "fabrication of data or references" being one of the primary reasons for retraction. Retraction not only means the invalidation of a single paper but also triggers scrutiny by journal editors of the author's other papers. At many universities and research institutions, a retraction record can directly affect a researcher's promotion review, project application eligibility, and even job security. An even more insidious cost is the damage to academic reputation: in the highly interconnected academic community, once a paper containing false citations is discovered by peers and spread on social media, its negative impact is difficult to eliminate through subsequent corrections. For researchers who use AI to assist their writing, "the AI generated the false citation" does not constitute an excuse for exemption—the authors of a paper bear academic responsibility for all its content, which is a widely held consensus in academia. The value of automated citation verification tools lies precisely in reducing the cost of fulfilling this responsibility to a minimum.

An automated, integratable verification tool can significantly reduce the academic risks posed by human negligence.

The Significance of Open Source

This project is released as open source, meaning any researcher, institution, or developer can freely deploy, review, and improve it. For an academic community that emphasizes transparency and reproducibility, the open-source nature is especially crucial—users can clearly understand the verification logic, avoiding the new uncertainties brought by "black-box" validation, while also being able to integrate different literature databases according to their own disciplinary needs. Humanities and social sciences, minority-language journals, or regional publications often fall outside the coverage of mainstream databases; the open-source architecture allows the community to make targeted extensions and adaptations for these blind spots, rather than passively waiting for commercial tools to add support.

Practical Applications and Limitations

Applicable Scenarios

Such tools are best used in conjunction with AI clients that support the MCP protocol (such as Claude Desktop). When conducting literature reviews or writing the introduction or methodology sections of a paper, researchers can have the AI automatically complete verification while generating citations, greatly reducing the workload of subsequent manual checking.

Boundaries to Keep in Mind

Automated verification is not a panacea. It can effectively identify completely nonexistent false citations, but for deeper problems such as "the citation exists but its content does not support the argument," human judgment is still required. In addition, the quality of the tool's verification depends heavily on the coverage of the databases it integrates—obscure journals, non-English literature, or the latest preprints may have coverage blind spots, and the actual experience of tool effectiveness may differ considerably between researchers in science/engineering and those in the humanities and social sciences.

The Gulf Between "Citation Exists" and "Citation Is Appropriate" Automated verification tools solve the problem of a citation's "existence," but the deeper challenge in academic writing is "relevance" and "accuracy"—that is, whether an article truly supports the argument for which the author cites it. This kind of problem requires understanding the substantive content of the paper, which currently far exceeds the capability boundaries of automated tools. In addition, there is a category of "semantic confusion" errors: the model cites a genuinely existing article but distorts or exaggerates its conclusions. For example, a paper that only demonstrates a certain effect under specific controlled experimental conditions is cited by the AI as proof of the effect's universality. Such problems cannot be detected at the automated verification level; they require researchers to personally read the original text and critically evaluate its argumentative logic. Therefore, the correct posture for using the tool is: treat automated verification as the first line of defense for filtering out obvious errors, not as a reason to replace deep reading. A citation list that has passed automated verification means "these articles exist," not "these articles support your argument"—a distinction that is crucial for maintaining academic rigor.

When using it, one should maintain rational expectations, viewing automated verification as an aid to reduce low-level errors, rather than an ultimate solution that replaces professional judgment.

Conclusion: Making AI-Assisted Academic Writing More Trustworthy

This open-source MCP server represents a direction worth watching: not restricting the use of AI in academic settings, but equipping it with the necessary "safety valve." As AI becomes deeply integrated into the knowledge production process, similar citation verification layers will become increasingly indispensable.

For users who rely on AI-assisted research, rather than nervously checking citations one by one after a paper is finished, it is better to establish an automated protection mechanism from the very beginning. Though small, this project points toward a more mature and more responsible paradigm for AI-assisted academic writing. Interested researchers might consider keeping an eye on its open-source repository and trying to incorporate it into their daily workflow.