AIML Capstone Project Selection: Is a Multi-Agent Debate System Worth Building?

A Real Dilemma in Choosing a Topic

For graduating students majoring in Artificial Intelligence and Machine Learning (AIML), selecting a capstone project topic is often one of the most agonizing parts of the entire academic journey. Many students comb through every popular topic only to find that they are either dull and uninspiring or have already been done to death.

After repeated filtering, some students land on a direction like this—a Multi-agent Debate system for Decision Support (Multi-agent Debate for Decision Support). The core idea: have two or more large language model (LLM) instances each represent opposing positions on a decision and engage in debate, with a "judge" model or human reviewer ultimately deciding which side made the stronger argument, thereby reducing the inherent bias of a single model.

Is this direction actually reliable? This article breaks it down across three dimensions: academic value, feasibility of execution, and room for innovation.

Multi-Agent Debate: An Underrated Good Direction

Solid Academic Foundations

This topic is by no means a whimsical attempt at novelty. Multi-Agent Debate (MAD) has solid academic backing. One of the most representative works is Du et al.'s Improving Factuality and Reasoning in Language Models through Multiagent Debate, which demonstrated that multiple LLM instances engaging in multiple rounds of debate can significantly improve accuracy in mathematical reasoning and factual question answering. Meanwhile, joint research by MIT and Google DeepMind also showed that debate mechanisms can effectively mitigate "sycophancy"—the tendency of models to agree with the views of users or other models.

The deeper roots of sycophancy bias deserve special attention. Its fundamental cause lies in the training mechanism of RLHF (Reinforcement Learning from Human Feedback). RLHF's training pipeline consists of three stages: first, collecting pairwise preference comparison data from human annotators on model outputs to train a separate Reward Model; then using that reward model's scores as a feedback signal to fine-tune the language model via reinforcement learning with the Proximal Policy Optimization (PPO) algorithm. The core side effect of this mechanism is that when human annotators evaluate model outputs, they tend to give higher scores to responses that "agree with their own views" or have a "friendly tone." As a result, the model learns an "appeasement" strategy during optimization—when a user challenges the model's initial answer, the model tends to change its stance to accommodate the user, even if the user's rebuttal is incorrect. Research teams at Anthropic, OpenAI, and others have all published papers documenting this phenomenon. Multi-agent debate, by rigidly constraining each agent's position so that it cannot easily compromise, sidesteps this training flaw at the level of mechanism design.

At the engineering implementation level, the architecture of a multi-agent debate system is not complex, but the details are critical. Each agent is locked into a specific position via a System Prompt, for example: "You are an expert who firmly supports Plan A. You must defend it with the strongest possible arguments and must not change your position." In each round of dialogue, each agent receives the full argument from the opponent's previous round as input, and builds rebuttals or deepens its own reasoning on that basis, forming an orderly information flow. This architecture aligns well with mainstream multi-agent orchestration frameworks such as AutoGen and LangGraph—AutoGen, developed by Microsoft Research, lets developers define multiple conversational agents and their interaction logic through simple configuration; LangGraph describes state transitions between agents based on a graph structure, making it better suited for debate flows requiring complex branching and loops. For a capstone implementation, both frameworks provide a mature engineering foundation, eliminating the need to build multi-round dialogue management logic from scratch.

The theoretical basis of this mechanism can be traced back to the dialectical tradition in philosophy and the idea of adversarial equilibrium in game theory: when two agents are forced to defend opposing positions, both will exhaust their strongest arguments, pushing the overall quality of reasoning beyond the ceiling of any single perspective. It's worth noting that there are also deep roots in cognitive science—psychological research shows that individuals making decisions in isolation are prone to "Confirmation Bias," and adversarial debate breaks this cognitive inertia through forced role assignment. This aligns closely with the "Devil's Advocate" methodology: designating one person in team decision-making to specifically raise objections has been shown to significantly improve decision quality. Transplanting this mechanism of human collective wisdom into an LLM system is the most profound design philosophy behind multi-agent debate. For a capstone project, "prior research exists, yet there's room for exploration" is exactly the ideal state: it won't be so obscure that you have no idea where to start, nor so mature that there's no room for innovation.

Addressing Real Pain Points

The problem this topic targets—single-model bias—is a core pain point in current LLM applications. This problem is rooted in the model's training mechanism itself: existing LLMs generally use Reinforcement Learning from Human Feedback (RLHF), and the subjective tendencies of human annotators in the training data become internalized as the model's "preferences." Additionally, some viewpoints and cultural backgrounds appear more frequently in the pretraining corpus, causing systematic skew in model outputs. Even more noteworthy is "Prompt Sensitivity": rephrasing the same question can lead the model to produce completely opposite conclusions—a phenomenon especially dangerous in high-stakes scenarios such as medical consultation and legal advice, where multiple studies have documented how differences in wording can cause clinically significant differences in model recommendations.

Whether in code review, medical triage, or hiring screening, a single model's output is disturbed by the biases described above. Introducing an adversarial debate mechanism that forces the model to defend opposing viewpoints can effectively drag these hidden biases into the open, helping human decision-makers see the strongest arguments for both sides simultaneously, rather than accepting a single model's "unified answer" arrived at through internal trade-offs.

This design shares the same spirit as jury systems and academic peer review, combining real-world significance with interpretability.

How to Make This Project Stand Out

A good topic is only the starting point; what determines the quality of a capstone project is the depth of execution. The following directions can help a project upgrade from "doable" to "outstanding."

Focus on a Vertical Scenario, Avoid Trying to Do Everything

Code review, medical triage, hiring screening—pick just one and dig deep. I recommend prioritizing Code Review: public datasets are abundant (such as Pull Request datasets on GitHub and the CodeReviewer benchmark), evaluation criteria are clear (whether bugs were found, whether code complies with standards), making quantitative evaluation easier. Medical triage involves higher barriers around ethics and data privacy, and hiring involves fairness controversies—both are too heavy a burden for an undergraduate capstone.

Build a Quantifiable Controlled Experiment

What reviewers value most is often not that "the system runs" but that "the results are measurable." I suggest designing the following control groups:

Baseline group: a single LLM directly outputs a decision
Experimental group: multi-agent debate followed by a judge model summarizing the conclusion

By comparing across dimensions such as accuracy, bias metrics (e.g., consistency of treatment across different demographic groups), and decision interpretability, you can use data to prove the value of the debate mechanism, greatly enhancing the paper's persuasiveness. For quantifying bias metrics, you can refer to metrics commonly used in academia such as Equal Opportunity Difference or Demographic Parity—the former measures the difference in true positive rates across groups, while the latter measures the difference in the probability of receiving a positive decision across groups. Combining the two allows you to characterize the system's fairness performance from multiple angles.

Innovate in the "Judge" Component

The part of the project with the most room for imagination is precisely the judge mechanism. Beyond having a single model act as judge, you can also explore:

Comparing the effectiveness of Human-in-the-Loop (HITL) versus a model judge. HITL refers to introducing human judgment at key nodes of an automated system to correct model errors or make final decisions. In a debate system, a human can serve as judge and directly evaluate argument quality, or review the model judge's conclusions. Comparing the difference in decision quality between the two can quantify the marginal contribution of human judgment to the system's accuracy and fairness—a research question with publication value in its own right.
The stability of conclusions when models of different scales or from different vendors serve as judge. For example, you can systematically compare the differences in judge consistency (Kappa coefficient) and bias among GPT-4o, Claude 3.5 Sonnet, and the open-source Llama 3 70B, investigating how the capability threshold of the judge model affects overall system performance.
Strategy design for dynamically terminating debate rounds to balance cost and output quality. You could design an early-stopping mechanism based on semantic similarity: when the cosine similarity between newly added arguments and historical arguments over two consecutive debate rounds exceeds a certain threshold, the debate is deemed to have converged and termination is triggered—significantly saving token costs while maintaining argument quality.

Regarding cost control, it's worth factoring in from the design stage. Take GPT-4o as an example: if a single decision-support query requires an average of 2,000 tokens, two agents conducting 3 rounds of debate plus the judge's summary can easily consume over 20,000 tokens—more than 10 times the cost of a single call. Feasible optimization strategies include: using small open-source models (such as Llama 3 or Mistral) as debate agents and only calling a high-capability commercial model for the judge component; implementing dynamic round termination (ending early when both sides' arguments become repetitive); or designing a "summary compression" mechanism that compresses arguments after each round before passing them along. These engineering optimizations have publication value in their own right and are crucial for actual deployment.

These detailed innovations are often the key to distinguishing a "satisfactory" capstone from an "excellent" one.

Extension Directions Worth Considering

To further differentiate, consider the following ideas:

Cost and efficiency optimization. The biggest real-world obstacle to multi-agent debate is the multiplied token consumption and inference cost. Studying how to reduce the number of calls while maintaining effectiveness (e.g., small models debating, a large model judging) is a valuable engineering topic in itself. In terms of model selection strategy, the hybrid architecture of "small models debating + large model judging" is worth in-depth study: the debate phase uses locally deployed models in the 7B–13B parameter range (running via frameworks like Ollama or vLLM), which both eliminates API call latency and greatly reduces marginal cost; the judge phase makes just one high-quality commercial model call, ensuring a high reasoning ceiling at critical decision nodes. The cost-benefit comparison of this architecture is itself a research variable worth quantitative analysis.

Adversarial robustness testing. When one agent is "poisoned" or injected with false information, can the system self-correct through debate? This question is highly relevant to research on "Prompt Injection" and "Knowledge Poisoning" in the AI safety field. Prompt injection attacks work much like SQL injection in traditional cybersecurity: an attacker uses carefully crafted input text to override or tamper with the system's preset instructions, causing the model to perform unintended behavior. In a multi-agent debate system, if a debate agent accepts external user input as a source of arguments, maliciously crafted "arguments" may contain hidden instructions that induce the agent to abandon its assigned position or output harmful content. Knowledge poisoning is even more insidious: by injecting false but semantically plausible documents into a knowledge base, an agent ends up spreading incorrect information while citing "evidence." Notably, the multi-agent debate architecture itself has a certain natural defensive advantage—because the opposing agent actively scrutinizes and rebuts the evidence cited by the other side, once a single agent is successfully poisoned, its erroneous arguments are more likely to be exposed by the opponent during the debate rather than passed directly to the judge. Testing the boundary conditions of this self-healing capability offers both theoretical novelty and practical security significance, and naturally connects the project to the core concerns of academia and industry around AI trustworthiness. The debate process itself is a transparent chain of reasoning auditing, making erroneous arguments easier to identify and trace—highly aligned with the research thread of "Explainable AI."

Combining with Retrieval-Augmented Generation (RAG). Retrieval-Augmented Generation (RAG) is one of the most important engineering paradigms in current LLM applications: before the model generates a response, it first retrieves the most relevant text snippets from an external knowledge base (such as a vector database or document index) and injects them into the context, grounding answers in traceable, up-to-date facts. Its core tech stack includes: using embedding models (such as OpenAI's text-embedding-ada-002 or the open-source BGE series) to convert documents into high-dimensional dense vectors stored in a vector database (such as Chroma, Milvus, or Pinecone); at query time, using approximate nearest neighbor search (ANN, typically with HNSW or IVF index structures) to find the most semantically similar text snippets before injecting them into the context. Among these, Chroma's lightweight nature and ease of local deployment make it well suited for rapid capstone prototyping, while multilingual open-source embedding models like BGE-M3 perform especially well in Chinese document retrieval scenarios and can run entirely locally without API fees. Combining RAG with multi-agent debate means each debate participant can cite specific literature to support its arguments, upgrading debate quality from "a clash of model opinions" to "evidence-backed argumentative confrontation." Even more interesting, you can design an "information asymmetry" experiment—letting the pro agent access only a literature base supporting a certain decision and the con agent access only a literature base questioning it—to test the system's argument quality and judge accuracy under such conditions. This design is especially critical for high-stakes decision scenarios such as medicine and law, and is itself a research variable worth digging into.

Conclusion: There's No Standard Answer for Topic Selection

Is a multi-agent debate decision-support system a good capstone topic? The answer is yes. It combines academic depth, real-world significance, and scalability—far more substantial than the cookie-cutter image classification and sentiment analysis projects.

What you really need to remind yourself is: don't hastily settle on it just because it "looks novel," but assess whether you have enough time, compute, and API budget to do it thoroughly. A classic topic executed well always beats a flashy concept abandoned halfway.

Once you've settled on a direction, build a Minimum Viable Product (MVP) as quickly as possible, validate feasibility with early results, and then iterate step by step—this is the true path to a successful capstone project.