LangGraph Four-Way Agent Showdown: Plan-and-Execute Costs 2.3× More With No Advantage

Title

LangGraph Four-Way Agent Showdown: Plan-and-Execute Costs 2.3× More With No Advantage

Meta Description

A pre-registered LangGraph comparison of four architectures: under the same model, tools, and budget, ReAct, Plan-and-Execute, Supervisor, and Draft-Verify show no significant accuracy difference on 240 legal Q&A tasks, yet Plan-and-Execute costs 2.3× more with no benefit.

Article Content

A developer built four agent architectures on LangGraph using the same model, tools, and budget, then ran a pre-registered comparison test on 240 questions about Japanese legal statutes. The results were surprising: the lowest-cost design and the highest-cost design showed no statistically significant difference in accuracy, while the Plan-and-Execute architecture consumed more than 2.3× the tokens without delivering any detectable benefit.

Experiment Design: Four Architectures, Controlled Variables

The author implemented four agent designs in LangGraph, holding all other conditions constant: the same three tools (search_articles, get_article, verify_citation), the same base prompt, the same local model (qwen3.5 9B, Q4_K_M quantization, temperature 0), and a budget cap of 24,000 tokens and 20 model calls per question.

• A, ReAct: a single model-tool loop.
• B, Plan-and-Execute: a planner writes up to three steps, an executor loops through each step, and finally a synthesizer writes the answer.
• C, Supervisor: a supervisor routes between a researcher loop and a writer.
• D, Draft-Verify: a ReAct loop first produces a draft, then every citation is symbolically verified, with up to two revisions.

The test set contained 120 questions, asked in both Japanese and English, for a total of 240 questions. Question types were divided into topical, adjacent, and paired, and the questions did not directly name statute numbers (e.g., "What is the article after Article X?"), so search capability was critical. Scoring was fully symbolic, with no language-model judge: a citation extractor, a registry existence check, and registry-based gold article matching.

Pre-Registration and Noise Baseline

Before running the test set, the author submitted a pre-registration document including input hashes, metric definitions, the paired test method (exact McNemar test per question, with a threshold of 0.05/6 = 0.0083 for six pairwise comparisons), and five hypotheses. Prompts were tuned on 36 separate development questions, using the same rounds for each design and with no design-specific modifications.

To measure the noise floor, the author ran the development set twice under identical settings: the same design on the same 36 questions varied in correctness by 1 to 5 questions, and only 2 to 5 answers had identical text. This means that even with an identical configuration, there is non-negligible random fluctuation.

The McNemar test is a statistical test for paired binary data, commonly used to compare the performance difference between two methods on the same set of samples. It focuses on paired samples where the two methods disagree, constructing a 2x2 contingency table and computing a chi-square statistic (or exact binomial test) to determine whether the direction of disagreement is systematic. In this experiment, each question was answered by two designs, and correctness formed paired data, making the McNemar test suitable for detecting significant differences in accuracy between two designs. The author also applied Bonferroni correction, dividing the significance threshold by the number of comparisons (6 pairwise comparisons, threshold set to 0.05/6 ≈ 0.0083) to control the false-positive risk from multiple comparisons.

Core Results: No Significant Accuracy Difference, Clear Cost Divergence

The results on the 240 questions are as follows:

Design	Accuracy	Avg tokens per question	Tokens per correct answer	p50 latency (s)
ReAct	59.6% [53.1, 65.8]	4,663	7,826	14
Plan-execute	61.7% [55.2, 67.8]	10,535	17,084	40
Supervisor	65.0% [58.6, 71.0]	7,494	11,530	31
Draft-verify	63.7% [57.3, 69.8]	4,905	7,694	16

No pair of designs reached the statistical significance threshold for accuracy, with the smallest p-value at 0.0525. But the cost differences were clear: ReAct and Draft-Verify both averaged lower cost than Plan-and-Execute and Supervisor, with non-overlapping bootstrap confidence intervals and no detectable drop in accuracy. Plan-and-Execute was dominated across the board by the other three designs.

The extra tokens were mainly consumed by tool-loop rounds (more searches and statute reading, each re-sending a growing context) rather than planning text—90% of the tokens were spent in the executor loop.

Four of the five hypotheses held as expected: accuracy gaps were all within 10 percentage points; the cost ordering was A < D < C < B, with B at least twice A; the fabricated citation rate was below 5%; and ReAct and Draft-Verify skipped tools in fewer than 25% of runs. The only partially supported hypothesis was that the author originally expected Plan-and-Execute to perform better on paired questions, but it did not stand out (39 of 80 questions correct, the same as Supervisor).

Limitations of Statistical Power

When designs disagree on between 9% and 23% of questions, a pair of designs needs a gap of roughly 6 to 9 percentage points to be judged different. Supervisor's 5-point lead over ReAct (p=0.085) was neither confirmed nor ruled out. This means the experiment's statistical power was limited, and small-to-medium real differences cannot be excluded.

Statistical power refers to the probability that an experiment can detect a true difference when one exists. Power is affected by sample size, effect size, and the significance threshold. In this experiment, due to the limited sample size (240 questions) and a disagreement rate between designs of 9% to 23%, the experiment could only reliably detect large differences (about 6-9 percentage points). Therefore, even if Supervisor is 5 percentage points higher than ReAct, it may fail to reach statistical significance due to insufficient power. The author explicitly notes that this non-significant result should not be interpreted as "no difference," but rather as "the current experiment is insufficient to distinguish medium or small differences."

The Seemingly Victorious Draft-Verify

Draft-Verify got 10 more questions correct than ReAct (16 questions correct only for it, 6 correct only for ReAct). This looks like the verification mechanism at work. But deeper inspection revealed: of those 16 questions, 5 involved ReAct producing fabricated citations, and 1 was unanswered. Removing fabricated citations is exactly what the verifier does—using the same registry as the scoring—so its low 0.4% fabrication rate is a property of the experimental setup, not an achievement. Of the remaining 10 questions, ReAct cited real articles but missed the gold article, which the verifier could not detect. The two designs are indistinguishable, and their disagreement (22 of 240 questions) is roughly equivalent to a single design's own run-to-run noise.

"Correct" Is Just a Citation Check

The first record the author printed exposed the problem: the question asked for the article after Article 730 of the Civil Code, and ReAct answered Article 731—the number was correct—and claimed that this article stipulates the termination of support obligations. But Article 731 actually stipulates the minimum age for marriage. The model had only seen Article 730 from the search results and then guessed the next number.

Because the tools are deterministic, the author replayed the tool calls for every record and checked whether each gold article was actually returned to the model. This was true for 91% to 99% of correct answers. But for the remainder (13 of ReAct's 143, 12 of Supervisor's 156, 14 of Draft-Verify's 153, and 1 of Plan-Execute's 148), it was not: the number was correct but the content was fabricated. No metric in this study checked whether the answer accurately described the content of the correctly cited article. Under the stricter standard of "correct and gold article read," the paired tests still did not reach significance (smallest p=0.036), so the conclusion still holds—but the number the author treats with the most caution is precisely the one called "correct."

Symbolic scoring refers to methods that evaluate answer correctness through rules or deterministic procedures rather than relying on language-model judgment. The symbolic scoring in this experiment involved three steps: extracting the cited article numbers from the answer, checking whether those numbers exist in the legal statute registry, and whether they match the pre-annotated gold articles. This approach avoids subjectivity and model bias, but it only verifies the existence and matching of citation numbers, not whether the answer accurately describes the article's content. Thus, symbolic "correctness" may overestimate actual legal accuracy, as the author found cases of correct numbers with fabricated content among correct answers.

Errors Encountered During the Experiment

• A scoring bug that looked like a model result: The initial development-set scoring used a pattern requiring 第740条, but the model wrote 第 740 条 and 民法709条. 28 answers were scored as "no citation." After fixing, they were re-scored from the text.
• An early item extractor bug: The prefix of each branch citation (第二条の二) was counted as a second phantom citation. All 1,800 published answers for that item were re-scored and the numbers corrected.
• A metric measuring headings: The pre-registered "unsupported citation" rate was 86% to 90% for all designs, because answers cited the statute heading from the question, and headings are explanatory text rather than article body. The author reported but did not use this metric.
• A picky tool: get_article accepted 415-2 but not 415条の2 or 250の6, returning "no such article" for articles that exist. 11 of Plan-Execute's 34 failed lookups were of this type. The tool was the same for all designs and was not changed after pre-registration.

Conclusions and Limitations

The core message of this experiment is: under controlled variables, pre-registration, and symbolic scoring, more complex agent architectures (planner, supervisor, verifier) did not bring detectable accuracy improvements, while costs increased significantly. Plan-and-Execute in particular proved inefficient—the extra tokens came mainly from tool loops rather than planning intelligence.

But the author also explicitly lists limitations: these results apply only to synthetic questions, a single model, a single prompt set, and a single budget; the questions are highly regular, and English questions directly cite Japanese headings; "correct" is only a citation check, not legal accuracy. Different prompts, different models, or different budgets could all change the ranking. The code, pre-registration document, report, post-hoc analysis (annotated), and per-question records are all publicly available on GitHub and Hugging Face.

One-line Summary

Four agent architectures show no significant accuracy difference on 240 legal questions, but complex architectures cost more.

Paragraph Summary

A developer implemented four agent architectures on LangGraph (ReAct, Plan-and-Execute, Supervisor, Draft-Verify) and ran a pre-registered comparison test on 240 Japanese legal questions using the same model, tools, and budget. The results show no statistically significant accuracy difference across designs (smallest p=0.0525), but clear cost differences: Plan-and-Execute consumes more than 2.3× the tokens of ReAct and is dominated across the board. Extra tokens are mainly consumed by tool loops rather than planning. Draft-Verify appears to lead, but its advantage mainly comes from the verifier removing fabricated citations, and the remaining gap is indistinguishable from run noise. In addition, symbolic "correct" scoring only checks citation numbers, and some correct answers contain fabricated content. The experiment's statistical power is limited and cannot rule out small-to-medium real differences.