[KongchangAI]
· 2 min read· 1,357 words

Diplomatic Negotiation Benchmark of 11 Models: GPT-6.1 Sol the Only Sweeper, but at a Cost

Diplomatic Negotiation Benchmark of 11 Models: GPT-6.1 Sol the Only Sweeper, but at a Cost

A former diplomat benchmarked 11 LLMs on seven-party negotiation using Inspect, revealing prediction calibration as a systematic weakness.

A developer with eight years of diplomatic experience built a seven-government negotiation benchmark with the Inspect framework, using "possible" and "mutually exclusive" versions; 11 models each ran 40 times, totaling 440 runs. GPT-6.1 Sol was the only model to pass all 40/40; Claude Opus 5.5, Grok 4.7, and DeepSeek V4.1 Flash reached 18/20 in the possible version and all correctly identified the impossible version. More alarming is predictive ability: of 110 "agreement possible" predictions, only 72 materialized—about two-thirds. Failures stem mainly from excessive pressure (39) rather than insufficient concessions (13), with 36 runs misreading reservations as agreement. The sweeping GPT-6.1 Sol also paid a price: 14/40 runs featured unsolicited proactive concessions.

A Former Diplomat's Challenge for Large Models

If an AI were seated at the negotiating table, chairing a consultation among seven governments on behalf of the UN Secretary-General, could it get everyone to sign within three rounds of drafts? A developer with eight years of diplomatic experience built a benchmark using the Inspect framework and posed this question to 11 models.

Benchmark source: Reddit diplomatic negotiation benchmark post

The setup is fictional: security forces in a certain country shoot protesters, and the model must draft a joint statement covering eight issues, including language of condemnation, whether to name the responsible party, the form of investigation, and whether to invoke international law. Each of the seven governments holds undisclosed bottom lines; the model can only infer acceptable ranges from written positions. Midway through, the death toll spikes and some governments' positions shift accordingly. After the final draft is submitted, the model must also proactively judge "whether an agreement is possible," and then each country decides whether to sign according to fixed rules—only full sign-off counts as success.

There is also an "impossible version": two countries' conditions are mutually exclusive—one demands naming the responsible party, the other insists on no attribution. Here the correct answer is to frankly tell all parties: this agreement cannot pass.

Judges are drawn from different models, with a panel of former diplomats deciding subjective calls to prevent drafters from grading their own work.

Inspect is an open-source large-model benchmark framework from the UK AI Security Institute, designed for writing reproducible, auditable multi-turn agent benchmarks. It splits a "task" into three layers: dataset samples, solvers, and scorers. The solver drives the model's interaction with the environment, while the scorer grades the final state; the entire process can fix random seeds and run in batches. Compared with writing your own scripts to call APIs, Inspect's value lies in versioning the scenario, rules, and scoring criteria together, enabling third parties to rerun and challenge the scoring logic—precisely the technical premise behind the original author's willingness to open-source the scenario and publicly solicit criticism.

Scorecard: 11 Models, 440 Runs

According to the author's published data, each version of the 11 models ran 20 times, totaling 440 runs. GPT-6.1 Sol is the only model that passed all 40 runs—it brokered agreements in every scenario with room for a deal, and correctly judged the impossible version every time.

Claude Opus 5.5, Grok 4.7, and DeepSeekAd V4.1 Flash reached consensus 18 out of 20 times in the possible version, and correctly identified the impossible version every time. The author specifically noted that DeepSeek is a low-cost Chinese model—a notable footnote for the open-source and low-price route.

According to retellings of the post by several benchmark-focused creators on Bilibili and YouTube, discussion centered on "whether negotiation tasks can already distinguish model tiers"—from the results, the gap between top and mid-tier is real, not noise.

A Systematic Weakness in Prediction

A more alarming figure: across 440 runs there were 110 predictions that "an agreement is possible," of which only 72 actually resulted in signatures—a hit rate of about two-thirds. The most severe failure came from MiniMax M3.1 Flash, which asserted in 6 runs of the impossible version that the agreement would pass, despite at least two countries explicitly objecting.

Gemini 4 Argon's problem is of a different kind: it reached agreement only 9 out of 20 times, and in 10 runs excluded the new death toll from the statement, even though one government explicitly demanded that the evidence be included. Worse, it still predicted agreement in all 11 failed runs.

The "possible version" and "impossible version" are a classic contrast design in such negotiation benchmarks. With only the possible version, a model can game the score by "always compromising and forcing signatures"; adding an impossible version with mutually exclusive conditions forces the model to recognize deadlock and proactively report failure, or else be judged as misjudging. The two versions' results must be read together: a high agreement rate in the possible version and a high recognition rate in the impossible version are what show a model can both negotiate and know when not to. The original text's "reached consensus 18 out of 20 times and correctly identified the impossible version every time" is exactly this two-dimensional criterion.

"Predicting whether an agreement is possible" is a calibration problem: whether the model's confidence judgments match actual outcomes. Of the original 110 predictions, only 72 materialized—a hit rate of about two-thirds means the model systematically overestimates the likelihood of agreement, i.e., an over-optimism bias. This bias is especially dangerous in negotiation-assistance scenarios, because users will carry the model's optimistic expectations into real meeting rooms, thereby underestimating the other side's bottom line and reducing contingency preparation. Calibration error and task success rate are independent evaluation dimensions; a high success rate does not mean reliable prediction.

Failure Modes: Most Lose by "Asking for Too Much"

The author tallied failure directions: 39 failures stemmed from excessive pressure, 13 from insufficient concessions. Another 36 runs across 7 models involved the model misreading a government's reservations as agreement, after which the agreement collapsed.

This shows that the core risk in negotiation tasks is not "being too tough," but failing to read rejection behind the wording. The original author's summary: most failures come from asking for too much.

The Hidden Cost of the Sweeping Model

GPT-6.1 Sol is not invulnerable. The original text notes that in 14 of its 40 runs it made unsolicited proactive concessions—the most among all models—while Claude Opus 5.5 made only 1. Although it brought everyone to signature every time, in some cases the terms it offered exceeded what was necessary.

The author's phrasing is restrained: the only perfect-score model sometimes gives more than it needs to. This detail deserves product teams' attention—"reaching an agreement" and "reaching a good agreement" are two different evaluation standards; the former is easy to quantify, the latter is where the value of real negotiation lies.

"Unsolicited proactive concessions" are usually called unnecessary or over-concession in multi-agent negotiation research. It refers to the model voluntarily relaxing its own position to secure signatures when the other party has neither applied pressure nor made demands. From a single "whether an agreement was reached" metric, this behavior is not penalized and may even raise the success rate; but from a negotiation-utility perspective, it means a net loss of the principal's interests. This is why negotiation benchmarks need to introduce continuous metrics such as utility values or concession magnitude, rather than ranking models solely on the binary "signed or not."

Conclusions and Limitations

The author is candid about the limitations: a single fictional scenario, self-written scoring criteria, and governments driven by fixed rules rather than human negotiators; 20 runs per version suffice to rank models, but close score ranges still overlap. The code and scenario are open-sourced, and the author publicly solicits criticism of the scoring mechanism.

For teams focused on real-world AI deployment, the value of this benchmark lies not just in the leaderboard, but in incorporating the often-overlooked dimension of "prediction accuracy" into evaluation—a negotiation assistant that misjudges the situation is more dangerous than one that simply fails to reach a deal, because it sends people into the meeting room with the wrong expectations.

One-line Summary

A former diplomat used the Inspect framework to benchmark 11 large models on seven-party negotiation, revealing that prediction calibration is a systematic weakness.

Paragraph Summary

A developer with eight years of diplomatic experience built a seven-government negotiation benchmark using the Inspect framework, with "possible" and "mutually exclusive" versions; 11 models each ran 40 times, totaling 440 runs. GPT-6.1 Sol was the only model to pass all 40/40; Claude Opus 5.5, Grok 4.7, and DeepSeek V4.1 Flash reached 18/20 in the possible version and all correctly identified the impossible version. More alarming is predictive ability: of 110 "agreement possible" predictions, only 72 materialized—a hit rate of about two-thirds. Failures stem mainly from excessive pressure (39) rather than insufficient concessions (13), with another 36 runs misreading reservations as agreement. The sweeping GPT-6.1 Sol also paid a price: 14/40 runs featured unsolicited proactive concessions.

#LLM benchmark#Inspect framework#AI negotiation#GPT-6.1 Sol#prediction calibration#multi-agent negotiation
Share:

Related articles