Stickblade Arena: A New Benchmark That Pits LLMs Against Each Other in a Physics-Based Combat Arena

Why We Need an LLM Benchmark That "Fights"

Evaluating the reasoning capabilities of large language models has always been challenging. Current mainstream "reasoning" benchmarks suffer from two chronic problems: first, static question sets whose answers have often already leaked into training data, meaning models might simply be "reciting" correct answers; second, the "LLM-as-judge" approach, where research has repeatedly shown that this method measures similarity between models more than their actual quality — judge models tend to give higher scores to answers that match their own style.

The LLM-as-judge method was first adopted at scale in MT-Bench and Chatbot Arena, with the core idea of having a powerful language model (typically GPT-4) score other models' outputs for quality. However, multiple studies (such as the 2023 "Judging LLM-as-a-Judge" paper) have revealed systematic biases: judge models tend to prefer longer responses (length bias), responses similar to their own generation style (self-preference bias), and responses appearing earlier in a list (position bias). More fundamentally, when the judge model itself has insufficient understanding of a reasoning problem, it cannot reliably evaluate other models' reasoning quality, creating a ceiling for this method when measuring frontier reasoning capabilities.

A developer (Reddit user Cometbuster4969) built an unconventional benchmark to address this: Stickblade Arena. Its core idea is to transform two LLMs into combat characters in a 2D physics arena, putting reasoning capabilities to the test through real physical collisions.

Stickblade Arena Project Introduction

Stickblade Arena's Design Principles: Game Theory in a Physics Engine

World State and Action Loop Mechanism

Stickblade Arena is built on the pymunk physics engine. pymunk is a Python wrapper around the Chipmunk2D physics library, specialized for 2D rigid body physics simulation with support for collision detection, gravity simulation, friction calculation, joint constraints, and more. Unlike simple rule engines, every collision in pymunk undergoes real momentum conservation and energy transfer calculations, meaning character movement trajectories, weapon swing arcs, and post-collision rebound effects are all driven by underlying physics equations rather than preset animations or lookup tables.

Each turn, both competing LLMs receive a JSON-formatted world state containing extremely rich information:

Health points (HP) and positions of both players
Weapon geometry and cooldowns
Damage received in the previous turn
Remaining ammunition
Hazard zones in the arena

Models respond with a JSON-formatted action command. The crucial point is that all actions are resolved through the physics engine. For example, if a swing misses, it's because the opponent actually dodged via a dash — this is a real physics-level "miss," not some rule-lookup result. Matches end upon KO, HP advantage at the time limit, or a draw.

The elegance of this design lies in the fact that answers cannot be memorized in advance. Every match is a dynamically generated, novel situation that depends on spatial reasoning and real-time decision-making, fundamentally circumventing the training data leakage problem. The spatial reasoning here requires models to understand geometric relationships like distance, direction, and motion trajectory prediction in a 2D coordinate system — a known weakness of current LLMs, since language models are primarily trained on text data and lack systematic training in physical intuition for continuous spaces. The partially observable environment (models may not see the opponent's full intentions or hidden state) further increases decision complexity, requiring models to make reasonable inferences under uncertainty.

Identity-Masked Human Voting

The evaluation phase is equally carefully designed. When human reviewers watch match replays, both models' identities are masked. Reviewers simply vote for which side "fought smarter." An even more important detail: votes complete the Elo update before identities are revealed, so voting cannot be contaminated by a model's fame or reputation — you won't subconsciously give extra points just because you know it's GPT.

Objective Data Leaderboard

Running parallel to human voting is a fully objective data leaderboard recording:

Wins / Losses / Draws
Average damage dealt
Hits / Attack attempts
Timeouts

This dual-track system allows "subjective perception" and "objective performance" to be cross-validated, laying the groundwork for subsequent analysis.

Six-Axis Elo Rating: Different Physical Constraints Test Different Reasoning Skills

Stickblade Arena's most research-valuable design is its six-axis Elo rating system. The primary key for ratings isn't simply "model," but a six-tuple:

(model, blade_zone_enabled, weapon, mode, arena, is_blind)

The Elo rating system was originally designed by physicist Arpad Elo for chess, with the core idea of dynamically updating player ratings based on match outcomes. Each player has a numerical rating; before a match, the expected win rate is calculated based on the rating difference, and after the match, both ratings are adjusted based on the deviation between actual results and expectations — an upset victory earns more points, while losing to a weaker opponent costs more. In the AI evaluation space, Chatbot Arena pioneered bringing Elo into LLM rankings through crowdsourced voting for pairwise model comparisons.

The aggregated total Elo is merely the marginalization result across all dimensions, while what the author truly studies are the independent ratings on each dimension. Marginalization is a fundamental operation in probability theory, referring to obtaining marginal distributions from a joint distribution by summing or integrating over certain variables. In the six-axis Elo context, each model performs differently across the five conditional dimensions (blade zone, weapon, mode, arena, blind status), and the total Elo is the comprehensive score obtained after "integrating out" these conditional dimensions. The risk of this approach is that if match sample sizes are severely imbalanced across dimensions, or certain dimension combinations were never tested, marginalization results may be misleading.

The underlying assumption is quite insightful: different physical constraints separately test different reasoning skills — spatial planning ability, uncertainty handling in partially observable environments, and resource management ability.

In other words, a model might excel at spatial game theory in close-quarters combat but perform poorly in ranged weapon duels that require "patient waiting." A single composite score would mask these fine-grained capability differences.

Key Findings: A Scripted Bot Beat 30% of LLMs

Why Baselines Are Critical

The benchmark's participant roster includes 24 entries: 17 LLMs from OpenAI, Groq, and OpenRouter's free tier, plus 4 non-LLM baselines (random strategy, greedy attack, keep distance, scripted heuristic) and 2 simulated strategies. The author specifically emphasizes that baselines are critical — without them, a low-Elo LLM would be indistinguishable from a "however bad" strategy.

Three Noteworthy Experimental Findings

With n=443 matches and 106 votes (lifetime vote conversion rate of 23.9%), several findings are quite illuminating:

Correlation between human vote Elo and objective win rate: The cross-weapon Spearman rank correlation coefficient is approximately ρ≈0.71. The Spearman rank correlation coefficient measures the strength of the monotonic relationship between two variables, ranging from -1 to 1. Unlike the Pearson correlation coefficient, it doesn't require a linear relationship between variables but is calculated based on rankings. ρ≈0.71 means there's a fairly strong positive correlation between human vote rankings and objective win rate rankings, but far from perfect alignment — this indicates that humans' intuition about "smartness" largely aligns with objective data, but considerable room for deviation remains. With a sample size of 106 votes, the confidence interval for this correlation coefficient remains quite wide, meaning the conclusion has statistical significance but precision still needs more data to improve.
Greatest divergence with bow-type weapons: Humans reward "smart waiting" behavior, a strategy that doesn't show up at all in raw damage data. This precisely reveals the blind spot of purely objective metrics — in ranged weapon duels requiring patience and timing, "not attacking" is sometimes the optimal strategy, but damage statistics only see zero output.
Scripted bot dominance: A scripted heuristic bot of merely 100 lines of code (bot:pro) beat approximately 30% of the LLM roster on the objective leaderboard, and about 10% on subjective perception. The author states plainly: "This gap is basically where the signal in this benchmark lives."

This conclusion is quite pointed — if a simple script can beat 30% of large language models, then these models' capabilities in real-time spatial reasoning and decision-making may be far less impressive than their scores on static reasoning benchmarks would suggest.

Reproducibility and Known Limitations

The author maintains candor regarding the project's transparency and limitations. Complete match logs can be exported via /api/export in JSON/JSONL format, and the prompt version for each match is permanently recorded. Since the physics engine involves RNG collisions, the system is non-deterministic, but random seeds are recorded, and deterministic replays based on the same seed are on the roadmap.

Publicly acknowledged limitations include:

The voting population is self-selected (website visitors), not a calibrated panel of expert reviewers
Bot baselines are not strategically optimal (no reinforcement learning-trained baselines yet)
The HuggingFace dataset snapshot cron job hasn't launched yet, and the "frozen evaluation package" currently cannot be reproduced outside the platform

Conclusion: A New Anti-Training-Leakage Paradigm for LLM Evaluation

The significance of Stickblade Arena lies not in its current specific rankings, but in providing a physics-grounded evaluation paradigm. When reasoning must unfold in a dynamic, partially observable environment with physical consequences, models have nowhere to "recite answers," nor do similar judges stand to vouch for them.

This evaluation paradigm resonates with the recent trend of "Embodied AI" in AI research — an increasing number of researchers believe that true intelligence cannot exist divorced from physical world constraints. While Stickblade Arena is merely a simplified 2D environment, the philosophy it represents — "making models prove themselves under physical constraints" — may better expose the true boundaries of model reasoning capabilities than traditional text-based Q&A benchmarks.

The two points where the author most seeks feedback are precisely the key controversies of this design: first, whether the marginalization treatment of six-axis Elo is reasonable; second, whether the "identity reveal as reward" voting incentive mechanism introduces voting bias. These two questions touch on the core challenges of any crowdsourced evaluation system and deserve in-depth discussion by the entire community.

Project URL: https://stickblade-arena.vercel.app Code Repository: https://github.com/Cometbuster4969/STICKBLADE-ARENA