VSArena Open-Sources VLA Evaluation Benchmark: A Robot Leaderboard Driven Purely by Vision and Language

An Open-Source Benchmark Dedicated to VLA Evaluation

The robot learning community is undergoing a paradigm shift from "privileged state" to "pure perception-driven" approaches — and fairly evaluating the results of this transition has long been a challenge. Recently, a developer shared a significant update to the VSArena project on Reddit: it now features a genuine VLA (Vision-Language-Action) track, with the complete code repository and documentation publicly available.

The core of this update isn't about new features — it's a meaningful correction to evaluation philosophy. The project author explicitly stated that this change directly addresses feedback from earlier community discussions, particularly the debate around "how observation spaces should be partitioned."

Core Design of the VLA Track: Policies Only See Camera Feeds and Language

Strict Observation Space Isolation

The biggest change in this VSArena update is a thorough partitioning of the observation space. In the new VLA track, the policy model can only receive a single 128×128 RGB camera image plus a natural language stacking instruction. In other words, the precise cube poses are never passed to the policy.

This is critically important. In many robot simulation benchmarks, models can directly read the 3D coordinates of objects — known as "privileged state." While this makes training easier to converge, it doesn't reflect the reality of robots that can only perceive their environment through cameras. The VSArena VLA track forces policies to work under conditions where they can "see but not read coordinates," much closer to real-world deployment.

VLA (Vision-Language-Action) models are a frontier architecture in embodied AI that integrates visual perception, language understanding, and robot action planning into a single unified model. Unlike traditional perception-planning-execution pipelines, VLA models typically use a pretrained large vision-language model as a backbone, taking camera images and natural language instructions as input and directly outputting joint angles or end-effector displacement commands. Representative works include Google's RT-2 and OpenVLA. The core challenge for these models is precisely that they must estimate the spatial positions of objects from pixel-level visual information, rather than relying on pre-calibrated coordinate data — which is why the definition of observation space boundaries is so crucial during evaluation.

Decoupling the Scoring System from Policy Inputs

Interestingly, the scoring system internally still uses ground-truth poses to evaluate spatial accuracy and task completion. But the author emphasizes that these ground-truth poses are "judge-only" and will never flow into the policy's inputs.

This design — "ground truth for scoring, perception for policy" — ensures both objective accuracy in grading and maintains the challenge of the task. It reflects a principle that mature benchmarks should uphold: the rigor of evaluation standards should not come at the cost of sacrificing task authenticity.

Explicit Separation Between the VLA Track and the State Track

The author deliberately kept a "state-based" track that uses privileged poses, but positioned it as an independent debug track that will not be written into the public ELO rankings.

The reasoning behind this decision is clear. The author mentioned wanting the distinction between "VLA vs state" to be explicit and immediately obvious, rather than something users have to dig deep to discover. In the past, many benchmarks mixed settings of different difficulty levels into a single leaderboard, making it hard for readers to determine whether a high score reflected strong perceptual ability or simply that the model "cheated" by reading object coordinates. VSArena's track separation institutionally eliminates this kind of confusion.

In other words, every score on the public ELO leaderboard represents pure vision-language-driven capability, while the debug track provides researchers with a sandbox environment convenient for development and debugging.

Architectural Division: Client-Side Physics vs. Server-Side Scoring

The Studio Demo Environment Does Not Contribute to Rankings

In earlier discussions, the community raised concerns about the reliability of "client-side physics." After all, if the simulation runs in the user's browser, the physics engine results could be tampered with or inconsistent — clearly unsuitable for generating public rankings.

The author responded clearly: the in-browser demo environment, Studio, is "spectator/dev-only", clearly labeled as such, and will not submit any results to the public leaderboard. Studio runs on Rapier/WASM and achieves a smooth 60fps experience, making it a great entry point for intuitively understanding the tasks — but its role ends there.

Public ELO Rankings Come from the Server-Side Evaluation Framework

The genuine public ELO rankings will only come from a hosted evaluation harness, with all scoring completed server-side. This architecturally guarantees the fairness of the rankings — the scoring logic is neither affected by client-side environments nor manipulable by submitters.

However, the author also candidly acknowledges that this server-side evaluation framework is not yet live — it's the one remaining piece needed before the project can "truly be open to community submissions." This means that although the code and documentation are public, formal community submissions will need to wait a bit longer.

The ELO rating system was originally designed by Hungarian-American physicist Arpad Elo for chess and has since been widely adopted for competitive AI model evaluation (such as Chatbot Arena's rankings of large language models). Its core idea is to measure capability through relative comparisons between models rather than absolute scores: after each match, the winner gains points from the loser, with the amount depending on the expected win probability gap between the two. The advantages of ELO lie in its ability to continuously accommodate new entrants, automatically handle imbalances in match counts, and maintain some robustness against single anomalous results. Introducing ELO rankings into robot benchmarks means a model's capability is measured by its performance relative to other policies, rather than solely by absolute scores on a fixed test set — particularly suitable for early-stage projects where task design continues to evolve.

VSArena Public Repository and Documentation

The VSArena code repository and documentation are now fully public, hosted on GitHub (NovaCoding-G/VSArena). Several documents are particularly worth noting:

• docs/harness.md: Scoring writing guide, detailing how spatial accuracy and task completion are evaluated;
• docs/sdk.md: Submission protocol, explaining how to integrate with the evaluation framework;
• Studio online demo: Runs in-browser on the client-side Rapier/WASM engine at 60fps.

The author described the project as still being in a "solo development, early stage, not ready for heavy promotion" phase — a notably pragmatic stance. He openly invited the community to continue providing feedback, particularly hoping to hear thoughts on what the scoring protocol might still be missing.

A Rigorous Evolution Model Driven by Community Feedback

The significance of this VSArena update may lie not only in the technical changes themselves, but in the healthy open-source development model it demonstrates. The separation of VLA and state tracks, and the division of responsibilities between client and server, weren't designed in a vacuum — they were direct responses to community criticism.

As VLA research heats up rapidly, the credibility of evaluation benchmarks becomes increasingly important. A benchmark with an unclear observation space or manipulable scoring produces rankings that lose their reference value. VSArena's proactive tightening of rules, explicit track separation, and migration of scoring to the server side raise the bar for the project's launch — but also lay the foundation for its future credibility.

For developers focused on embodied AI and VLA models, this kind of open-source benchmark that emphasizes "pure perception, cheat-proof, and reproducible" evaluation is well worth following.