VSArena
An open-source robotics evaluation benchmark for VLA (Vision-Language-Action) models. Policies are restricted to using only RGB camera images and natural language instructions, with access to privileged state information prohibited. Rankings are determined via ELO scoring, aiming to fairly evaluate purely perception-driven robot policy capabilities.
Timeline (last 90 days)
The challenge of applying the ELO mechanism to robot strategy evaluation lies in the limited number of task scenarios and sparse submission samples, resulting in high score variance in the early stages.
ONISCOR has launched VSArena 1.0, an open browser-based benchmark for embodied AI.
VSArena's core task is to have policy models complete object stacking in the order cyan → orange → magenta.
VSArena's physics simulation runs directly in the browser, powered by the Rapier physics engine.
In VSArena, policy models communicate with the hosted evaluation framework via WebSocket, receiving observation data and sending back action commands.
VSArena introduces an ELO rating system for VLA policies, and the public ELO leaderboard is exclusively for VLA policies.
VSArena results are verified with signatures, using a digest plus Ed25519 digital signature to ensure traceability and tamper-resistance.
VSArena distinguishes between two paths: demonstrations (Studio mode) and official submissions, with Studio mode demonstrations not written to the official leaderboard.
The official submission process for VSArena is officially stated to take approximately only 10 minutes.
VSArena provides a Studio mode trial entry at vsarena.vercel.app/simulation and an official submission entry at vsarena.vercel.app/submit.
13 more timeline events
All Facts (20)
VSArena has publicly released an eval-integrity document on GitHub detailing its signing and anti-cheating mechanisms, and has opened a Discord channel.
50%UnverifiedThe challenge of applying the ELO mechanism to robot strategy evaluation lies in the limited number of task scenarios and sparse submission samples, resulting in high score variance in the early stages.
50%UnverifiedVSArena's core task is to have policy models complete object stacking in the order cyan → orange → magenta.
50%UnverifiedVSArena's physics simulation runs directly in the browser, powered by the Rapier physics engine.
50%UnverifiedIn VSArena, policy models communicate with the hosted evaluation framework via WebSocket, receiving observation data and sending back action commands.
50%UnverifiedVSArena introduces an ELO rating system for VLA policies, and the public ELO leaderboard is exclusively for VLA policies.
50%UnverifiedVSArena results are verified with signatures, using a digest plus Ed25519 digital signature to ensure traceability and tamper-resistance.
50%UnverifiedVSArena distinguishes between two paths: demonstrations (Studio mode) and official submissions, with Studio mode demonstrations not written to the official leaderboard.
50%UnverifiedThe official submission process for VSArena is officially stated to take approximately only 10 minutes.
50%UnverifiedVSArena provides a Studio mode trial entry at vsarena.vercel.app/simulation and an official submission entry at vsarena.vercel.app/submit.
50%UnverifiedIn VSArena's VLA track, the policy model can only receive a single 128×128 RGB camera image and a natural language stacking instruction.
50%UnverifiedIn VSArena's VLA track, the exact pose of the blocks is not passed to the policy model.
50%UnverifiedVSArena's scoring system internally uses ground-truth poses to evaluate spatial accuracy and task completion, but these ground-truth poses are visible only to the judge and do not flow into the policy's input.
50%UnverifiedVSArena retains a state-based track that uses privileged poses, but positions it as a separate debugging track that does not contribute to the public ELO rankings.
50%UnverifiedVSArena's in-browser demo environment, Studio, is intended for observation and development only and does not submit scores to the public leaderboard.
50%UnverifiedVSArena's public ELO rankings come exclusively from a hosted server-side evaluation framework, with all scoring completed on the server.
50%UnverifiedVSArena's code repository and documentation are hosted on GitHub at NovaCoding-G/VSArena.
50%UnverifiedVSArena is a solo-developed project in an early stage.
50%UnverifiedVSArena's track-separation design (explicitly distinguishing VLA from state) is a direct response to community feedback.
50%UnverifiedONISCOR has launched VSArena 1.0, an open browser-based benchmark for embodied AI.
50%