ViBench 2.0
A benchmark targeting long-horizon software construction capability, evaluating an agent's engineering ability to build a complete product step by step from scratch without a global view of the task.
Timeline (last 90 days)
In the ViBench 2.0 benchmark, even the best models had only 49-65% of their builds pass all checks.
ViBench 2.0 has been included in Fireworks AI's Specialized Intelligence Index.
The core constraints of ViBench 2.0 are 'long-horizon' and 'no lookahead'.
ViBench 2.0 has been included in FireworksAI_HQ's Specialized Intelligence Index, accompanied by official Fireworks run results.
ViBench 2.0 is a benchmark for long-horizon software building, aimed at evaluating agents' ability to build a complete product from scratch without a global view.
ViBench 2.0 shifts the evaluation focus from 'can it write correct code' to 'can it build a correct system'.
The release of ViBench 2.0 provides the AI programming field with an evaluation dimension closer to real practice, revealing the capability gap between code generation and system building.
ViBench 2.0's difficulty setting is discriminative: it neither lets top models easily score full marks nor is so low that capability differences cannot be reflected.
ViBench 2.0 simulates real development scenarios: the agent starts from an empty project, implements only one feature at a time, and cannot foresee subsequent requirements.
All Facts (9)
ViBench 2.0 is a benchmark for long-horizon software building, aimed at evaluating agents' ability to build a complete product from scratch without a global view.
70%UnverifiedViBench 2.0 has been included in Fireworks AI's Specialized Intelligence Index.
50%UnverifiedIn the ViBench 2.0 benchmark, even the best models had only 49-65% of their builds pass all checks.
50%UnverifiedViBench 2.0's difficulty setting is discriminative: it neither lets top models easily score full marks nor is so low that capability differences cannot be reflected.
50%UnverifiedViBench 2.0 shifts the evaluation focus from 'can it write correct code' to 'can it build a correct system'.
50%UnverifiedThe release of ViBench 2.0 provides the AI programming field with an evaluation dimension closer to real practice, revealing the capability gap between code generation and system building.
50%UnverifiedViBench 2.0 simulates real development scenarios: the agent starts from an empty project, implements only one feature at a time, and cannot foresee subsequent requirements.
50%UnverifiedThe core constraints of ViBench 2.0 are 'long-horizon' and 'no lookahead'.
50%UnverifiedViBench 2.0 has been included in FireworksAI_HQ's Specialized Intelligence Index, accompanied by official Fireworks run results.
50%