CXBench
A benchmark project that evaluates large language model capabilities through the medium of Go (board game) matches. Dynamic game scenarios are used to assess seven core reasoning abilities: state tracking, local-versus-global trade-offs, long-horizon planning, branch reasoning, adversarial inference, error recovery, and uncertainty management.
Timeline (last 90 days)
CXBench is a project that uses the game of Go to evaluate the capabilities of large language models.
The author decomposed a single Go move decision into seven capability dimensions: state tracking, local versus global trade-offs, long-range planning, branch reasoning, adversarial reasoning, error recovery, and uncertainty management.
Testing revealed that asking the model to reason two more steps ahead from the same board position does not necessarily yield better results, and sometimes the final move is actually worse.
During testing, cases emerged where a model claimed 95% confidence in a move but scored 0 points according to the evaluation criteria.
CXBench uses a smaller 9×9 board to reduce complexity while preserving adversarial dynamics.
CXBench problems do not necessarily come from existing game records; board positions can be regenerated and dedicated Go programs can be used to verify answer quality in order to reduce data contamination.
The focus of CXBench evaluation is on distinguishing between two failure modes: whether the model failed to see the danger, or whether it saw it but made a calculation error afterward.
The author believes that stability and adaptability beyond scores are the key metrics for measuring whether a model is reliable.
All Facts (8)
CXBench is a project that uses the game of Go to evaluate the capabilities of large language models.
50%UnverifiedThe author decomposed a single Go move decision into seven capability dimensions: state tracking, local versus global trade-offs, long-range planning, branch reasoning, adversarial reasoning, error recovery, and uncertainty management.
50%UnverifiedTesting revealed that asking the model to reason two more steps ahead from the same board position does not necessarily yield better results, and sometimes the final move is actually worse.
50%UnverifiedDuring testing, cases emerged where a model claimed 95% confidence in a move but scored 0 points according to the evaluation criteria.
50%UnverifiedCXBench uses a smaller 9×9 board to reduce complexity while preserving adversarial dynamics.
50%UnverifiedCXBench problems do not necessarily come from existing game records; board positions can be regenerated and dedicated Go programs can be used to verify answer quality in order to reduce data contamination.
50%UnverifiedThe focus of CXBench evaluation is on distinguishing between two failure modes: whether the model failed to see the danger, or whether it saw it but made a calculation error afterward.
50%UnverifiedThe author believes that stability and adaptability beyond scores are the key metrics for measuring whether a model is reliable.
50%