Elo
A rating system for assessing the relative skill levels of players, originally developed for chess. It iteratively adjusts scores based on the deviation between match outcomes and expected outcomes, and is widely used in competitive rankings and AI model evaluation.
Timeline (last 90 days)
In highly subjective tasks, pairwise comparison is gradually replacing absolute scoring as the mainstream evaluation paradigm, and its statistical results can be directly fed into the Bradley-Terry model or an Elo system to output a globally transitive ranking.
Chatbot Arena uses the Elo rating system to convert a large number of voting results into model rankings, and Elo was originally used for chess.
In a blind paired Elo study, PWM-WROP ranked first among continuation-style models and third overall, behind only a statistical tie between two reference video generation models.
Knowledge work benchmarks are typically Elo-based and used to measure a model's overall performance on professional tasks such as code review, technical documentation, and system design.
Code Arena is a competitive arena focused on front-end code generation that uses blind user voting to produce Elo scores.
Chatbot Arena uses real-user blind comparative evaluation, having users simultaneously converse with two anonymous models and vote, with rankings determined by an Elo scoring system.
The Elo system assumes capability is a one-dimensional scalar and cannot reflect differences in model performance across various task dimensions; LMSYS subsequently introduced a category-based ranking mechanism to address this shortcoming.
LMSYS Chatbot Arena is an evaluation platform led by UC Berkeley that uses an Elo rating system similar to chess, allowing real users to vote in pairwise comparisons without knowing the identities of the models.
The LMSYS team typically requires each model to accumulate at least several thousand comparative votes before considering its Elo rating statistically significant.
When the number of match samples is sufficiently large, Elo ratings converge to the maximum likelihood estimate.
2 more timeline events
All Facts (12)
Text Arena uses an Elo-like scoring mechanism to dynamically calculate comprehensive rankings based on head-to-head model competitions
85%UnverifiedIn highly subjective tasks, pairwise comparison is gradually replacing absolute scoring as the mainstream evaluation paradigm, and its statistical results can be directly fed into the Bradley-Terry model or an Elo system to output a globally transitive ranking.
50%UnverifiedChatbot Arena uses the Elo rating system to convert a large number of voting results into model rankings, and Elo was originally used for chess.
50%UnverifiedIn a blind paired Elo study, PWM-WROP ranked first among continuation-style models and third overall, behind only a statistical tie between two reference video generation models.
50%UnverifiedKnowledge work benchmarks are typically Elo-based and used to measure a model's overall performance on professional tasks such as code review, technical documentation, and system design.
50%UnverifiedCode Arena is a competitive arena focused on front-end code generation that uses blind user voting to produce Elo scores.
50%UnverifiedChatbot Arena uses real-user blind comparative evaluation, having users simultaneously converse with two anonymous models and vote, with rankings determined by an Elo scoring system.
50%UnverifiedLMSYS Chatbot Arena is an evaluation platform led by UC Berkeley that uses an Elo rating system similar to chess, allowing real users to vote in pairwise comparisons without knowing the identities of the models.
50%UnverifiedThe Elo system assumes capability is a one-dimensional scalar and cannot reflect differences in model performance across various task dimensions; LMSYS subsequently introduced a category-based ranking mechanism to address this shortcoming.
50%UnverifiedWhen the number of match samples is sufficiently large, Elo ratings converge to the maximum likelihood estimate.
50%UnverifiedThe Elo system is based on the Bradley-Terry model, which assumes that the probability of player A defeating player B depends only on the difference between their ratings, specifically as a logistic function.
50%UnverifiedThe LMSYS team typically requires each model to accumulate at least several thousand comparative votes before considering its Elo rating statistically significant.
50%