Chatbot Arena
An AI model evaluation platform created by the LMSYS team at the University of California, Berkeley. It employs a blind pairwise comparison mechanism in which user preference votes generate ELO-based leaderboard rankings, making it one of the most authoritative general-purpose AI benchmarks.
Core Facts
Chatbot Arena, operated by LMSYS, ranks models through blind evaluation votes by real users on anonymous models, using an ELO rating system derived from chess.
By 2024, Chatbot Arena had accumulated over 1 million human preference votes.
Zheng et al. (2023) systematically proposed the LLM-as-a-Judge method in their research on MT-Bench and Chatbot Arena.
Timeline (last 90 days)
Chatbot Arena (LMSYS) partially mitigates the bias limitations of LLM-as-a-judge through large-scale blind human comparisons, but its cost makes it difficult to integrate into developers' daily workflows
The LLM-as-a-Judge approach is widely adopted in benchmarks such as MT-Bench and Chatbot Arena.
LMArena's predecessor is related to Chatbot Arena.
Major labs are eager to warm up new models in Arena in anonymous variant form before official release, in order to collect real user feedback and accumulate Elo scores.
Ghost testing is an anonymous grey-scale validation method used before a large model's official release, in which the model participates in blind testing in Arena under the identity of an anonymous variant.
Gemini 4.0 Pro has begun anonymous grey-scale testing (ghost testing) in the large model competition arena, Arena.
Chatbot Arena (formerly LMSYS Arena) was developed by a team at UC Berkeley.
The LLM-as-Judge paradigm was popularized by research such as MT-Bench and Chatbot Arena, and is capable of capturing semantic equivalence, logical reasoning, and factual consistency.
Chatbot Arena was developed by UC Berkeley and establishes a ranking system by having users compare the outputs of different models.
The H3 Acceleration Arena draws on the Chatbot Arena evaluation approach from the large language model domain, adopting a blind-test voting mechanism.
30 more timeline events
All Facts (20)
Chatbot Arena, operated by LMSYS, ranks models through blind evaluation votes by real users on anonymous models, using an ELO rating system derived from chess.
85%VerifiedBy 2024, Chatbot Arena had accumulated over 1 million human preference votes.
80%VerifiedZheng et al. (2023) systematically proposed the LLM-as-a-Judge method in their research on MT-Bench and Chatbot Arena.
80%VerifiedChatbot Arena is a crowdsourced live evaluation platform operated by LMSYS, where users converse with two anonymous models simultaneously and vote for the better one.
75%VerifiedArena (Chatbot Arena) is an AI model evaluation platform created by the LMSYS team at UC Berkeley, which uses a blind battle mechanism to generate an ELO score leaderboard.
75%VerifiedLMSys Chatbot Arena was created by the LMSYS organization at UC Berkeley and has collected more than one million human votes since its launch in 2023.
75%VerifiedLM Arena (formerly Chatbot Arena) is an anonymous head-to-head large model evaluation platform developed by a team at the University of California, Berkeley.
75%VerifiedThe Chatbot Arena project was launched by the LMSYS team at UC Berkeley in 2023.
70%VerifiedThe LLM-as-a-Judge evaluation paradigm was first applied at scale by the LMSYS team in MT-Bench and Chatbot Arena.
65%VerifiedChatbot Arena is maintained by LMSYS Org and uses the ELO rating system.
65%UnverifiedChatbot Arena was introduced by a research team at UC Berkeley in 2023 and is one of the earliest platforms to implement large-scale LLM battle evaluation
90%UnverifiedAs of 2024, Chatbot Arena has accumulated over one million human votes
85%UnverifiedWebDev Arena draws on the human preference voting mechanism of Chatbot Arena, ranking participants by ELO score through blind-test voting.
70%UnverifiedArena-type platforms use a pairwise blind-testing mechanism in which users vote for the better response without knowing the identity of the models.
60%UnverifiedThe LLM-as-judge approach was first adopted at scale in MT-Bench and Chatbot Arena.
60%UnverifiedChatbot Arena works by having a user submit a question, the system randomly selects two anonymous models to generate responses, and the user votes for the better answer without knowing which model produced it
60%UnverifiedChatbot Arena (LMSYS) partially mitigates the bias limitations of LLM-as-a-judge through large-scale blind human comparisons, but its cost makes it difficult to integrate into developers' daily workflows
50%UnverifiedThe LLM-as-a-Judge approach is widely adopted in benchmarks such as MT-Bench and Chatbot Arena.
50%UnverifiedLMArena's predecessor is related to Chatbot Arena.
50%UnverifiedMajor labs are eager to warm up new models in Arena in anonymous variant form before official release, in order to collect real user feedback and accumulate Elo scores.
50%