Verbosity bias is a widely studied issue in the research community: human evaluators tend to favor longer, more nicely formatted responses, and a 2024 LMSYS study found that controlling for output length caused the Elo rankings of some models to change significantly.