Hype Trap
A financial stock-selection benchmark designed to detect momentum-echo behavior in large language models. It analyzes the correlation between a model's stock scores and historical price returns to determine whether the model exhibits excessive dependence on past price performance.
Timeline (last 90 days)
A developer created a benchmark called "Hype Trap" to test whether AI models would favor a stock simply because it had already risen sharply, even if the price increase has no predictive power for future performance.
The benchmark started with 15,606 raw scoring records, which after cleaning yielded 351 qualifying cases, with placeholder scores accounting for 31% of all scoring rows being excluded.
None of the 9 models could clearly outperform the "fixed guess of 31%" baseline, with all models' AUC confidence intervals containing 0.5.
Asking the model in the prompt to classify the type of news caused 3 models running the no_read test to lower their average predictions by 1.5 to 6.0 points, making them track price gains more closely, indicating that prompt wording changes model behavior.
All Facts (4)
A developer created a benchmark called "Hype Trap" to test whether AI models would favor a stock simply because it had already risen sharply, even if the price increase has no predictive power for future performance.
50%UnverifiedThe benchmark started with 15,606 raw scoring records, which after cleaning yielded 351 qualifying cases, with placeholder scores accounting for 31% of all scoring rows being excluded.
50%UnverifiedNone of the 9 models could clearly outperform the "fixed guess of 31%" baseline, with all models' AUC confidence intervals containing 0.5.
50%UnverifiedAsking the model in the prompt to classify the type of news caused 3 models running the no_read test to lower their average predictions by 1.5 to 6.0 points, making them track price gains more closely, indicating that prompt wording changes model behavior.
50%