Reproducibility Crisis
The phenomenon in scientific research whereby other researchers find it difficult to reproduce published results; in machine learning it is often caused by issues such as selective reporting, p-hacking and benchmark overfitting.
Core Facts
Timeline (last 90 days)
Selective reporting systematically exaggerates a method's actual effectiveness and is one of the important causes of the 'reproducibility crisis' in machine learning.
The lack of rigorous replication validation for a massive volume of papers has led to a reproducibility crisis, with incorrect or irreproducible conclusions entering the literature database.
All Facts (3)
In 2015, the Open Science Collaboration attempted to replicate 100 psychology studies, and only about 36% were successfully replicated.
80%UnverifiedSelective reporting systematically exaggerates a method's actual effectiveness and is one of the important causes of the 'reproducibility crisis' in machine learning.
50%UnverifiedThe lack of rigorous replication validation for a massive volume of papers has led to a reproducibility crisis, with incorrect or irreproducible conclusions entering the literature database.
50%