DataBugBench
A benchmark designed to evaluate the detection and repair of silent bugs in Python data science code. It focuses on assessing large language models' ability to identify and fix silent logic errors in Pandas, NumPy, and Scikit-Learn codebases.
Timeline (last 90 days)
DataBugBench is a benchmark for evaluating large language models' debugging ability on invisible bugs in data science.
DataBugBench specifically constructs diagnostic tests targeting Pandas, NumPy, and Scikit-Learn code.
DataBugBench focuses on five categories of hidden hazards in data engineering: Pandas SettingWithCopyWarning unintended modification, groupby aggregation losing the grouping key, NumPy implicit broadcast dimension mismatch, feature-scaling data leakage, and non-contiguous indices after NaN deletion.
GPT-4o and Qwen 2.5 Coder 32B tied for second place in DataBugBench, both scoring 4/5 with an accuracy of 80%.
Models can almost always fix explicit syntax errors with ease, but frequently fail on invisible logical errors such as data leakage.
DataBugBench uses only a single sample per question type (denominator of 5), making the sample size small and the conclusions more suitable as trend references than as rigorous rankings.
All Facts (6)
DataBugBench is a benchmark for evaluating large language models' debugging ability on invisible bugs in data science.
50%UnverifiedDataBugBench specifically constructs diagnostic tests targeting Pandas, NumPy, and Scikit-Learn code.
50%UnverifiedDataBugBench focuses on five categories of hidden hazards in data engineering: Pandas SettingWithCopyWarning unintended modification, groupby aggregation losing the grouping key, NumPy implicit broadcast dimension mismatch, feature-scaling data leakage, and non-contiguous indices after NaN deletion.
50%UnverifiedGPT-4o and Qwen 2.5 Coder 32B tied for second place in DataBugBench, both scoring 4/5 with an accuracy of 80%.
50%UnverifiedModels can almost always fix explicit syntax errors with ease, but frequently fail on invisible logical errors such as data leakage.
50%UnverifiedDataBugBench uses only a single sample per question type (denominator of 5), making the sample size small and the conclusions more suitable as trend references than as rigorous rankings.
50%