Real-SWE
An AI programming capability benchmark based on private enterprise codebases, designed to mitigate data contamination by using internal code repositories that have never been publicly released. It measures a model's reasoning and debugging abilities in unfamiliar and complex code environments.
Timeline (last 90 days)
Real-SWE advocates evaluating AI models on private, real-world, enterprise-level codebases.
Private enterprise codebases have never been made public, and therefore can almost certainly not appear in a model's pre-training corpus.
Using private enterprise codebases inherently faces a reproducibility dilemma, as external researchers cannot independently verify the evaluation process and results.
Although private data avoids contamination, it also introduces new risks of opacity.
All Facts (4)
Real-SWE advocates evaluating AI models on private, real-world, enterprise-level codebases.
50%UnverifiedPrivate enterprise codebases have never been made public, and therefore can almost certainly not appear in a model's pre-training corpus.
50%UnverifiedUsing private enterprise codebases inherently faces a reproducibility dilemma, as external researchers cannot independently verify the evaluation process and results.
50%UnverifiedAlthough private data avoids contamination, it also introduces new risks of opacity.
50%