Real-SWE: Benchmarking AI Coding Models on Real Enterprise Codebases

Why We Need a New AI Coding Benchmark

Most mainstream benchmarks for evaluating AI coding capabilities — such as SWE-bench — are built on open-source projects. This creates a fundamental problem: those open-source repositories have very likely already been included in large models' training data. When a model scores well on these benchmarks, it's genuinely hard to tell whether that reflects real problem-solving ability or simply memorized reproductions of training data.

Real-SWE aims to address exactly this pain point. It advocates evaluating AI models on private, real-world, enterprise-grade codebases. These codebases share one critical characteristic — they have never been made public, making it nearly impossible for them to appear in a model's pretraining corpus. This allows for a much purer measurement of a model's reasoning and repair capabilities when faced with unfamiliar code.

How Enterprise Codebases Change the Evaluation Equation

Open-source projects and internal enterprise code differ significantly in structure. Enterprise codebases tend to be larger, more deeply interconnected, and riddled with internal conventions, legacy logic, and undocumented tribal knowledge. For AI models, handling this kind of code can't rely on pattern-matching against previously seen examples — it requires genuinely understanding context.

This difference makes evaluation results far more reflective of real engineering scenarios. A model that scores highly on open-source benchmarks may still struggle to locate and fix a bug inside a company's sprawling internal microservices system. Real-SWE's value lies in pulling the evaluation environment from the "ideal laboratory" back to the "real factory floor."

The Core Problem of Data Contamination

Data contamination has been a recurring concern in the evaluation community. When there is overlap between training and test sets, model performance is systematically overestimated. Using private codebases as the test foundation cuts off this overlap at the source — and this is arguably the most compelling methodological contribution of Real-SWE.

Real-World Challenges and Community Discussion

The traction Real-SWE gained on Hacker News (31 points, 19 comments) signals genuine community interest, though reasonable skepticism exists as well.

Using private enterprise codebases creates an inherent tension around reproducibility: because the code isn't public, outside researchers cannot independently verify the evaluation process or its results. This conflicts with the openness and reproducibility principles that traditional academic benchmarks uphold. Striking a balance between protecting enterprise code privacy and maintaining benchmark credibility is a fundamental question this type of benchmark must answer.

Further questions remain: Are the selected enterprise codebases representative? How is task difficulty calibrated? Are scoring criteria sufficiently objective? Private data sidesteps contamination but introduces a new kind of opacity risk.

Implications for AI Coding Tools in the Enterprise

As AI coding assistants enter enterprise development workflows at scale, buyers increasingly need evaluation references that reflect real production environments — not just vendor rankings on public leaderboards. Evaluation approaches like Real-SWE, centered on authentic enterprise scenarios, speak directly to this need.

For development teams, it offers a dimension of judgment closer to actual usage: when a model encounters a completely unfamiliar, structurally complex internal codebase, can it actually help? That's more meaningful than chasing scores on public benchmarks that have been repeatedly optimized against.

In the longer run, the evolution of benchmarks is shifting from "can the model solve known problems" toward "can the model handle unknown scenarios." Real-SWE represents a meaningful attempt in that direction — even if it still needs to deliver more complete answers on transparency and reproducibility.