SWE-Smith Expands to JavaScript: 6,099 Synthetic Bugs Powering Multilingual Code Repair

Introduction: From Python to Multilingual Synthetic Bug Generation

SWE-bench, a critical benchmark for evaluating large language models' code repair capabilities, has long been limited to Python. SWE-bench was released by Princeton University's NLP group in 2023 as a code repair evaluation benchmark. It extracts tasks from real GitHub Issues and their corresponding Pull Requests, requiring models to generate correct code patches given a problem description. What makes it unique is that it evaluates repository-level code comprehension and repair ability, rather than simple function-level code generation—models must locate the problematic file, understand contextual dependencies, and generate patches that pass tests. Since its release, SWE-bench has become the gold standard for measuring AI programming assistants' practical capabilities, with organizations like Anthropic and OpenAI using it as a core evaluation metric.

Now, the SWE-Smith Multilingual team has taken a critical step—extending SWE-Smith's synthetic bug generation capabilities to JavaScript, with plans to cover TypeScript, Java, Rust, Golang, and C++ in the coming months.

The core results of this expansion are impressive: the team generated and validated 6,099 synthetic bug patches across 74 popular JavaScript repositories, achieving a 17% validation pass rate. This data will provide valuable multilingual data support for training and evaluating code repair models.

How SWE-Smith Works

SWE-Smith is a tool for synthesizing realistic bugs in software repositories. Its core approach applies procedural modifications to working code—such as flipping operators, swapping function arguments, removing conditional checks—to create bugs that cause test failures.

This method originates from the classic software engineering theory of mutation testing. Mutation testing was first proposed by Richard Lipton in 1971, with the core hypothesis that if a test suite can detect small artificial mutations (mutants) in code, it can also detect real bugs. Common mutation operators include Arithmetic Operator Replacement (AOR), Relational Operator Replacement (ROR), and Conditional Operator Replacement (COR). SWE-Smith transforms this theory from test quality assessment into training data generation—mutants that are detected by tests constitute high-quality bug-fix training pairs.

The advantage of this approach is that generated bugs are realistic (based on actual codebases), and each bug has corresponding test cases to verify its existence and repair. This provides high-quality "fail→pass" test pairs for training code repair models and lays the foundation for building evaluation benchmarks.

Repository-Level Output Analysis

Common Characteristics of High-Yield Repositories

Among the 74 JavaScript repositories, output differences were dramatic. The top-performing repository was josdejong/mathjs, generating 845 validated patches with a pass rate of 62.8%. This is no coincidence—mathjs contains extensive arithmetic operations and comprehensive unit tests, making it ideal ground for procedural bug generation.

The top five repositories include:

Repository	Generated	Validated	Pass Rate
josdejong/mathjs	1,346	845	62.8%
novnc/noVNC	1,384	715	51.7%
Automattic/mongoose	1,440	653	45.3%
bootstrap-vue/bootstrap-vue	1,079	492	45.7%
foliojs/pdfkit	797	408	51.2%

High-yield repositories consistently share two characteristics: rich mutable code patterns (such as mathematical operations and conditional logic) and robust test coverage. Test coverage plays a dual role here: it's both a prerequisite for mutants to be detected (mutations at positions not covered by tests cannot be discovered) and a guarantee for verifying repair correctness.

Half the Repositories Yielded Zero Results

Interestingly, half of the 74 repositories contributed zero validated patches. The specific reasons fall into two categories:

• 20 repositories failed at the pre-validation stage, where test results couldn't be correctly parsed
• 20 repositories generated too few patches (fewer than 50), making validation costs outweigh benefits

This reveals an important issue: not all repositories are suitable for procedural bug generation. Well-known projects like jQuery, Express, and Three.js actually yielded zero output, likely due to their code structure, test framework compatibility, or build system complexity. The fragmentation of test frameworks in the JavaScript ecosystem (Jest, Mocha, Vitest, Karma, Jasmine, etc.) and the diversity of build toolchains (Webpack, Rollup, esbuild, Vite, etc.) make unified test execution and result parsing extremely challenging. This also provides important reference for future repository selection strategies.

Effectiveness Comparison of 14 Modifiers

The team implemented 14 procedural modifiers for JavaScript, with significantly varying effectiveness.

Top Five Most Effective Modifiers

Modifier	Description	Validated Patches	Pass Rate
func_pm_op_flip	Flip binary operators	1,117	30.8%
func_pm_arg_swap	Swap function arguments	722	32.7%
func_pm_remove_assign	Remove assignment statements	632	25.4%
func_pm_remove_cond	Remove conditional checks	604	21.9%
func_pm_ctrl_invert_if	Swap if/else branches	490	35.7%

Operator flipping (op_flip) leads by far with 1,117 validated patches—nearly double the second place. This makes sense: flipping === to !==, < to >=, && to ||—these modifications have extremely broad applicability and can find targets in virtually any repository. Notably, JavaScript's distinction between strict equality operators (===/!==) and loose equality operators (==/!=), along with type coercion, means operator flipping can produce more diverse bug patterns in JS than in strongly-typed languages.

Notable Long-Tail Modifiers

Some seemingly niche modifiers demonstrated surprisingly high pass rates:

• func_pm_ctrl_shuffle (shuffle statements within loops): Although it only generated 388 patches, its pass rate of 36.7% ranked first
• func_pm_ctrl_invert_if (swap if/else branches): A 35.7% pass rate indicates that conditional logic inversion is a highly effective bug injection method
• The svg/svgo repository achieved an overall pass rate of 73.7%, suggesting that SVG processing code is particularly well-suited for procedural mutation

The team also noted that the HAS_LOOP filter condition for the func_pm_ctrl_shuffle modifier may be overly strict, limiting its applicability, with room for optimization in future iterations. From a mutation testing theory perspective, Statement Order Mutation is a higher-order mutation operator that can expose implicit order dependencies in code—bugs that are also common in real development, such as using variables before initialization or accessing resources after release.

Engineering Infrastructure: Modal Cloud Pipeline

Previously, large-scale bug generation required days of local computation. The team has now migrated the entire pipeline to the Modal cloud platform, reducing processing time from days to hours.

Modal is a serverless cloud computing platform designed specifically for machine learning and data-intensive workloads. Unlike traditional cloud services, Modal allows developers to define computational tasks in pure Python, with the platform automatically handling containerization, GPU scheduling, parallel scaling, and result collection. Its core advantages include millisecond-level cold starts, billing based on actual compute time (precise to the second), and native support for custom Docker images. For scenarios like SWE-Smith that require running numerous repository test suites in isolated environments, Modal's container isolation and elastic scaling capabilities are particularly critical—each patch validation runs in an independent container, avoiding environment pollution and dependency conflicts.

The pipeline consists of three stages:

1. Patch Generation: Apply procedural modifications to repository code, identify mutable positions through AST (Abstract Syntax Tree) parsing, then apply corresponding modifiers to generate candidate patches
2. Patch Validation: Run each repository's test suite in isolated containers, confirming that tests fail after mutation (bug exists) and pass with original code (repair is valid)
3. Result Storage: Upload validated datasets to HuggingFace, including complete information such as bug patches, fix patches, and related test cases

This end-to-end cloud pipeline design enables rapid infrastructure reuse when expanding to additional programming languages.

SWE-gen: Extracting Tasks from Real PRs

Beyond synthetic patches, the team is also using the SWE-gen tool to automatically extract verifiable tasks from merged GitHub PRs. SWE-gen can detect programming languages, build systems, and test frameworks, reconstruct bug states by reversing PRs, and validate through "fail→pass" tests.

SWE-gen's PR reversal technique is an ingenious data engineering method. The principle is: a merged PR represents the transition from a bug state to a fixed state. By reversing this diff (i.e., rolling back the fixed code to its pre-fix state), the bug state can be reconstructed. Then, test cases added or modified in the PR are used to verify that the bug is indeed reintroduced (tests fail) and that the original code indeed fixes it (tests pass). The advantage of this approach is its ability to capture complex real-world bug patterns, including cross-file modifications, API changes, and logical errors requiring deep contextual understanding—scenarios that simple procedural mutations struggle to simulate.

Currently, SWE-gen-JS has generated 1,000 tasks from 30 popular JS/TS open-source repositories. These tasks derived from real bug fixes can capture multi-file changes and complex test setups, complementing scenarios that procedural modifiers struggle to simulate. Combining SWE-Smith's synthetic data with SWE-gen's real-world data achieves a balance between training data scale (the advantage of synthetic data) and diversity (the advantage of real data).

Future Outlook

The team has outlined several key directions in their roadmap:

• Language Expansion: After JavaScript, TypeScript, Java, Rust, Golang, and C++ will be added successively. Each language presents unique challenges—TypeScript's type system provides additional dimensions for mutation, Rust's ownership model means certain mutations are directly rejected by the compiler, and Java's object-oriented features require class-level mutation operators
• Repository Selection Optimization: Learning from the 20 zero-output repositories to improve repository selection heuristics
• Pre-validation Checks: Ensuring at least some pre-validation tests pass before adding repository configurations
• LM-Driven Bug Generation: Running language model-based bug generation methods for all added repositories, leveraging LLMs' understanding of code semantics to generate more complex bugs that more closely resemble real developer errors
• Training Dynamics Research: Deeply understanding how programming language and repository selection affect model training and downstream coding task performance, exploring possibilities for cross-language transfer learning

This project advances through open collaboration, and interested developers can join the team via Slack. The team has explicitly welcomed community contributions, particularly in novel bug generation strategies.

Conclusion

SWE-Smith's expansion to JavaScript marks an important milestone in multilingual code repair benchmark construction. The 6,099 validated synthetic bug patches, 14 proven modifiers, and cloud-based generation pipeline establish a solid foundation for rapid expansion to additional languages. As multilingual data accumulates, we have good reason to expect next-generation code repair models to demonstrate stronger generalization capabilities in cross-language scenarios. From a broader perspective, this work is addressing a core bottleneck in AI-assisted programming—the scarcity of high-quality, multilingual training data—with implications that extend far beyond any single benchmark.