Trusting AI-Written Code: A Five-Day Extreme Test of a Rust Package Manager

A software engineer with 27 years of experience had Claude write a Rust package manager in an afternoon, then spent five days convincing himself he could trust it. The story of jpm isn't about what AI can write — it's about how to verify what AI writes.

An Afternoon of Vibe Coding

jpm is a JavaScript package manager that works in npm, pnpm, yarn, and bun projects. Its author didn't write or read a single line of code, and even had the AI write its own TLS, crypto, and HTTP modules to shrink the binary. The initial version was done in a single afternoon — in the author's own words, "a true vibe coded project."

The author's background is worth noting: he primarily works in higher-level languages like C#, Elixir, and JavaScript, and aside from helping his father debug a disk controller in C as a kid, he has almost no low-level language experience. So building a 60,000-plus-line project in Rust, in his view, would have been "impossible for me to do on my own in that timeframe." He's been using Node.js professionally since Node.js 0.8 (2012), and knows npm, pnpm, and bun inside out.

The project was triggered by an episode of the Syntax podcast. Hosts Wes, Scott, and CJ talked about upm, a fast, tiny package manager written in JavaScript by Pooya. The author's thought was: since so many tools are being rewritten in Rust, maybe he could build something smaller and faster.

Why jpm

The core question discussed on the podcast is exactly what every engineer is asking right now: can AI build real software, or only demos? Most listeners believed AI could generate anything with a single prompt. The author's assessment: "Spoiler — it can. But without human guidance and software engineering understanding, the result is bad."

That judgment runs through the entire project. Claude produced a working version in about 45 minutes, but it relied on a huge number of third-party Rust crates, had a bloated binary, and had only been tested against a very small package.json file. The author then had it analyze where the size came from, and by the end of the night, there were custom HTTP, JSON, crypto, and TLS modules — and a noticeably smaller binary.

Five Days of Hardening: Verification Is the Real Work

The part vibe coding projects often skip — testing against the real world. The author had Claude write a script to install and build 166 large public projects (starting with 100), then ran the same operations with each project's original package manager for comparison, including running their test suites where possible. The same flow was also executed on Windows Command Prompt, PowerShell, and macOS.

Five days later, the result was 261 merged PRs and a hardened package manager.

The author's steering shows up in a series of concrete decisions: when Claude said a faster lockfile format would only save a few milliseconds, the author still insisted on it; when Claude wanted global storage as an optional feature, the author pushed back — "if 99% of packages can use it, why make it optional?" — so it became the default; the author insisted on not auto-running dependency install scripts unless explicitly listed — his reasoning being that now that AI does most of the work, you can have AI review a package's scripts for problems before approving them.

Two Stands and One Concession

Claude explicitly pushed back on the author's decisions twice, and both times the author held his ground.

The first was TLS. To shrink the binary, the author wanted to implement TLS and crypto himself. Claude said, "this is code you should never write yourself." The author countered that this was outdated engineering thinking: in Rust, Heartbleed-style memory vulnerabilities don't exist, and it can be tested as heavily as any other code. Claude agreed it had overestimated the risk and listed the test suites that could prove it: Wycheproof, NIST vectors, x509-limbo, and BoGo.

But the author also admitted his doubts: "Claude is a sycophant — saying it overestimated the risk might just be it playing along with me." Still, it passed tests including ones converted from real bugs in other projects. The author says that regardless of who wrote it, he now trusts it more.

The second was HTTP/2. Claude tested with curl first and said it wasn't worth doing, because jpm's entire install flow over HTTP/1.1 was already faster than curl's HTTP/2 download. The author insisted anyway: curl has to handle every case, we only need to send lots of GET requests to one registry, and to beat bun we have to break from convention. Two hours later, Claude had a working client.

This time the author was wrong. Benchmarks showed: HTTP/2 was about 17% faster on a home connection, but equal or worse on GitHub runners, up to 2x slower on cold-start nuxt installs, used 8% to 15% more CPU on large tarballs, and added 60KB to the binary. So all 3,671 lines of code were deleted.

The reverse also happened. jpm can install Node for users, and the author wanted to add a toggle for Node signature verification, thinking it would slow down installs. Claude measured only 15 to 19 milliseconds and pointed out that "a toggle means nobody will turn it on." In the end, signature verification is on by default, and the author is happy with that.

Wycheproof is a set of cryptographic test vectors open-sourced by Google, specifically designed to expose edge-case errors in crypto library implementations — invalid curve points, malformed signatures, or extremely long inputs. It covers common algorithms like AES, ECDH, RSA, and Ed25519, and every test case is annotated with its expected result. It matters because cryptographic code is most prone to errors on "impossible-looking" inputs, and those errors often slip past ordinary unit tests. Combined with NIST's official test vectors, x509-limbo (an X.509 certificate parsing test suite), and BoGo (BoringSSL's TLS interoperability test suite), they can systematically verify whether a self-implemented TLS/crypto stack holds up at both the protocol and implementation layers. The author used these four categories of tests to address Claude's concern that "writing your own TLS is too risky."

Why Trust Code Nobody Typed

The short answer: Wycheproof, node-semver, pnpm, yarn and bun's own tests, and 166 real repositories. Bugs found along the way included: node-semver's own tests found 25 cases where jpm's judgment disagreed — for example, treating ^9007199254740991.0.0 as a valid range; npm-package-arg's tests initially passed only 117 of 154; a security scan found that the .npmrc of a cloned repository could send the registry token through its own proxy, and a proof of concept captured Bearer USER-SECRET-TOKEN; on real repositories, dependency-link differences between jpm and pnpm dropped from 1,320 to 183, and differences with npm dropped from 614 to 14; on macOS, it simply wouldn't compile before testing.

The author doesn't think this proves the code is bug-free: "After this many years in the industry, I know that no matter how many tests you have, nothing is bug-free." His plan is to keep using jpm across all his own projects, accept fix PRs including those submitted by AI, but require that they prove what they're fixing with tests.

node-semver is the library in the npm ecosystem responsible for parsing and comparing semantic version ranges — the rules for ^, ~, >=, and prerelease version handling are all defined by it. Since virtually every package manager has to determine "does a given dependency version satisfy a range," node-semver's test suite has become the de facto compatibility standard. After plugging into that test suite, jpm found 25 cases of disagreement, such as treating ^9007199254740991.0.0 as a valid range — exposing differences in numeric limits and overflow handling. npm-package-arg, meanwhile, handles parsing the various dependency specifiers in package.json (git URLs, tarballs, aliases, etc.); its test pass rate starting at 117/154 shows that parsing package names and dependency specifiers is far more complex than it appears. Reusing these existing test suites effectively turns years of community-accumulated edge cases into acceptance criteria.

Current State and Weaknesses

jpm's binary is compressed to 2MB. It's not always the fastest, but it's in the top tier, and security is one of its selling points: no scripts run without approval, packages aren't upgraded to the latest version until they've been published for at least a day, and there's security hardening around .npmrc. On a GitHub Actions runner, the median cold-start install of nuxt (591 packages): jpm 1.22s, bun 1.28s, pnpm 1.72s, npm 18.54s.

The author is candid about the benchmark's limitations: only three repositories were tested, nuxt is slowed by Defender scanning on Windows, and on macOS bun and aube are faster on warm installs.

Cold-start install and warm install are two easily confused scenarios in package manager benchmarks. Cold start means clearing the cache and downloading and extracting all dependencies from scratch, which is closer to the real experience on a CI runner or a new machine; warm install relies on a local global store or cache, where many packages already exist, so it measures linking and verification overhead. The author chose to measure cold-start nuxt (591 packages) on a GitHub Actions runner precisely because CI environments are closest to "no cache" and are reproducible. On Windows, Defender's real-time scanning significantly slows file writes, and on macOS bun and aube are faster on warm installs — both showing that a single number can't represent all workloads. When putting jpm's 1.22s next to bun's 1.28s and pnpm's 1.72s in the same table, you have to state the test platform, cache state, and repository size at the same time — otherwise the conclusion will be misread.

The Real Lesson for AI Programming

The jpm story provides a clear division of labor. AI handles generation and directional exploration, while human value concentrates on judging which tests can prove what, when to stand firm, and when to concede. The author didn't write code, but he made a great many small decisions and used test data to overturn his own judgment.

This may be more worth discussing than "can AI write code": when the cost of generation approaches zero, the cost of verification becomes the key variable determining engineering quality.

One-line Summary

A 27-year veteran had Claude generate jpm, then spent five days testing it against 166 projects to prove it trustworthy.

Paragraph Summary

A software engineer with 27 years of experience used Claude to generate the Rust package manager jpm in an afternoon, then spent five days running install, build, and test comparisons against 166 real projects, merging 261 PRs to complete hardening. The article emphasizes that AI can generate code, but human value lies in judging which tests can prove what, when to hold firm, and when to concede. Claude opposed the author's decisions twice: self-implemented TLS was justified by tests like Wycheproof, while HTTP/2 was proven wrong by the data and all 3,671 lines of code were deleted. jpm's binary is compressed to 2MB, leading bun and pnpm with a median CI cold-install of nuxt at 1.22s, and it fixed security issues such as .npmrc leaking tokens. The author argues that AI-generated fixes must be proven with tests, and accepts AI-submitted PRs only when they include tests.