Claude Ports Rust Code: 9,628 of 10,000 Results Don't Match — Where's the Problem?

Title

Claude Ports Rust Code: 9,628 of 10,000 Results Don't Match — Where's the Problem?

Meta Description

Bilibili creator tests Claude Code porting a Python function to Rust: zero compile warnings, but 9,628 of 10,000 test cases mismatch. Three traps the compiler can't see, and why you shouldn't add features during a port.

Article Content

The Anthropic team revealed in an interview that their API is being migrated from Python to Rust, with most of the code written by Claude — and nobody on the team even knows Rust. This raises a key question: how do you prove that AI-migrated code isn't wrong?

A Bilibili creator, "1024 Engineering Notes," ran a small-scale reproduction experiment with Claude Code: porting a Python function that assembles streaming model output into a web response, line by line, to Rust. It compiled on the first try with zero warnings — but the first round of testing showed that out of 10,000 cases, over 9,600 didn't match. The problem wasn't Rust — it was the "not written anywhere" default behavior in the old Python code.

The value of this experiment is that it pulls the abstract debate about "is AI-written code reliable?" back into verifiable engineering details.

Experiment Design: Narrowing the Comparison to a Single Function

Fully replicating Anthropic's "shadow traffic" verification isn't realistic — that would require copying production requests to the new system and comparing results. The creator had no production traffic, but narrowed the comparison to one function: it takes the characters the model spits out piece by piece and assembles them into the streaming responses that pop onto the web page.

Inputs were randomly generated, deliberately stuffed with Chinese characters, emoji, backslashes, and control characters copied from terminals. The two outputs were compared byte by byte.

The key to this design is "byte comparison" rather than "compare after parsing." The creator later concluded: if you parse to JSON first and then compare, the first two traps would slip right through. Equal after parsing does not mean equal bytes.

Three Traps the Compiler Can't See

Trap 1: The Default JSON Escaping Switch

In the first round's 9,600+ mismatches, not a single line of the Rust code was wrong. The problem was a default switch in Python's JSON generation: it converts Chinese characters, emoji, and the like into backslash-U-plus-number notation. Claude Code ported it faithfully, but this default value isn't written anywhere in the Python code — the AI couldn't find it.

The result: both sides parse to the same string, but the bytes differ. Anyone who signs the response body or uses it as a cache key will hit a bug here. After fixing this trap and rerunning, errors dropped by only forty-something — a single test case often steps on several traps at once, and as long as one remains unfixed, it still counts as wrong.

The JSON standard (RFC 8259) only requires escaping quotes, backslashes, and control characters U+0000 through U+001F; all other characters (including Chinese and emoji) can be output directly as raw UTF-8. But Python's json.dumps defaults to ensure_ascii=True, converting all non-ASCII characters to \uXXXX form — a legacy default set for compatibility with ASCII-only transport channels, not a JSON spec requirement. Rust's serde_json does the opposite, outputting raw UTF-8 by default. Both parse to the same string, but the byte sequences differ. This also explains why many teams explicitly specify ensure_ascii=False at the serialization layer, and why, when porting across languages, serialization configuration must be verified separately as a contract — you can't expect to read it out of the code logic.

Trap 2: Short Forms vs. Long Encodings

The second round's blame fell on the backspace and form feed characters. Python writes them as two-character short forms; Rust writes them as long encoding sequences. The third round still had 9,400+ mismatches — this time the error was in the final usage number.

For the same message, Python calculates 103, Rust calculates 142 — nearly 40% more. The cause: both languages have a length property with identical spelling. The compiler won't complain, but Python counts characters while Rust counts bytes. It's like taking attendance: one side counts how many people showed up, the other counts how many seats they occupy — one Chinese character takes three seats per person, so the more Chinese text there is, the more likely Rust's usage count is inflated. In a real API, usage is used for billing.

JSON permits two equivalent escape forms: short forms like \b (backspace), \f (form feed), \n, \r, \t, and generic long forms like \u0008, \u000c, etc. Both parse to identical results, but output different byte counts. JSON libraries across languages have their own default strategies for which form to use — this is latitude the spec allows, so compilers naturally won't complain. In any system that uses response body bytes directly for hashing, signing, ETags, or cache keys, this kind of difference causes the old and new systems to produce different keys for the same logical content — leading to cache misses or signature verification failures — and these problems are completely invisible when comparing at the parse layer.

Python's len() applied to str returns the number of Unicode code points, because Python 3 strings are sequences of code points; Rust's str::len() returns the number of UTF-8 encoded bytes. The same-named "length" has completely different semantics. For billing, rate limiting, truncation, and similar scenarios, you need to decide explicitly which convention to use: OpenAI and other APIs measure tokens based on tokenizers rather than characters or bytes, but if a system internally uses length for quotas or log statistics, inconsistent conventions will cause observable discrepancies. When porting, wrap an explicit char_count or byte_len function to avoid calling the language's built-in length property directly.

Trap 3: The Test Tool Buries Its Own Mine

After fixing the first three traps, the fourth round still had 39 mismatches. This time the fault was in the creator's own comparison tool: an empty string got mixed into the input format and became an empty field, so Rust thought there was no fragment at all. The same applies in production — how requests are copied and paired is all newly written code, which can wrongly blame the new system and also let real errors through on its behalf.

After fixing the input format, the fifth round had zero mismatches. The creator then tried five more random seeds, each running 20,000 cases — 110,000 total — with not a single byte off.

Don't Add New Features During the Port

The experiment also ran an extended verification: simulating the old system continuing to evolve, adding a field on the Python side while leaving Rust untouched — all 10,000 cases went red.

The interview mentioned that during the migration, Anthropic only moved code and added no features. The creator's understanding of why: once new features are mixed in, the porting errors you should catch get buried in a sea of red, and you can't tell which failures were introduced by the port and which by the new feature. If your old system is still adding features, the comparison needs to be hooked into every commit so changes show up immediately.

Shadow traffic verification is the standard practice for this kind of migration: copy production requests to the new system, record its responses without returning them to users, then compare against the old system's responses field by field or byte by byte. Its effectiveness depends on two premises — the old and new systems process the same input in the same time window, and the comparison baseline remains stable during this period. If the old system's behavior also changes during that time, shadow comparison failures can't be attributed, and the experiment design itself fails. This is why migration windows typically require the old system to enter feature freeze, and why comparison should be a CI gate rather than one-time acceptance.

Performance: One Function 7x Faster Doesn't Mean the Whole API

The creator also measured speed: running the same batch of cases twenty times, the single function was about 7x faster in Rust. But the interview said the whole API uses 40-50% less CPU.

This function only does string concatenation — the work Rust benefits from most. The whole service clearly isn't all this kind of work. The creator's advice is pragmatic: before switching languages, measure what proportion of your request time is CPU-bound, then act.

Defining "What Counts as the Same" Is the Real Skill

Claude Code finished writing this Rust code quickly; the creator spent all his time defining "the same." A few lessons worth recording:

• Compare byte by byte; parsing to JSON first would let the first two traps slip through;
• Equal after parsing doesn't count; equal bytes is what counts;
• The input pool must include Chinese, emoji, control characters, and empty strings — if you only use English words, you won't catch any of the first three traps;
• For each discrepancy, first determine whether the port is wrong or the test itself is wrong.

This brings us back to the original question from the interview: how do you prove AI-written code isn't wrong? The answer isn't "it compiled" or "it passed the tests" — it's whether you can define "the same" down to the byte level, and acknowledge that the comparison tool itself can be wrong.

One-line Summary

AI-ported code compiles with zero warnings yet 96% mismatches — the truth lies in byte-level default behaviors.

Paragraph Summary

Anthropic migrating its API from Python to Rust with most code written by Claude raises the question: how do you prove AI-written code isn't wrong? A Bilibili creator reproduced a small experiment with Claude Code: porting a streaming output assembly function from Python to Rust. It compiled with zero warnings, but in the first round of 10,000 cases, over 9,600 had byte mismatches. The problem wasn't Rust logic but Python's unwritten default behaviors — the JSON escaping switch, character vs. byte length conventions, and flaws in the comparison tool itself. After fixes, 110,000 cases showed zero differences. The experiment also shows that adding features to the old system during migration masks errors, and that a single function being 7x faster can't be extrapolated to the whole API — the real skill is defining "the same" down to the byte level.