How Python's json.dumps Default Parameter Causes Claude to Burn 24% More Tokens

A developer migrating a support ticket classifier from TypeScript to Python discovered something strange: same model, same system prompt, same 2,000 tickets — yet input tokens were inexplicably 24% higher. After an afternoon of debugging, the culprit turned out to be a single default parameter in json.dumps: ensure_ascii=True. It was silently rewriting every non-ASCII character in the prompt into hexadecimal escapes — wasting money and causing Claude to misread Japanese customer names.

The Root Cause: What ensure_ascii Actually Does

Python's json.dumps enables ensure_ascii=True by default, escaping all non-ASCII characters as \uXXXX sequences. Characters outside the Basic Multilingual Plane (like most emoji) become two-escape surrogate pairs. The output is still valid JSON, and parsing it back gives you the original data — this is a transport-safety feature from an era when pipes were untrusted and UTF-8 couldn't be safely transmitted.

>>> import json
>>> json.dumps({"city": "München", "name": "東京"})
'{"city": "M\\u00fcnchen", "name": "\\u6771\\u4eac"}'
>>> json.dumps({"city": "München", "name": "東京"}, ensure_ascii=False)
'{"city": "München", "name": "東京"}'

By contrast, JavaScript's JSON.stringify doesn't perform this escaping — which is exactly why the developer fell into this trap when the migrated code looked "line-for-line identical." For a JSON file or HTTP request body, this is completely harmless; the receiver parses it and gets 東京 either way. The problem only arises when JSON stops being a "transport medium" and becomes the "content itself."

Understanding this issue requires knowing how Unicode relates to tokenization. Large language models use Byte Pair Encoding (BPE) or similar algorithms to split text into tokens — frequently occurring character sequences in training data get merged into single tokens. 東京 is likely one or two tokens in Claude's vocabulary. But \u6771\u4eac is twelve ASCII characters made up of backslashes, letters, and digits — each character or character pair gets split independently, causing token counts to explode. More importantly, while models have seen plenty of \uXXXX-style JSON escapes during pretraining (e.g., in code repositories), processing them is an entirely different attention pathway than reading native Japanese text. The model must dynamically "decode" these escapes at inference time to understand their meaning, introducing extra reasoning overhead and error probability.

Why Only the Prompt Breaks, Not the API Call

The critical difference: a prompt is a string, and nothing interprets it as JSON before the model reads it. Here's the problematic line of code:

prompt = f"""Classify this ticket and extract the customer's name and city.

<ticket>
{json.dumps(ticket, indent=2)}
</ticket>"""

When the SDK serializes the entire request body and sends it over HTTP, that outer layer is fine. But the inner 東 is already part of a string value by then, so the SDK escapes the backslash, the API decodes once, and the model receives six literal characters: \, u, 6, 7, 7, 1. The model doesn't get 東 — it gets a hexadecimal puzzle to solve "in its head."

The TypeScript version did the same thing with JSON.stringify(ticket, null, 2), but it doesn't touch non-ASCII characters — that's the underlying reason no one noticed during migration.

There's an easy-to-miss "double serialization" trap here. When sending a request to the Anthropic API, the entire request body is serialized by the SDK into a JSON object for HTTP transport. When the SDK serializes this outer JSON, it escapes backslashes in the prompt string again — \u6771 becomes \\u6771. The API server parses the outer JSON, converts \\u6771 back to \u6771, and passes that six-character string to the model as-is. So the model truly sees a literal backslash followed by five characters, not a Chinese character. This "unwrap one layer but not the inner layer" behavior is perfectly valid per the JSON spec — and it's precisely why the bug survives: any intermediate tooling printing the request body sees "valid JSON" and raises no alarms.

How Many Tokens Are Actually Being Wasted

The developer used Anthropic's count_tokens endpoint to measure both rendering approaches across the same 2,000 tickets. Average input tokens per request rose from 1,180 to 1,461 — a 24% increase. Japanese ticket payloads alone ballooned by roughly 3.1×.

Ticket Language	Share	Payload Tokens (Escaped vs. Raw)
English	38%	1.00×
German	22%	1.06×
Spanish/Portuguese	13%	1.09×
Japanese	17%	3.1×
Other (French, Polish, emoji-heavy)	10%	~2×

The pattern is clear: German tickets only occasionally escape ü or ß, so the loss is a few percentage points. But in Japanese tickets, every single character gets escaped into ~10 ASCII hex characters with terrible tokenization. Emoji suffer the most per character — a single 😀 becomes a twelve-character surrogate pair escape.

At roughly 40,000 tickets per month, the extra 281 tokens per request adds up to approximately 11 million input tokens — money spent entirely on making the model "read hexadecimal."

It Doesn't Just Burn Money — It Misreads Names

The more insidious issue is accuracy. The bug was actually discovered through an accuracy problem: a customer service rep noticed a Chinese character in an extracted customer name that didn't exist anywhere in the original message. The developer grepped the source text for the extracted name: zero hits.

He then scored 340 Japanese tickets, counting any ticket where an extracted name or city string didn't appear verbatim in the source as an error:

• Escaped prompts: 29 out of 340 had at least one character error (8.5%)
• Raw UTF-8 prompts: only 3 out of 340 (0.9%)
• Classification labels: virtually no difference between versions

The developer's hypothesis: common characters like 東京 appear frequently in escaped form in training data, so the model decodes them reliably. But rare kanji in personal names are uncommon, and the model falls back to a character with a similar code point but completely different appearance and meaning. "Close in hex" doesn't mean "visually similar" — one digit off in 髙 gives you an entirely different character, with nothing in the surrounding context to tip the model off.

He's also upfront about limitations: single model, one run per version, 340 samples, exact-match scoring. This is a strong signal on his own data, not a benchmark — but the directional finding is unambiguous.

Where Else Does ensure_ascii Hide in Python LLM Pipelines

Anywhere you serialize data into prompt text is a potential landmine. After fixing the first instance, the developer found three more:

1. pandas.DataFrame.to_json() defaults to force_ascii=True — same bug, different parameter name.
2. Few-shot examples loaded from files and re-dumped: the loader correctly read UTF-8 JSON, but called json.dumps again during formatting, re-escaping every Japanese example.
3. Tool results: an agent inserted tool output into tool_result via json.dumps(result), passing search results with French product names to the model with café encoded as caf\u00e9.

Places that are already safe: Pydantic v2's model_dump_json() outputs raw UTF-8, and orjson.dumps() always outputs UTF-8. If you're using either of those, you're probably fine — but it's still worth a check.

A few other serialization entry points worth watching in the Python ecosystem: subclasses of json.JSONEncoder inherit the default parameters, so if your team has a custom encoder that doesn't override ensure_ascii, you're still affected. The standard library's http.server and urllib call json.dumps internally when constructing responses/request bodies — if tool calls forward data through these modules, escaping can happen silently. flask.jsonify defaulted to JSON_AS_ASCII=True before Flask 2.2, which changed it to False. In frameworks like LangChain or LlamaIndex, built-in document serialization tooling generally handles encoding correctly, but custom Tool func return values formatted with bare json.dumps can still bypass the framework's protections.

How to Fix It — and Prevent Recurrence

The core fix is to pass ensure_ascii=False everywhere JSON ultimately enters a prompt string, then add a test to catch any future helper functions that reintroduce the problem. The developer centralized prompt serialization into one function:

def to_prompt_json(obj) -> str:
    # Compact separators, raw UTF-8. This goes to a model, not a wire.
    return json.dumps(obj, ensure_ascii=False, separators=(",", ":"))

Dropping indent=2 in favor of compact separators saved another ~6% in payload tokens. Then a CI guard using a fixture containing Japanese, German, and emoji to test every prompt template:

import re

ESCAPE = re.compile(r"\\u[0-9a-fA-F]{4}")

def test_prompts_contain_no_unicode_escapes():
    for name, render in PROMPT_TEMPLATES.items():
        prompt = render(MULTILINGUAL_FIXTURE)
        assert not ESCAPE.search(prompt), f"{name} leaks \\u escapes"

This test caught the pandas call on its very first run. There's also a quick audit grep:

grep -rn "json.dumps(" src/ | grep -v "ensure_ascii=False"
grep -rn "to_json(" src/ | grep -v "force_ascii=False"

After the fix, the Python version's average input tokens dropped to 1,148 — 3% lower than the original TypeScript version (thanks to compact separators) — and Japanese extraction errors fell back to 3 out of 340.

The Real Lesson Worth Remembering

What makes this scary isn't that ensure_ascii exists — it's that every layer looks correct: the JSON is valid, requests succeed, logs look normal (because the developer was logging the ticket itself, not the rendered prompt), and the model answers confidently. The only visible symptoms were a slightly larger bill and one mismatched name that a human happened to catch.

The developer's advice is simple but powerful: for at least a sample of requests, log the exact string the model actually receives and read it with your own eyes. Most prompt bugs reveal themselves in five seconds once you look at the real content.