Building Free Urdu AI Voice with Gemini TTS: Three Bugs We Learned the Hard Way

Building a text-to-speech (TTS) system for non-English languages is often far more complex than it sounds. A Pakistani team adding voice narration to Sitey, an Urdu writing app, ran into a series of classic but easy-to-overlook engineering problems. They turned every AI response into readable Urdu audio, made it shareable as WhatsApp voice messages, and offered the feature free on the web — but getting from "it speaks" to "it actually works" took considerable effort. Here's what they learned, with lessons applicable to any developer building voice features for multilingual audiences.

Choosing a Voice: Listen With Your Ears, Not Benchmarks

The team initially tried another Urdu TTS service, but the founder wasn't satisfied after hearing it. So they ran a small test: feeding the same four sentences to two Gemini TTS models and the device's built-in Android Urdu speech engine.

The four sentences were deliberately designed:

• A standard Urdu script sentence
• The same sentence in Roman Urdu (Urdu written with Latin characters — the dominant way locals text)
• A sentence in English
• A mixed Urdu-English sentence, because that's how people actually talk

They collected the .wav files generated by each Gemini voice into a single folder and had real humans listen through them one by one. The entire test cost less than ten cents. The winner was Aoede on gemini-3.8-flash-lite-tts — it handled all four sentence types and correctly pronounced the brand name "Sitey."

The mixed-language sentence was the real litmus test. Many voices performed fine on pure Urdu but completely fell apart the moment an English brand name appeared mid-sentence. The lesson: don't evaluate TTS against official benchmarks alone — test with the way real users actually speak, and have real humans do the listening.

Bug 1: The Voice Read the Instructions Out Loud

During testing, the team discovered an embarrassing issue: style instructions embedded in the text (something like "read this in a warm tone") were being read aloud by certain voices as if they were content.

The fix was straightforward — send only the text you want spoken to the model, and pass style configuration through API parameters rather than in the body:

body: JSON.stringify({
  contents: [{ parts: [{ text }] }],
  generationConfig: {
    responseModalities: ['AUDIO'],
    speechConfig: { voiceConfig: { prebuiltVoiceConfig: { voiceName: 'Aoede' } } },
  },
}),

The same problem appeared in the app in a different form. A greeting card titled "Short and sweet" would have its title read aloud; emoji got rendered as bizarre syllables (a birthday cake 🎂 became "chha"). The reader now accepts only the message body, strips emoji first, and has unit tests covering this behavior.

This bug reveals a common pattern: the model faithfully processes every byte of text you give it. Any labels, headings, or emoji mixed in can and will be vocalized.

Bug 2: Playback Produced Nothing but Noise

Older example code returned raw 16-bit PCM data, requiring developers to manually prepend a 44-byte WAV header. The team did exactly that — and got a burst of harsh noise when playing back on a phone.

The root cause: Gemini now returns a complete WAV file directly, with a C2PA content provenance block appended after the audio data. The team had wrapped this already-complete file in a second WAV header, causing the phone to interpret the inner header and the provenance block as audio waveform data.

The fix was to check the first four bytes and pass the data through unchanged if it's already a valid WAV:

const bytes = fromBase64(part.inlineData.data)
const isWav = bytes.length > 12 && String.fromCharCode(...bytes.subarray(0, 4)) === 'RIFF'
if (isWav) return { ok: true, audio: part.inlineData.data, usage }
// Otherwise it's raw PCM: read sample rate from mime type and add our own header.

This is a particularly insidious trap: API return formats evolve across versions, and "add a header" logic written for old examples becomes the source of bugs under newer behavior. When handling binary audio, inspect the byte header before deciding how to process it — don't blindly apply a template.

C2PA (Coalition for Content Provenance and Authenticity) is an open technical standard for embedding verifiable provenance and history into digital content (images, audio, video), helping identify whether content was AI-generated. Gemini TTS appends a C2PA metadata block to the end of returned WAV files to mark the audio as AI-generated. This tail data is not audio samples, but it does exist in the byte stream. When a developer mistakes the return for raw PCM and prepends another WAV header, the player interprets the original WAV header and the C2PA block as audio waveform data — producing harsh noise. This also means: even if a future API version stops appending C2PA, defensive byte-header detection remains the right approach — it's far more robust than assuming the return format never changes.

Bug 3: WhatsApp Truncated the End of Voice Messages

After users sent the generated WAV as WhatsApp voice messages, the end of the message was cut off.

The team didn't bother pinpointing whether WhatsApp itself or the trailing C2PA block was the culprit — they didn't need to. The app now re-encodes the audio on-device to AAC format as an .m4a using Android's built-in encoder, fully draining the encoder to the end of stream before sharing. Device tests verify that both 3-second and 11.7-second voice messages maintain their full duration. On a tablet, a message with a true duration of 4.60 seconds outputs at 4.61 seconds.

The team's advice is direct: if you're sharing generated audio to a chat app, test the duration of the file as received on the other end — not just whether it plays back.

AAC (Advanced Audio Coding) is currently the most widely used lossy audio compression format on mobile, and the default encoding for voice messages in apps like WhatsApp and WeChat. Unlike WAV's uncompressed PCM data, AAC uses a frame-based structure where each frame is independently encoded and requires complete frames to decode correctly. "Fully draining the encoder to the end of stream" is a critical step: encoders typically maintain internal buffers, and if the stream is closed before all frames are written out, the last few frames are discarded, truncating the audio. Android's MediaCodec API provides an explicit BUFFER_FLAG_END_OF_STREAM signal to trigger this drain process — skipping it is one of the most common pitfalls in mobile audio encoding.

Staying Free Without Letting Costs Spiral

The voice feature runs on a Cloudflare Worker that holds the Gemini API key — neither the app nor the web client ever touches it directly. A few constraints keep costs manageable:

• The web client gets 5 voice generations per day, capped at 500 characters each, with Cloudflare Turnstile bot protection; the app gets 40 per day.
• A daily dollar cap is set across all AI calls, with a separate monthly cap for Gemini.
• If Gemini is busy, rate-limited, or offline, the app falls back to the phone's built-in Urdu voice engine. The quality is lower, but it still speaks.
• Only call counts and costs are stored — never text or audio.

At the Lite TTS pricing at time of writing, input costs $0.50 per million tokens and audio costs $6.00 per million tokens. A short Urdu narration costs a fraction of a cent. This pattern of key isolation + layered rate limits + graceful degradation is a solid reference architecture for any small team that wants to offer AI features for free without worrying about runaway bills.

Cloudflare Turnstile is Cloudflare's non-intrusive bot verification solution, designed as a replacement for traditional image CAPTCHAs. It analyzes browser behavior, device fingerprints, and other signals in the background to determine whether a visitor is human — most of the time requiring no user interaction at all. For AI features exposed to anonymous users, Turnstile is a low-friction way to block automated scripts from burning through quotas. Compared to Google reCAPTCHA, it has more relaxed privacy requirements and integrates with Cloudflare Workers with near-zero configuration. Combined with per-day usage limits, these two layers together form the foundation of "let real users try it free while preventing abuse from causing bill shock."

If We Were Starting Over, Here's What We'd Do

The team's final takeaways can serve almost directly as a checklist for shipping multilingual TTS:

• Test with real mixed-language sentences, in the way users actually speak, and have real humans listen.
• Only send the text you want spoken to the model — don't mix in style instructions or labels.
• Inspect the leading bytes of audio before wrapping or converting it.
• Check audio duration after it goes through WhatsApp, not before.

These pitfalls may seem minor, but each points to the same underlying truth: the hard part of TTS is rarely "making it speak" — it's format compatibility, language adaptation, and end-to-end delivery quality across the real pipeline. For languages beyond English, these details rarely have off-the-shelf answers. You have to find them through real-world testing, one bug at a time.