How Solo Developers Can Build a Multilingual AI Voice Chat App with Flask

Why Is a Chatbot That Can "Talk" So Hard to Build?

Most chatbots on the market can only type. Developer Emre Guzel wanted more than that: characters should sound like the character itself, stay in character throughout the entire conversation, and respond in the user's own language. So he set out to build IboChat—a Flask-based app where each character can reply in text and also speak aloud, supporting 24+ languages, with users able to create custom characters in minutes.

The project is live (chat.emreguzel.ca), and the author has written a full retrospective on the tech stack and the pitfalls he hit. What makes this article worth reading isn't "yet another AI app"—it's how a solo developer welded together a voice pipeline, a character system, and cost control under the constraint of a single GPU with 4GB of VRAM.

Tech Choices: Deliberately "Boring"

The author's stack can only be described as plain:

• Flask, single file, serving pages, JSON APIs, and audio endpoints at once
• GLM (glm-4.5-air) via the Z.ai API as the "brain"
• faster-whisper running locally for speech-to-text
• SQLite + Werkzeug hashing for account management, auto-created on startup
• Docker deployment behind Cloudflare

No React, no build step, no microservices. This isn't a lack of technical skill—it's about saving complexity for what truly matters: the voice pipeline and the character system. For a personal project, fewer infrastructure layers mean faster iteration and lower maintenance costs. This trade-off is worth reflecting on for many projects that reach for microservices at the drop of a hat.

GLM (glm-4.5-air) is the large language model series from Zhipu AI, accessed via the OpenAI-compatible API provided by Z.ai. "OpenAI-compatible" means the endpoint paths, request bodies, and response formats essentially follow OpenAI's Chat Completions spec, so developers can reuse existing SDKs or HTTP clients directly—just swap the base URL and API key. glm-4.5-air is the lightweight, low-cost tier in the series, suited for high-frequency, short-reply conversational scenarios. For a personal project, choosing a compatible endpoint means minimal changes when switching model providers later, which is one of the prerequisites that let the author iterate cheaply.

The Journey of a Message: From Input to "Speaking"

The user enters text or voice. If voice, faster-whisper converts it to text on the server CPU. The backend then builds the prompt in exactly the same way: character persona prompt, recent history window, user's language code.

There's an easily overlooked engineering detail here: the model never sees a "bare message." It's precisely this discipline in prompt construction that prevents characters from drifting into a generic assistant tone mid-conversation. The reason many roleplay apps fail is exactly that prompt assembly is too casual, letting the model's own assistant tendencies override the persona after several turns.

Once the reply comes back in character, the real challenge begins: making it say it out loud.

The Three-Tier Voice Pipeline: No Single Engine Can Do It All

The author had three requirements: sound like the character, support any language, and never crash. No single engine satisfies all three, so the /speak endpoint tries three paths in order:

1. GPT-SoVITS—best voice cloning, synthesis takes about 5–18 seconds, but only supports English, Chinese, Japanese, and Korean, and permanently occupies ~3.9GB of VRAM once loaded.
2. OmniVoice Studio—supports any language, takes about 30–40 seconds, called via an OpenAI-compatible endpoint.
3. Edge TTS—Microsoft's free neural voices, no cloning capability, but supports 24 languages, zero latency, and never fails.

The last one is the safety net, ensuring the app always has something to say. This fallback design approach is especially important in voice products: users tolerate "the voice isn't quite right" far more than "no response."

GPT-SoVITS is an open-source few-shot voice cloning and synthesis project that combines GPT-style autoregressive modeling with the SoVITS acoustic model—just a few seconds to tens of seconds of target audio suffices to clone a voice. Its advantages are high voice similarity and good naturalness; the cost is that inference requires loading large model weights, with high VRAM usage and slow first load, and the official pretrained models mainly cover Chinese, English, Japanese, Korean, and a few others. The article's note that it "permanently occupies ~3.9GB of VRAM once loaded" shows it's a resident-memory model—coexisting with another voice model on a 4GB GPU is nearly impossible, which is the root cause of the lifecycle management problem discussed later.

Edge TTS refers to calling the online neural text-to-speech service built into Microsoft's Edge browser (often accessed via unofficial client libraries like edge-tts). It needs no local model and no GPU, returning audio directly from text, so latency is extremely low and language coverage is broad—but it only offers preset voices and can't clone a user-specified voice. Placing it at the end of the pipeline as a "safety net" essentially uses a reliably available fallback to absorb load failures, timeouts, and unsupported-language errors from the first two tiers, thereby eliminating the worst experience—"no response"—from the system.

The Bug That Took Three Days to Fix: One GPU, Two Voice Models

GPT-SoVITS and OmniVoice share the same 4GB GPU. For a while, loading one would starve the other of resources, resulting in corrupted audio output. The author initially thought it was a CUDA installation problem and reinstalled PyTorch twice.

The real fix wasn't in hardware but in orchestration:

• Flask loads GPT-SoVITS on demand, never at startup
• A watchdog thread kills it after 120 seconds of idleness
• VRAM is cleared before each OmniVoice call to prevent the two models from competing

This case shows that the core conflict in running multiple models in a small-VRAM environment is often not insufficient compute but missing lifecycle management. On-demand loading plus idle reclamation is more realistic than upgrading hardware.

CUDA VRAM is a separate memory pool on NVIDIA GPUs managed by the driver, where model weights, activations, and intermediate inference tensors must reside. Unlike system memory, allocated CUDA VRAM isn't automatically returned to other processes, and VRAM released by one process may be temporarily unavailable due to caching mechanisms. When two model processes request VRAM in sequence, if the earlier loader doesn't explicitly free or clear its cache, the later one will hit out of memory or be forced into extremely small batch sizes—manifesting as corrupted audio or inference errors. So "clearing VRAM" and "killing processes after idleness" in the article aren't nice-to-haves but necessary operations when multiple models share a single card.

Cold Start: The Silence of Death

Loading a voice model takes about 30–50 seconds. If a user sends a message during this window, the request hangs. The initial implementation kept the request open, resulting in a terrible user experience.

The current approach: the server first probes whether the model is ready; if not, it immediately falls back to the next voice tier instead of leaving the user waiting in silence. What the user hears is a slightly inferior voice rather than a timeout error. Behind this decision is a clear judgment about experience priorities—availability over audio quality.

The Small Battles: Rate Limiting, Autoplay, and Voice Cloning

• Rate limiting: AI endpoints trigger rate limits under load. The author uses exponential backoff retries, so chatty users only feel a pause rather than seeing an error.
• Mobile autoplay: Browsers refuse to autoplay audio before the user touches the page, so the first reply only plays after the first tap. Annoying but unsolvable—you just have to accept it.
• Cloning voices from YouTube: The workflow is automated—yt-dlp grabs clean clips, ffmpeg trims them, OmniVoice generates voiceprints, cached per character. The hard part is finding clips that are "clean enough," which cost the author an entire evening.
• The endpoint is still called /api/gemini: Early versions used Google Gemini and now proxy to Z.ai. The name wasn't changed because doing so would break his own frontend—a little tombstone of architectural evolution.

Cost: LLM Is a Rounding Error, Voice Is Nearly Free

For anyone budgeting a similar project, the author offers a pragmatic conclusion: with a cheap model (glm-4.5-air level) and text-only replies per message, the LLM cost of conversations is merely a "rounding error." On the voice side, Edge TTS is free, and the two GPU models consume only electricity.

This means the barrier to entry for AI voice chat apps is lower than you'd think. The real cost isn't in API bills but in engineering complexity—how to make multiple voice engines work together within limited VRAM.

Next Steps and Open Questions

IboChat currently labels its characters explicitly as AI parody. The author plans to add long-term cross-session memory and better mid-conversation language switching.

He also poses an open question at the end: if you've also built a voice + AI project, how did you resolve the "one GPU, two models" conflict? This question has real value for the indie developer community—big-company solutions usually assume ample compute, while personal projects must find a path within constraints.

From a single-file Flask app to three-tier voice fallback, IboChat's build process demonstrates a pragmatic philosophy of AI app development: don't chase architectural sophistication—design reliability into every failure path.

One-line Summary

A solo developer built a multilingual character chat app with a three-tier voice fallback pipeline on 4GB of VRAM.

Paragraph Summary

The article recaps solo developer Emre Guzel's entire process of building IboChat. The project uses a single-file Flask app integrating the GLM model, faster-whisper speech recognition, and a SQLite account system, deliberately avoiding React and microservices to reduce maintenance costs. The core challenge is voice: no single engine satisfies voice cloning, multilingual support, and stability simultaneously, so he designed a three-tier fallback pipeline of GPT-SoVITS, OmniVoice, and Edge TTS. The author also documents the bug where two voice models competed for VRAM on a single 4GB GPU, ultimately solved through on-demand loading, idle reclamation, and VRAM clearing, emphasizing availability over audio quality.