Connect Retell AI to Open-Source Models: Build Your Own LLM Bridge in 90 Lines of Code

Retell AI's voice agents bill LLM calls by the minute by default: frontier models cost between $0.045 and $0.16 per minute, stacked on top of the voice engine fees. But Retell leaves a Custom LLM interface open, letting you bypass that billing line—during a call, it opens a WebSocket to your server, pushes real-time transcripts, and speaks back whatever you stream in return. Plugging an open-source model into this slot takes only a ~90-line bridge service.

According to the official blog, this bridge receives Retell's events and responds with streaming completions from llama-3.1-8b-instruct, with token costs well under one cent per minute. Below we break down the protocol, the bridge code, local testing without a real phone call, and an honest cost accounting.

Which components you need to wire together

Retell doesn't call a chat completions URL directly. At the start of a call, it opens a WebSocket to your server, and both sides exchange JSON events. The full chain consists of:

• Retell slot: Custom LLM, WebSocket protocol, Retell initiates the connection to your server on each call
• Protocol: Retell sends transcript events; you stream back response events
• Bridge: Located in the integrations/retell directory of ecohash-examples, about 90 lines
• Default model: llama-3.1-8b-instruct, $0.10 / 1M tokens for both input and output
• Measured data: 205 ms TTFT (p95), ~143 tokens/sec per stream
• Still on Retell's side: Voice engine, STT, TTS, phone lines (using its built-in providers)
• API base URL: https://api.ecohash.com/v1

How WebSocket differs from ordinary HTTP requests lies in the connection's persistence and bidirectionality. A normal chat completions call is a "question and answer": the client sends one HTTP request, the server returns a complete response, and the connection closes. A WebSocket, after the initial handshake, keeps a long-lived connection open, letting both sides send messages at any time without repeatedly establishing connections. This is exactly what real-time voice needs: a call may last several minutes and contain multiple turns, and Retell can keep pushing transcripts over the same connection, while the bridge can spit token increments back even as the main model is still streaming. Moreover, WebSocket is naturally suited to handling interruptions—the server can receive and process a higher-priority event from Retell without waiting for the current request to fully complete.

How the components work together

The event flow follows a clear handshake and turn mechanism:

1. Your server sends a config event and the first utterance to be spoken.
2. When the caller finishes a turn, Retell sends response_required, with the real-time transcript and a response_id.
3. The bridge forwards the transcript to EcoHash's /v1/chat/completions with stream: true.
4. Each token increment returns to Retell as a response event, and the final content_complete closes the turn. If the caller interrupts, a new request with a larger response_id arrives, and the stale reply is discarded.

response_id and the interruption mechanism is the most easily overlooked detail in this protocol. Every time Retell asks the model to respond, it assigns a monotonically increasing response_id. When the caller suddenly interjects before the model finishes speaking, Retell immediately sends a response_required event carrying a larger response_id. The bridge needs to maintain a "current latest id" state: once a new id arrives, it stops pushing the old reply's tokens to Retell, because that content is already stale. Without this check, the user will hear the model continue reciting the previous turn's lines, and the interruption experience fails completely. This is also one of the biggest engineering differences between streaming scenarios and ordinary request-response calls.

Bridge implementation

The complete service is in integrations/retell/server.py. The core logic is implemented with FastAPI's WebSocket route: after accepting the connection, it first sends the config and the opening line, then asynchronously iterates over the JSON events coming from Retell. Once it receives response_required or reminder_required, it maps the transcript into messages, calls the streaming completion, pushes each delta chunk back as a response event, and finally appends a terminator event with content_complete: True.

Running it is straightforward: install fastapi uvicorn openai, export an API Key starting with eco_ created from console.ecohash.com, then run uvicorn server:app --host 0.0.0.0 --port 8000. For the first test, expose it publicly with ngrok.

Pointing Retell at the bridge

In the Retell dashboard, click Create an Agent, open Other options, choose Custom LLM, then set the Custom LLM URL to your bridge address, e.g. wss://your-host.example.com/llm-websocket. Retell will append the call id at the start of the call and establish the connection.

At this point, the agent header's cost estimate only covers voice and telephony, no longer including the LLM line—which is precisely the point of this approach. Other agent configurations (voice, transcriber, phone number) remain unchanged. Retell's official custom LLM setup guide covers the dashboard details, and the bridge is exactly the part that guide leaves as an exercise.

Testing without making a call

The bridge speaks a documented protocol, so you can fake Retell in the terminal with any WebSocket client. The test script connects to ws://localhost:8000/llm-websocket/test-1, first reads the config and opening line, then sends a response_required event with a response_id and transcript, looping to concatenate the returned content until content_complete is true.

According to the original article, this loop was actually run while writing: the bridge completed the config handshake, spoke the first line, and streamed back a coherent 18 chunks from llama-3.1-8b-instruct, with ping and interruption handling all working as documented. This shows that apart from the phone network itself, protocol-level correctness can be fully verified offline.

Cost calculation

With Retell's built-in models, the LLM is billed by the minute; after switching to a Custom LLM, that line disappears from the Retell bill and you instead pay your own provider by token. The prices listed in the original article (dated 2026-07-23) are:

• GPT 5.5 (via Retell): $0.16/min
• GPT 5.4 or Claude 4.6 Sonnet (via Retell): $0.08/min
• GPT 4.1 (via Retell): $0.045/min
• llama-3.1-8b-instruct (EcoHash via bridge): ~$0.0003 token cost/min

This EcoHash figure assumes a typical customer service call of about 2000 input tokens and 250 output tokens per minute, at $0.10/1M; even for an agent with ten times the context, the token cost remains under one cent. Retell's other billing items—voice engine at $0.055/min, plus TTS and telephony—are unchanged, and the savings occur only on the LLM line. The trade-off is that you need to host a small server.

TTFT (Time To First Token) is a key metric for evaluating voice agent latency, referring to the time from sending a request to receiving the model's first output token. For voice interaction, TTFT directly determines the user-perceived "response speed": the human ear tolerates roughly under 300 milliseconds of conversational gap, beyond which the other party seems to "zone out." The 205 ms (p95) mentioned in the article means 95% of requests return their first token within 205 milliseconds, which is acceptable. In addition to the model's TTFT, the network round trip to the bridge's server also adds up, so deploying the server in a region closer to Retell's data centers and choosing a fast model that supports streaming are the two main ways to control end-to-end latency.

Trade-offs and limitations

• The Custom LLM slot only covers the "brain." Retell's STT and TTS still go through its built-in providers, and third-party open-source TTS cannot be plugged in here. The original article mentions that Vapi offers an open TTS slot, which takes a different path.
• You host a WebSocket server on the critical path of the call. The server should be as close to Retell's region as possible and return the first token quickly; llama-3.1-8b-instruct on EcoHash has a TTFT of 205 ms (p95), and your hosting adds extra latency, so measure before making promises.
• Voice replies should be short. The system prompt must strictly require one-sentence answers, otherwise the agent will ramble.
• The cost table uses list prices from a specific date and stated token assumptions; recheck before doing business calculations.

FAQ

Does Retell support directly entering an OpenAI-compatible base URL? No. The Custom LLM slot is a WebSocket protocol, not a base URL field, which is why a bridge is needed.

Which models can the bridge use? Any chat model on EcoHash. llama-3.1-8b-instruct is a reasonable default for voice scenarios; gpt-oss-20b offers stronger capability at $0.05 input / $0.18 output per 1M tokens.

What happens when the caller interrupts? Retell sends a new response_required with a higher response_id, the bridge tracks the latest id, and discards the old reply that is still streaming.

Can I add tool calling or RAG? Yes, and that's exactly the point of owning the bridge: insert retrieval, tool calls, or guardrails between the transcript and the model. The same key can also be used for /v1/embeddings and rerankers.

Does it affect Retell's voice quality or latency budget? Voice still runs on Retell's engine. Your contribution to latency is the bridge hosting plus model TTFT, so host the server near Retell and choose a fast model.

How do I transcribe or analyze calls after the fact? Process recordings with whisper-large-v3-turbo, then summarize with a chat model, using the same API Key.

Getting started

Overall, the value of this approach isn't in "saving a few cents," but in decoupling the voice agent's inference layer from closed billing: you can freely swap models, add retrieval, insert tools, and build guardrails, while voice and telephony infrastructure continue to be handled by Retell. For teams that need to control costs or deeply customize conversation logic, this 90-line bridge is a low-barrier entry point.