AssemblyAI's In-House Support Agent Joey: Resolution Rate Jumps from 10% to 80%

When 1,000 Daily Signups Are Supported by Just One Engineer

AssemblyAI is a company focused on voice AI infrastructure. It builds its own Speech-to-Text foundation models and has been expanding into a full Voice Agent technology stack. If you've ever used a meeting recorder like Fireflies, called a corporate phone support line, or placed an order at a fast-food restaurant via voice, there's a good chance AssemblyAI's models were running behind the scenes.

At the AI Engineer conference, AssemblyAI's Forward-Deployed Engineer (FDE) Matt Lawler shared a strikingly stark number: the company currently receives roughly 1,000 API signups per day, yet until just two days before his talk, he was the only person on the entire team responsible for onboarding new users.

This supply-demand imbalance is a reality faced by virtually every voice AI company. An FDE's job is to deeply embed themselves in a customer's business — committing code directly to their repositories, meeting with them weekly or even daily, and understanding their specific use cases. It's a high-value service model, but the problem is straightforward: one person simply cannot serve thousands of customers in a one-to-many fashion. Scaling by hiring more FDEs is neither economical nor practical.

Matt put forward a perspective worth considering for every FDE: your daily goal should be to automate yourself out of your job. If you want to serve customers better and deliver a quality experience at scale, you can't afford to be the bottleneck.

Why Off-the-Shelf Chatbots Could Only Solve 10%

Faced with an overwhelming volume of inquiries, AssemblyAI's initial approach was the same as most teams — they purchased an off-the-shelf support chatbot, pointed it at their documentation, and hoped it would automatically handle all the basic questions.

The results were disappointing: the out-of-the-box bot resolved only about 10% of conversations. With 1,000 sessions per day, that meant the bot handled 100 while the team still had to manually deal with the remaining 900 tickets. A 10% end-to-end resolution rate was nowhere near good enough.

Even more damaging was the iteration speed problem. The team had no access to the system prompt, no visibility into the tool-calling logic, and no control over the RAG retrieval infrastructure. Every time they wanted to adjust something, the vendor's response was, "that feature is on the roadmap." For a company that needs to respond quickly to customers, this kind of dependency was unacceptable.

So the team made a decision: rather than hire a new team member, they would build one — and they named this AI support agent Joey. Now, whether you reach out via the website chat widget, contact sales, or email the support address, Joey is the first to greet you, not a human.

Joey's Four-Layer Architecture: Building an AI That Actually Works Like an FDE

Joey is built on the Claude Agent SDK, which gives it capabilities far beyond a typical chatbot: it can manage its own infrastructure, has a file system, can write and debug code, call tools, and even grant itself new capabilities. In other words, it operates more like a real FDE than the standard Q&A bot sitting in the bottom-right corner of a website. It also remembers past conversations with you, and all metrics are tracked through Pylon — because it's managed as a genuine member of the support team.

The entire architecture breaks down into four core components:

Documentation Localization: Markdown Is Claude's Best Friend

The team gives Joey all official documentation as local Markdown files. Whenever the documentation system updates, these files sync to Joey's local file system. An unexpected benefit: even if the entire documentation site goes down, Joey can still provide accurate answers about the latest features based on its local files, with no human intervention needed.

Retrieval Augmentation: Embedding with Voyage

To deliver accurate answers faster, the team uses Voyage embeddings, allowing Joey to surface the most relevant documentation first and reduce search time. When it encounters something confusing or needs to connect multiple sources, it can also perform deep agentic searches across the file system, entering what might be called a "deep thinking mode."

RAG (Retrieval-Augmented Generation) is an architectural pattern that combines an external knowledge base with a large language model: an embedding model first converts document chunks into vectors stored in a vector database. When a user asks a question, the query is similarly vectorized, the most similar document fragments are retrieved, and those fragments are injected into the LLM's context to generate a response. Voyage is a company specializing in embedding models, whose models outperform general-purpose embeddings in semantic similarity retrieval for specialized domains like code and technical documentation. Joey uses Voyage rather than OpenAI or open-source embeddings to achieve higher retrieval precision across extensive API docs and changelogs, reducing hallucinated answers caused by irrelevant document fragments.

Behavioral Guidelines: A 30,000-Line Claude.md

All major pages on the website — changelogs, pricing, feature descriptions — are converted to Markdown with URLs preserved, so Joey can cite sources for customers to verify. More critically, there's a ~30,000-line Claude.md instruction file containing all guardrail rules and communication guidelines. Whenever Joey gives an incorrect answer, the team updates this file, and all subsequent conversations immediately improve.

Claude.md is a configuration file format conventionally used by Anthropic's AI coding assistant Claude Code — essentially a Markdown file placed in the project root directory that Claude reads as priority "context rules" when working on that project. The team extended this mechanism to Joey's runtime environment, writing all business guardrails, communication style requirements, prohibited behaviors, and product knowledge into this file. At 30,000 lines, it has grown far beyond a simple system prompt — it's closer to a continuously iterated "employee handbook." Each time Joey's responses drift, only this file needs updating; no retraining or redeployment is required, and all subsequent conversations immediately benefit. This "prompt-as-configuration" approach is currently one of the most lightweight engineering practices for rapidly iterating on AI agent behavior.

Deployment: Live in 30 Seconds with Railway

The team didn't want to wrestle with EC2 or manage instance types, so the entire system was containerized with Docker and deployed on Railway. Every time an update is pushed to the repository, the new version goes live in approximately 30 seconds, with no manual monitoring of deployments. They've even had experiences like this: while watching a conversation in real time, they spotted a bug, immediately wrote and deployed a PR fix, and the current session was updated on the fly — all while the customer had no idea a hot patch had just been applied.

The Claude Agent SDK (Anthropic's Claude Agents framework) is an orchestration layer for developers that allows Claude models to run in "agent" mode: rather than passively answering questions, it can actively plan steps, call external tools (such as reading/writing file systems, executing code, making API calls), and adjust actions based on intermediate results across multi-step loops. Compared to single completion calls, agent mode gives the model a longer "action chain within the context window," enabling it to autonomously decompose complex tasks and complete them incrementally. This is also the underlying reason Joey can deeply retrieve documentation, autonomously debug code, and even grant itself new capabilities — it's not running a single Q&A exchange, but a continuous loop of reasoning and action.

From 10% to 80%: Results in One Week, Just $700/Month

The results were remarkable. With what Matt described as a "fairly scrappy" initial implementation, Joey raised the end-to-end resolution rate from 10% to 80% in its very first week of deployment, with total costs (tokens plus inference/infrastructure) running only about $700 per month.

Matt was careful to emphasize that this 80% figure wasn't massaged. Some teams claim to "resolve 80%" but only for a specific category of tickets. Joey's situation is different: users cannot reach a human directly unless they explicitly ask Joey to escalate. That means it genuinely handles 100% of all inbound tickets, only routing the 20% that truly require human attention — things like rate changes, data deletion requests, agreement signing, or matters requiring legal involvement.

What's more, the things Joey can't handle form a clear and actionable backlog for the team. Can't send a BAA (Business Associate Agreement)? Give it a citable link. Can't discuss pricing? Now if you go to the website and tell it how many hours of audio you need, it will actually quote you a price — and you can negotiate with Joey.

Dogfooding: Using Their Own API to Give Joey a Voice

Another thing Matt wanted to validate through Joey was eating their own dog food. AssemblyAI is in the voice business — customers use it to build voice agents, so the support bot should naturally be a voice agent too.

The team used their own Voice Agent API to add voice capabilities to Joey. Switch to voice mode on the website, and you can talk to Joey directly. The entire solution is voice-in, voice-out, connecting speech-to-text, LLM, and text-to-speech through a single WebSocket connection — all optimized for voice scenarios, with real-time low latency, automatic handling of pauses, interruptions, and barge-in, and a built-in voice requiring no additional vendor integrations. In the live demo at the conference, Joey smoothly handled a real question about whether a BAA needs to be signed before building a medical transcription tool, accurately explaining that signing a BAA automatically opts the account out of model training.

There's a delightful meta-experience at play here: while customers are talking to Joey about how to build a voice agent API, they're simultaneously using that very API — getting a direct feel for what the product experience is like. The team also plans to give Joey a phone number this week so customers can call in for real-time support at any time.

WebSocket is a bidirectional persistent connection protocol built on TCP. Unlike the traditional HTTP request-response model, it allows the server and client to continuously exchange messages after a single handshake, without repeatedly establishing new connections. In voice agent scenarios, this is critical: audio data needs to be transmitted as a real-time stream, with the ASR (Automatic Speech Recognition) module transcribing as it receives, the LLM generating responses as it receives text, and TTS (Text-to-Speech) synthesizing audio as it receives the response — three stages forming a streaming cascade, compressing total latency to within a range acceptable for human conversation (typically under 500 milliseconds). AssemblyAI encapsulates this entire pipeline within a single WebSocket connection, saving developers from the engineering cost of stitching together multiple vendor APIs and handling stream synchronization issues.

A Note for FDEs: Don't Wait for Customers to Ask for Help — Build the Product Yourself First

Matt closed by distilling the core methodology of the talk: FDEs must continuously dogfood their own products. Teaching customers to get started isn't enough — for every new feature and product launch, you should personally use it, build with it, and ship something with it, then bring all the knowledge you've accumulated to guide your customers.

He admitted that in adding voice capabilities to Joey, he personally hit every pitfall — latency, turn-taking, interruption handling. Precisely because he used the exact same API his customers use, he can now offer advice with far more empathy and reliability.

"Go build something," he said. "If you're an FDE, go build what your customers are building — don't just wait to clear obstacles for them. In the process, you'll very likely end up making the customer experience better at the same time."