AssemblyAI in Action: How AI Agent Joey Resolves 80% of Support Tickets

From 10% to 80%: One Engineer's Fix for Being Crushed by Volume

AssemblyAI's Forward-Deployed Engineer (FDE) Matt Lawler shared a compelling case study at a tech conference: how they built an AI customer support agent named Joey from scratch using the Claude Agent SDK, pushing their end-to-end ticket resolution rate from 10% all the way to 80%.

The problem was straightforward. AssemblyAI's core business is voice AI infrastructure — they build their own speech-to-text foundation models and are expanding into a full voice agent stack. As demand for voice AI exploded, they were receiving around 1,000 API registration requests per day. Two days before the talk, Matt was still the only onboarding engineer at the company. One person handling 1,000 new signups a day simply isn't sustainable.

The FDE role itself is defined by depth of service — committing code to customer repos, weekly or monthly syncs, deeply understanding each customer's specific use case. That one-on-one model delivers high quality, but it can't scale. Matt offered a perspective worth sitting with for any FDE: your daily goal should be to automate yourself out of a job. If you want to deliver better customer experiences at scale, you can't be the bottleneck.

Why an Off-the-Shelf Bot Only Resolved 10%

The team's initial instinct — like most companies — was to buy a ready-made support bot, point it at the docs, and expect it to handle the basics. The results were disappointing: end-to-end resolution sat at roughly 10%. Out of 1,000 conversations, the bot handled 100, and the team still had to deal with the other 900.

Even worse was the iteration speed. The vendor didn't expose the system prompt, tool calls, or RAG infrastructure. Every time the team wanted to change something, the vendor's response was "it's on the roadmap." For a team that needs to move fast and respond to product changes, a black box like that simply doesn't work.

So they decided to build their own — instead of hiring a new employee, they'd build one. Joey was born.

Joey's Four-Part Architecture

Joey is now the first line of defense for all inbound conversations: whether you reach out via the website chat widget, contact sales, or email support, you meet Joey before any human. It's designed to be an "infinitely scalable FDE" — it remembers your conversation history, all its metrics are tracked through Pylon (because it's treated as a full member of the support team), and it's built on the Claude Agent SDK. It can manage its own infrastructure, has a file system, can read, write, and debug code, call tools, and even give itself new capabilities.

The entire architecture breaks down into four core components:

Docs Are Fed to Claude as Local Markdown

Markdown is the best format for working with Claude. The team converts all documentation, changelogs, pricing pages, and other key content into Markdown and checks it out locally into the repository. Whenever the documentation system updates, these files update too. This has an interesting side effect: even if the entire docs site goes down, Joey can still answer questions accurately from its local files.

Retrieval Uses Voyage Embeddings

To ensure answer quality and speed, the team partnered with Voyage to use their embeddings, allowing Joey to quickly locate the most relevant documentation and reduce search time. Because all docs are local files, when Joey encounters a complex question requiring multiple sources, it can agentically deep-dive through the file system to piece together an answer.

Voyage is an AI company focused on embedding models, with core products optimized for Retrieval-Augmented Generation (RAG) scenarios. The basic principle of RAG is: split documents into chunks, convert each chunk into a high-dimensional vector using an embedding model and store it; when a user asks a question, vectorize the question the same way, find the semantically closest document chunks in vector space, then feed those chunks along with the question into a language model to generate an answer. Compared to stuffing all documents into a context window, RAG can handle knowledge bases far larger than context limits while reducing token costs and hallucination risk. AssemblyAI chose Voyage over general-purpose embedding solutions primarily because Voyage has targeted optimizations for technical documentation, code, and API reference retrieval accuracy — better suited for developer support use cases.

Deployed on Railway, New Versions Live in 30 Seconds

The team didn't want to touch EC2 or manage instance types, so they deployed Joey directly on Railway. The benefit is blazing-fast iteration: spot a bad conversation in Slack, write a PR, deploy, and a new version of Joey is live within 30 seconds. They've even managed to monitor a conversation in real time, catch a bug on the spot, deploy a fix, and the customer never noticed.

A 30,000-Line CLAUDE.md

All instructions are delivered via a CLAUDE.md file which, according to Matt, contains roughly 30,000 lines of guardrails and guidance for interacting with customers. Whenever Joey gives a wrong answer or delivers a poor experience, the team updates this file — and all subsequent conversations improve. The whole thing is packaged as a Docker file and deployed as a containerized service on Railway with a single command.

The Claude Agent SDK (Anthropic's agent development toolkit) is a framework that allows developers to deploy Claude as an autonomous agent, enabling it to manage tool calls, maintain multi-turn conversation state, execute code, and call external APIs. Unlike calling the Claude API directly, the Agent SDK's core design philosophy is to let the model "autonomously plan steps" when completing complex tasks — for example, deciding when to retrieve documents, when to call a tool, and when to escalate to a human. This architecture is particularly well-suited for customer support, since user questions often aren't singular but require bridging multiple knowledge sources and operational steps. Pylon is a customer communication and ticket tracking platform designed for B2B customer success teams, unifying conversations from Slack, email, chat widgets, and other channels into a single view with metrics tracking.

CLAUDE.md is a convention file format Anthropic designed for Claude agents — essentially a "job description" and "behavior handbook" for the agent. When a Claude Agent runs in a project or repository, it reads this file first to understand its role definition, behavioral guardrails, communication style requirements, and handling rules for specific scenarios. At 30,000 lines, the team has encoded a large volume of edge-case logic, customer communication playbooks, product knowledge, and escalation criteria — forming a continuously evolving "institutional memory." The advantage of this approach to driving agent behavior through a plain-text file is that every time CLAUDE.md is updated and deployed, all subsequent conversations immediately reflect those changes — no model retraining required, decoupling rule updates from model capabilities.

Results: 80% Resolution Rate, ~$700/Month

In the very first week of deployment, a fairly rudimentary implementation pushed the end-to-end resolution rate from 10% to 80%, at a total cost of roughly $700 per month (tokens plus infrastructure).

Matt was careful to emphasize the integrity of that number. Some teams game their metrics — claiming 80% on a cherry-picked subset of tickets. Joey is different: unless you explicitly ask to escalate, you can't reach a human directly. Joey handles 100% of inbound tickets and only escalates 20% with legitimate reasons — things like rate changes, data opt-out from training, signing agreements, or needs that require FDE or legal involvement.

Even better, the things Joey can't handle turn into a clear "team capability expansion backlog." Can't issue a BAA? Give it a resource link it can reference. Can't discuss pricing? Teach it to quote — now you can go to the website, tell Joey your usage hours, and it'll give you a quote on the spot. You can even negotiate with it.

Eating Your Own Dog Food: Why Dogfooding Matters

The most interesting part of this story isn't how much headcount was saved — it's the dogfooding philosophy. AssemblyAI is in the voice business, and their customers use its API to build voice agents, so Matt gave Joey voice capabilities.

Joey now supports voice mode: speech in, speech out, with a single WebSocket connection threading together speech-to-text, LLM, and text-to-speech — all optimized for voice scenarios. It handles pauses, interruptions, barge-in, and other interaction nuances, has its own voice, and tool-calling works identically to text mode. The team also plans to add a phone number so users can call in for real-time support anytime.

There's a meta layer here: when you're talking to Joey via voice about how to build a voice agent on its API, you're already using the product — you're seeing firsthand what your own product experience actually feels like.

Matt's core takeaway is blunt and clear: if you're an FDE, you should be constantly eating your own dog food. Don't just teach customers how to get started — actually build the same kinds of products your customers are building. He personally ran into the real walls of latency, turn-taking, and interruption handling while adding voice to Joey, and that's what gave him genuine insight into what customers are experiencing — and made him capable of giving much better advice.

For any company with an FDE team or considering AI support automation, Joey offers a replicable template: an open, controllable architecture, a local Markdown knowledge base, a fast-iteration deployment pipeline, and an escalation backlog that drives continuous team capability growth.

Dogfooding is a common tech industry term, derived from the English idiom "eating your own dog food," meaning that a company's employees actually use their own products. The concept carries special strategic weight for FDEs: their job is to embed deeply in customer environments and help customers successfully deploy the product, but FDEs who don't use the product themselves can only offer theoretical advice — they never truly feel the friction points customers encounter. WebSocket is the technical foundation for Joey's voice mode — it's a protocol that establishes full-duplex communication over a single TCP connection, allowing continuous, low-latency, bidirectional data transmission between server and client, making it ideal for real-time voice streams that are extremely sensitive to latency. Traditional HTTP request-response patterns can't support voice continuity, but WebSocket maintains a persistent connection that continuously transmits audio data frames.