After a Call Interruption, Why the Voice Agent's Memory Must Be Rewritten

Imagine a scheduling assistant offering two available times: "Tuesday at 10, or Thursday at 2." The caller suddenly interjects to ask about parking right after hearing "Tuesday at 10." If the model then follows up with, "Which of these two times would you like?"—it's relying on information the caller never actually heard. The audio indeed stopped, but the conversation state didn't keep up.

This scenario comes from a continuously evolving voice agent engineering issue documented by developer Asaday in the SmartVoiceX project docs. It points to a fact overlooked by most teams: generated text does not equal delivery receipt.

Generation complete ≠ playback complete

Twilio provides a clear division of responsibility in its voice agent interruption handling guide: ConversationRelay handles audio playback and interruption detection, while the model's conversation history is maintained by the application itself. Twilio explicitly recommends that after an interruption occurs, the conversation history must be updated to reflect "the actual audio content that was played."

This isn't a newly released feature, but an engineering issue continuously exposed as teams integrate streaming voice into real workflows. The core conflict is that a model generates a sentence much faster than it plays back the full sentence. When the caller interrupts mid-sentence, the model's generation is already complete, but the caller has only received a portion.

Asaday's approach is to maintain two separate records: the full generated response for debugging, and the actually played portion for shaping the next turn of conversation. The key point is that neither equals "the caller understood or accepted a proposal."

For scheduling scenarios, this distinction directly changes the recovery strategy: the assistant can first answer the parking question, then introduce the options that haven't been heard yet, rather than assuming both time options have entered the shared conversation context.

ConversationRelay is Twilio's voice agent middleware layer that separates session-layer responsibilities—such as sending and receiving call audio, speech recognition, text-to-speech synthesis, and interruption detection—from application logic, streaming events and text to the backend via WebSocket. It is not itself a language model, nor does it maintain conversation history, so the application must bear the decision of "what the next turn's prompt should contain." This is why Twilio emphasizes "conversation history maintained by the application itself": ConversationRelay only reports what audio events occurred, and how those events should change the model's understanding of the shared context is a semantic judgment belonging to the upper-layer application. Understanding this boundary of responsibility is the prerequisite for handling interruption issues.

How to use interruption events, and how to avoid over-interpretation

According to ConversationRelay's WebSocket reference documentation, the interruption message contains two key fields: utteranceUntilInterrupt (the utterance content played up to the interruption point) and durationUntilInterruptMs (the duration played). These are input signals for the application, not evidence of the caller's understanding or confirmation.

Asaday's suggested process is: before requesting the next model response, first reconcile the interrupted assistant turn—keep the generated text for auditing, mark that turn as interrupted, and pass the played portion along with the interruption context to the model. Simply deleting the entire turn may lose information the caller actually received.

Another easily overlooked detail is the forwarding of stale output. Merely canceling the current response is not enough: callbacks already queued may still be delivering tokens. The application can assign a unique identifier to each response to reject output from responses that have been superseded. Asaday labels this as a "suggested concurrency protection measure," not an official Twilio-guaranteed field.

The core invariant is practical: once a response has been superseded, its remaining speech should not appear again after the caller's new request. Tests should verify this property at the boundary where text leaves the application, including cases of delayed output.

Backchannel refers to the short responses a listener makes while the other party is continuously speaking, such as "uh-huh," "right," "I see," etc. It is acoustically an interruption of the speaker, just like barge-in (intentional interruption), but the pragmatic intent is entirely different: backchannel means "please continue," barge-in means "please stop, it's my turn." If a voice agent applies the same stop rule to both, it will misjudge the caller as needing a turn switch when they are merely indicating they are listening, causing the conversation rhythm to become fragmented. ConversationRelay therefore provides backchannel sensitivity control, allowing applications to distinguish these two types of events through thresholds and duration. For scheduling scenarios, this distinction is especially important—the user may express following along with "mm-hmm" while the assistant lists available times, and interrupting generation at this point would lose options not yet played.

Calendar state and voice state must be separated

Interrupting a sentence does not undo an external operation.

Suppose the calendar write has already succeeded, but the caller interrupts halfway through the confirmation message playback. At this point the application faces two facts: the appointment exists, and the confirmation message was only partially delivered. Trimming the assistant's speech cannot erase the successful write, nor can it trigger a duplicate appointment when the conversation resumes.

Asaday's solution is to have the next turn's response query the recorded operation results and clarify the existing appointment. If the caller wants a different time, that's a new workflow decision; if the write is still pending, the application should preserve that uncertainty rather than improvising results based on the assistant's wording.

In his Twilio practice notes, he describes the phone system as an "integration boundary," where business state and results are held by the application itself. Interruption handling is precisely the concrete reason for maintaining this separation—voice text can describe an operation, but it is not the source of truth for the operation.

Integration boundary here refers to the interface layer between the voice application and external systems (such as calendars, scheduling APIs, CRMs). Its meaning is: the voice side is only responsible for initiating calls and describing results, while the real business state is authoritatively held by external systems. This brings a typical problem—external writes may complete asynchronously or partially fail, and voice text is generated when the write result is not yet determined. If the application treats the assistant's wording as the source of truth, once the write later fails, it will produce an appointment that is "confirmed but nonexistent." The correct approach is to have the next turn's response query the actual write result, preserve uncertainty for incomplete operations, rather than inferring state from the previous turn's speech. This is consistent with the distributed systems principle of "don't use log text as a substitute for a state machine."

Adjust turn strategy based on interruption type

Before historical reconciliation, there is an even more prior distinction: does the caller intend to take a turn?

Twilio's guide categorizes interruptions into three types: intentional barge-in, brief acknowledgments (such as "uh-huh"), and background noise. Its discussion of sensitivity and backchannel control shows that a single "stop when sound is detected" rule creates a clumsy agent. But over-suppression makes it hard to correct a wrong date or stop an unwanted explanation.

For the scheduling assistant, Asaday suggests testing three scenarios: explicit correction, brief acknowledgment during explanation, and the caller starting a new question. Each scenario requires different observation metrics—whether the agent yields the turn at the appropriate time, whether the history is accurate after recovery, and whether the underlying appointment is correct. Using only "reduced interruption count" as a success metric is misleading.

This also affects how teams evaluate voice products. Fluent greetings and low-latency responses leave many critical behaviors untested. Callers can change direction mid-sentence, while generation, playback, and API operations are each at different stages. A valuable call record should allow operators to reconstruct these stages without guessing "which words reached the playback side" or "which operation completed." This is why interruption handling becomes an operational feature: the next turn's response, and the person reviewing the call afterward, both need a truthful record of "what has happened so far."

Turn-yielding refers to the agent determining, after detecting the caller speaking, whether it should stop the current output and hand the floor back to the user. It is not a binary switch, but a set of policies: when to stop immediately, when to finish speaking, when to respond with an acknowledgment phrase and continue. A reasonable yielding strategy typically combines multiple signals—where the interruption occurs (mid-sentence or at the end), the duration of speech overlap, whether the user's utterance constitutes a complete semantic unit, and whether the current output contains operational information that cannot be discarded (such as confirmation codes, times). For the scheduling assistant, if the output is interrupted midway and already involves a calendar write, a hasty yield could result in an incomplete confirmation message; but continuing to speak would seem disrespectful to the user. Therefore, the yield decision needs to be designed in coordination with historical reconciliation and state queries, rather than triggered solely by acoustic detection.

One-line Summary

Voice agents must distinguish generation from playback, reconcile conversation history with actually delivered text, and separate voice state from business state.

Paragraph Summary

The article focuses on the engineering problem of voice agents handling user interruptions, with the core idea that "generated text does not equal delivery receipt." When a caller inserts a new question mid-way through the assistant's speech, the model has already generated the full response, but the user only heard part of it; if the next turn still assumes all content has entered the shared context, errors occur. Using Twilio ConversationRelay as an example, the article explains that the utteranceUntilInterrupt and durationUntilInterruptMs fields in its interruption message are merely input signals, not evidence of user understanding. The application should reconcile the interrupted turn before requesting the next turn, keep the generated text for auditing, mark that turn as interrupted, and pass the played portion to the model. It must also prevent residual tokens from superseded responses from being delivered, and let external operation states like calendars remain independent of voice state. Interruptions should also be distinguished as intentional barge-in, brief acknowledgment, and background noise, with separate turn strategies and evaluation metrics.