Generating Realistic Exam Listening with TTS: Multi-Voice, Pauses, and Timestamps Explained

The cruelest part of the listening section in a language exam is this: the recording won't wait for you. It plays at its own pace—you finish one question and the next begins immediately. That's exactly why, when creating practice materials for learners, a bare audio file is far from enough—it must include instructions, reading time, and a second playback, just like a real exam.

ExamReady is a platform for German B1/B2 exam preparation, and I built the listening practice feature on it from scratch. This article discusses how I generate speech with TTS, concatenate audio with ffmpeg, and extract timestamps during the build process so the app can display line-by-line text during playback and highlight the currently spoken content.

(Note: ExamReady is an independent project and is not officially endorsed by ÖSD, ÖIF, or the Goethe-Institut.)

Breaking Down the Problem: What Listening Practice Actually Needs

I needed a set of listening exercises that simultaneously satisfies the following conditions:

• Multiple speakers with clearly distinguishable voices
• Includes the exam framework: instruction announcements, reading time, second playback
• After learners answer, they can view line-by-line text, ideally with real-time highlighting of the currently spoken sentence

I didn't want to manually edit each recording in a DAW (Digital Audio Workstation). Every time the script changes, I'd have to re-export, and it's hard to track where each sentence is located in the audio. I wanted the audio to be a build artifact—automatically generated from a text script, not the result of manual splicing.

This shift in thinking is crucial: manage audio as code. The script is the single source of truth, and the audio is just its compiled output. This way, changing a sentence only requires modifying the script and rebuilding, and all associated data (timestamps, text alignment) update in sync.

Step 1: Describe the Recording as Data

Each recording consists of a series of "segments." A segment is either speech (specifying voice and text), silence (specifying seconds), or a chapter marker.

type Segment =
  | { kind: "speech"; voice: string; text: string }
  | { kind: "silence"; seconds: number }
  | { kind: "chapter"; title: string };

const recording: Segment[] = [
  { kind: "chapter", title: "Instructions" },
  { kind: "speech", voice: "narrator", text: "You will hear a short conversation twice." },
  { kind: "silence", seconds: 5 }, // example value
  { kind: "chapter", title: "Conversation" },
  { kind: "speech", voice: "speakerA", text: "..." },
  { kind: "speech", voice: "speakerB", text: "..." },
];

The most important design here is: everything the learner hears is in this list, including silence. Silence is not left to the TTS engine to decide arbitrarily. If the exam gives reading time, that time is an explicit segment.

Voices are just names mapped to TTS voice configurations. A two-person conversation uses two different voices, and narration is assigned to a third voice. This structure makes "who is speaking" part of the data, rather than something considered during post-editing.

TTS (Text-to-Speech) is the technology that automatically converts written text into natural speech. Modern TTS is based on deep learning and can generate pronunciation, intonation, and rhythm that are close to human. In language learning scenarios, the advantages of TTS are: speech for arbitrary text can be generated on demand, multiple voices (e.g., different genders, accents) are supported, and the output format is uniform and controllable. However, the reading pace of TTS is determined internally by the model and typically cannot insert silence precise to the second between sentences—this is exactly why the article emphasizes that silence must be handled as an independent segment. In addition, TTS may pronounce numbers, dates, and proper nouns unexpectedly, requiring manual listening and proofreading.

Step 2: Generate Speech Segment by Segment

For the speech parts, I use ElevenLabs' TTS. Each speech segment generates an independent audio clip. The pseudocode for the build step is as follows:

clips = []
for seg in script.segments:
    if seg.kind == "speech":
        clips.append(tts(voice=VOICES[seg.voice], text=seg.text))
    elif seg.kind == "silence":
        clips.append(make_silence(seg.seconds))
    # chapters produce no audio, only a marker

Generating segment by segment rather than requesting the entire conversation at once has a practical benefit: I can precisely control the start and end points of each voice, and I can insert silence between any two sentences. If I request the whole thing at once, the pause rhythm is determined by the TTS engine's model, and it's difficult to insert reading time precise to the second.

The cost is longer build times as segments increase, and the need to ensure all segments have consistent formats (mentioned later). But the control gained is worth it.

Step 3: ffmpeg Concatenation and Timestamp Generation

After obtaining all segments, ffmpeg concatenates them into a single file. The truly critical part is: the build process also outputs the start time of each segment.

The approach is to measure the actual duration of each segment and maintain a cumulative timeline:

t = 0.0
timeline = []
for seg, clip in zip(script.segments, clips):
    timeline.append({"segment": seg.id, "start": round(t, 2)})
    t += duration_of(clip)  # measured from the real audio, not estimated

The core detail is in the comment: durations are measured from the real audio, not estimated. If you guess durations based on text length, timestamps will drift, and highlighting will misalign after a few sentences. Only by reading the actual duration of each generated segment can the timeline stay accurate.

The concatenation itself is a single ffmpeg call that reads the segment list in order:

ffmpeg -f concat -safe 0 -i clips.txt -c copy listening.mp3

The premise for using stream copy is that all segments have consistent formats and parameters, so it's important to make the silence segments match the speech segment parameters. If parameters are inconsistent, re-encoding is a safer fallback.

ffmpeg is an open-source cross-platform audio/video processing tool widely used for format conversion, editing, concatenation, streaming, and more. Its -f concat mode allows concatenating multiple media files into one file in list order, while -c copy (stream copy) means not re-encoding but directly copying the original audio/video streams, making it extremely fast with no quality loss. However, stream copy requires all input segments to have exactly the same encoding format, sample rate, channel count, and other parameters; if parameters differ, concatenation will fail or produce anomalies, in which case you need to remove -c copy and let ffmpeg re-encode. The article uses ffmpeg precisely for its efficiency and scriptability, enabling the audio build to be fully automated.

Two Major Uses of Timestamps

The start times output by the build become data attached to the recording, and the app uses them in two scenarios:

1. Line-by-line text: After exam-style playback ends, learners can read the text line by line, with each line linked to its corresponding position in the audio.
2. Real-time highlighting: During playback, the app compares the current playback time with start times and highlights the matching text.

Because the timeline comes from the same script that generated the audio, text and audio won't drift apart. Modify a line of text and rebuild, and everything is regenerated together. This "single source of truth" design eliminates the most common source of errors in manual alignment.

A timestamp here refers to the starting playback moment of each speech segment in the final audio file, usually in seconds or milliseconds. With precise timestamps, the application can map audio playback progress to text lines: when playback reaches a certain point, highlight the corresponding text line; or allow users to click a text line to jump to the corresponding position in the audio. The accuracy of timestamps directly determines the quality of highlight synchronization. Since the duration of each segment generated by the TTS engine is unknown, you must read the duration from the actual generated audio file (e.g., via ffprobe or an audio library) and then accumulate to get the timeline. Any estimation based on text length or speaking rate will introduce errors, and the errors will accumulate as segments increase, eventually causing severe highlight misalignment.

Baking the Exam Framework into the File

I put the exam framework directly into the file itself: instruction announcements, reading pauses, second playback. This is a deliberate choice. Learners should experience the same rhythm as the exam during practice, not a clean recording whose pace they control.

The cost is that the file is less flexible. If you want a version without pauses, you have to build a second file. For exam practice, I think this is the right trade-off; and because everything is generated from scripts, a second build is cheap.

A Few Pitfalls to Watch Out For

Here are a few reminders, not hard-earned lessons:

• TTS voices are not real exam audio. They are clear and consistent, which is good for practice, but real recordings may have different accents, speaking rates, and background noise. I won't claim the generated audio is exactly what you'll hear on exam day.
• Check pronunciation. Names, numbers, dates, and times are very important in listening questions, and TTS may produce pronunciations you didn't expect. It's worth listening to the output yourself rather than trusting blindly.
• The script is the single source of truth. Don't patch the audio by hand. If something is wrong, fix the script and rebuild, otherwise timestamps will no longer match.

Key Takeaways

1. Describe recordings as data: speech segments with voice and text, silence, chapters.
2. Make silence explicit. It's part of the exam and shouldn't be left to the engine.
3. Measure segment durations from the generated audio, never estimate from text.
4. Output the timeline from the build. Line-by-line text and real-time highlighting come almost for free.
5. Listen to the output yourself. A pipeline can run without errors and still misread numbers.

If you've done something similar—whether for exams or any audio that needs to stay synchronized with text—I'd love to know how you manage timelines.

Background Supplement

A DAW (Digital Audio Workstation) is professional software for recording, editing, mixing, and producing audio, such as Audition, Pro Tools, Logic Pro, etc. In a DAW, users can manually cut, move, and splice audio clips, and precisely adjust silence and volume. Traditionally, producing exam listening audio often requires editing sentence by sentence in a DAW and inserting pauses—a tedious process that's hard to reuse. The article proposes treating audio as a "build artifact," describing segments with code scripts and automatically generating audio, effectively replacing manual operations in a DAW with programming, thereby achieving a repeatable, version-controllable pipeline. This approach is particularly suitable for scenarios that require frequent text modifications or rhythm adjustments.