Self-Hosted Whispersync: Automatic Progress Sync Between E-Books and Audiobooks

People who switch back and forth between e-books and audiobooks have probably all run into the same frustration: you read a few chapters on your Kobo, get in the car and switch to the audiobook, and have no idea where to pick up. Amazon's Whispersync solves this problem — as long as you buy everything from Amazon. For readers committed to self-hosting and avoiding single-ecosystem lock-in, that path has never been an option.

A Reddit user has open-sourced a project called Concordance, built specifically to bring progress sync to self-hosted book stacks. It connects KOReader, Calibre-Web-Automated (CWA), and Audiobookshelf (ABS) to enable bidirectional progress alignment between e-books and audiobooks.

What Problem It Solves

The author's use case is relatable: reading on a Kobo at home, listening to audiobooks in the car. Two systems each track their own progress with no connection between them, resulting in constantly "losing your place."

Concordance aims to make the handoff seamless — read a few chapters on your Kobo, open the audiobook, and it starts playing from just before where you left off. Listen to a stretch on a walk, then open the e-book, and KOReader prompts you to jump to the corresponding page.

It's worth emphasizing that the author explicitly describes this as an early alpha. So far it has only run on the author's own library and hardware, and the "write" functionality has only been active for two days across two books. In his own words: "Everything works, but currently means 'works on one person's collection.'" It's a refreshingly honest project description.

Amazon's Whispersync for Voice, launched in 2011, lets users seamlessly switch between Kindle e-books and Audible audiobooks with cloud-synced progress. This requires both the e-book and audiobook to be purchased from Amazon, with Amazon having done proprietary text-audio alignment work on both versions in the background. For users who buy DRM-free e-books, borrow from libraries, or manage their libraries in Calibre, Whispersync is completely unavailable. The self-hosting community has long relied on open-source tools like KOReader, Calibre, and Audiobookshelf, but these tools operate independently with incompatible progress data formats and storage locations. Concordance attempts to fill this gap without depending on any commercial platform.

Hierarchical Alignment: Why a Single Percentage Isn't Enough

The technical core of the project is alignment precision. The author notes that using a single "percentage through the whole book" for mapping falls far short — narration speed drifts, and by the end of a book the cumulative error can be several minutes.

Concordance uses a hierarchical strategy to address this:

Chapter Proportion Matching

It first maps e-book internal files to audiobook chapters by comparing proportions. This mechanism handles front matter, production credits tracks, and cases where a single chapter is split across multiple files.

Character-Level Positioning

KOReader reports your current exact character position, not a rough percentage, so the system knows precisely how deep into a chapter you are.

Forced Alignment

During overnight runs, a forced alignment container timestamps each word in the chapters you're about to reach, then maps your reading position to approximately the corresponding word. Each mapping result receives a "tier" based on computational precision, and only sufficiently accurate positions are actually written — imprecise results are reported but not applied.

One thoughtful design detail: when writing audiobook progress, the target position is intentionally set 2.5 minutes early. Skipping forward in audio is easy; scrubbing backward is a miserable experience.

Forced alignment is a classic technique in speech processing. Given an audio recording and its text transcript, it automatically calculates precise start and end timestamps for each word (or even each phoneme) on the audio timeline. It works in the opposite direction from automatic speech recognition (ASR) — ASR "hears audio and guesses text," while forced alignment "knows the text and finds the timing" — making it generally faster and more accurate than ASR. Implementations typically rely on Hidden Markov Models (HMMs) or more recent Transformer-based acoustic models (such as Montreal Forced Aligner or WhisperX). In Concordance's context, the text comes from e-book chapter content and the audio comes from the corresponding audiobook chapter; the forced alignment engine translates "you've read to the Nth character" into "X minutes and Y seconds into the audiobook." This also explains the high computational cost — the model must analyze audio acoustic features frame by frame and align them against the text sequence, a compute-intensive operation.

Narration speed drift is a commonly overlooked source of systematic error in audiobook progress conversion. Professional audiobook narrators can vary their pace by 20–30% between tense and relaxed passages, and pauses, re-reads, and editorial cuts mean the average speed derived from "total word count / total duration" continuously diverges from the actual speed of any given local segment. If you use a linear mapping based on whole-book percentage, these local deviations accumulate rather than cancel out — the further you are into the book, the larger the conversion error. Concordance's approach of re-anchoring at the chapter level and then doing fine-grained alignment within each chapter essentially breaks a long-range linear error problem into many independent short-interval problems, keeping the cumulative error within each interval at an acceptable level.

Deployment Requirements and Performance Overhead

To run this setup, you'll need:

• Calibre-Web-Automated: for storing e-books, read via KOReader using CWA's sync plugin
• Audiobookshelf: for storing audiobooks
• A Linux host with Docker for the alignment step, plus Python 3.11+

The alignment step has non-trivial overhead: it's pure CPU computation and runs slowly, at roughly 22 CPU-minutes per hour of audio, with peak memory usage around 3 GB per job. The upside is that it only aligns chapters ahead of where you currently are, so a single overnight run typically keeps pace, and jobs won't start if memory is insufficient. If your server is already busy at night, this pipeline may create resource contention.

Getting Started Carefully: Read Before You Write

The author has designed a deliberately cautious onboarding path, which is a real plus for an early-stage project:

After cloning the repo and running pip install -e ., fill in your CWA app password and ABS API key in .env, then run concordance --progress-only. This command prints every matched book, the progress positions on both sides, and what it "intends to do" — without sending data anywhere or writing anything.

Only after you add a book's Calibre ID to a whitelist (one book at a time) will anything actually be written. The author recommends starting with a book you're not currently reading, verifying the target position looks correct, and only then adding books you're actively reading.

Known Gotchas

The author lists a number of "things that will bite you," with an admirable level of candor:

• KOReader only — sync via the native Kobo reader is not supported.
• Chapter matching can go wrong. Reports include confidence levels, and weak matches are worth manually checking before whitelisting.
• For poorly produced audiobooks that pack an entire book into a handful of giant "Chapter 4" chunks, the project includes a tool for rebuilding chapter markers — but it has "only actually been run on one book." Read its proposal before applying it, and there's a restore command as a fallback.
• An alignment run might look fine overall but quietly go wrong in one section. Suspect intervals are rejected rather than trusted, so actual coverage will be spottier than "this book is aligned" might imply.
• Each configuration supports only one CWA user and one ABS user.
• The default alignment model uses a CC-BY-NC license — fine for personal use, but not for commercial purposes.

Project Status and How to Contribute

Concordance is MIT-licensed and open to issues and PRs. What the author most needs is for people to point it at someone else's library and report what breaks — --progress-only writes nothing and costs nothing beyond a bit of CPU. Bad chapter matches and unpairable books are exactly the problems one developer can't discover alone.

The author is also currently developing against the Calibre-Web NextGen fork and hasn't tested compatibility with the original upstream CWA — community feedback on that is welcome too. He's even explicitly stated: "'I tried it and nothing happened' is a genuinely useful bug report."

For enthusiasts who've long been working within the self-hosted e-book and audiobook ecosystem, Concordance offers a solution to a problem that has had essentially no answer until now. It's still rough around the edges, but the technical approach — hierarchical alignment, character-level positioning, precision-tiered writes, read-only preview mode — is quite solid. If you're already running the KOReader + CWA + ABS stack, it's at least worth running --progress-only to see what it finds.