Parsing and Rebuilding EPUB with Python: Practical Lessons from an AI Book Translation Project

Building an AI book translation service sounds like a straightforward task: users upload an EPUB, you call an LLM to translate it, and then they download the translated version. But once you actually start, you realize the real challenge is preserving the original book's formatting, fonts, images, and structure completely intact throughout the translation process. Based on practical experience from the LectuLibre project, this article shares the pitfalls encountered and solutions developed when processing EPUB files with Python.

Why EPUB Processing Is More Complicated Than It Seems

An EPUB is essentially a ZIP archive containing XHTML files, CSS styles, images, and metadata. To translate a book, four steps are required: extract text from each XHTML file in reading order, chunk the text and send it to an LLM for translation, replace the original text with the translation while preserving the DOM structure and inline tags, and finally rebuild the EPUB with updated metadata, navigation, and all resources.

A naive approach would be: unzip, use regex to replace text, and re-zip. But this immediately breaks down when encountering nested tags, sentences split across tags, or special characters. A real parser is essential.

Tool Selection: The ebooklib + lxml Combination

The author evaluated several options. ebooklib is a high-level library that handles the ZIP container, manifest, spine, and table of contents, but does not parse HTML content itself; epubfile is poorly maintained; Calibre's ebook-convert is powerful but command-line oriented and unsuitable for embedding in a web service; using zipfile + lxml directly offers full control but requires implementing the EPUB specification yourself.

The final choice was ebooklib for the container, lxml for the content, striking the best balance between convenience and control.

EPUB Specification Overview: An EPUB is essentially a ZIP package following the OCF (Open Container Format) specification, which must contain a mimetype file (content application/epub+zip, uncompressed), META-INF/container.xml (pointing to the OPF package file), and an OPF file (defining manifest, spine, and metadata). The manifest lists all resources (XHTML, CSS, images, etc.) with their IDs and paths; the spine defines the reading order by referencing manifest IDs via itemref. In addition, EPUB 3 includes a navigation document (Nav) and an optional NCX (for EPUB 2 compatibility). Understanding these structures is fundamental to using ebooklib: ebooklib's EpubBook object automatically parses container.xml and the OPF, exposing interfaces like book.spine and book.get_items(), thereby avoiding the tedious work of manually handling ZIP and XML.

Namespace Pitfalls When Reading Content

Loading an EPUB and iterating through documents in spine order is not difficult—item.content returns a byte string that can be passed to lxml.etree.fromstring. But the first major pitfall appears immediately: XHTML files declare a default namespace xmlns="http://www.w3.org/1999/xhtml", and writing tree.xpath('//p') directly returns nothing. The namespace must be explicitly mapped:

ns = {'xhtml': 'http://www.w3.org/1999/xhtml'}
paragraphs = tree.xpath('//xhtml:p', namespaces=ns)

The author wrote a helper function for this, or always carried the namespace mapping. This was the first "pitfall" exposed in the entire project.

XHTML and XML Namespaces: XHTML is an application of XML, and its root element typically carries a default namespace declaration xmlns="http://www.w3.org/1999/xhtml". This means all unprefixed elements (such as <p>, <div>) belong to that namespace. When querying with XPath 1.0, unprefixed names by default match elements in "no namespace", so //p cannot match any node. A namespace mapping must be provided to the XPath engine, for example using namespaces={'xhtml': 'http://www.w3.org/1999/xhtml'} and querying //xhtml:p. Similarly, when creating new elements or setting attributes in lxml that involve namespaces, the Clark notation {namespace}tag must be used. This is a common pitfall when handling any XML-based document format (such as EPUB, SVG).

Extracting and Replacing Text: Balancing Structure and Semantics

Ideally, inline tags like <em> and <span> should be preserved because they carry semantic information. The team tried serializing the inner content of block-level elements into compact markup strings, having the LLM preserve tags during translation, and then parsing back into the DOM—but this approach was complex and error-prone.

The MVP phase chose a simpler approach: translate only the plain text of each paragraph, then remove all inline child elements and set the translation directly on the block-level element. The cost is losing formatting such as italics and bold in the translation, but it buys stability and development speed. This trade-off was explicitly documented.

There is a detail when replacing: if you create a new element to replace the old one, the original attributes are lost; the correct approach is to keep the element itself, delete all child nodes, and then set .text, so that tags, attributes, and tail text are all preserved.

Batch Translation and Rebuild Workflow

Translating paragraph by paragraph is too slow and loses context. The actual pipeline is: extract all paragraphs from the entire book in spine order, send batches of 50 paragraphs to the LLM, and map the results back to the corresponding elements in each XHTML file in the same order. A global ordered list of block elements across files is maintained, and after translation is complete, they are written back file by file to ensure the reading order is unchanged.

In terms of performance, a 300-page novel of about 100,000 words takes approximately 2-3 seconds for parsing and rebuilding (excluding LLM translation time) on a standard VPS. The bottleneck is ZIP compression/decompression in read_epub and write_epub; lxml DOM operations take only milliseconds. Compared to the minutes-long LLM translation time, this overhead is negligible.

Details Not to Be Overlooked During Rebuild

Images and CSS are independent EPUB entries that require no modification, and ebooklib automatically preserves them when reading and writing. However, modified XHTML entries must have their serialized output written back to item.content, and care must be taken with the xml:lang attribute and DOCTYPE declarations—ebooklib sometimes adds or removes them inconsistently, and the team ultimately fixed this by post-processing the serialized string.

As for the table of contents, since only text is changed and not structure, the TOC usually needs no modification; however, language metadata and the title need updating, done via book.set_title(), book.set_language(), and add_metadata().

EPUB Metadata and Navigation Documents: EPUB metadata is stored in the <metadata> element of the OPF file, including title (dc:title), language (dc:language), author (dc:creator), etc. ebooklib provides convenient methods like set_title(), set_language(), and add_metadata(), but note that these methods may not automatically update all related fields (for example, dc:title may have multiple <dc:title> elements for main title and subtitle). The navigation document (EPUB 3 Nav or EPUB 2 NCX) defines the table of contents structure, usually represented by the toc element. Since translation only modifies text content and not structure, links and hierarchy in the TOC usually need no adjustment, but if the book title or chapter titles are translated, the text nodes in the navigation document may need to be updated accordingly. ebooklib's book.toc property can be used to access and modify the table of contents, but caution is needed to avoid breaking link relationships.

Five Hard Cases in the Real World

1. Namespaces: Always use namespace mappings.
2. Self-closing tags: lxml may turn <p></p> into <p/>, which causes errors in some readers. The team ultimately found that many EPUBs are not strictly valid XML, so they switched to a more lenient HTML parser.
3. Encoding: Most EPUBs are UTF-8, but occasionally encodings like Windows-1252 are encountered, which ebooklib handles poorly; chardet must be introduced for detection.
4. Fixed-layout EPUBs: Fixed-layout books where each page is an image cannot be handled with a text-based approach and require detection and fallback to OCR or PDF-like workflows; this remains an open problem.
5. Corrupted EPUBs: Many EPUBs in the wild are not validly formatted. The team implemented a fallback strategy—first attempt strict XML parsing, and if that fails, use the lxml.html parser.

Key Takeaways and Remaining Challenges

Several core lessons: use ebooklib to manage the container and lxml to manipulate content; do not reinvent the ZIP/OPF layer; use regression tests with 50+ books from different sources; preserve the original DOM as much as possible—even deleting an empty span can break the layout; plan for inline tag handling early, as stripping tags is only a temporary solution.

The team is currently exploring tag-preserving translation approaches—sending serialized HTML to the LLM and asking it to return translated HTML—but reliability is inconsistent. How to preserve the semantics of inline formatting such as bold, italics, and links during translation remains an open question for the community.