lxml
A high-performance Python XML/HTML parsing library built on libxml2, supporting XPath, namespace handling and DOM operations, commonly used for processing structured documents.
Timeline (last 90 days)
The tech stack for web scraping typically includes the request library requests and the parsing libraries BeautifulSoup or lxml.
When handling XHTML files, because of the default namespace xmlns="http://www.w3.org/1999/xhtml", XPath queries such as //p return empty results, and a namespace mapping must be provided explicitly (for example namespaces={'xhtml': 'http://www.w3.org/1999/xhtml'} and query //xhtml:p).
After evaluation, the author chose ebooklib to manage the container and lxml to manipulate content, considering this the best balance between convenience and control.
When replacing paragraph text, the correct approach is to keep the original element, delete all child nodes and then set the .text attribute, rather than creating a new element to replace the old one, so that tags, attributes and tail text are preserved.
For a 300-page novel of about 100,000 words, parsing plus rebuilding (excluding LLM translation time) takes about 2-3 seconds on a standard VPS; the bottleneck is ZIP compression and decompression, and lxml's DOM operations take only milliseconds.
lxml may turn <p></p> into <p/>, which causes errors in some readers, so the team ultimately switched to a more tolerant HTML parser to handle the many EPUBs that are not strictly valid XML.
When processing real-world EPUBs, it is recommended to use fault-tolerant parsing libraries, such as Python's lxml with the recover=True option or BeautifulSoup's lxml-xml parser
Lazy imports means that heavy dependencies such as torch, nltk, and lxml are only loaded when the relevant functionality is actually used, thereby reducing the package's startup overhead and memory footprint.
All Facts (8)
The tech stack for web scraping typically includes the request library requests and the parsing libraries BeautifulSoup or lxml.
50%UnverifiedWhen handling XHTML files, because of the default namespace xmlns="http://www.w3.org/1999/xhtml", XPath queries such as //p return empty results, and a namespace mapping must be provided explicitly (for example namespaces={'xhtml': 'http://www.w3.org/1999/xhtml'} and query //xhtml:p).
50%UnverifiedAfter evaluation, the author chose ebooklib to manage the container and lxml to manipulate content, considering this the best balance between convenience and control.
50%UnverifiedWhen replacing paragraph text, the correct approach is to keep the original element, delete all child nodes and then set the .text attribute, rather than creating a new element to replace the old one, so that tags, attributes and tail text are preserved.
50%UnverifiedFor a 300-page novel of about 100,000 words, parsing plus rebuilding (excluding LLM translation time) takes about 2-3 seconds on a standard VPS; the bottleneck is ZIP compression and decompression, and lxml's DOM operations take only milliseconds.
50%Unverifiedlxml may turn <p></p> into <p/>, which causes errors in some readers, so the team ultimately switched to a more tolerant HTML parser to handle the many EPUBs that are not strictly valid XML.
50%UnverifiedWhen processing real-world EPUBs, it is recommended to use fault-tolerant parsing libraries, such as Python's lxml with the recover=True option or BeautifulSoup's lxml-xml parser
50%UnverifiedLazy imports means that heavy dependencies such as torch, nltk, and lxml are only loaded when the relevant functionality is actually used, thereby reducing the package's startup overhead and memory footprint.
50%