Deep Dive into Pandoc Conversion: What Survives When Converting Between Markup Languages

Why "Lossy" Markup Language Conversion Deserves Your Attention

With the release of Pandoc 3.10.2, this "Swiss Army knife of document conversion" has once again sparked discussion in the tech community. Whether converting Markdown to HTML, performing the reverse operation, or transforming various plain text files into PDF, Pandoc can handle nearly anything. But a long-underestimated core question has surfaced: When converting between different markup languages, how much content actually "survives," and how much is silently lost in the process?

This is far from a trivial technical detail. For teams and individuals who need to manage documents across platforms and formats, conversion loss directly impacts content maintainability and long-term value. This article takes a deep look at Pandoc's conversion mechanism, explores the design philosophy of modern markup languages, and explains why AST (Abstract Syntax Tree) is becoming the critical factor determining conversion quality.

Pandoc's Core Mechanism: An AST-Centered Conversion Architecture

Why AST Is the Soul of Pandoc's Conversion

The secret behind Pandoc's ability to freely convert between dozens of markup formats is that it doesn't simply perform "text replacement." Instead, it parses input documents into a unified Abstract Syntax Tree (AST). This intermediate representation layer strips away the shell of specific syntax, retaining only the document's structure and semantics—headings, paragraphs, lists, emphasis, links, tables, and other core elements.

The Abstract Syntax Tree is a core concept in compiler theory, originally widely applied in the compilation and interpretation of programming languages. An AST represents the syntactic structure of source code or documents as a tree data structure, where each node represents a syntactic construct (such as an expression, statement, paragraph, or heading). The "abstract" qualifier means it omits concrete syntactic details (like the # symbols in Markdown or <h1> tags in HTML), preserving only semantically meaningful information. In the document conversion domain, introducing AST means elevating documents from "string operations" to "structured operations"—similar to how database queries are more reliable than directly manipulating text files. Pandoc's AST is defined using Haskell's algebraic data types, with core types including Block (block-level elements like paragraphs, headings, and code blocks) and Inline (inline elements like bold text, links, and footnotes). This strict type system ensures type safety throughout the conversion process.

In simple terms, Pandoc's workflow has two steps:

• Input format → Parsed into AST
• AST → Rendered as output format

The direct benefit of this architecture is that any element that can be fully expressed by the AST can flow "losslessly" between target formats that support that element. The essence of conversion loss typically occurs at two points—when certain features of the source format cannot be fully parsed into the AST, or when the target format cannot express certain nodes in the AST.

Markup Languages Designed Around AST Achieve Higher Conversion Fidelity

A key finding deserves emphasis: Modern markup languages that were built from the ground up around a clear syntax tree have a significant advantage in Pandoc conversion.

Traditionally, many markup languages (especially early Markdown dialects) had vaguely defined syntax, inconsistent handling of edge cases, and parsing results that were difficult to standardize—naturally leading to "distortion" during conversion. Languages with explicit parsing rules and predictable results achieve significantly higher fidelity when converting to and from other formats.

This also illustrates an important principle: when evaluating "what survives conversion," a language's underlying design matters more than its surface syntax.

Markdown's Historical Baggage and Djot's Revolutionary Exploration

From Markdown Fragmentation to the Birth of Djot

Pandoc's author, John MacFarlane, was also one of the early participants in Markdown standardization efforts 15 years ago. However, Markdown's "standardization" process actually devolved into considerable chaos—CommonMark, GitHub Flavored Markdown (GFM), MultiMarkdown, and various other dialects emerged one after another, with subtle but annoying syntactic differences between them.

Markdown was originally released by John Gruber in 2004, with its specification existing only as an informal document and a Perl script implementation, lacking any formalized syntax definition. This led to wildly different outputs across implementations when input contained edge cases like nested list indentation, inline HTML mixing, or nested link references. The CommonMark project, launched in 2014, attempted to resolve this through a strict specification containing over 600 test cases, but it also sparked community controversy—Gruber himself remained reserved about "standardization," arguing that Markdown's flexibility was part of its charm. GFM added extensions like tables, task lists, and strikethrough on top of CommonMark, while MultiMarkdown introduced footnotes, metadata, math formulas, and other academic writing features. The coexistence of these dialects means that "Markdown file" is effectively a semantically ambiguous concept.

It was precisely this deep reflection on fragmentation that led MacFarlane to introduce Djot in recent years—a new markup language positioned as the "spiritual successor to Markdown." Djot attempts to retain Markdown's readability advantages while thoroughly resolving its historical baggage of syntactic ambiguity and parsing inconsistency, providing a more rigorous rule system better suited for precise machine processing.

Djot was formally proposed by MacFarlane in 2022, with its design following several core principles: First, eliminating the need for backtracking during parsing—in Markdown, parsing emphasis markers (* and _) requires complex backtracking algorithms to determine opening and closing positions, while Djot achieves near-linear-time parsing through more explicit rules. Second, Djot strictly distinguishes markup methods for block-level and inline elements, avoiding the ambiguous relationship between list items and paragraphs found in Markdown. Third, Djot introduces a universal attribute syntax (similar to HTML's class and id), allowing any element to carry metadata without falling back to raw HTML. From an implementation perspective, Djot's reference parser is written in Lua and outputs a standardized AST representation, with its specification document formally defining the priority and applicable conditions for each parsing rule.

Three Dimensions an Ideal Markup Language Must Balance

An ideal markup language should achieve a good balance across the following dimensions:

• Readability: Humans should be able to intuitively understand the plain text source when reading it
• Writeability: The writing process should feel natural and fluid without adding cognitive burden
• Machine-consumability: Programs should be able to parse it precisely without ambiguity

Additionally, it needs to be comprehensive enough to express all types of elements from offline documents to online content. Natural tension exists between these three—over-optimizing for machine-friendliness may sacrifice writing experience (e.g., XML is machine-parsing friendly but terrible for humans to write and read), while over-simplification limits expressive power (e.g., plain Markdown cannot natively express footnotes, definition lists, or math formulas). Djot's emergence represents a substantive attempt at this balance exploration, trying to prove that strict parsing rules and excellent writing experience are not irreconcilable.

Developer's Practical Guide: Format Selection and Conversion Best Practices

Validate Your Toolchain with Dogfooding

"Dogfooding" (eating your own dog food) is the best way to verify tool reliability—actually using the formats and tools you advocate in real projects. Only when you truly write documents in a markup language, go through the conversion workflow, and face actual losses can you genuinely understand its strengths and limitations.

Practical Tips for Reducing Pandoc Conversion Loss

For developers and content creators who regularly handle document format conversion, the following strategies can significantly improve conversion quality:

• Prioritize formats with clear syntax rules: If content needs frequent conversion, choosing languages with explicit parsing rules (like CommonMark or Djot) can dramatically reduce loss risk
• Understand the AST's expressive boundaries in advance: Before converting, confirm which elements both the source and target formats support, and avoid using features that cannot be expressed in the target format. For example, Markdown's table syntax loses alignment information when converted to plain text, and footnotes may be downgraded to parenthetical notes when converting to formats that don't support footnotes
• Establish reproducible conversion workflows: Leverage Pandoc's template and filter mechanisms to make complex conversions controllable, maintainable, and traceable. Pandoc filters are essentially programs that transform the AST: Pandoc passes the parsed AST in JSON format to an external filter program, the filter modifies it and returns a new AST, and Pandoc then renders the modified AST into the target format. This "pipeline" architecture allows users to implement complex document transformations without modifying Pandoc's core code. Pandoc supports two filter interfaces: traditional JSON filters (writable in any language) and higher-performance Lua filters (running embedded within the Pandoc process, avoiding JSON serialization overhead)
• Always preserve plain text source files: Regardless of whether the final output is HTML, PDF, or DOCX, you should maintain a readable plain text markup source as your "single source of truth"—this is the fundamental guarantee of content's long-term value

The philosophy of using plain text markup files as the "Single Source of Truth" originates from version control practices in software engineering. Unlike binary formats such as DOCX or PDF, plain text files can be efficiently tracked for differences, merged, and history-reviewed by version control systems like Git. This means every document modification has a complete audit trail, and conflicts during multi-person collaboration can be resolved using standard diff/merge tools. Furthermore, plain text formats can be opened and edited without depending on any specific software, offering extremely high long-term readability—a plain text file from 20 years ago can still be read easily today, but a proprietary format file from 20 years ago may no longer open. This "future-proof" characteristic is especially important for academic publishing, legal documents, technical documentation, and other content that needs long-term preservation.

Markup Languages Continue to Evolve—Understanding Conversion Logic Is Crucial

The release of Pandoc 3.10.2 reminds us once again that document format conversion is far from a solved problem. Markdown's fifteen years of fragmented evolution shows that the path to standardization is full of twists and turns, while the emergence of new languages like Djot demonstrates the community's relentless pursuit of better solutions.

Whether you're a content creator or a developer, understanding "what survives conversion" and the AST design philosophy behind it can help you make wiser choices in the complex markup language ecosystem. The perfect markup language that ideally balances readability, writeability, and machine-readability may still be on its way, but every exploration pushes the entire document tooling ecosystem steadily forward.