The HTML-to-Markdown Indentation Trap: Why Code Blocks Inside Lists Disappear

A seemingly normal conversion result hides a silent loss

Converting HTML to Markdown usually looks fine: headings are fine, lists are fine, links are fine. But there is a class of bug that is extremely insidious—the output throws no errors, isn't truncated, and yet content quietly vanishes.

A developer spent nearly a week debugging his own HTML-to-Markdown converter before tracking down a problem hidden inside nested lists: fenced code blocks inside list items were being converted into ordinary paragraphs. Top-level code blocks were fine, lists were fine, but the most common structure in GitHub READMEs—"a code block wrapped inside a list item"—was being silently flattened.

Indentation is not a formatting detail, but a grammar contract

The problem lies in the "re-indentation" step

After converting the DOM to Markdown, the author ran a re-indentation pass to align nested content with the list depth. That's where the problem was: Markdown requires that a fenced code block inside a list item be indented to align with the column where the list item's content starts—for an unordered list - marker, that's 4 spaces, or the marker width plus one.

But his implementation re-indented based on "visual column position," not AST depth. As a result, a code fence at list depth 2 was indented by 3 spaces instead of 4. The CommonMark parser then treated that fence as a "lazy continuation" of the preceding paragraph—not a syntax error, but the code block simply disappeared.

The fix is plain, but the direction is right

The real fix isn't fancy: explicitly track the list item content column during DOM traversal, rather than handling it after the fact. Compute the required indentation for each block based on the parent list marker, and apply that indentation to every line of the fence, not just the first.

This kind of bug only surfaces when a human manually diffs the output against the source—and suddenly you notice a function definition is missing. The author admits this is already the second time he's been burned by "treating Markdown as a plain-text format," when it is essentially a whitespace-sensitive grammar.

Lazy continuation is an easily overlooked parsing rule in the CommonMark spec. Inside a list item, if a line isn't properly indented to the list item's content column but can still be parsed as a continuation of paragraph text, the parser won't error out—it will keep treating it as part of the preceding paragraph. That means an under-indented fenced code block opening line ``` gets swallowed into the paragraph, causing the entire code block to collapse into plain text. This is not a syntax error, so most converters, linters, and rendering pipelines emit no warning at all. To avoid this silent failure, you must understand that Markdown's whitespace rules are essentially declaring structural hierarchy—the amount of indentation directly determines node ownership.

Why this class of bug is especially dangerous for RAG and Agent pipelines

In his summary, the author emphasizes one key judgment: if you're building a "document-to-text" pipeline for an agent's context window, this kind of bug is among the most insidious, because the output looks reasonable.

No errors, no truncation—just content silently missing. And your RAG pipeline will, without any awareness, take a README missing its usage examples and embed it—retrieval, recall, and generation all work, but the foundation for the answer is already incomplete.

The core lesson is just one sentence: never do post-processing indentation on generated Markdown. Do it right once at the emission stage, from the parse tree, while the nesting depth is still known.

The author ultimately packaged the whole converter into an online tool for direct use.

Embedding is the process of mapping text into high-dimensional vectors, and RAG (retrieval-augmented generation) pipelines rely on it to compute semantic similarity between queries and document chunks. When code examples in a README are silently flattened into ordinary paragraphs, the embeddings can still be computed successfully—the model won't detect the missing content. The problem is that code examples usually carry the most specific and actionable semantic information; once missing, retrieved chunks become semantically biased toward descriptive prose and lose the actual usage. The answer a user gets may be grammatically correct and fluent in tone, yet lack critical executable details. What's trickier is that this kind of degradation can't be detected by monitoring embedding dimensions or vector similarity thresholds—it can only be exposed by semantically comparing the source document with the conversion result.

Three actionable recommendations for developers building documentation pipelines

1. Base indentation on AST depth, not visual columns: Determine the required indentation when each node is emitted, and apply it to all lines of that block.
2. Make "code blocks inside list items" a mandatory test case: This structure is extremely common in READMEs, tutorials, and API docs, yet most likely to be flattened during conversion.
3. Diff the output against the source, not just eyeball it: This kind of silent loss throws no errors—only comparison can catch "a piece of code has disappeared."

Markdown indentation was never a formatting preference; it is a contract. The price of breaking the contract is often the content itself.