From Substack Newsletters to a RAG Knowledge Base: A Clean Markdown Pipeline in Practice

Why Substack Is High-Quality RAG Corpus

Substack newsletters are some of the highest-quality expert content on the internet, making them excellent knowledge sources for RAG chatbots or AI research agents. But feeding Substack pages directly into an embedding model causes problems: the HTML is stuffed with "Subscribe now" buttons, share widgets, and image wrappers, and this noise pollutes vector retrieval results. Below is a clean pipeline from archive listing to chunked ingestion, along with hard-won lessons.

Why Substack Is High-Quality RAG Corpus

Substack authors are typically deep practitioners in their fields, their articles are well-structured with clear viewpoints, and they publish at a steady cadence. Compared to fragmented forum discussions, newsletters are better suited as the knowledge foundation for retrieval-augmented generation. The challenge lies in extraction: Substack body content is wrapped in a large amount of UI elements, and using a generic HTML-to-Markdown tool directly yields a pile of meaningless button text and nested links.

The core idea of this pipeline is: first fetch structured metadata via the API, then separately grab the body content, then use turndown for targeted cleaning, and finally chunk semantically while preserving citation metadata.

RAG (Retrieval-Augmented Generation) is an architecture that combines an external knowledge base with a large language model: when a user asks a question, the system first retrieves the most relevant text fragments (chunks) from the corpus, then feeds these fragments as context to the model to generate an answer. Its value lies in compensating for the timeliness and domain-depth limitations of the model's parametric knowledge, and in making answers traceable. RAG's effectiveness depends heavily on corpus quality—if the knowledge source contains UI noise such as navigation bars and subscribe buttons, the embedding vectors get diluted by this meaningless text, and retrieval may recall a bunch of "Subscribe now" fragments, directly polluting the generated output. Therefore the "clean—chunk—ingest" pipeline determines the ceiling of the RAG system, which is the fundamental reason this article emphasizes fetching structured body content via the API rather than scraping the rendered HTML directly.

Step 1: Paginate Through the Article List

Every Substack publication—including custom domains like lennysnewsletter.com—provides an archive JSON endpoint:

curl "https://www.lennysnewsletter.com/api/v1/archive?sort=new&offset=0&limit=50"

Increment offset to page through until an empty array [] is returned. Each entry contains fields such as title, slug, post_date, audience, wordcount, reaction_count, publishedBylines, and postTags, but not the body content. This step is only for building the article index and filtering scope.

Step 2: Fetch the Body and Respect the Paywall

Once you have the slug, request the single-article endpoint:

curl "https://www.lennysnewsletter.com/api/v1/posts/{slug}"

The body_html in the response is the body HTML. There is a key field here, audience: a value of everyone means a free article; paid articles return only preview content. The original author explicitly recommends respecting this restriction—for paid articles, ingest metadata only, and do not attempt to bypass it.

The quality of body_html determines the complexity of subsequent cleaning. Substack body content embeds subscription widgets, button wrappers, and icon containers, all of which need to be stripped during conversion.

A paywall is a mechanism by which content platforms restrict access to body content based on subscription status. Substack uses the audience field in the single-article endpoint to distinguish visibility: everyone means a public article, while other values (such as only_paid, founding) indicate access is limited to paid subscribers, and the endpoint returns only a preview fragment rather than the full body_html. For a RAG pipeline, this means you cannot assume every article's full text is available; in engineering terms, you should ingest audience as metadata alongside the article, keep only index information such as title, summary, and date for paid articles, and mark them as "metadata only" during retrieval. Bypassing the paywall not only involves copyright and platform-terms risks but also casts doubt on the legitimacy of the knowledge base, so "metadata only" is both a compliant and robust default strategy.

Step 3: Convert to Clean Markdown with turndown

Using turndown for HTML-to-Markdown conversion is a mature approach, but running it directly leaves junk behind. The original author summarized two practical lessons:

Lesson 1: Use addRule to discard junk nodes, not remove(). turndown's built-in paragraph rule matches <p class="button-wrapper"> first, so the remove() you register never fires. The correct approach is to add a custom rule that matches by node name and class regex in the filter, then returns an empty string:

td.addRule('junk', {
  filter: (n) => ['FORM', 'BUTTON', 'SVG'].includes(n.nodeName.toUpperCase())
    || /subscription-widget|button-wrapper|captioned-button|icon-container/.test(n.getAttribute?.('class') || ''),
  replacement: () => '',
});

Lesson 2: Unwrap image link wrappers. Every Substack image is wrapped in <a class="image-link">, which converts into an ugly [![](img)](img) nested form. Use a rule to strip the link layer, keeping only the image itself:

td.addRule('imageLink', {
  filter: (n) => n.nodeName === 'A' && /image-link/.test(n.getAttribute('class') || ''),
  replacement: (content) => `\n\n${content.trim()}\n\n`,
});

These two rules plus the default conversion are basically enough to produce semantically clean Markdown body content.

turndown is a JavaScript library that converts HTML to Markdown, driven internally by a set of "rules": each rule contains a filter (determines whether a given DOM node applies) and a replacement (returns the replaced Markdown string). During conversion, turndown matches rules one by one according to registration order and node type, and its built-in rules already cover common elements such as paragraphs, headings, links, and images. addRule is used to append custom rules, which can precisely target nodes by node name, class, or attribute; while remove() directly deletes matching nodes, but it relies on built-in rules not handling that node first—once the built-in paragraph rule matches <p class="button-wrapper"> first, the node has already been converted before reaching your deletion logic, rendering remove() useless. Understanding the order of rule matching is the key to writing effective cleaning rules.

Step 4: Chunk in a Heading-Aware Way

Chunking strategy directly affects retrieval quality. The original article recommends: split paragraphs by blank lines, carry the nearest # heading into each chunk, cap each chunk at about 2,000 characters, and use about 200 characters of overlap between chunks. This heading-aware chunking lets each chunk carry its contextual belonging, avoiding retrieval of isolated fragments.

Metadata is equally important: store url, title, postDate, and heading for each chunk. This way the bot can accurately cite sources when answering, and users can trace back to the original text. For scenarios that require citing sources, this is more valuable than simply improving recall.

Chunking is the process of splitting long documents into small fragments suitable for embedding and retrieval; chunk size and boundaries directly determine retrieval precision. If chunks are too large, vectors mix multiple topics and semantics get averaged out; if too small, context is lost and answers are incomplete. Common strategies include fixed-length splitting, splitting by paragraph/sentence boundaries, and the "heading-aware" splitting used in this article—carrying the nearest Markdown heading as a context prefix into each chunk, so that isolated fragments can still indicate which section they belong to. Overlap between chunks is used to prevent a key sentence from being cut exactly at a boundary and thus incomplete on both sides. Combined with metadata such as url, title, and heading, the model can accurately annotate sources after a retrieval hit, enabling traceable question answering.

Step 5: Incremental Updates Keep It Fresh

The Substack archive endpoint returns newest-first by default. When running a daily incremental task, you only need to page from the first page until you encounter an article ID you've already seen, then stop. This avoids re-scraping everything, keeps costs manageable, and lets new articles enter the knowledge base promptly.

For a personal knowledge base or a small research agent, this pipeline is very cheap to run; for multi-publication aggregation scenarios, you can build separate indexes per domain and then fuse them at the retrieval layer.

Ready-Made Tools and Reference Implementations

If you don't want to write code yourself, the original article provides two entry points:

• Endpoint documentation and full code: github.com/Mersetti/substack-api-guide
• No-code version (supports Markdown/text/HTML output, RAG chunking, filtering, new-article monitoring, paywall-safe mode): Substack to Markdown on Apify

The Apify version is good for quickly validating ideas, while writing your own pipeline is better suited for deeply customizing chunking logic and metadata fields.

Who This Pipeline Is For

Developers building AI research agents, industry-intelligence bots, and personal knowledge bases can all reuse it directly. Substack's content quality plus a clean structured pipeline results in far less noise than scraping generic web pages for RAG. The only boundary you need to hold is paid content—use metadata only, don't touch the body, which is both a compliance requirement and a sign of respect for creators.

What content sources are you using to build your RAG knowledge base? Feel free to share in the comments.