How Many Tokens Does It Cost to Feed a Webpage to an LLM? 20 Real Pages Tested

If you're building a RAG pipeline or an Agent that can read webpages, every page eventually turns into input tokens: at the embedding stage, when stuffed into the context window, and again on every query. What form you hand to the model directly determines your cost. Rather than estimating, someone ran a systematic comparison across 20 real pages.

Core Finding: Raw HTML Is the Most Expensive Form

The experiment selected 20 public German-language pages, mixing 6 start/overview pages (dense with menus and recommendation slots) and 14 article pages (news, encyclopedia, documentation, legal text, recipes, guides), sourced from Wikipedia, MDN, tagesschau.de, heise online, the German Environment Agency, gesetze-im-internet.de, Chefkoch, DHL, Siemens, and others. Each page was saved in four forms: raw HTML, full-page-to-text Markdown, main-content-only Markdown, and LLM-optimized Markdown further cleaned for LLMs.

The results show that the median page in the sample had roughly 128,000 tokens of raw HTML—a single page could fill a 128k context window. The LLM-optimized Markdown form, by contrast, saves a median of 98% of tokens versus raw HTML, and 63% versus full-page-to-text.

What is a token, and why does it determine cost?

A token is the smallest unit of text that a large language model processes. Models don't read characters directly; they first split text into token sequences per tokenizer rules, then convert them into vectors for computation. An English word typically maps to one or two tokens, while an HTML tag like <div class="nav"> might be split into a dozen or more. Both billing and context window limits are measured in tokens: APIs charge by input and output token count, and the maximum context length the model can "remember" is also counted in tokens. So different representations of the same page—raw HTML versus cleaned Markdown—differ little semantically but can vary by dozens of times in token count, which maps directly to your bill and how many pages you can process. Understanding this is the prerequisite for judging whether a RAG pipeline is efficient.

How Big Is the Token Gap Across the Four Forms

In total, the raw HTML of the 20 pages exceeds 3.71 million tokens, full-page-to-text drops to about 243,000, main-content-only Markdown to about 110,000, and LLM Markdown further compresses to about 95,000. On a median page basis, raw HTML is about 128,051 tokens, full-page text about 10,917, main-content-only about 3,241, and LLM Markdown about 2,948.

This means that simply "converting HTML to plain text" cuts overhead by an order of magnitude, and "main content extraction + cleaning" compresses the remaining overhead by more than half again.

Savings Come From Two Stages: Main Content Extraction Is the Main Lever

Broken down, the bulk of the savings comes from main content extraction—dropping navigation, footers, cookie notices, recommendation slots—which alone contributes a median 51% savings versus full-page text. The subsequent extra cleaning (removing residual noise, adding a short source header) contributes about 9% more. This breakdown is crucial: most of the gain actually comes from any competent main content extractor, not from some secret cleaning technique.

Main content extraction and boilerplate

A webpage typically consists of "main content" and "boilerplate." Main content is what the user actually wants to read—article paragraphs, recipe steps, or legal clauses; boilerplate is the navigation bar, breadcrumbs, footer links, cookie consent banners, ad slots, related recommendations, and other structures repeated across every page. Main content extraction is the process of automatically identifying and keeping only the main content, discarding boilerplate. It is not the same as "stripping HTML tags"—the latter only removes angle brackets, leaving menu and footer text in the result. Mainstream approaches include heuristics based on DOM density and text density, Readability-style libraries, and in recent years semantic judgment using small models. Extraction quality directly sets the ceiling on token savings and also affects retrieval quality: once boilerplate text enters the vector index, it competes with genuinely relevant content for similarity scores.

Page Type Determines the Magnitude of Savings

The savings ratio varies widely: versus full-page text, savings range from 18% to 90%. Portal pages and overview pages save the most, while long, clean full articles save the least.

Page type	vs raw HTML	vs full-page text
All 20 pages	98%	63%
Start pages (6)	99%	64%
Article pages (14)	98%	59%

Looking at individual pages, a guide page from the German consumer advice center saved only 17.6%, while a Chefkoch start page saved 90.3%, and the Wikipedia Big Data entry saved 37.8%. A median hides this variation, so plan by range in practice.

Saving Tokens Isn't Just About Saving Money

On a median page estimate, full-page text is about 10,900 tokens and LLM Markdown about 2,900 tokens, saving roughly 8,000 input tokens per page per pass. At an indicative price of about $1 per million input tokens, that's about $8 saved per 1,000 pages per pass—a RAG pipeline pays this once at the embedding stage, and an Agent pays it again every time it stuffs the page into the context window.

But cost is only one aspect. Fewer tokens mean less noise in the context, so the model doesn't have to dig through irrelevant content; the vector index also contains less irrelevant text competing with the target passage. The impact on retrieval quality may be more worth watching than the bill.

Tokenizer differences and o200k_base

The same text yields different token counts under different tokenizers. o200k_base is the tokenizer OpenAI introduced for the GPT-4o series, with a vocabulary of about 200,000, and its compression efficiency for multilingual (especially non-English) text is notably better than the earlier cl100k_base. The article counts tokens with o200k_base, so absolute token counts change when the model switches: with a tokenizer that has a smaller vocabulary, the same page may have more tokens; with a tokenizer more "friendly" to HTML, raw HTML's inflation may also differ. But research repeatedly shows that the relative ratios between different representation forms are far more stable than absolute values—the conclusion that raw HTML is one to two orders of magnitude more expensive than cleaned Markdown will not be reversed by changing tokenizers. For budgeting, planning by ratio and validating with a specific tokenizer is the more prudent approach.

The Boundaries of the Test and Caveats

The author is the founder of Ragfeed—a Firecrawl-compatible, Germany-hosted web data API, and the four variants above were generated with it, with "LLM Markdown" being its llmMarkdown format. The scripts, page list, and raw data are all public on GitHub, so you can reproduce it yourself, or even compare using another Firecrawl-compatible API (the --ohne-llm flag).

The limitations of the experiment itself are also clearly stated: the sample is only 20 pages, all German-language sites, and English-language sites might differ; token counting is based on o200k_base (GPT-4o's tokenizer), so absolute values change with the tokenizer, though ratios change far less; pages change, and this is a snapshot from October 1, 2026; "full-page text" is their own full-page Markdown conversion, not any specific third-party tool; the experiment did not benchmark other extraction services or libraries, comparing page representation forms rather than products.

The final caveat is especially important: token count does not equal answer quality. The author also ran a separate retrieval comparison based on title-based chunking versus fixed-length chunking (58 questions, one embedding model), which is a separate, smaller-scale study.

Practical Advice for RAG and Agent Developers

Never throw raw HTML directly at the model. Even the leanest page in the sample (a legal clause, 1,265 tokens of HTML) lost 72%. A typical page's HTML is 100,000 to 500,000 tokens, while the actual content is only a few thousand tokens—the rest is all scripts, inline styles, tracking code, and tags.

"Converting to text" is not the same as trimming. HTML-to-text removes tags, but menus, footer links, cookie notices, and recommendation slots all remain, and on a median page this is still more than three times the actual content tokens.

Confirm your extractor is really doing main content extraction. Every tool should be checked as to whether it truly drops boilerplate, and its output spot-checked—an extractor that cuts too aggressively is worse than one that cuts too little.

Plan your budget by page type, not by a single median. Portal pages and overview pages benefit the most, long, clean articles the least, so plan by range rather than average.