HTML to Markdown for LLMs and RAG

The hard part of RAG on live web content is not fetching the page. It is making the page useful once you have it. Raw HTML is built for browsers: layout wrappers, nav trees, cookie banners, ad slots, and class names that mean nothing to a model. Feed that into an embedding step and you are paying to vectorize chrome.

Markdown is the better interchange format for LLMs. Headings stay hierarchical. Paragraphs stay readable. Links keep their destinations. Code blocks stay fenced. The model gets structure without DOM noise.

CaptureWeb's /markdown endpoint converts any URL into that shape: boilerplate removed, main content kept, one credit per successful render.

TL;DR: HTML-to-Markdown for LLMs is not a formatting preference. It is a token and quality decision. CaptureWeb strips nav, ads, footers, and sidebars server-side, preserves headings/links/tables/code, and typically cuts payload size by ~90% on docs-style pages. Same facts, far less context window burned.

Why raw HTML fails agents

When you paste a full HTML document into a prompt or embed it verbatim, most tokens are not content. They are scaffolding: mega-menus, footer sitemaps, "Accept cookies" copy, social share rows, and nested div stacks from the site's component library.

On a typical documentation page we see HTML responses in the 10,000–15,000 token range where the underlying article is often under 1,000 tokens. The sidebar alone can cost more than the answer your user asked for.

That hurts in three places at once:

• Embeddings cluster nav labels next to real paragraphs
• Retrieval surfaces cookie text and footer links as "relevant" chunks
• Generation spends capacity reading markup before reasoning

Pipeline-oriented scrapers optimize for completeness. Agent pipelines optimize for signal per token. Those are different products solving different problems.

What good HTML-to-Markdown conversion keeps

"Strip boilerplate" only helps if you do not strip meaning. Useful LLM Markdown preserves:

• Heading hierarchy so chunk boundaries follow document structure
• Inline and reference links with targets intact for citation and follow-up fetches
• Lists and tables so comparisons and API fields stay structured
• Fenced code blocks so examples do not collapse into prose
• Paragraph breaks that match how humans skim the page

What should disappear: repeated nav links, legal footers, comment widgets, tracking pixels rendered as text, and decorative images that add no semantic value to Q&A.

What CaptureWeb removes automatically

The /markdown endpoint runs after a real render (see below), then converts the page through a readability-style extraction pass:

• Global navigation, breadcrumbs, and footers
• Cookie and consent banners (when they sit outside the main article)
• Ads, chat widgets, and newsletter popups that are not part of the article body
• Duplicate link blocks that appear in both sidebar and body
• Layout-only wrappers that do not carry semantic content

Need the untouched HTML for archival or selector-based scraping? Use /content. Need every format from one browser session? Use /snapshot (screenshot, PDF, HTML, and Markdown for 2 credits).

From URL to Markdown in four steps

1. POST the URL to /v1/markdown with your API key.
2. Render the page. CaptureWeb walks a smart render chain: fast fetch when static HTML is enough, headless browser when JavaScript must run, stealth when bot protection blocks naive requests.
3. Extract main content. The DOM is reduced to the article body, then rewritten as clean Markdown.
4. Return JSON. You get a Markdown string ready to chunk, embed, or pass straight into an agent tool call.

Speed, cost, and context: the three benefits that matter

1. Speed

Smaller strings move faster through every downstream step: chunking, embedding API calls, vector upserts, retrieval, and final generation. Fewer chunks also means fewer nearest-neighbor lookups at query time.

Repeat URLs hit cache at 0 credits. Agents that re-read the same docs, pricing page, or changelog every session should not pay full price on every turn.

2. Cost

Model vendors meter tokens, not intentions. If your retriever injects 12,000 tokens of HTML wrapper around 800 tokens of answer, you pay for all 12,000 on every generation pass that touches that chunk.

Clean Markdown keeps embeddings dense. On docs-style pages, teams typically see on the order of 90% fewer tokens versus raw HTML for the same informational content. That shows up on both embedding bills and completion bills.

3. Context quality

Agents fail in quiet ways when context is polluted. They cite footer links as product features. They summarize cookie policies. They treat sidebar labels as API parameters because those strings sit near real content in embedding space.

Markdown aligned to main content removes that noise. The model reads what a human skimmer would read, not what the front-end framework emitted.

Before and after

Minimal API call

curl -X POST https://api.captureweb.dev/v1/markdown \
  -H "Authorization: Bearer pk_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://docs.stripe.com/api"}'

Example response shape:

{
  "url": "https://docs.stripe.com/api",
  "markdown": "# Authentication\n\nUse Bearer tokens...\n\n[Errors](/docs/errors)",
  "title": "Authentication",
  "word_count": 142,
  "token_estimate": 218,
  "source": "jina",
  "cached": false
}

Chunk the markdown field, embed, store. When the page changes, fetch again. When it has not, cache keeps the call free.

Building a corpus, not just one page

Single-page extraction is the starting point. Production RAG usually needs a site:

• Use /crawl to walk a docs tree or blog archive asynchronously
• Render each URL once, convert to Markdown, push into your vector store
• Re-crawl on a schedule; cache makes incremental updates cheap

Pair with the MCP server so Claude, Cursor, or Codex can call markdown as a tool without you wiring HTTP in every agent framework. Setup notes live in agents.md.

Common questions

Is Markdown better than HTML for LLM input?

For reading, summarizing, and retrieval: yes. HTML carries presentation detail models cannot use. Markdown carries semantics they can. Keep HTML when you need exact DOM structure or visual replay.

Do JavaScript-heavy pages work?

Yes. If a static fetch returns an empty shell, CaptureWeb escalates to a full browser render automatically. You send the same request either way.

What about pages behind login?

Pass a session cookie or authenticated context when your use case requires it. Public URL extraction is the default path for RAG over docs, marketing sites, and changelogs.

What to avoid

• Embedding full HTML and hoping the retriever learns to ignore chrome
• Hand-rolling tag strippers that break on every new site template
• Paying three vendors for Markdown, screenshots, and PDFs from the same browser load
• Skipping cache on URLs your agents request every session

Start with clean input

If you are building RAG on live web data, convert HTML to Markdown at the boundary. Your vector store gets denser vectors, your agents get cleaner context, and your token line item stops smuggling nav bars into every answer.