Docs ingestion
Walk documentation trees and push Markdown into your vector store on completion.
Async full-site ingestion
Crawl an entire docs site
Building a RAG corpus or monitoring site? Crawl discovers pages, renders each one, and returns results via webhook when the job finishes.
Works with your agent stack
Use cases
From homepage to full corpus
Single-page extraction is the start. Crawl follows internal links up to your depth and page limits, processing each URL through the render chain.
Blog archives
Capture years of posts in one async job instead of scripting pagination yourself.
Webhook driven
Get notified when the crawl finishes so your pipeline indexes without polling.
Technical approach
Async by design
Crawl returns a job ID immediately. Pages render in the background with the same smart chain as single-url endpoints.
Depth and page caps
Control max_depth and max_pages so crawls stay bounded on large sites.
Include/exclude patterns
Regex filters skip admin paths and focus on the sections you care about.
1 credit per page
Pay per successful render. Failed pages do not burn credits.
Integration
Corpus pipeline
Start crawl
POST seed URL with depth limits. Receive job_id.
View docs →Webhook callback
Your server receives completion payload with URLs and extracted content.
Async mode →Index Markdown
Chunk and embed crawl results into your vector database.
RAG use case →001Is crawl synchronous?
No. You get a job ID and results via webhook or polling when the job completes.
002Can I crawl only /docs paths?
Yes. Use include_pattern and exclude_pattern to scope the crawl.
003What format is each page?
Configure the crawl to extract Markdown, HTML, or screenshots per page depending on your pipeline.
Explore more
Web capture for every workflow
Capture at scale
Directories & marketplaces
Thumbnail every listing. Crawl categories until the full catalog is covered.
Learn more →AI & agentsRAG pipelines
Feed your vector database with fresh, clean web content stripped of boilerplate.
Learn more →AI & agentsAI agents & MCP
Give your AI agent live eyes on the web. One MCP install, ten tools.
Learn more →