Cloudflare Crawl Endpoint
developers.cloudflare.com
March 10, 2026
1 min read
Summary
The new /crawl endpoint in Browser Rendering allows users to crawl entire websites with a single API call. It discovers and renders pages in a headless browser, returning results in formats such as HTML, Markdown, and structured JSON.
Key Takeaways
- The new /crawl endpoint allows users to crawl entire websites with a single API call using Browser Rendering, currently in open beta.
- Crawled content can be returned in multiple formats, including HTML, Markdown, and structured JSON.
- Users can configure crawl depth, page limits, and include or exclude specific URL paths through crawl scope controls.
- The service honors robots.txt directives and supports incremental crawling to optimize performance and reduce costs.
Community Sentiment
NegativePositives
- The Cloudflare Crawl endpoint utilizes a headless Chrome instance, enabling efficient rendering and fetching of publicly accessible URLs, which can enhance web scraping capabilities.
Concerns
- Concerns arise that Cloudflare's dual role as a provider of scraping countermeasures and a scraping service creates a conflict of interest, potentially undermining trust in their services.
- There is skepticism about Cloudflare's intentions, with some viewing their scraping service as a way to control and monetize access to web content, likening it to a 'mob outfit'.
- The pricing structure for Browser Rendering on the Workers Paid plan raises accessibility concerns, as it limits usage to paying customers, potentially excluding smaller developers.
Source
developers.cloudflare.com
Published
March 10, 2026
Reading Time
1 minutes
Relevance Score
67/100
🔥🔥🔥🔥🔥
Why It Matters
This page is optimized for focused reading: quick context up top, a clean summary block, and a direct path to the original source when you want the full story.