Cloudflare Crawl Endpoint
developers.cloudflare.com
March 10, 2026
1 min read
🔥🔥🔥🔥🔥
67/100
Summary
The new /crawl endpoint in Browser Rendering allows users to crawl entire websites with a single API call. It discovers and renders pages in a headless browser, returning results in formats such as HTML, Markdown, and structured JSON.
Key Takeaways
- The new /crawl endpoint allows users to crawl entire websites with a single API call using Browser Rendering, currently in open beta.
- Crawled content can be returned in multiple formats, including HTML, Markdown, and structured JSON.
- Users can configure crawl depth, page limits, and include or exclude specific URL paths through crawl scope controls.
- The service honors robots.txt directives and supports incremental crawling to optimize performance and reduce costs.
Community Sentiment
Positives
- The Cloudflare Crawl endpoint utilizes a headless Chrome instance, enabling efficient rendering and fetching of publicly accessible URLs, which can enhance web scraping capabilities.
Concerns
- Concerns arise that Cloudflare's dual role as a provider of scraping countermeasures and a scraping service creates a conflict of interest, potentially undermining trust in their services.
- There is skepticism about Cloudflare's intentions, with some viewing their scraping service as a way to control and monetize access to web content, likening it to a 'mob outfit'.
- The pricing structure for Browser Rendering on the Workers Paid plan raises accessibility concerns, as it limits usage to paying customers, potentially excluding smaller developers.