web-crawling api-development developer-tools data-extraction

Cloudflare Crawl Endpoint

Crawl entire websites with a single API call using Browser Rendering Â· Changelog

developers.cloudflare.com

March 10, 2026

1 min read

🔥🔥🔥🔥🔥

67/100

Summary

The new /crawl endpoint in Browser Rendering allows users to crawl entire websites with a single API call. It discovers and renders pages in a headless browser, returning results in formats such as HTML, Markdown, and structured JSON.

Key Takeaways

The new /crawl endpoint allows users to crawl entire websites with a single API call using Browser Rendering, currently in open beta.
Crawled content can be returned in multiple formats, including HTML, Markdown, and structured JSON.
Users can configure crawl depth, page limits, and include or exclude specific URL paths through crawl scope controls.
The service honors robots.txt directives and supports incremental crawling to optimize performance and reduce costs.

Read original article

View story →

Coverage & discussions

Hacker News · developers.cloudflare.com

Cloudflare Crawl Endpoint ↗

Community discussions

Hacker News discussion ↗

Historical count: 5 · 5 saved in sample · Historical sample

Analysis ready

Community Sentiment

Negative

Positives

The Cloudflare Crawl endpoint utilizes a headless Chrome instance, enabling efficient rendering and fetching of publicly accessible URLs, which can enhance web scraping capabilities.

Concerns

Concerns arise that Cloudflare's dual role as a provider of scraping countermeasures and a scraping service creates a conflict of interest, potentially undermining trust in their services.
There is skepticism about Cloudflare's intentions, with some viewing their scraping service as a way to control and monetize access to web content, likening it to a 'mob outfit'.
The pricing structure for Browser Rendering on the Workers Paid plan raises accessibility concerns, as it limits usage to paying customers, potentially excluding smaller developers.