web-crawling
Latest stories covering web-crawling, with summaries and links to related coverage.
3 articles · page 1 of 1
Connected topics
TIME Is Serving AI Bots a Different Website, with Ads Built In
news
4 min read
TIME serves two different versions of its website: one for human readers and a stripped-down markdown version for AI crawlers that includes embedded ads. GPT-5.6 Sol xhigh utilizes more than double the tokens per session compared to GPT-5.5 xhigh in Codex workflows.
Cloudflare Crawl Endpoint
tool
1 min read
The new /crawl endpoint in Browser Rendering allows users to crawl entire websites with a single API call. It discovers and renders pages in a headless browser, returning results in formats such as HTML, Markdown, and structured JSON.
Crawling a billion web pages in just over 24 hours, in 2025
tool
15 min read
Crawling 1.005 billion web pages took just over 25.5 hours and cost $462. Advances in technology, such as multi-core CPUs and NVMe solid-state drives, have significantly improved web crawling efficiency since 2012.