Crawling a billion web pages in just over 24 hours, in 2025
andrewkchan.dev
February 23, 2026
15 min read
🔥🔥🔥🔥🔥
57/100
Summary
Crawling 1.005 billion web pages took just over 25.5 hours and cost $462. Advances in technology, such as multi-core CPUs and NVMe solid-state drives, have significantly improved web crawling efficiency since 2012.
Key Takeaways
- A web crawler successfully crawled 1.005 billion web pages in 25.5 hours at a cost of $462.
- The crawler operated without executing JavaScript, focusing solely on parsing HTML to maintain consistency with older web crawls.
- The crawler adhered to politeness protocols, including respecting robots.txt and enforcing delays between requests to avoid overwhelming servers.
- The design utilized a cluster of independent nodes, each handling all crawler functions for a shard of domains, optimizing for budget and efficiency.
Community Sentiment
Positives
- Crawling a billion web pages in just over 24 hours showcases significant advancements in web scraping technology, potentially enabling faster data collection for various applications.
- Achieving 35k requests per second on a single node with optimized Rust demonstrates the potential for high-performance web crawling solutions that can handle massive datasets efficiently.
Concerns
- The challenges of circumventing anti-bot measures like Cloudflare highlight ongoing issues in web scraping, which can hinder access to valuable data and increase operational costs.
- Per-domain politeness queuing becomes complex at scale, indicating that existing crawlers may struggle with efficiency and compliance when targeting large volumes of pages.
