Crawling a billion web pages in just over 24 hours, in 2025
andrewkchan.dev
February 23, 2026
15 min read
Summary
Crawling 1.005 billion web pages took just over 25.5 hours and cost $462. Advances in technology, such as multi-core CPUs and NVMe solid-state drives, have significantly improved web crawling efficiency since 2012.
Key Takeaways
- A web crawler successfully crawled 1.005 billion web pages in 25.5 hours at a cost of $462.
- The crawler operated without executing JavaScript, focusing solely on parsing HTML to maintain consistency with older web crawls.
- The crawler adhered to politeness protocols, including respecting robots.txt and enforcing delays between requests to avoid overwhelming servers.
- The design utilized a cluster of independent nodes, each handling all crawler functions for a shard of domains, optimizing for budget and efficiency.
Community Sentiment
MixedPositives
- Crawling a billion web pages in just over 24 hours showcases significant advancements in web scraping technology, potentially enabling faster data collection for various applications.
- Achieving 35k requests per second on a single node with optimized Rust demonstrates the potential for high-performance web crawling solutions that can handle massive datasets efficiently.
Concerns
- The challenges of circumventing anti-bot measures like Cloudflare highlight ongoing issues in web scraping, which can hinder access to valuable data and increase operational costs.
- Per-domain politeness queuing becomes complex at scale, indicating that existing crawlers may struggle with efficiency and compliance when targeting large volumes of pages.
Source
andrewkchan.dev
Published
February 23, 2026
Reading Time
15 minutes
Relevance Score
57/100
🔥🔥🔥🔥🔥
Why It Matters
This page is optimized for focused reading: quick context up top, a clean summary block, and a direct path to the original source when you want the full story.