AI is changing the world. Don't stay behind. Clear summaries, community insight, delivered without the noise. Subscribe to never miss a beat.

Crawling a billion web pages in just over 24 hours, in 2025

andrewkchan.dev

February 23, 2026

15 min read

🔥🔥🔥🔥🔥

57/100

Summary

Crawling 1.005 billion web pages took just over 25.5 hours and cost $462. Advances in technology, such as multi-core CPUs and NVMe solid-state drives, have significantly improved web crawling efficiency since 2012.

Key Takeaways

A web crawler successfully crawled 1.005 billion web pages in 25.5 hours at a cost of $462.
The crawler operated without executing JavaScript, focusing solely on parsing HTML to maintain consistency with older web crawls.
The crawler adhered to politeness protocols, including respecting robots.txt and enforcing delays between requests to avoid overwhelming servers.
The design utilized a cluster of independent nodes, each handling all crawler functions for a shard of domains, optimizing for budget and efficiency.

Read original article

Community Sentiment

Mixed

Positives

Crawling a billion web pages in just over 24 hours showcases significant advancements in web scraping technology, potentially enabling faster data collection for various applications.
Achieving 35k requests per second on a single node with optimized Rust demonstrates the potential for high-performance web crawling solutions that can handle massive datasets efficiently.

Concerns

The challenges of circumventing anti-bot measures like Cloudflare highlight ongoing issues in web scraping, which can hinder access to valuable data and increase operational costs.
Per-domain politeness queuing becomes complex at scale, indicating that existing crawlers may struggle with efficiency and compliance when targeting large volumes of pages.