Story
Crawling a billion web pages in just over 24 hours, in 2025
1 report
1 website
1 discussion
Latest report
Overview
Crawling 1.005 billion web pages took just over 25.5 hours and cost $462. Advances in technology, such as multi-core CPUs and NVMe solid-state drives, have significantly improved web crawling efficiency since 2012.
Summary from andrewkchan.dev.
Reports
One report so far. Additional coverage will appear here when it is linked to this story.
Community reactions
Each reaction summarizes sampled comments from the linked discussion.
Hacker News · andrewkchan.dev
Hacker News discussion ↗
Historical count: 5 · 5 saved in sample · Historical sample
Analysis ready
Read this discussion’s reaction
Based on 5 sampled comments from this thread. Analyzed 2026-02-23.
- Praise: Crawling a billion web pages in just over 24 hours showcases significant advancements in web scraping technology, potentially enabling faster data collection for various applications.
- Praise: Achieving 35k requests per second on a single node with optimized Rust demonstrates the potential for high-performance web crawling solutions that can handle massive datasets efficiently.
- Concern: The challenges of circumventing anti-bot measures like Cloudflare highlight ongoing issues in web scraping, which can hinder access to valuable data and increase operational costs.
- Concern: Per-domain politeness queuing becomes complex at scale, indicating that existing crawlers may struggle with efficiency and compliance when targeting large volumes of pages.