Crawling a billion web pages in just over 24 hours, in 2025
Crawling 1.005 billion web pages took just over 25.5 hours and cost $462. Advances in technology, such as multi-core CPUs and NVMe solid-state drives, have significantly improved web crawling efficiency since 2012.
andrewkchan.dev
15 min
2/23/2026
Crawling a billion web pages in just over 24 hours, in 2025
Crawling 1.005 billion web pages took just over 25.5 hours and cost $462. Advances in technology, such as multi-core CPUs and NVMe solid-state drives, have significantly improved web crawling efficiency since 2012.
andrewkchan.dev
15 min
2/23/2026
Crawling a billion web pages in just over 24 hours, in 2025
Crawling 1.005 billion web pages took just over 25.5 hours and cost $462. Advances in technology, such as multi-core CPUs and NVMe solid-state drives, have significantly improved web crawling efficiency since 2012.
andrewkchan.dev
15 min
2/23/2026
No more articles to load