data-processing
Latest stories covering data-processing, with summaries and links to related coverage.
2 articles · page 1 of 1
Connected topics
DuckDB – Data power tools for your laptop, now in Clojure (2023)
tool
8 min read
DuckDB provides in-memory column-major data processing tools optimized for laptops, now available in Clojure. The tech.ml.dataset (TMD) platform allows users to handle large datasets by operating on samples or filtering subsets, and supports data persistence with nippy, arrow, or par.
Crawling a billion web pages in just over 24 hours, in 2025
tool
15 min read
Crawling 1.005 billion web pages took just over 25.5 hours and cost $462. Advances in technology, such as multi-core CPUs and NVMe solid-state drives, have significantly improved web crawling efficiency since 2012.