DuckDB provides in-memory column-major data processing tools optimized for laptops, now available in Clojure. The tech.ml.dataset (TMD) platform allows users to handle large datasets by operating on samples or filtering subsets, and supports data persistence with nippy, arrow, or par.

Crawling 1.005 billion web pages took just over 25.5 hours and cost $462. Advances in technology, such as multi-core CPUs and NVMe solid-state drives, have significantly improved web crawling efficiency since 2012.