big-data
Latest stories covering big-data, with summaries and links to related coverage.
3 articles · page 1 of 1
Connected topics
Palantir Wants to Reinstate the Draft
opinion
7 min read
Palantir has proposed a 22-point manifesto advocating for the reinstatement of the military draft, arguing that societal participation in military service should be mandatory. The company is known for its big data analysis tools used in various applications, including predictive policing and military operations.
S3 Files
tool
30 min read
Moving large datasets can be a frustrating process for many professionals. Andy Warfield's experience at UBC involved assisting genomics researchers who generated vast amounts of sequencing data, highlighting the inefficiencies in data transfer.
Crawling a billion web pages in just over 24 hours, in 2025
tool
15 min read
Crawling 1.005 billion web pages took just over 25.5 hours and cost $462. Advances in technology, such as multi-core CPUs and NVMe solid-state drives, have significantly improved web crawling efficiency since 2012.