S3 Files
allthingsdistributed.com
April 7, 2026
30 min read
🔥🔥🔥🔥🔥
60/100
Summary
Moving large datasets can be a frustrating process for many professionals. Andy Warfield's experience at UBC involved assisting genomics researchers who generated vast amounts of sequencing data, highlighting the inefficiencies in data transfer.
Key Takeaways
- S3 Files was developed to address the challenges of moving large datasets efficiently, particularly in genomics research.
- The integration of S3 with serverless compute allows researchers to run complex analyses in parallel and scale resources down when not in use.
- A significant issue faced by researchers was the friction at the storage boundary, as existing tools required local Linux filesystems, leading to data management complications.
- The solution aimed to enhance velocity, repeatability, and performance in genomic data analysis by leveraging cloud capabilities.
Community Sentiment
Concerns
- The high costs associated with EFS, such as $0.06/GB for writes, could deter users from adopting S3 Files for write-heavy applications, impacting its usability.
- S3's immutability means that even minor changes to large files require re-uploading the entire file, limiting S3 Files' effectiveness for certain workflows.
- The absence of atomic rename support in S3 Files poses significant challenges for applications like Claude Code, as it could lead to inefficient data management and increased overhead.
- The need for a managed bridging solution to local NVMe storage highlights the limitations of current AWS offerings, suggesting that performance could be significantly improved.
