Run a 1T parameter model on a 32gb Mac by streaming tensors from NVMe
github.com
March 24, 2026
6 min read
Summary
Hypura is a storage-tier-aware LLM inference scheduler designed for Apple Silicon, allowing users to run large models that exceed their Mac's memory. It optimally distributes model tensors across GPU, RAM, and NVMe storage based on access patterns and hardware capabilities to prevent system crashes.
Key Takeaways
- Hypura is a storage-tier-aware LLM inference scheduler designed for Apple Silicon that allows models exceeding physical memory to run without crashing the system.
- The system optimizes tensor placement across GPU, RAM, and NVMe based on access patterns, achieving significant I/O reduction and enabling efficient model execution.
- Hypura supports multiple inference modes, including full-resident, expert-streaming, and dense FFN-streaming, automatically selecting the best mode based on model size and available hardware resources.
- Benchmarks show that Hypura can run large models like Mixtral 8x7B and Llama 70B on a 32 GB Mac Mini, which would otherwise crash under naive loading methods.
Community Sentiment
MixedPositives
- The project demonstrates an innovative approach to utilizing NVMe storage, effectively extending RAM capabilities for running large models, which could enhance accessibility for users with limited hardware.
- Using a smart method to manage memory allocation could optimize performance, allowing for more efficient use of resources during model inference.
Concerns
- Concerns about the potential stress on NVMe drives during intensive model generation highlight the risks to hardware longevity, which could deter users from adopting this approach.
- The overhead associated with mmap in similar designs raises questions about the efficiency of this method, suggesting it may not perform as well as hoped in practice.
Related Articles
Right-sizes LLM models to your system's RAM, CPU, and GPU
Mar 1, 2026
Flash-MoE: Running a 397B Parameter Model on a Laptop
Mar 22, 2026
Rust implementation of Mistral's Voxtral Mini 4B Realtime runs in your browser
Feb 10, 2026
LLM Neuroanatomy II: Modern LLM Hacking and Hints of a Universal Language?
Mar 24, 2026
A CPU that runs entirely on GPU
Mar 4, 2026
Source
github.com
Published
March 24, 2026
Reading Time
6 minutes
Relevance Score
59/100
Why It Matters
This page is optimized for focused reading: quick context up top, a clean summary block, and a direct path to the original source when you want the full story.