llms developer-tools ai-inference machine-learning-models

Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

GitHub - sqliteai/waste: Run the full 2.78-trillion-parameter Kimi K3 model beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine.

github.com

July 31, 2026

12 min read

🔥🔥🔥🔥🔥

63/100

Summary

WASTE is an embeddable C inference engine that allows the Kimi K3 model, which has 2.78 trillion parameters, to run on consumer laptops by streaming activated weights directly from NVMe storage. It operates without third-party runtime dependencies and manages memory by keeping the model trunk in memory while streaming selected experts from disk.

Key Takeaways

The Kimi K3 model has 2.78 trillion parameters and runs on a consumer laptop with 64 GB of RAM, utilizing an embeddable C inference engine called WASTE.
WASTE streams activated weights directly from NVMe storage, allowing the model to operate without fitting entirely in RAM, achieving a speed of approximately 0.49–0.54 tokens per second.
The Kimi K3 model, after conversion, occupies 982 GiB and is designed to activate only about 4% of its parameters per token, keeping the majority of its weight on disk.
The implementation demonstrates that large-scale models can be run on consumer hardware without relying on external APIs, enabling local processing of data.

Read original article

View story →

Coverage & discussions

Hacker News · github.com

Run Kimi K3 using 29 GB of RAM at 0.50 tok/s ↗

Community discussions

Hacker News discussion ↗

Historical count: 5 · 5 saved in sample · Historical sample

Analysis ready

Community Sentiment

Negative

Positives

The 3-bit quantization claim has potential, sparking interest in how this could optimize performance without a massive resource footprint.
Some commenters are excited about using the model for long-running tasks, turning idle time on laptops into productive sessions.

Concerns

The project documentation feels like a jumble of contradictions, leaving users questioning the model's actual capabilities and precision.
Critics are skeptical about the headline claim of achieving 0.50 tok/s with only 29GB of RAM, labeling it as nonsensical based on existing knowledge of similar models.
There's a palpable frustration over the quality of the README, with users noting that LLM-generated documentation often lacks clarity and context for human readers.