llms openai developer-tools apple-silicon

TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

GitHub - SharpAI/SwiftLM: ⚡ Native MLX Swift LLM inference server for Apple Silicon. OpenAI-compatible API, SSD streaming for 100B+ MoE models, TurboQuant KV cache compression, + iOS iPhone app.

github.com

April 1, 2026

7 min read

🔥🔥🔥🔥🔥

47/100

Summary

SharpAI's SwiftLM is a native MLX inference server optimized for Apple Silicon, utilizing Metal and Swift for performance. It features an OpenAI-compatible API, supports SSD streaming for 100B+ MoE models, and enables direct loading of HuggingFace format models without a Python runtime.

Key Takeaways

SwiftLM is a native Swift inference server designed for Apple Silicon, providing an OpenAI-compatible API without the need for a Python runtime.
The server supports SSD streaming for 100B+ Mixture of Experts (MoE) models, enabling zero-copy streaming directly from NVMe SSD to GPU.
SwiftLM implements a hybrid TurboQuant architecture that achieves high-quality KV cache compression with minimal accuracy loss, processing dequantization natively in Metal shaders.
A companion iPhone and iPad app allows users to download and run MLX models directly on-device, featuring a user-friendly interface and model catalog.

Read original article

View story →

Coverage & discussions

Hacker News · github.com

TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS ↗

Community discussions

Hacker News discussion ↗

Historical count: 5 · 5 saved in sample · Historical sample

Analysis ready

Community Sentiment

Mixed

Positives

TurboQuant KV compression achieves a remarkable 4.3× cache compression, significantly enhancing the efficiency of running large models on constrained hardware like the M5 Pro.
The implementation of SSD Expert Streaming allows for handling massive 122B parameter models without performance degradation, showcasing innovative approaches to model deployment.

Concerns

The community is flooded with similar projects that lack substantial benchmarks or unique insights, raising concerns about the originality and practical value of these developments.
There is skepticism regarding the quality of some contributions, with claims that many are simply generated by AI without meaningful innovation or improvement.