Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
neomindlabs.com
July 15, 2026
10 min read
🔥🔥🔥🔥🔥
59/100
Summary
Gemma 4, a 26-billion-parameter open-weights mixture-of-experts model, runs on a 13-year-old Xeon server without a GPU at a speed of approximately five tokens per second. The server is a repurposed HP StoreVirtual storage box originally designed for disk storage.
Key Takeaways
- Gemma 4, a 26-billion-parameter mixture-of-experts model, runs at approximately five tokens per second on a 13-year-old Xeon server without a GPU.
- The server used for this experiment is a repurposed HP StoreVirtual storage box with dual Ivy Bridge Xeons and DDR3 memory, costing under $300.
- The successful operation of Gemma 4 required modifications to the code to accommodate the older CPU architecture, specifically avoiding reliance on newer instruction sets like AVX2 and FMA3.
- The project demonstrates that it is possible to run modern AI models on outdated hardware, providing an alternative to subscription-based AI services.
Community Sentiment
Positives
- Running advanced models like Qwen3.6-35B locally at 7-9 tokens/second on a 16GB Mac is a game changer for accessibility — it shows just how far we’ve come in AI efficiency.
- The emergence of models like Prism's Bonsai 27B, running at 44+ tokens/sec on a consumer laptop, suggests we're on the brink of a new era in AI capabilities.
- With local models processing tokens at impressive speeds, many are excited about the potential for rapid advancements in AI applications without the need for expensive cloud services.
Concerns
- The cost comparison of running models locally versus using inference providers raises eyebrows — local setups can be 30x more expensive, which is a significant barrier for many.
- Some commenters express skepticism about the feasibility of running >200B MoE models on basic hardware, pointing to challenges in parameter compression and speed limitations.
- Concerns about the actual performance of older hardware when running these models highlight the potential pitfalls of relying on outdated technology for cutting-edge AI tasks.
