Running a 28.9M parameter LLM on an $8 microcontroller
github.com
July 25, 2026
3 min read
🔥🔥🔥🔥🔥
61/100
Summary
A 28.9 million parameter language model generates text on the ESP32-S3 microcontroller, which costs about $8. The model operates entirely on the chip without server communication, displaying output on a small screen at approximately 9 tokens per second.
Key Takeaways
- The model has 28.9 million parameters and runs entirely on the ESP32-S3 microcontroller without server connectivity.
- It utilizes Google's Per-Layer Embeddings to store most parameters in flash memory, allowing it to operate efficiently with limited SRAM.
- The model was trained on the TinyStories dataset, enabling it to generate coherent short stories but not to answer questions or follow instructions.
- The model achieves a speed of approximately 9 tokens per second and has a size of 14.9MB at 4-bit precision.
Community Sentiment
Positives
- It's impressive what you can achieve with a microcontroller for just $5; the possibilities for running AI models are expanding.
- The use of per-layer embedding tricks opens fascinating avenues for deploying TTS models on low-power devices like the ESP32.
- Running a 28.9M parameter model on a microcontroller showcases how far AI deployment has come, making powerful tech more accessible.
Concerns
- Running models on microcontrollers feels constrained and impractical; they might not be suitable for serious AI tasks.
- The performance metrics are concerning; one user reported 10 seconds per token, which is painfully slow for real applications.
- There's skepticism about scaling to larger models on constrained hardware, which raises doubts about the practicality of these microcontroller setups.
