An SLM trained on $8 ESP32-S3
github.com
August 5, 2026
12 min read
🔥🔥🔥🔥🔥
45/100
Summary
Carloscodix/qapla is a char-level transformer model trained from scratch on an $8 ESP32-S3 chip, capable of running the full training loop with backpropagation implemented in C. This project demonstrates that training a model does not always require expensive GPUs or datacenters, as a low-cost microcontroller can suffice.
Key Takeaways
- The ESP32-S3 microcontroller can train a char-level transformer model from scratch without relying on external GPUs or datacenters.
- The training process includes forward pass, backpropagation, and weight updates executed entirely on the chip.
- The project explores the potential of on-device training for applications where data is not available until the device is deployed, such as monitoring machinery or soil conditions.
- Constraints of the ESP32-S3, including limited memory and model size, necessitate a focus on small, specific tasks and clean, structured training data.
Community Sentiment
Positives
- This project is a fascinating blend of human ingenuity and AI assistance — it’s a testament to how collaboration can push boundaries in AI development.
- The idea of clustering ESP32-S3s opens up exciting possibilities for distributed AI tasks, which could democratize access to powerful computing.
- The emphasis on manually derived gradients shows a unique approach that may inspire others to explore unconventional methods in AI training.
Concerns
- Skepticism looms over the practicality of using Klingon as a training target — some see it as a gimmick rather than a serious application.
- Doubts arise about the effectiveness of the project, with some commenters questioning whether the approach truly demonstrates meaningful AI capabilities.
