AI is changing the world. Don't stay behind. Clear summaries, community insight, delivered without the noise. Subscribe to never miss a beat.

Privacy

Contact

Back to all news

transformers retro-computing ai-development developer-tools

Soul Player C64 – A real transformer running on a 1 MHz Commodore 64

GitHub - gizmo64k/soulplayer-c64: A real 25k-parameter transformer running on a Commodore 64!

github.com

April 20, 2026

5 min read

🔥🔥🔥🔥🔥

44/100

Summary

A transformer model with approximately 25,000 parameters is implemented on an unmodified Commodore 64, utilizing hand-written 6502/6510 assembly. This 2-layer decoder-only architecture features real multi-head causal self-attention, softmax, and RMSNorm, and can be loaded from a floppy disk.

Key Takeaways

A 25,000-parameter transformer model has been successfully implemented on a Commodore 64 using hand-written 6502/6510 assembly code.
The model features a 2-layer architecture with real multi-head causal self-attention and can process approximately one token every 60 seconds.
The implementation includes a key breakthrough in softmax score normalization, allowing for meaningful attention weights in the integer-based model.
Users can train their own models and create custom chat interactions using a provided emotional support corpus and a simple training script.

Read original article

Community Sentiment

Mixed

Positives

The project demonstrates an interesting application of transformer architecture on retro hardware, showcasing creativity in utilizing limited resources.
Running the model under enhanced conditions like SuperCPU with warp mode could significantly improve performance, making it more engaging for users.

Concerns

The model's output is criticized for producing broken and nonsensical sentences, raising doubts about the effectiveness of the architecture at such a small scale.
With only 25K parameters, the model's capabilities are severely limited, leading to a lack of meaningful interactions and comparisons to outdated technologies.