LeMario: Training a JEPA World Model on Super Mario Bros
benjamin-bai.com
July 14, 2026
11 min read
🔥🔥🔥🔥🔥
49/100
Summary
LeMario is a Joint-Embedding Predictive Architecture (JEPA) model trained on Super Mario Bros to learn world dynamics from pixels and actions. The model successfully passed all initial tests.
Key Takeaways
- The LeMario model is based on a Joint-Embedding Predictive Architecture (JEPA) that learns world dynamics from pixels and actions in Super Mario Bros.
- The model successfully generalized to held-out episodes and predicted five-step futures better than strong baselines, but struggled with navigating toward distant goals.
- The architecture includes a vision encoder and an action encoder that compresses visual frames and button states into latent representations for prediction.
- The causal predictor uses six transformer blocks and incorporates Adaptive LayerNorm Zero (AdaLN-Zero) to modulate the influence of actions on frame predictions.
Community Sentiment
Positives
- This paper shines a light on JEPA's potential, especially how it can predict horizontal position effectively after minimal training — a promising step for model-based control.
- The insights shared by the author about the training process are invaluable, showcasing a real-world application of AI concepts in gaming.
Concerns
- Planning in latent space can be noisy and unrepresentative of actual game positions, which raises concerns about control and optimization in dynamic environments.
- Chunking planning into intermediate goals feels limiting, undermining the essence of model-based control where flexibility is key.
