MTG Bench: Testing how well LLMs can play Magic

mtgautodeck.com

June 11, 2026

5 min read

🔥🔥🔥🔥🔥

45/100

Summary

Fable 5 successfully plays a scry land and examines the top card of the deck, while Gemini 3.5 performs a complex turn involving scry, discover, and tutor effects. The benchmark tests the capability of LLMs to play Magic: The Gathering without relying on a rules engine, suggesting that high-performing LLMs should not require one.

Key Takeaways

The MTG Bench tests the performance of large language models (LLMs) in playing Magic: The Gathering, highlighting both successful and failed simulations.
LLMs demonstrated better performance in evaluating the legality of simulated turns than in executing legal turn simulations.
Using a remote MCP server for LLM API calls allows for cost savings by reducing the number of cached input token charges.
The benchmark penalizes models that excessively call tools, as mistakes in card drawing cannot be easily undone in the context of Magic: The Gathering gameplay.

Read original article

View story →

Coverage & discussions

Hacker News · mtgautodeck.com

MTG Bench: Testing how well LLMs can play Magic ↗

Community discussions

Hacker News discussion ↗

Historical count: 5 · 5 saved in sample · Historical sample

Analysis ready

Community Sentiment

Mixed

Positives

This benchmark is timely and addresses the shortcomings of existing benchmarks, highlighting the complexity of Magic's mechanics and edge cases.
Testing LLMs with maximum thinking time and web search capabilities showed they could follow rules better than average players, suggesting potential for advanced AI in gaming.
The exploration of letting LLMs play Magic like humans could lead to more engaging gameplay experiences, emphasizing natural interaction over rigid rule enforcement.

Concerns

Concerns were raised about the reliability of LLM grading, suggesting that using a rules engine might yield more consistent results than relying on LLMs alone.
The scoring method based on a simple prompt may not adequately capture the quality of the simulation, indicating potential biases in evaluation.