NanoGPT Speedrun Frontier
primeintellect.ai
August 22, 2026
4 min read
48/100
Summary
Prime Intellect ran 153 autonomous nanoGPT optimizer speedruns across 18 frontier-model entries and reported the best validated result for each. Fable 5, using Claude Code at the high setting, ranked first with a validated record of 2,726 and an 81.7% gap closed toward the stated human record of 2,600 from a baseline of 3,290. Its run used 3,010 steps and lasted 8.7 days. Opus 5 placed second at 2,920, closing 53.6% of the gap, followed by Kimi K3 at 2,930 with Prime Agent and 2,974 with Kimi Code. Opus 4.8 recorded 3,018, while GPT-5.6 Sol and GPT-5.6 Sol Pro reached 3,042 and 3,058. Sonnet 5 reached 3,105, and GPT-5.6 Luna reached 3,110. The remaining validated records ranged from 3,120 for Grok 4.5 and Qwen3.8 Max to 3,240 for Kimi K2.7. GLM 5.3 had no recorded result. The results use different agent harnesses and configurations, including Claude Code, Codex, Prime Agent, Kimi Code, Grok CLI, Qwen Code, Pi, and Muse Code. Prime Intellect also provides 41 curated full agent trajectories containing tool calls, subagents, and scratchpads.
Key Takeaways
- Fable 5 achieved the best reported validated nanoGPT optimizer record, 2,726, using Claude Code at the high setting.
- The benchmark identifies a human record of 2,600 and a baseline of 3,290; Fable 5 closed 81.7% of that gap.
- Prime Intellect conducted 153 autonomous runs across 18 frontier-model entries.
- Opus 5 ranked second with a 2,920 record, while Kimi K3 ranked third with a 2,930 record using Prime Agent.
What the discussion said
The thread spent less time celebrating a frontier-model leaderboard than trying to decode what the benchmark actually measures. Several readers found the write-up needlessly opaque: it did not plainly define an autonomous run, the optimizer objective, frozen verification, or why repeated fixed-seed trials represent research ability. Others reconstructed the setup as agents iteratively altering a small model’s training recipe, testing it against a target loss, and retaining successful changes. That made the evaluation sound genuinely useful rather than mystical, but commenters felt the article should have done that explanatory work itself. There is real interest in the task as an AI R&D proxy. Readers liked that it turns optimization work into a concrete, automatically checkable competition, and one low-cost model’s near-Sonnet result was taken as evidence that cheap, fast agents can be highly effective when verification is crisp. The poor result from one competing model drew attention, though the discussion resisted treating it as a clean verdict on raw intelligence. Long-horizon agent training, harness quality, waiting behavior, effort settings, prompt wording, and nonidentical experimental configurations may all move the score. Commenters also wondered whether stronger memory for tentative findings would separate agents more than their ability to discover the same obvious ideas. Overall, the benchmark is seen as promising, but its presentation and comparability need tightening before model rankings deserve much confidence.
Where opinion split
The main dispute is whether the leaderboard reflects models' autonomous research ability or mostly the quality and configuration of the agent harness. Supporters see a difficult, repeatable optimize-test loop with automatic validation as a meaningful R&D task; skeptics argue that unequal settings, long-horizon training exposure, and weak experimental memory can dominate the apparent model differences.
Community Sentiment
Positives
- A concrete loss target and automated verification make this a useful proving ground for agents that must propose, test, and refine ML training changes rather than merely describe them.
- Cheap, fast models can punch near far more expensive systems on tightly verifiable optimization work, because they can cheaply generate and discard many failed experiments.
- Turning practical model-training optimization into a competitive speedrun gives autonomous-research claims a measurable outcome instead of relying on polished demonstrations.
- The substantial compute budget and repeated trials signal an unusually serious attempt to evaluate long-horizon AI research behavior.
Concerns
- The benchmark explanation obscures its core mechanics, leaving readers to infer what a run, fixed seed, frozen validator, and optimizer success actually mean before judging the result.
- One model's weak showing may say as much about its long-horizon agent training and harness as its underlying reasoning, making the ranking easy to overread.
- Different effort settings, serial program versions, waiting behavior, and possibly unequal parallelism undermine clean apples-to-apples comparisons across models.
- A single repeated goal prompt may reward agents that exploit this exact workflow while missing how prompt framing or better retention of weak experimental signals changes research performance.
