Sakana Fugu is a multi-agent system that autonomously enhances a small GPT's training recipe using AutoResearch, which iteratively edits training code and conducts experiments. The AI agent completed 123 experiments over approximately 14 hours on a single H100 GPU, tracking improvements in validation bits-per-byte (BPB).

The autoresearch repository allows an LLM agent to optimize hyperparameters by directly editing training code. A study compares classical hyperparameter optimization algorithms with LLM-based methods for tuning a small language model's hyperparameters.

Karpathy's Autoresearch utilizes a constrained optimization loop with a large language model (LLM) agent. The author applied Autoresearch to legacy code from eCLIP while managing household tasks.

Claude Code was given access to 16 GPUs on a Kubernetes cluster and submitted approximately 910 experiments over 8 hours. It determined that scaling model width was more significant than any single hyperparameter and achieved a 2.87% improvement in validation performance, reducing val_bpb from 1.003 to 0.974.