Benchmarks
Source reports covering Benchmarks. Stories will appear when analysis and community reactions are ready.
8 reports · page 1 of 1
Connected topics are temporarily unavailable. You can still browse stories below.
Source reports
Blatant AI slop just won a 25k USD DeepMind Kaggle Grand Prize
research
1 min read
High-quality benchmarks are being designed to assess cognitive abilities of frontier models, focusing on reasoning, acting, and judgment rather than just recall. These benchmarks aim to measure progress toward artificial general intelligence (AGI).
How We Broke Top AI Agent Benchmarks: And What Comes Next
research
18 min read
Automated scanning reveals that top AI models frequently achieve high benchmark scores that do not accurately reflect their capabilities. The reliance on these benchmarks has led to a misrepresentation of model performance in the AI industry.
ARC-AGI-3
research
1 min read
ARC-AGI-3 is the first interactive reasoning benchmark designed to evaluate human-like intelligence in AI agents. It requires agents to explore novel environments, acquire goals dynamically, build adaptable world models, and learn continuously, with a perfect score indicating performance that matches or exceeds human efficiency in every game.
Study: Self-generated Agent Skills are useless
research
2 min read
SkillsBench is a benchmarking framework designed to evaluate the effectiveness of agent skills across 86 tasks in 11 domains. It includes curated skills and deterministic verifiers to assess their impact on large language model (LLM) agents during inference.
Frontier AI agents violate ethical constraints 30–50% of time, pressured by KPIs
research
2 min read
A new benchmark evaluates outcome-driven constraint violations in autonomous AI agents to enhance safety and alignment with human values. This benchmark addresses limitations of existing safety assessments that mainly focus on harmful actions.
A real-world benchmark for AI code review
8 min read
Qodo's code review benchmark 1.0 provides a rigorous methodology to objectively measure and validate the performance of AI-powered code review systems. The benchmark addresses limitations in existing methods that rely on backtracking from fix commits.
Advancing AI Benchmarking with Game Arena
5 min read
Google DeepMind and Kaggle launched Game Arena, a public benchmarking platform for AI models to compete in strategic games, starting with chess. The platform aims to develop AI capable of making decisions in environments with incomplete information.
Which AI Lies Best? A game theory classic designed by John Nash
3 min read
Gemini 3 utilizes the So Long Sucker game, a benchmark for AI deception, negotiation, and trust, originally designed by John Nash and others in 1950. The game involves four players using colored chips and requires betrayal for a player to win, enabling the assessment of AI capabilities that traditional benchmarks do not evaluate.