AI benchmarks
Source reports covering AI benchmarks. Stories will appear when analysis and community reactions are ready.
13 reports · page 1 of 1
Connected topics are temporarily unavailable. You can still browse stories below.
Source reports
90% of US working hours are spent on tasks not covered well by AI benchmarks
research
9 min read
Mercor has published a Benchmark Map that measures how well 29 selected AI benchmarks cover tasks across the US labor force. The map uses the Department of Labor’s O*NET taxonomy, which divides the workforce into 923 occupations, 22 job families, and 18,797 tasks. A task is counted as covered when an LLM judge maps at least five benchmark prompts to it; one prompt can map to multiple tasks. Only 208 of 923 occupations have at least one task meeting that threshold, while six of 22 job families have none. Mercor estimates that 90% of US working hours are spent on tasks with little or no coverage in its initial benchmark set, making economy-wide assessments of AI capability difficult. The organization says benchmark coverage disproportionately favors AI-exposed tasks, with O*NET tasks rated exposed to AI about six times more likely to appear than unexposed tasks. Tasks in occupations with larger wage bills and tasks rated most important are also overrepresented. Mercor found that reweighting APEX benchmark scores so each O*NET task counts equally changed model results little for investment banking, law, accounting, and software engineering. Less than one-third of relevant tasks in those four occupations had sufficient APEX coverage. Mercor characterizes its estimates as provisional because its benchmark list is incomplete, selection criteria involve judgment, and excluding saturated benchmarks can understate coverage.
Models Are Getting Dumber on Purpose
opinion
6 min read
Reasoning scores for AI models are increasing while per-token compute is decreasing. GLM-5.2 achieves 99.2% on AIME 2026 with 40 billion parameters, Qwen3.5 scores 91.3% with 17 billion parameters, and DeepSeek V4-Flash operates with 13 billion parameters, contrasting with GPT-4's rumored 280 billion parameters which struggled with AIME problems.
Anthropic: Introducing The Conceptual Reasoning Index
research
6 min read
The Conceptual Reasoning Index introduces a suite of three benchmarks to evaluate AI capabilities in understanding situations, planning for the future, and developing risk mitigations. Access to the primary conceptual dataset, LMCA, is available upon request.
DeepSeek V4 Flash 0731
research
21 min read
DeepSeek V4 Flash 0731 achieves scores of 89.0% on ARC-AGI-1 Semi-Private at a cost of $0.02 per task and 61.4% on ARC-AGI-2 Semi-Private at $0.04 per task. The model also records scores of 87.0% and 56.0% for the High variant, and 84.0% and 46.0% for the Low variant on the respective benchmarks.
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
research
2 min read
AI benchmarks are essential for assessing model performance and informing deployment choices. However, they often reach saturation, reducing their effectiveness in distinguishing between models and limiting their long-term utility.
What's the largest software project AI can complete on its own?
research
4 min read
MirrorCode is a benchmark designed to evaluate AI models on long-horizon coding tasks by requiring them to reimplement entire programs end-to-end without access to the original source code. Solutions generated by AI must match the output of the original program to be considered successful.
Any text-to-SQL benchmark should address difficulties of real-world data stores
research
1 min read
Text-to-SQL benchmarks must consider the complexities and challenges posed by real-world data stores. Effective evaluation should include factors like data variability, schema complexity, and query performance.
Are AI Labs Pelicanmaxxing?
research
11 min read
Simon Willison has tested major LLM releases using the prompt “Generate an SVG of a pelican riding a bicycle,” creating an informal benchmark in AI. The results often receive significant engagement on platforms like Hacker News, sparking discussions about the benchmark's usefulness.
Kimi K3, and what we can still learn from the pelican benchmark
tool
6 min read
Moonshot AI has launched Kimi K3, their most capable model to date with 2.8 trillion parameters. It is currently accessible via their website and API, with an open weight release scheduled for July 27, 2026.
The gap between open weights LLMs and closed source LLMs
opinion
2 min read
A prediction indicates that a new Frontier Open Source LLM will be released on December 3, 2026. The analysis compares the performance gap between open weights and closed source LLMs by examining historical benchmarks.
GLM-5.2 is the new leading open weights model on Artificial Analysis
tool
3 min read
GLM-5.2 has become the leading open weights model on the Artificial Analysis Intelligence Index, scoring 51. It matches the size of GLM-5.1 with 744 billion total parameters and 40 billion active parameters, but surpasses it by 11 points in the Intelligence Index v4.1.
FrontierCode
tool
13 min read
FrontierCode is a new benchmark designed to evaluate the quality of AI-generated code in production environments. It aims to raise standards beyond mere correctness to assess models' ability to produce high-quality code.
Claude Opus 4.6 accuracy on BridgeBench hallucination test drops from 83% to 68%
news
1 min read
CLAUDE OPUS 4.6 IS NERFED. BridgeBench just proved it. Last week Claude Opus 4.6 ranked #2 on the Hallucination benchmark with an accuracy of 83.3%. Today Claude Opus 4.6 was retested and it fell to #10 on the leaderboard with an accuracy of only 68.3%. A 98% increase in hallucination. bridgebench.ai just confirmed that Claude Opus 4.6 has reduced reasoning levels and is nerfed. Bài đăng Cuộc trò ...