DeepSeek V4 Flash 0731 achieves scores of 89.0% on ARC-AGI-1 Semi-Private at a cost of $0.02 per task and 61.4% on ARC-AGI-2 Semi-Private at $0.04 per task. The model also records scores of 87.0% and 56.0% for the High variant, and 84.0% and 46.0% for the Low variant on the respective benchmarks.

AI benchmarks are essential for assessing model performance and informing deployment choices. However, they often reach saturation, reducing their effectiveness in distinguishing between models and limiting their long-term utility.

MirrorCode is a benchmark designed to evaluate AI models on long-horizon coding tasks by requiring them to reimplement entire programs end-to-end without access to the original source code. Solutions generated by AI must match the output of the original program to be considered successful.

Text-to-SQL benchmarks must consider the complexities and challenges posed by real-world data stores. Effective evaluation should include factors like data variability, schema complexity, and query performance.

Simon Willison has tested major LLM releases using the prompt “Generate an SVG of a pelican riding a bicycle,” creating an informal benchmark in AI. The results often receive significant engagement on platforms like Hacker News, sparking discussions about the benchmark's usefulness.

Moonshot AI has launched Kimi K3, their most capable model to date with 2.8 trillion parameters. It is currently accessible via their website and API, with an open weight release scheduled for July 27, 2026.

A prediction indicates that a new Frontier Open Source LLM will be released on December 3, 2026. The analysis compares the performance gap between open weights and closed source LLMs by examining historical benchmarks.

GLM-5.2 has become the leading open weights model on the Artificial Analysis Intelligence Index, scoring 51. It matches the size of GLM-5.1 with 744 billion total parameters and 40 billion active parameters, but surpasses it by 11 points in the Intelligence Index v4.1.

FrontierCode is a new benchmark designed to evaluate the quality of AI-generated code in production environments. It aims to raise standards beyond mere correctness to assess models' ability to produce high-quality code.

CLAUDE OPUS 4.6 IS NERFED. BridgeBench just proved it. Last week Claude Opus 4.6 ranked #2 on the Hallucination benchmark with an accuracy of 83.3%. Today Claude Opus 4.6 was retested and it fell to #10 on the leaderboard with an accuracy of only 68.3%. A 98% increase in hallucination. bridgebench.ai just confirmed that Claude Opus 4.6 has reduced reasoning levels and is nerfed. Bài đăng Cuộc trò ...