Themata.AI
Themata.AI

Popular tags:

#developer-tools#ai-agents#discussion#trending#llms#ai-ethics#claude#code-generation#ai-safety#openai

AI is changing the world. Don't stay behind. Clear summaries, community insight, delivered without the noise. Subscribe to never miss a beat.

© 2026 Themata.AI • All Rights Reserved

Archive

|

Topics

|

Privacy

|

Cookies

|

Contact
🕒 Latest🔥 Top

Filtering by tag:

job-automationClear
A benchmark map of the US economy | Mercor
ai-benchmarkslabor-market-analysiseconomic-impact-of-aijob-automation
Research

90% of US working hours are spent on tasks not covered well by AI benchmarks

Mercor has published a Benchmark Map that measures how well 29 selected AI benchmarks cover tasks across the US labor force. The map uses the Department of Labor’s O*NET taxonomy, which divides the workforce into 923 occupations, 22 job families, and 18,797 tasks. A task is counted as covered when an LLM judge maps at least five benchmark prompts to it; one prompt can map to multiple tasks. Only 208 of 923 occupations have at least one task meeting that threshold, while six of 22 job families have none. Mercor estimates that 90% of US working hours are spent on tasks with little or no coverage in its initial benchmark set, making economy-wide assessments of AI capability difficult. The organization says benchmark coverage disproportionately favors AI-exposed tasks, with O*NET tasks rated exposed to AI about six times more likely to appear than unexposed tasks. Tasks in occupations with larger wage bills and tasks rated most important are also overrepresented. Mercor found that reweighting APEX benchmark scores so each O*NET task counts equally changed model results little for investment banking, law, accounting, and software engineering. Less than one-third of relevant tasks in those four occupations had sufficient APEX coverage. Mercor characterizes its estimates as provisional because its benchmark list is incomplete, selection criteria involve judgment, and excluding saturated benchmarks can understate coverage.

mercor.com

🔥🔥🔥🔥🔥

9 min

8h ago

90% of US working hours are spent on tasks not covered well by AI benchmarks

Mercor has published a Benchmark Map that measures how well 29 selected AI benchmarks cover tasks across the US labor force. The map uses the Department of Labor’s O*NET taxonomy, which divides the workforce into 923 occupations, 22 job families, and 18,797 tasks. A task is counted as covered when an LLM judge maps at least five benchmark prompts to it; one prompt can map to multiple tasks. Only 208 of 923 occupations have at least one task meeting that threshold, while six of 22 job families have none. Mercor estimates that 90% of US working hours are spent on tasks with little or no coverage in its initial benchmark set, making economy-wide assessments of AI capability difficult. The organization says benchmark coverage disproportionately favors AI-exposed tasks, with O*NET tasks rated exposed to AI about six times more likely to appear than unexposed tasks. Tasks in occupations with larger wage bills and tasks rated most important are also overrepresented. Mercor found that reweighting APEX benchmark scores so each O*NET task counts equally changed model results little for investment banking, law, accounting, and software engineering. Less than one-third of relevant tasks in those four occupations had sufficient APEX coverage. Mercor characterizes its estimates as provisional because its benchmark list is incomplete, selection criteria involve judgment, and excluding saturated benchmarks can understate coverage.

mercor.com

🔥🔥🔥🔥🔥

9 min

8h ago

90% of US working hours are spent on tasks not covered well by AI benchmarks

Mercor has published a Benchmark Map that measures how well 29 selected AI benchmarks cover tasks across the US labor force. The map uses the Department of Labor’s O*NET taxonomy, which divides the workforce into 923 occupations, 22 job families, and 18,797 tasks. A task is counted as covered when an LLM judge maps at least five benchmark prompts to it; one prompt can map to multiple tasks. Only 208 of 923 occupations have at least one task meeting that threshold, while six of 22 job families have none. Mercor estimates that 90% of US working hours are spent on tasks with little or no coverage in its initial benchmark set, making economy-wide assessments of AI capability difficult. The organization says benchmark coverage disproportionately favors AI-exposed tasks, with O*NET tasks rated exposed to AI about six times more likely to appear than unexposed tasks. Tasks in occupations with larger wage bills and tasks rated most important are also overrepresented. Mercor found that reweighting APEX benchmark scores so each O*NET task counts equally changed model results little for investment banking, law, accounting, and software engineering. Less than one-third of relevant tasks in those four occupations had sufficient APEX coverage. Mercor characterizes its estimates as provisional because its benchmark list is incomplete, selection criteria involve judgment, and excluding saturated benchmarks can understate coverage.

mercor.com

🔥🔥🔥🔥🔥

9 min

8h ago

No more articles to load