90% of US working hours are spent on tasks not covered well by AI benchmarks
mercor.com
October 16, 2026
9 min read
70/100
Summary
Mercor has published a Benchmark Map that measures how well 29 selected AI benchmarks cover tasks across the US labor force. The map uses the Department of Labor’s O*NET taxonomy, which divides the workforce into 923 occupations, 22 job families, and 18,797 tasks. A task is counted as covered when an LLM judge maps at least five benchmark prompts to it; one prompt can map to multiple tasks. Only 208 of 923 occupations have at least one task meeting that threshold, while six of 22 job families have none. Mercor estimates that 90% of US working hours are spent on tasks with little or no coverage in its initial benchmark set, making economy-wide assessments of AI capability difficult. The organization says benchmark coverage disproportionately favors AI-exposed tasks, with O*NET tasks rated exposed to AI about six times more likely to appear than unexposed tasks. Tasks in occupations with larger wage bills and tasks rated most important are also overrepresented. Mercor found that reweighting APEX benchmark scores so each O*NET task counts equally changed model results little for investment banking, law, accounting, and software engineering. Less than one-third of relevant tasks in those four occupations had sufficient APEX coverage. Mercor characterizes its estimates as provisional because its benchmark list is incomplete, selection criteria involve judgment, and excluding saturated benchmarks can understate coverage.
Key Takeaways
- Mercor’s map covers 923 O*NET occupations and finds that only 208 have a task represented by at least five prompts from its 29 selected AI benchmarks.
- Mercor estimates that 90% of American working hours involve tasks with little or no benchmark coverage in its initial dataset.
- Benchmarks disproportionately represent tasks considered AI-exposed, high-importance, and associated with occupations that have larger wage bills.
- O*NET task weighting produced results similar to simple prompt averages on Mercor’s APEX tests for four occupations, but fewer than one-third of relevant tasks had enough coverage for inclusion.
- Mercor says its coverage estimates are provisional because the benchmark sample is incomplete and benchmark selection and task tagging introduce potential biases.
What the discussion said
The thread largely accepted the paper’s core warning: benchmark scores say far less about real work than the industry’s sales pitch suggests. Commenters kept returning to the unmeasured labor around software and knowledge work: reconciling competing interests, chasing dependencies, QA, presentations, ticket management, and simply noticing problems before they become failures. Faster code generation does not magically turn an idea into a production feature, because coding was often only one small link in a slow organizational chain. The sharper debate was over why AI targets certain jobs first and what that implies. Several readers rejected the idea that benchmarks are deliberately aimed at elite work, arguing that programming and mathematics are unusually tractable because correctness can be cheaply verified with compilers and theorem provers. Others saw an obvious economic tilt toward automating high-wage digital tasks, while noting that customer support may be vulnerable despite lower pay. Physical service work remains difficult mainly because reliable manipulation and deployment-level robotics lag language models. There was genuine enthusiasm for AI taking over routine build and debugging work, and for large coding productivity gains. But the dominant mood was caution: gains in a narrow task can expose slower, more human bottlenecks rather than deliver sweeping job replacement or economy-wide productivity miracles.
Where opinion split
The central dispute is whether AI is aimed at high-status, high-pay work by economic design or simply reaches it first because digital tasks offer clean feedback and automatic verification. One side sees profit-seeking firms prioritizing valuable white-collar labor; the other argues that coding, math, and support are advancing where models can be trained and checked, while physical work waits on robotics.
Coverage & discussions
Hacker News · mercor.com
Community discussions
Community Sentiment
Positives
- AI can clear routine builds and debugging after implementation, letting developers keep momentum instead of spending their attention on repetitive tool wrangling.
- Large gains in coding and scripting productivity suggest AI is already valuable in tightly specified digital work, especially where outputs can be tested automatically.
- Compiler- and prover-checked domains give AI unusually strong training feedback, making dependable progress on programming and mathematical tasks more plausible than broad workplace automation.
- Automating costly information work could lower prices for consumers if competitive pressure passes productivity gains through rather than merely boosting margins.
Concerns
- Existing benchmarks overrepresent economically prominent tasks while missing the messy coordination, oversight, and exception handling that consume most real working hours.
- Generating code faster does not shorten delivery cycles when stakeholder conflict, QA, approvals, and operational follow-through remain the actual bottlenecks.
- Language models do not replace the judgment needed to frame problems and design systems; overusing them may even displace the hands-on work where developers think best.
- Physical service work remains largely outside current AI’s reach because dependable manipulation in uncontrolled environments is far harder than producing text or code.
