ai-agents benchmarks llms developer-tools

Study: Self-generated Agent Skills are useless

SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks

arxiv.org

February 16, 2026

2 min read

🔥🔥🔥🔥🔥

64/100

Summary

SkillsBench is a benchmarking framework designed to evaluate the effectiveness of agent skills across 86 tasks in 11 domains. It includes curated skills and deterministic verifiers to assess their impact on large language model (LLM) agents during inference.

Key Takeaways

SkillsBench is a benchmark consisting of 86 tasks across 11 domains designed to evaluate the effectiveness of agent skills in large language models (LLMs).
Curated skills increase the average pass rate by 16.2 percentage points, with varying effects across domains, from +4.5pp in Software Engineering to +51.9pp in Healthcare.
Self-generated skills do not provide any average benefit, indicating that models struggle to create effective procedural knowledge independently.
Focused skills with 2-3 modules outperform comprehensive documentation, and smaller models equipped with skills can achieve performance comparable to larger models without skills.

Read original article

View story →

Coverage & discussions

Hacker News · arxiv.org

Study: Self-generated Agent Skills are useless ↗

Community discussions

Hacker News discussion ↗

Historical count: 5 · 5 saved in sample · Historical sample

Analysis ready

Community Sentiment

Mixed

Positives

The finding that curated skills provide a significant positive benefit (+16.2pp) suggests that LLMs excel at utilizing existing procedural knowledge rather than generating new skills.
Using LLMs to distill information from research can enhance skill creation, leading to more effective and relevant outcomes tailored to specific workflows.

Concerns

Self-generated skills have been shown to provide a negative benefit (-1.3pp), indicating that LLMs struggle to produce useful procedural knowledge independently.
The observation that layering LLM outputs leads to diminishing returns highlights a critical limitation in the current approach to automating tasks with LLMs.