Study: Self-generated Agent Skills are useless
arxiv.org
February 16, 2026
2 min read
Summary
SkillsBench is a benchmarking framework designed to evaluate the effectiveness of agent skills across 86 tasks in 11 domains. It includes curated skills and deterministic verifiers to assess their impact on large language model (LLM) agents during inference.
Key Takeaways
- SkillsBench is a benchmark consisting of 86 tasks across 11 domains designed to evaluate the effectiveness of agent skills in large language models (LLMs).
- Curated skills increase the average pass rate by 16.2 percentage points, with varying effects across domains, from +4.5pp in Software Engineering to +51.9pp in Healthcare.
- Self-generated skills do not provide any average benefit, indicating that models struggle to create effective procedural knowledge independently.
- Focused skills with 2-3 modules outperform comprehensive documentation, and smaller models equipped with skills can achieve performance comparable to larger models without skills.
Community Sentiment
MixedPositives
- The finding that curated skills provide a significant positive benefit (+16.2pp) suggests that LLMs excel at utilizing existing procedural knowledge rather than generating new skills.
- Using LLMs to distill information from research can enhance skill creation, leading to more effective and relevant outcomes tailored to specific workflows.
Concerns
- Self-generated skills have been shown to provide a negative benefit (-1.3pp), indicating that LLMs struggle to produce useful procedural knowledge independently.
- The observation that layering LLM outputs leads to diminishing returns highlights a critical limitation in the current approach to automating tasks with LLMs.
Related Articles
Frontier AI agents violate ethical constraints 30–50% of time, pressured by KPIs
Feb 10, 2026
Evaluating AGENTS.md: are they helpful for coding agents?
Feb 16, 2026
SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI
Mar 8, 2026
Speed at the cost of quality: Study of use of Cursor AI in open source projects (2025)
Mar 16, 2026
Psychometric Jailbreaks Reveal Internal Conflict in Frontier Models
Feb 5, 2026
Source
arxiv.org
Published
February 16, 2026
Reading Time
2 minutes
Relevance Score
64/100
Why It Matters
This page is optimized for focused reading: quick context up top, a clean summary block, and a direct path to the original source when you want the full story.