Handbook.md shows that long policy documents do not reliably govern agents
arxiv.org
July 29, 2026
2 min read
🔥🔥🔥🔥🔥
60/100
Summary
HANDBOOK.md introduces a benchmark specifically designed to evaluate long-context agentic instruction following in language-model agents. This benchmark assesses how well agents adhere to a system prompt, policy file, or skills document during their operations.
Key Takeaways
- The HANDBOOK.md benchmark consists of 65 agentic tasks designed to evaluate how language-model agents follow long, binding policy documents in professional environments.
- The benchmark spans five domains: finance, medical billing, insurance, logistics, and HR, with tasks modeled on the behavior of enterprise employees following company handbooks.
- The best-performing model configuration achieved a pass rate of 36.2% under strict grading criteria, while most configurations remained below 25%.
- Common failure patterns include agents allowing plausible in-environment requests to override standing policies and losing rule details over long task durations.
Community Sentiment
Positives
- Local inference could magically fix LLM defects, giving users control and potentially transforming how we manage complex models.
- Models like Kimi K3 are pushing the boundaries of frontier performance, suggesting exciting advancements in LLM capabilities.
- Using a limited context window (like 50%) seems to lead to better outcomes, showing that managing input can enhance model effectiveness.
Concerns
- Claiming 1M tokens in context is misleading; users face severe limitations due to quantization and poor sampling methods.
- Local models still exhibit the same defects as cloud models, debunking the idea that local hosting is a panacea for LLM issues.
- The inability of models to retain information over extended interactions is a significant flaw, undermining their effectiveness in following complex policies.
Related Articles
Evaluating AGENTS.md: are they helpful for coding agents?
Feb 16, 2026
Study: Self-generated Agent Skills are useless
Feb 16, 2026
Frontier AI agents violate ethical constraints 30–50% of time, pressured by KPIs
Feb 10, 2026
A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI
Jul 13, 2026
Coding agents think ahead of time
Jul 14, 2026