llms ai-agents benchmarking instruction-following

Handbook.md shows that long policy documents do not reliably govern agents

HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following

arxiv.org

July 29, 2026

2 min read

🔥🔥🔥🔥🔥

63/100

Summary

HANDBOOK.md introduces a benchmark specifically designed to evaluate long-context agentic instruction following in language-model agents. This benchmark assesses how well agents adhere to a system prompt, policy file, or skills document during their operations.

Key Takeaways

The HANDBOOK.md benchmark consists of 65 agentic tasks designed to evaluate how language-model agents follow long, binding policy documents in professional environments.
The benchmark spans five domains: finance, medical billing, insurance, logistics, and HR, with tasks modeled on the behavior of enterprise employees following company handbooks.
The best-performing model configuration achieved a pass rate of 36.2% under strict grading criteria, while most configurations remained below 25%.
Common failure patterns include agents allowing plausible in-environment requests to override standing policies and losing rule details over long task durations.

Read original article

View story →

Coverage & discussions

Hacker News · arxiv.org

Handbook.md shows that long policy documents do not reliably govern agents ↗

Community discussions

Hacker News discussion ↗

Historical count: 5 · 5 saved in sample · Historical sample

Analysis ready

Community Sentiment

Mixed

Positives

Local inference could magically fix LLM defects, giving users control and potentially transforming how we manage complex models.
Models like Kimi K3 are pushing the boundaries of frontier performance, suggesting exciting advancements in LLM capabilities.
Using a limited context window (like 50%) seems to lead to better outcomes, showing that managing input can enhance model effectiveness.

Concerns

Claiming 1M tokens in context is misleading; users face severe limitations due to quantization and poor sampling methods.
Local models still exhibit the same defects as cloud models, debunking the idea that local hosting is a panacea for LLM issues.
The inability of models to retain information over extended interactions is a significant flaw, undermining their effectiveness in following complex policies.