Please Do Not A/B Test My Workflow
backnotprop.com
March 14, 2026
3 min read
Summary
Anthropic is conducting A/B tests on Claude Code, which has impacted user workflows. The author expresses frustration over the degradation of their workflow due to these tests.
Key Takeaways
- Anthropic is conducting A/B tests on Claude Code that have negatively impacted user workflows, leading to frustration among users.
- Users of Claude Code, who pay $200/month, seek greater transparency and configurability in the tool's functionality.
- Changes to core features, such as plan mode, have been implemented without user notice, affecting the quality of outputs.
- The engineer responsible for the A/B test acknowledged that limiting plan lengths was intended to reduce rate-limit hits, but early results indicated minimal impact.
Community Sentiment
MixedPositives
- LLMs can significantly enhance productivity by quickly generating boilerplate content, which can guide users in their thought processes more efficiently than traditional methods.
- The ability of newer LLM models to perform effectively with less direction indicates advancements in AI capabilities, potentially improving user experience.
Concerns
- LLMs currently lack reliability and replicability in results, raising concerns about their suitability for professional workflows, especially when subjected to A/B testing.
- The risk of LLMs producing misleading or irrelevant information can derail users, emphasizing the need for a solid understanding of the subject matter to utilize them effectively.
- Open source tools may struggle to compete with proprietary models like Claude Code due to their inability to conduct A/B testing, which limits data-driven design improvements.
Related Articles
Anthropic tries to hide Claude's AI actions. Devs hate it
Feb 16, 2026
How I use Claude Code: Separation of planning and execution
Feb 22, 2026
I was banned from Claude for scaffolding a Claude.md file?
Jan 22, 2026
AI makes the easy part easier and the hard part harder
Feb 8, 2026
Why Developers Keep Choosing Claude over Every Other AI
Feb 26, 2026
Source
backnotprop.com
Published
March 14, 2026
Reading Time
3 minutes
Relevance Score
56/100
Why It Matters
This page is optimized for focused reading: quick context up top, a clean summary block, and a direct path to the original source when you want the full story.