claude anthropic ai-agents developer-tools

Please Do Not A/B Test My Workflow

backnotprop.com

March 14, 2026

3 min read

🔥🔥🔥🔥🔥

56/100

Summary

Anthropic is conducting A/B tests on Claude Code, which has impacted user workflows. The author expresses frustration over the degradation of their workflow due to these tests.

Key Takeaways

Anthropic is conducting A/B tests on Claude Code that have negatively impacted user workflows, leading to frustration among users.
Users of Claude Code, who pay $200/month, seek greater transparency and configurability in the tool's functionality.
Changes to core features, such as plan mode, have been implemented without user notice, affecting the quality of outputs.
The engineer responsible for the A/B test acknowledged that limiting plan lengths was intended to reduce rate-limit hits, but early results indicated minimal impact.

Read original article

View story →

Coverage & discussions

Hacker News · backnotprop.com

Please Do Not A/B Test My Workflow ↗

Community discussions

Hacker News discussion ↗

Historical count: 5 · 5 saved in sample · Historical sample

Analysis ready

Community Sentiment

Mixed

Positives

LLMs can significantly enhance productivity by quickly generating boilerplate content, which can guide users in their thought processes more efficiently than traditional methods.
The ability of newer LLM models to perform effectively with less direction indicates advancements in AI capabilities, potentially improving user experience.

Concerns

LLMs currently lack reliability and replicability in results, raising concerns about their suitability for professional workflows, especially when subjected to A/B testing.
The risk of LLMs producing misleading or irrelevant information can derail users, emphasizing the need for a solid understanding of the subject matter to utilize them effectively.
Open source tools may struggle to compete with proprietary models like Claude Code due to their inability to conduct A/B testing, which limits data-driven design improvements.