Don't trust large context windows
garrit.xyz
June 14, 2026
2 min read
🔥🔥🔥🔥🔥
53/100
Summary
Large language models (LLMs) exhibit a division in their context window, with a "smart zone" where performance is optimal and a "dumb zone" where attention and memory retention decline significantly. The effective cutoff for reliable memory and attention occurs around 100,000 tokens, regardless of the advertised size of the context window.
Key Takeaways
- Large context windows in language models are primarily marketing numbers, as effective context is much smaller than advertised, typically around 100k tokens.
- Performance of language models degrades as the context window fills, leading to a "dumb zone" where the model forgets earlier information.
- Tools like Claude Code implement auto-compaction to summarize history, but this occurs after the model has already degraded in performance.
- Structuring workflows around small, named artifacts can help maintain effective context and keep sessions in the "smart zone."
Community Sentiment
Positives
- Implementing constraints on tool calls allows for effective management of context size, enabling prolonged conversations without hitting token limits, which is crucial for maintaining continuity in complex tasks.
- Users report that newer versions of Opus perform well beyond 100k tokens, indicating improvements in model capabilities and robustness for handling larger contexts.
- Adopting a structured approach, such as creating PRDs for features, helps maintain the AI's focus and reduces the likelihood of it drifting off-topic, enhancing overall productivity.
Concerns
- Despite claims of improved performance, some users experience basic recall mistakes even at lower token counts, raising concerns about the reliability of the Opus models.
- The current memory systems in AI models are criticized for being ineffective, as they often fail to utilize stored information properly, leading to repeated mistakes.
- There is a perception that large context windows can lead to diminishing returns, with some users advocating for smaller contexts to achieve better results.