You can't solve computer use by ignoring the interface
steelmanlabs.com
July 30, 2026
5 min read
🔥🔥🔥🔥🔥
44/100
Summary
Agentic computer use is crucial for maximizing real-world AI impact, particularly in software development. The best model on the OSWorld-V2 benchmark for long-horizon computer tasks currently achieves only 20.6% completion.
Key Takeaways
- The best-performing AI models achieve only 20.6% completion on OSWorld-V2 and 26.2% on Agents' Last Exam, highlighting significant limitations in current computer use capabilities.
- Many AI agents bypass user interfaces by making API calls instead of interacting directly with graphical interfaces, which can lead to unexpected side effects and inefficiencies.
- Human users achieve over 95% success on simple web tasks, while AI models struggle significantly, indicating a gap in the ability of AI to perform tasks that require human-like interaction with interfaces.
- Current approaches to improving AI computer use, such as increasing model size and token count, do not address the fundamental challenges of perception, planning, and interface interaction.
Community Sentiment
Positives
- Some commenters appreciate the mouse trail feature for helping them focus while reading, showing that not all UI changes are negative.
- There's optimism about evolving UI systems to be more efficient through API calls, reflecting a forward-thinking approach to AI interactions.
- The idea that AI can find and use APIs effectively impresses some users, highlighting the potential for smarter AI integrations.
Concerns
- Many users are frustrated with the UI changes, claiming they hinder usability and make the site harder to navigate.
- Skepticism lingers about the efficacy of API calls, with some pointing out that they don't always work, raising concerns about AI's reliability in those scenarios.
- Critics argue that the benchmarks mentioned in the article are flawed, as they don't account for how frontier models interact with UIs compared to APIs.
