We spent the last few months trying to understand why computer-use agents (Claude Computer-Use, OpenAI CUA, Gemini 2.5 Computer-Use) fail so inconsistently.
The pattern we kept seeing: same agent, same task, different OS theme = notably different results.
Claude Sonnet 4 scores 31.9% on OSWorld and Windows Agent Arena (2 of the most relevant benchmarks for computer-use agents) — but with massive variance. An agent trained on Windows 11 light mode fails on dark mode. Works on macOS Ventura, breaks on Monterey. Works on Win11, collapses on Vista.
The root cause: training data lacks visual diversity. Current benchmarks (OSWorld, Windows Agent Arena) rely on static VM snapshots with fixed configurations. They don't capture the reality of diverse OS themes, window layouts, resolution differences, or desktop clutter.
We built cua-bench — HTML-based simulated environments that render across 10+ OS themes (macOS, Win11, WinXP, Win98, Vista, iOS, Android). Define a task once, generate thousands of visual variations.
This enables:
- Oracle trajectory generation via a Playwright-like API (verified ground truth for training)
- Trajectory replotting: record 1 demo → re-render across 10 OS themes = 10 training trajectories
The technical report covers our approach to trajectory generation, Android/iOS environments, cross-platform HTML snapshots, and a comparison with existing benchmarks.
We’re currently working with research labs on training data generation and benchmarks, but we’d really value input from the HN community:
- What tasks or OS environments should be standardized to actually stress computer-use agents?
- Legacy OSes? Weird resolutions? Broken themes? Cluttered desktops? Modal hell?
Curious what people here think are the real failure modes we should be benchmarking.
as an infrastructure engineer the idea of being able to train computer use agents without provisioning infrastructure sounds amazing!
a common use case i run into is i want to be able to configure corporate vpn software on windows machines. is there a link for a getting started guide i could try this out with?
Yes, in a simulated environment you can do this today using plain JS and connecting to a real VPN, while driving the desktop UI. No infra provisioning needed.
The author's conclusion feels even more relevant today: AI automation doesn’t really remove human difficulty—it just moves it around, often making it harder to notice and more risky. And even after a human steps in, there’s usually a lot of follow-up and adjustment work left to do. Thanks for surfacing these uncomfortable but relevant insights
Same here. I even blamed it on switching between Italian and Spanish all the time and thought my brain was short-circuiting. But when you see the right key light up and a different letter shows up, something’s clearly off. Also: with battery saver on it’s basically unusable - the lag makes typing way worse. The video was oddly comforting. Turns out I’m not losing it.
Yeah, a lot of these corporate hackathons are basically just lead gen in disguise. "Use our SaaS product, maybe we’ll give you a t-shirt." They're more about getting conversions than actually teaching anything useful to the students.
This is a nice first step - web search makes sense, and it’s easy to imagine other tools being added next: filesystem, browser, maybe even full desktop control. Could turn Ollama into more than just a model runner. Curious if they’ll open up a broader tool API for third-party stuff too
Are you refering to the blocking of Cloudflare when La Liga matches are played? That affects sites that use Cloudflare, but it's not the fault of Dockerhub.
> Our vision is simple: we want to create a factory that can produce a gigawatt of new AI infrastructure every week.
The moat will be how efficiently you convert electricity into useful behavior. Whoever industrializes evaluation and feedback loops wins the next decade.
Microsoft has a dozen vertical Copilots to build, so picking the model with the best capability today makes sense. If Claude Code is stronger for dev productivity, using it in VS Code just raises the bar for everything else they ship
Really thoughtful piece. It reminds me of how Angular once dominated by default, until its complexity and inertia gave space for React. The same dynamic could be repeating now - React’s network effects create stability, but also risk suffocating innovation
The pattern we kept seeing: same agent, same task, different OS theme = notably different results.
Claude Sonnet 4 scores 31.9% on OSWorld and Windows Agent Arena (2 of the most relevant benchmarks for computer-use agents) — but with massive variance. An agent trained on Windows 11 light mode fails on dark mode. Works on macOS Ventura, breaks on Monterey. Works on Win11, collapses on Vista.
The root cause: training data lacks visual diversity. Current benchmarks (OSWorld, Windows Agent Arena) rely on static VM snapshots with fixed configurations. They don't capture the reality of diverse OS themes, window layouts, resolution differences, or desktop clutter.
We built cua-bench — HTML-based simulated environments that render across 10+ OS themes (macOS, Win11, WinXP, Win98, Vista, iOS, Android). Define a task once, generate thousands of visual variations.
This enables: - Oracle trajectory generation via a Playwright-like API (verified ground truth for training) - Trajectory replotting: record 1 demo → re-render across 10 OS themes = 10 training trajectories
The technical report covers our approach to trajectory generation, Android/iOS environments, cross-platform HTML snapshots, and a comparison with existing benchmarks.
We’re currently working with research labs on training data generation and benchmarks, but we’d really value input from the HN community: - What tasks or OS environments should be standardized to actually stress computer-use agents? - Legacy OSes? Weird resolutions? Broken themes? Cluttered desktops? Modal hell?
Curious what people here think are the real failure modes we should be benchmarking.