The computer use verification skill that every agent needs

In this post I’ll describe how to add computer and browser use to your agents to reproduce issues and verify fixes and new features.

For folks unfamiliar with computer and browser use, it’s a tool that allows agents to control a running application directly with mouse clicks and keyboard presses. Most of the major model providers have models that support this, and they are increasingly available in different harnesses, including Warp.

The value of computer use in your cloud software factory is less as a standalone agent, and more as a capability to provide to other agents in the factory flow. Specifically, computer use is valuable:

• During the triage phase to reproduce bugs before trying to fix them
• During implementation to verify fixes and verify that new features match specs
• During review to prove to a human reviewer that code actually matches expected behavior

The verification is particularly valuable as a way of lessening code review burden. If you can see a video of a feature working, you are more likely to trust the underlying code. Especially for pure UI changes that are low risk, having “proof” that a feature works from the user perspective is valuable and saves time.

Another less obvious benefit of computer use is that it lets an agent create a loop where it can debug changes on its own. This works especially well with spec-driven development when you have a detailed PRODUCT.md that the agent is trying to implement. At every implementation pass, it can run computer use to see how close the implementation is to spec, and continue until it’s good.

As with prior posts, you can use the implementation on your own repos by following the open-source demo. There’s a skill you can install and then invoke from your favorite coding agent to directly set up everything on your own repo:

npx skills add warpdotdev-demos/cloud-factory-demo --skill oz-cloud-factory-demo

Add computer use verification to your agents

Diving into computer-use, we create a new skill called verify-behavior that defines how to use computer and browser use.

The key aspects of the skill are:

• When to use computer use (desktop and mobile-native) vs. browser use (webapps)
• What to capture: video preferably, but screenshots are fine
• The two modes to use it in:
  • reproduce: try to confirm that a reported bug occurs
  • verify: verify a new behavior

In order for this skill to work, it needs to be run through a harness that supports computer and browser use. Since we are building a cloud software factory, this should be a harness that supports cloud agents. At this point, you can run the verify-behavior skill directly, but it’s actually more useful to hook it into the other agents in your factory as a capability they can take advantage of. To do this, we will update the existing Triage, Review and Implementation skills to encourage them to use the verify-behavior skill when it will help them do their jobs.

Note that the verify-behavior skill uses cloud subagents. You can also do computer use locally, but it tends to be a worse experience unless the platform you’re using can test in the background. Warp and Codex do support this, but it still tends to be best in my experience to do computer use on a cloud machine where there’s no chance of the agent doing anything weird with the apps running on your machine.

For complex or new behaviors, we ask our orchestrator to fan-out to verify all of the user stories independently and in-parallel. Computer use is typically single-threaded, so if you want better throughput, it’s best to fanout cloud agents across machines.

Computer use in action

I have a demo repo that I’ve used in a few of these posts that implements a simple agentic image editor. As a first user story, I’m going to show how you can use computer use to verify an issue. In this case, there’s a bug in the demo app: when an image is uploaded, its preview renders at thumbnail size rather than gallery size.

This is a trivial example, but for folks dealing with bug reports, having images automatically generated to verify the reproduction is a big unlock.

Moving on to a second use case, I asked an agent to implement a new feature in this repo end-to-end and verify it using computer use. The feature is a relatively straightforward UI change, adding “clear” and “replace” controls to the image editor.

It really is that simple – once you have the scaffolding set up you can use computer use to help lift the burden on PR review. In fact, for low-risk UI changes, you might decide that watching these videos is enough and you don’t need to look at the code at all.

To recap, we have created agents that:

• Triage new issues, optionally sending them to be implemented or spec’d
• Spec issues from a product and technical perspective
• Implement the actual code
• Review the code and improve review over time with an observer loop
• Added a verification capability that all of the other agents can use

These posts show how, with a simple set of primitives, you can start to build your own factory and truly automate more of your mundane work. The primitives are just:

• A cloud agent platform like Warp that supports computer-use
• A workflow tool like GitHub Actions
• A set of agent skills that you and an agent can tune