agent-desktop Explained

agent-desktop is an open-source CLI (and C library) from Vercel Labs that lets AI agents operate desktop apps through the operating system's accessibility tree. It uses no screenshots and no pixel matching: it returns an app's UI structure as JSON, and the agent clicks or types on a ref assigned to each element. The README calls it a "Computer use CLI for AI agents." It is Apache-2.0 licensed and has no LLM inside.

According to the GitHub release pages, v1.0.0 shipped on October 28, 2026, v1.1.0 on November 3, 2026, and the latest, v1.2.0, on November 5, 2026. One correction worth knowing: the MCP server (agent-desktop mcp) arrived in v1.2.0, not in 1.0 or 1.1. This article draws on the GitHub README and release notes and the official docs (agent-browser.dev/desktop) to cover what it does, how it works, how to install it, how it differs from other tools, and what to watch for on security (as of November 9, 2026).

- What it is: a Rust CLI and C library (libagent_desktop_ffi) that reads and operates desktop app UIs via the accessibility APIs: AX API on macOS, UI Automation on Windows, AT-SPI on Linux
- Approach: it reads the UI tree (JSON) instead of images and acts on refs. The observe-act-verify loop is run by the calling agent
- Availability: npm install -g agent-desktop or brew install agent-desktop. Apache-2.0. About 20,000 GitHub stars as of November 9, 2026

Its sibling project, agent-browser, targets the browser; agent-desktop extends the same idea to native desktop apps. The official docs live under the /desktop path of agent-browser.dev.

Per the README, the CLI has 60 command names, 56 of which are operational. The other four (key-down, key-up, mouse-down, mouse-up) refuse to run until a daemon exists that can release a held key or button. The README lists command names only; per-command descriptions are in the Commands page of the docs.

Every command prints a single JSON envelope with version, ok, command, and data. Errors are coded: a stale ref returns STALE_REF, and multiple candidates return AMBIGUOUS_TARGET. The README says to retry only when error.disposition.retry is safe.

Screenshot-based computer use shows the model an image and has it click at coordinates. It works on any app that has a visible UI, but it costs tokens and time to read images, and coordinates drift when resolution or layout changes. agent-desktop instead reads the UI structure that apps expose to the OS (buttons, text fields, menus). This is the accessibility tree, the same source screen readers use.

The README describes a three-step loop that repeats:

- snapshot: fetch the app's UI tree as JSON. Each element carries a ref_id and available_actions
- act: run click, type, and so on against a ref
- verify: snapshot again (or use get, is, wait) to check the result

A ref belongs to the snapshot that produced it. The README uses snapshot-qualified refs such as @s8f3k2p9:e8; a bare ref like @e8 returns INVALID_ARGS (bare refs were removed in v0.10.0). When the UI changes you get STALE_REF, so take a fresh snapshot. With several windows you get AMBIGUOUS_TARGET and pick one with --window-id.

The default is headless: ref actions go through the accessibility API without moving the pointer or taking focus. Add the global --headed flag only when physical input is needed. Accessibility APIs can report success for an action that did nothing, and the macOS docs state that observation is the only check of an effect, which is another reason to re-observe after each step.

macOS requires an Accessibility grant. According to the macOS docs, it goes to the app that launches agent-desktop (Terminal, iTerm2, VS Code, or your agent's host), plus the binary if it is listed separately. Screen Recording is needed only for screenshot, and Automation (System Events) only for a few commands such as opening Notification Center. agent-desktop permissions --guide opens the right System Settings pane, and no app restart is needed after granting.

On macOS headless, pointer actions such as double-click (unless the element offers AXOpen), triple-click, hover, and drag return POLICY_DENIED and need --headed. Quit shortcuts like cmd+q are also denied without --force; use close-app to quit an app. Each platform page lists what works headless on that OS.

The README steps are below. The npm package bundles native binaries and also installs with Bun, Yarn, or pnpm. Homebrew covers macOS and Linux; there is no official Scoop package yet.

The basic pattern is snapshot, act, snapshot again. This is the README quick start verbatim; -i returns interactive elements only, and you pass a ref from the output to the next command.

For dense apps, the README suggests mapping first with snapshot --skeleton --app Slack -i --compact, then drilling into a region with snapshot --root @s8f3k2p9:e3 -i --compact. For secrets, type, set-value, and clipboard-set accept --stdin so text stays out of process arguments. Because batch JSON does appear in process arguments, the README says to run secret steps as separate commands.

To use it over MCP, agent-desktop mcp serves every command as an MCP tool over stdio (added in v1.2.0). Tools are named after commands with a desktop_ prefix, such as desktop_snapshot, desktop_click, and desktop_session_start, and return the same JSON envelope. The README gives one config example:

This mcpServers shape is what you add to the config of clients that use it, such as Claude Code or Cursor. However, neither the README nor the docs give client-specific steps, so we could not confirm a Claude Code or Cursor setup from primary sources. For background on MCP itself, see our summary of the MCP roadmap.

Without MCP, you can have the agent call the CLI directly and use the skills command. agent-desktop skills get desktop (the core loop and rules) and agent-desktop skills get platform (OS differences) print agent-facing guides embedded in the binary. The README suggests telling the agent in AGENTS.md to run both before first use, or installing a static copy with npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop. A static copy can go stale after a CLI upgrade, so skills get is recommended for version-matched content.

The real turning point before 1.0 was v0.10.0 (October 18, 2026): it added Windows 10 1809+ support, but also made breaking changes, including C ABI major version 5 and removal of bare ref input. The v1.0.0 notes do not mention these, so anyone upgrading from 0.9.x or earlier should read the v0.10.0 notes.

Ways of letting an agent drive a screen fall into three broad camps. agent-desktop is an accessibility-tree approach aimed at native apps outside the browser.

In short, screenshot approaches work on anything visible but must interpret images, while accessibility approaches are lighter and faster thanks to structured text but assume the app exposes accessibility information properly. Custom-rendered apps or apps with poor accessibility support can yield thin UI trees. That is a general property; we could not verify agent-desktop's behavior app by app from primary sources.

For OpenAI's side, see Codex Computer Use on Windows and the GPT-5.4 computer use API explainer; for a desktop-app example, see Manus Studio.

agent-desktop hands an agent the power to operate a logged-in desktop. On macOS, once Accessibility is granted, what that app can control covers nearly the whole screen. The README itself warns that snapshots, clipboard reads, screenshots, command output, and debug artifacts may contain private content and should be kept out of public logs and uploads.

- Run it in a dedicated environment: for computer use, Anthropic recommends a dedicated VM or container with minimal privileges, no access to sensitive data such as login credentials, and internet access limited to an allowlist. The same thinking applies here
- Treat on-screen text as untrusted: OpenAI's guidance says to treat screen content as untrusted. Prompt injection through emails or web pages can affect tree-based tools too
- Have a human approve consequential actions: purchases, sending, deleting, and clicking consent
- Keep the quit protection: macOS denies quit shortcuts by default; use --force only when needed
- Pass secrets with --stdin: never put passwords in arguments
- Manage logs: decide where snapshots and traces are stored, and do not share them

Note that the repository's SECURITY.md covers how to report vulnerabilities (use GitHub private vulnerability reporting, not a public issue); it does not give end-user safe-operation guidance. Operating rules for work PCs are up to the user.

Do not expect a finished agent. agent-desktop is a building block: it contains no model and no planning logic, and is meant to be wired into your own agent or a coding agent.

- GitHub: vercel-labs/agent-desktop (README, release pages, SECURITY.md): https://github.com/vercel-labs/agent-desktop
- Official docs (Quick start, macOS, Skills): https://agent-browser.dev/desktop
- npm package agent-desktop (v1.2.0, Apache-2.0): https://www.npmjs.com/package/agent-desktop
- agent-browser: https://github.com/vercel-labs/agent-browser
- Playwright MCP: https://github.com/microsoft/playwright-mcp
- Anthropic computer use tool: https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool
- OpenAI computer use: https://developers.openai.com/api/docs/guides/tools-computer-use
- Not confirmed: the weekly-trending figure reported in the press, official Claude Code or Cursor MCP setup steps, and per-app behavior