Noureddine RAMDI / usecomputer: A native cross-platform CLI for AI-driven desktop automation with precise coordinate mapping

Created Mon, 04 May 2026 10:23:02 +0000 Modified Sat, 23 May 2026 20:41:27 +0000
984 Words
github-stars zig desktop-automation ai-agents cli cross-platform coord-map

remorses/usecomputer

usecomputer tackles a common frustration in AI-driven desktop automation: how to precisely map coordinates from downscaled screenshots back to real screen positions for mouse and keyboard actions. It offers a native Zig binary CLI that avoids Node.js overhead and provides robust cross-platform primitives for controlling mouse, keyboard, and screenshots, all designed as the execution layer for AI agents executing computer tasks.

what usecomputer does and its architecture

usecomputer is a cross-platform desktop automation CLI implemented in Zig, aimed at serving as the execution backend for AI agents performing computer-use tasks. Unlike many automation tools built on Node.js or other runtimes, usecomputer’s core is a native binary compiled from Zig, which provides a lean runtime footprint and better performance for low-level input and screen capture operations.

The tool exposes a set of native CLI commands that allow capturing screenshots, moving and clicking the mouse, dragging, scrolling, typing text, and pressing keys. These primitives are essential building blocks for AI agents to interact with the desktop environment programmatically.

A unique feature of usecomputer is its coord-map system. AI agents typically operate on screenshots scaled down to a maximum edge of 1568 pixels (mandated by the README), but actual mouse and keyboard commands need to be sent in real screen coordinates. The coord-map encodes the transformation between the screenshot coordinate space and the real screen coordinates, allowing the CLI to reverse-map pointer positions accurately.

This coord-map is expressed as a string format:

captureX,captureY,captureWidth,captureHeight,imageWidth,imageHeight

which encodes the capture area in screen space and the size of the screenshot image. The CLI commands that move the mouse or click accept a --coord-map flag, enabling them to convert the input coordinates from scaled screenshot space back to actual screen space.

The repo also includes integration examples for OpenAI’s computer tool and Anthropic’s computer-use API, positioning usecomputer as a foundational execution layer in AI agent toolchains.

the coord-map system: precise coordinate mapping for AI agents

What sets usecomputer apart is its practical solution to the coordinate mismatch problem that plagues many computer-use agents. When working with downscaled screenshots, any pointer coordinates generated by vision models or AI tools must be transformed back to the actual screen space to issue correct mouse or keyboard commands.

The coord-map system encapsulates this transformation in a compact format that the CLI understands. By passing the coord-map string along with pointer commands, usecomputer applies an inverse transform to ensure that the mouse clicks and movements occur at the intended pixel locations on the real screen.

This design avoids hardcoding scaling factors or making assumptions about DPI and resolution, which can vary wildly across platforms and configurations. Instead, it relies on explicit capture parameters and image sizes, making the automation robust and predictable.

Under the hood, the codebase is surprisingly clean for a systems-level Zig project, with clear separation between platform-specific input/output handling and coordinate transformation logic. The use of Zig brings benefits such as no runtime overhead, safety features, and easy cross-compilation to macOS, Linux, and Windows.

The tradeoff here is that usecomputer is focused strictly on the execution layer: it does not provide AI model integration or vision processing itself. It expects an upstream agent or tool to generate coordinates in the scaled screenshot space and then uses the coord-map to perform reliable input simulation. This keeps the core small and focused.

quick start with usecomputer

To get started with usecomputer, install it globally via npm:

npm install -g usecomputer

It requires platform-specific permissions:

  • macOS: Accessibility permission enabled for your terminal app
  • Linux: X11 session with DISPLAY set (Wayland via XWayland works too)
  • Windows: Must be run in an interactive desktop session (automation input is blocked on locked desktop)

Once installed, you can try basic mouse and keyboard commands:

usecomputer mouse position --json
usecomputer mouse move -x 500 -y 500
usecomputer click -x 500 -y 500 --button left --count 1
usecomputer type "hello"
usecomputer press "cmd+s"

For commands involving coordinates from downscaled screenshots, use the --coord-map flag with the appropriate mapping string to ensure correct pointer placement.

verdict: a focused execution layer for AI-driven desktop automation

usecomputer is a solid, no-nonsense tool for anyone building AI agents that need to control desktops at a low level with precise coordinate mapping. The native Zig implementation keeps the footprint small and performance reasonable, especially compared to Node.js-based automation stacks.

Its main strength is solving the coordinate transformation challenge with an explicit coord-map system, which is often overlooked but critical for real-world AI agent workflows involving screenshots.

Limitations include platform permission requirements and the need for an upstream AI or vision tool to generate coordinates in screenshot space — usecomputer does not handle the AI part itself. It’s not a general-purpose automation library with fancy features but rather a dependable execution backend.

If you’re building AI agents that interact with the desktop and care about correctness and minimal runtime dependencies, usecomputer is worth a look. The repo’s examples for OpenAI and Anthropic integrations also provide useful starting points for agent toolchains.

→ GitHub Repo: remorses/usecomputer ⭐ 277 · Zig