computer-automation

Computer Automation is a vision-driven desktop-automation skill built on Midscene that lets Claude control a local desktop (macOS, Windows, Linux) or a remote Windows desktop over RDP using natural-language commands. It operates entirely from screenshots with no DOM or accessibility labels, so it can interact with any visible on-screen element regardless of the underlying technology stack. It solves automation of desktop-native applications (Electron, Qt, native apps) that cannot run in a browser, and explicitly steers web-app tasks to a separate Browser Automation skill instead.

The skill drives the @midscene/computer CLI, where the agent runs one command at a time synchronously, reads the resulting screenshot, and decides the next action in a screenshot-analyze-act loop. An act call can move the mouse, click, double/right-click, drag, type or clear text, scroll, and press keys or shortcuts on the selected display. It requires a vision-capable model configured through environment variables (API key, model name, base URL, family) with examples for Gemini, Qwen, Doubao, and others, and Midscene auto-loads a local .env. RDP mode is activated by passing a host, username, and password to connect, routing all subsequent actions through a bundled RDP helper; supported flags cover port, domain, security protocol, an option to skip TLS certificate validation for trusted dev hosts with self-signed certs, admin-session attachment, and requested desktop resolution. The skill's critical rules require synchronous single-command execution, minimizing rather than closing windows, and always reporting results to the user.

It targets QA engineers, RPA builders, and power users automating desktop-native apps or remote Windows hosts. This is high-capability tooling: in local mode it takes over the user's real mouse and keyboard, and it can drive remote machines over RDP with supplied credentials — genuine capability that warrants care. Mitigating factors are that credential handling follows good practice (read passwords from environment variables or a secrets manager, never paste them into a shared transcript), the certificate-skip option is explicitly limited to trusted dev hosts, and the workflow rules avoid closing the user's applications; there is no exfiltration, persistence, or malicious behavior in the content.