harmonyos-device-automation

harmonyos-device-automation provides vision-driven automation of HarmonyOS NEXT devices using Midscene.js, driving the device over HDC (HarmonyOS Device Connector). It solves the problem of automating and QA-testing HarmonyOS apps without relying on the DOM or accessibility labels — the agent works entirely from screenshots, so it can interact with any visible element regardless of the underlying technology stack, using natural-language commands.

The skill runs each capability as a CLI command via `npx -y @midscene/harmony@1`, where every command maps to an MCP tool and the agent acts as the decision-making brain. Core commands include `connect` (optionally by deviceId), `launch` for a deterministic starting app or URL, `take_screenshot`, `act` for high-level natural-language interactions (tap, double-tap, long-press, type, scroll, drag, key presses, and system navigation like Back/Home/recent apps), `assert` for verifying that the visible screen satisfies a condition, and `runhdcshell` for lower-level device commands forwarded to `hdc shell`. Assertions can compare against reference images and emit custom failure messages for QA and CI logs. The skill enforces critical operating rules: never run Midscene commands in the background, run only one command at a time, and allow enough time for each command (AI inference plus screen interaction can take a minute or more) so the screenshot-analyze-act loop stays intact. It requires a vision-capable model configured through `MIDSCENE_MODEL_*` environment variables (examples include Gemini, Qwen 3.5, and Doubao Seed) and a working HDC installation.

The skill targets QA engineers and developers who need end-to-end, visual verification of apps on HarmonyOS/Huawei devices. Because it reasons from screenshots, it is especially useful when accessibility metadata is unavailable and when a human-like, screen-driven testing loop is required. Note that two-finger zoom is not available because the HarmonyOS automation layer does not expose multi-touch input.