android-device-automation

The Android Device Automation skill enables vision-driven automation of Android devices using the Midscene framework. It eliminates the need for DOM access or accessibility labels, instead relying entirely on screenshots to interpret and interact with on-screen elements. This approach allows for automation across any Android app or interface, regardless of the underlying technology stack, solving the challenge of consistent device testing and interaction in environments where traditional automation methods may fail.

This skill offers a range of capabilities for controlling Android devices via natural language commands executed through ADB. Users can perform taps, double-taps, long-presses, text input, scrolling, swiping in any direction, drag-and-drop, zoom gestures, and system navigation actions like Back, Home, or accessing recent apps. Each action is performed within a single Midscene `act` command, ensuring synchronous execution and accurate screenshot-based feedback. Environment setup includes configuring model-specific API keys and endpoints, supporting models such as Gemini, Qwen 3.5, and Doubao Seed 2.0 Lite.

The skill is ideal for QA engineers, mobile app testers, and developers who need to perform end-to-end testing or visual verification on Android devices. It is particularly useful for testing apps in scenarios where UI elements are dynamic or non-standard, enabling thorough testing and validation without dependency on traditional automation frameworks. Typical use cases include mobile app testing, automation of repetitive tasks on devices, verifying UI functionality, and generating reports based on captured screenshots and performed actions.