ios-device-automation
Vision-driven iOS device automation using Midscene CLI. Operates entirely from screenshots — no DOM or accessibility labels required. Can interact with all visible elements on screen regardless of technology stack. Control iOS devices with natural language commands via WebDriverAgent. Triggers: ios, iphone, ipad, ios app, tap on iphone, swipe, mobile app ios, ios device, ios testing, iphone automation, ipad automation, ios screen, ios navigate, test ios app, verify on iphone, QA on ipad, check
Security Assessment
Detected risks:
About ios-device-automation
The ios-device-automation skill enables vision-driven automation of iOS devices (iPhone and iPad) using natural language commands. Unlike traditional automation tools that rely on DOM structures or accessibility labels, this skill operates entirely from screenshots, allowing it to interact with any visible element on screen regardless of the underlying technology stack. It uses Midscene CLI powered by Midscene.js to interpret visual content and execute actions through WebDriverAgent.
This skill supports comprehensive iOS interactions including tapping, swiping, typing, scrolling, dragging, pinch-to-zoom gestures, and system navigation (Home button, app switcher). Each automation command runs synchronously, capturing screenshots that the AI analyzes before deciding the next action. This screenshot-analyze-act loop ensures reliable automation even for complex UI elements or custom controls that traditional automation tools struggle with.
Ideal for QA engineers performing end-to-end testing on iOS applications, developers validating UI behavior across different screen states, and teams needing visual verification of mobile app functionality. The vision-driven approach makes it particularly valuable for testing apps with dynamic content, WebView components, games, or any interface where traditional selectors are unreliable or unavailable. Requires configuration of AI models with strong visual grounding capabilities such as Gemini, Qwen, or Doubao.
FAQ
What iOS versions and devices are supported?
The skill works with any iOS device that can connect via WebDriverAgent, including iPhones and iPads. Specific version compatibility depends on WebDriverAgent setup.
Do I need to modify my iOS app for automation?
No modifications are required. Since automation is vision-driven and works from screenshots, it can interact with any visible element regardless of how the app is built.
What AI models are required and how do I configure them?
You must configure a model with strong visual grounding capabilities via environment variables (MIDSCENE_MODEL_API_KEY, MIDSCENE_MODEL_NAME, MIDSCENE_MODEL_BASE_URL, MIDSCENE_MODEL_FAMILY). Supported models include Gemini-3-Flash, Qwen 3.5, Doubao Seed 2.0 Lite, and others listed in the Midscene documentation.
Can I run multiple automation commands in parallel?
No. Commands must run one at a time synchronously. Each command needs to complete and provide a screenshot before the next action is determined, as this maintains the screenshot-analyze-act workflow.
How long does each automation command take?
Typical commands need about 1 minute due to AI inference and screen interaction. Complex actions may take longer. Background execution is not supported as it breaks the visual feedback loop.
Install ios-device-automation
Quick Setup:
- Copy the skill folder to
.claude/skills/ - Claude will automatically detect and use the skill
Repository
web-infra-dev/midscene-skills