--- title: "iOSWorld & AndroidWorld benchmarks · mobilerun" description: "Explore a 98.24% mean official score across 133 iOSWorld tasks and 116 successful AndroidWorld trajectories. Replay every recorded action." url: "https://mobilerun.ai/benchmark/" --- # Benchmark Benchmark results for mobile-harness on iOSWorld and AndroidWorld: 249 tasks across iOS and Android, with the score and a replay of every recorded action for each task. The homepage calls them record-breaking benchmark results, with every run public. ## Results - **iOSWorld**: 98.24% mean official score across 133 iOSWorld tasks (124 full, 9 partial), in 26 apps. 124 tasks received full scores of 1.0 and 9 received partial scores greater than 0.0 and less than 1.0. The binary pass rate is 93.23% (124/133 tasks with a score of 1.0). Each task is weighted equally and partial credit contributes to the mean: 130.6608 ÷ 133 × 100 = 98.24%. Explore: https://mobilerun.ai/benchmark/?platform=ios#tasks - **AndroidWorld**: 116 successful AndroidWorld trajectories: every one of the 116 AndroidWorld tasks received an official score of 1.0 (100% official score). Explore: https://mobilerun.ai/benchmark/?platform=android#tasks - **Partial iOSWorld scores**: the 9 tasks below 1.0, as the results explorer shows them: mem-020 75.00%, mem-021 42.86%, mem-028 58.33%, mem-042 90.00%, mem-050 77.78%, multi-044 88.89%, multi-078 90.91%, multi-080 50.00%, multi-087 92.31%. The other 124 tasks scored 1.0. Each task page: https://mobilerun.ai/benchmark/ios// - **Evidence**: 8,807 recorded interactions (7,388 on iOSWorld, 1,419 on AndroidWorld). The trajectory collections contain a screenshot for every recorded interaction, plus instructions, tool operations, answers, judgments and timing. Two benchmarks, two scoring methods: AndroidWorld uses the official benchmark evaluation, iOSWorld the official rubric scored by GPT-5.4 mini (114 tasks); GPT-6 Astra high (19 tasks). How it compares: the public iOSWorld leaderboard (https://iosworld.io/, checked September 2026) reports pass rates on the same 133 tasks, with a GPT-5.4 Mini judge and at most 50 interaction steps per task. It is led by Claude Opus 4.6 (vision+XML) at 51.9%, then Claude Sonnet 4.6 at 46.6% and GPT-5.4 at 39.8% (both vision+XML). mobilerun's run is not a leaderboard submission and was set up differently: mobile-harness with app cards and shared memory, a session budget of 30 minutes and corrected data fixtures for 3 tasks. Its binary pass rate is 93.23%: 124 of 133 tasks with a full score of 1.0; the 98.24% mean includes partial credit. "Record-breaking" is relative to these published results. ## Evaluation setup | Configuration | AndroidWorld | iOSWorld | |---|---|---| | Device | Pixel 6 emulator | iPhone 17 Pro simulator | | OS | Android 13 / API 33 | iOS 26.5 | | Tasks | 116 | 133 across 26 apps | | Actor | gpt-6-astra / low | gpt-6-astra / low | | Execution budget | 30 minutes per session; existing action budgets | 30 minutes per session | | Scoring | Official benchmark evaluation | GPT-5.4 mini (114 tasks); GPT-6 Astra high (19 tasks) | | Recorded interactions | 1,419 | 7,388 | App cards and shared memory were enabled for both benchmarks. Seed 42 was used where supported. The benchmark devices were an emulator and a simulator, not hosted mobilerun phones. We patched AndroidWorld bugs and corrected confirmed iOSWorld app and data problems. Three iOS tasks used corrected data fixtures while keeping their goals and rubrics: mem-034 (https://mobilerun.ai/benchmark/ios/mem-034/), multi-046 (https://mobilerun.ai/benchmark/ios/multi-046/), multi-084 (https://mobilerun.ai/benchmark/ios/multi-084/). ## How mobile-harness was evaluated - **The harness.** mobile-harness (https://github.com/droidrun/mobile-harness) is a set of tools and instructions for running mobile tasks through agents such as Claude Code, Codex and OpenClaw. It combines Android and iOS controls with app cards, memory and recovery guidance, all in readable Markdown files. - **App knowledge.** App cards hold how each app works: where things are, how to save a change, how to check that it stuck. Memory holds what experience teaches: navigation shortcuts, reliable procedures, gestures that do not work on a particular screen. Most of the guidance was refined by a self-improvement agent that reviewed execution traces between evaluation rounds. - **Outcome checks.** The agent used accessibility trees to identify controls and read content, and screenshots to check the visible state, and read back saved changes before reporting completion. The official task score measures the outcome, not whether an action executed. ## Earlier results We previously got 91.4% on AndroidWorld with GPT-5 (https://mobilerun.ai/blog/posts/achieving-914-on-android-world-a-new-approach-to-mobile-ui-automation). That evaluation used a different model and setup, so the comparison does not isolate the effect of the harness or memory. For speed and lower token use, we would still choose a specialized mobile agent. ## Explore and verify - Results explorer: https://mobilerun.ai/benchmark/ - Methodology: https://mobilerun.ai/benchmark/method/ - Every task has a replay page: https://mobilerun.ai/benchmark/ios// for iOSWorld, https://mobilerun.ai/benchmark// for AndroidWorld - Per-task action data and replay: https://media.mobilerun.ai/benchmarks/2026-09-25-v1///task.json and replay.gif - iOSWorld, the benchmark and its public leaderboard: https://iosworld.io/ (paper: https://arxiv.org/abs/2606.09764) - AndroidWorld, the benchmark environment: https://google-research.github.io/android_world/ (code: https://github.com/google-research/android_world, paper: https://arxiv.org/abs/2405.14573) - Related app-knowledge research (arXiv, separate from these September 2026 evaluations): https://arxiv.org/abs/2601.19306 --- Source: https://mobilerun.ai/benchmark/ Full site as markdown: https://mobilerun.ai/llms-full.txt