New mobilerun VA is now in beta, a hosted 24/7 agent for mobile automation
Benchmark · AndroidWorld

91.4% on AndroidWorld.

91.4% success across 116 AndroidWorld tasks. Open source, GPT-5, direct accessibility API. Reproducible from the eval harness in our public repo.

View eval harness → Read the method Peer-reviewed research
SUCCESS RATE
91.4%

Measured, not implied.

106 of 116 tasks completed in the published mobilerun evaluation run.

TASKS
106 / 116

Tasks completed.

Calendar, Files, Notes, Markor, Audio, Browser, System, Expense, Recipe, Clock, and more.

ARCHITECTURE
Manager → Executor

Manager-Executor.

Dynamic feedback loops between planning and execution. Robust under long-horizon failure modes.

Leaderboard

How we compare.

A dated comparison of publicly reported AndroidWorld results. The current mobilerun run uses the accessibility API with vision fallback.

mobilerun framework
91.4%
AutoGLM-Mobile
84.5%
LX-GUIAgent
80.2%
K²-Agent
79.3%
Finalrun
76.7%
MobileUse-v2
75.0%
JT-GUIAgent-V2
67.2%
GUI-Owl-7B
66.4%

Scores are pulled from their public reports and the AndroidWorld leaderboard. The eval harness is in our public repo so anyone can reproduce. Updated 29.10.2025

Methodology

Why the a11y tree wins.

Smaller payloads, richer semantics, faster steps.

Per-step input Screenshot Accessibility tree Difference
Payload size ~1,024 KB ~2 KB 500× smaller
Semantic structure Pixels only Labels & roles Higher accuracy
Step latency High Low Faster agents
Vision fallback n/a When tree is incomplete Robust
Task breakdown

Where we succeeded.

Successful categories from the 106 / 116 run.

17

Calendar

Event creation, navigation, recurring schedules.

13

System

Settings, intents, system-level workflows.

13

Markor

Note editing, search, formatting.

10

Recipe

Multi-step content workflows with state.

9

Expense

Form completion, calculation, lookups.

+44

Audio, Browser, Clock, Files, Notes & more

Across the remaining task families.

Every task

All 115 tasks.

Open any task to replay its full execution trajectory.

Audio Recorder Record Audio Pass Audio Recorder Record Audio With File Name Pass Browser Maze Pass Browser Multiply Pass Camera Take Photo Pass Camera Take Video Pass Clock Stop Watch Paused Verify Pass Clock Stop Watch Running Pass Clock Timer Entry Pass Contacts Add Contact Pass Expense Add Multiple Pass Expense Add Multiple From Gallery Pass Expense Add Multiple From Markor Pass Expense Add Single Pass Expense Delete Duplicates Pass Expense Delete Duplicates2 Pass Expense Delete Multiple Pass Expense Delete Multiple2 Pass Expense Delete Single Pass Files Delete File Pass Files Move File Pass Markor Add Note Header Pass Markor Change Note Content Pass Markor Create Folder Pass Markor Create Note Pass Markor Create Note And Sms Pass Markor Create Note From Clipboard Pass Markor Delete All Notes Pass Markor Delete Newest Note Pass Markor Delete Note Pass Markor Edit Note Pass Markor Merge Notes Pass Markor Move Note Pass Markor Transcribe Receipt Pass Notes Is Todo Pass Notes Meeting Attendee Count Pass Notes Recipe Ingredient Count Pass Notes Todo Item Count Pass Open App Task Eval Pass Osm And Favorite Pass Osm And Track Pass Recipe Add Multiple Recipes Pass Recipe Add Multiple Recipes From Markor Pass Recipe Add Single Recipe Pass Recipe Delete Duplicate Recipes Pass Recipe Delete Duplicate Recipes2 Pass Recipe Delete Multiple Recipes Pass Recipe Delete Multiple Recipes With Constraint Pass Recipe Delete Multiple Recipes With Noise Pass Recipe Delete Single Recipe Pass Recipe Delete Single With Recipe With Noise Pass Retro Create Playlist Pass Retro Playing Queue Pass Retro Save Playlist Pass Save Copy Of Receipt Task Eval Pass Simple Calendar Add One Event Pass Simple Calendar Add One Event In Two Weeks Pass Simple Calendar Add One Event Relative Day Pass Simple Calendar Add One Event Tomorrow Pass Simple Calendar Add Repeating Event Pass Simple Calendar Any Events On Date Pass Simple Calendar Delete Events Pass Simple Calendar Delete Events On Relative Day Pass Simple Calendar Delete One Event Pass Simple Calendar Event On Date At Time Pass Simple Calendar Events In Next Week Pass Simple Calendar Events In Time Range Pass Simple Calendar Events On Date Pass Simple Calendar First Event After Start Time Pass Simple Calendar Location Of Event Pass Simple Calendar Next Event Pass Simple Calendar Next Meeting With Person Pass Simple Draw Pro Create Drawing Pass Simple Sms Reply Pass Simple Sms Reply Most Recent Pass Simple Sms Resend Pass Simple Sms Send Pass Simple Sms Send Clipboard Content Pass Simple Sms Send Received Address Pass Sports Tracker Activities Count For Week Pass Sports Tracker Activities On Date Pass Sports Tracker Activity Duration Pass Sports Tracker Longest Distance Activity Pass Sports Tracker Total Distance For Category Over Interval Pass Sports Tracker Total Duration For Category This Week Pass System Bluetooth Turn Off Pass System Bluetooth Turn Off Verify Pass System Bluetooth Turn On Pass System Bluetooth Turn On Verify Pass System Brightness Max Pass System Brightness Max Verify Pass System Brightness Min Pass System Brightness Min Verify Pass System Copy To Clipboard Pass System Wifi Turn Off Pass System Wifi Turn Off Verify Pass System Wifi Turn On Pass System Wifi Turn On Verify Pass Tasks Due Next Week Pass Tasks Due On Date Pass Tasks High Priority Tasks Pass Tasks High Priority Tasks Due On Date Pass Turn Off Wifi And Turn On Bluetooth Pass Turn On Wifi And Open App Pass Vlc Create Playlist Pass Vlc Create Two Playlists Pass Contacts New Contact Draft Fail Markor Transcribe Video Fail Osm And Marker Fail Recipe Add Multiple Recipes From Image Fail Recipe Add Multiple Recipes From Markor2 Fail Recipe Delete Duplicate Recipes3 Fail Retro Playlist Duration Fail Tasks Completed Tasks For Date Fail Tasks Incomplete Tasks On Date Fail

Run it yourself.

The eval harness is open source. Reproduce the result, or build your own benchmark on the same primitives.

View on GitHub → Read the method
Built in the open
Ask ChatGPT Ask Claude