Benchmark methodology.

Mobile tools for the agent
you already use.

mobile-harness is a set of tools and instructions for running mobile tasks through agents such as Claude Code, Codex, and OpenClaw. It combines Android and iOS controls with app cards, memory, and recovery guidance.

Its main advantage is ease of use: you can handle mobile tasks through the agent you already work with, alongside its usual tools. The guidance lives in readable Markdown files, so you can inspect and adapt it. Shared memory lets the agent reuse what it learns across tasks.

These files also support self-improvement. An agent can review execution traces and refine the app cards and memory, making what it learns available to later tasks.

Knowing the app helps
finish the task.

Both live in Markdown files. You can open the files the agent reads, change an unclear instruction, and keep your own overrides. When the agent does something unexpected, you have somewhere concrete to look.

Most of the guidance work was done by a self-improvement agent. It reviewed execution traces, identified where the task-solving agent struggled, and refined the relevant app cards and memory. It checked the revised procedures before the next evaluation round.

App cards held the main workflows, while memory captured lessons learned during execution. Together, they gave the agent a consistent approach to using each app and a way to carry useful knowledge between tasks.

Check that the change stuck.

A tool returning successfully does not tell you whether the task is done. Text can be entered without being saved. A tap can open the wrong item. A list can have more entries below the screen.

We used accessibility trees to identify controls and read content, alongside screenshots to check the visible state. The guidance also called for reading back saved changes before reporting completion.

Two platforms.
The same mobile harness.

App cards and shared memory were enabled for both benchmarks. Seed 42 was used where supported.

Benchmark and fixture corrections

We patched AndroidWorld bugs and corrected confirmed iOSWorld app and data problems. Three iOS tasks used corrected data fixtures while keeping their goals and rubrics: mem-034, multi-046, and multi-084.

Recorded evidence

The trajectory collections contain a screenshot for every recorded interaction, plus instructions, tool operations, answers, judgments, and timing. The website presents a compressed screenshot replay and action history for every task, with GIFs and action JSON available below the phone.

GIFs display screenshots at fixed intervals. Use the recorded execution durations when comparing speed.

What the numbers measure.

A different model.
A different setup.

We previously got 91.4% on AndroidWorld with Mobilerun, using GPT-5. This time, we wanted to see how well a standard coding agent could handle mobile tasks with mobile-harness. The earlier evaluation used a different model and setup, so the comparison does not isolate the effect of the harness or memory.

Does this mean we will not use specialized agents anymore? No. For speed and lower token consumption, we would still choose a specialized mobile agent. It gives us more control over which model handles each operation, how much context it needs, and how many steps it takes.