Why Mobile Is the Hardest and Most Valuable Environment for Computer-Use Agents

If you give an agent a laptop and a task like - find last month's invoice and email it to accounting, these days it will easily get it done.

Now try giving it a phone and tell it that the invoice is inside a banking app with no website. Login needs a code that arrives by SMS. The app checks whether it's running on a genuine device before it shows you anything. 

That's the gap we work in every day. Mobile is where agents struggle the most, and it's also where most of the work actually is.

The benchmarks say mobile is solved but it isn't.

If you judged mobile agents by the leaderboard, you would conclude the problem is close to solved. 

AndroidWorld is the standard benchmark, and the best agents now clear 99% on it. We have posted 100% on it ourselves.

The research community has noticed the saturation and MobileWorld was built in direct response, arguing that AndroidWorld's tasks are too short and too rarely cross between apps, and reporting much lower scores as a result. 

Those critiques are right and they stop short of the real problem. They are arguments about task complexity but the deeper issue is the environment itself.

AndroidWorld is designed to run in a controlled emulator environment, where tasks can be repeated and evaluated consistently. That’s exactly what a good benchmark should do.

But real-world mobile apps are much less predictable. And many of the problems that benchmarks leave out are the same ones that cause agents to fail in production:

• OTPs that expire before the agent can use them.
• Rate limits, security checks, and account suspensions
• Unstable networks, login portals, and proxy failures
• App layouts that change during a tasks
• Apps that detect and block emulators
• Devices that need to be reset before they can be used for the next task.

These problems are difficult to include in a benchmark because they make results harder to reproduce.

So, a high AndroidWorld score shows that an agent can handle tasks in a mobile interface under controlled conditions. But it doesn't necessarily mean the agent can complete those same tasks on a real phone, with a real network, and after using multiple accounts in a day.

That is the honest reading of our own result too. It measured the part of the problem that is measurable. The part that costs us engineering time is the other part.

So why call mobile the hardest?

These benchmarks run in controlled environments. Our 100% AndroidWorld result came from a Pixel 6 emulator running Android 13, while our iOSWorld run used an iPhone 17 Pro simulator running iOS 26.5.

AndroidWorld covers 20 apps, including open-source apps and built-in Android apps. Its tasks are designed to be reproducible and do not require login or authentication.

That makes it useful for measuring an agent's ability to navigate mobile interfaces and complete tasks. But it doesn't test every challenge agents face on real devices, such as account verification, network issues, or apps with additional security checks.

Make the tasks look more like real phone use and the scores drop. The MobileWorld team at Alibaba built 201 tasks with roughly twice the steps (27.8 vs 14.3 on average). 62.2% of them span more than one app, against 9.5% in AndroidWorld. The best agent setup they tested scored 51.7%. The best end-to-end model scored 20.9%.

A 97% benchmark score and a 52% one, both measuring "mobile agents." The difference is how much of the real phone shows up.

Why phones fight back

There is no DOM

A browser agent can read the page as structured HTML. A phone gives you two options: the accessibility tree, if the app's developers filled it in properly, or raw pixels. Custom-drawn views, games, and canvas-heavy apps often expose little of use in the a11y tree. The agent then has to read the screen like a picture and guess where to tap.

Every app is its own little operating system

There is no shared layout across apps. Each one ships its own navigation, gestures, and the idea of where "Save" goes. Then it changes every few weeks. A script that worked yesterday breaks tomorrow because a bottom sheet replaced a dialog.

The OS is built to keep things out

As we wrote in our Android agent post, "Android is a sandbox. Apps can't read other apps' screens or drive their UI without elevated permissions." iOS is stricter: "Apple's sandboxing means there's no official way to remotely drive a physical iPhone."

Apps check if the phone is real

Google's Play Integrity API lets an app ask whether it's "running on a genuine and certified Android device." On Android 13 and up, that verdict comes with hardware-backed proof of a locked bootloader. An emulator that doesn't pass Google Play integrity checks gets an empty verdict, and apps decide what to do with that. Plenty of them simply stop working.

Then there's the SMS code. Two-factor login wants a real number on a real device, and an emulator in a data center doesn't have one.

Small screens, real gestures

Long-press, swipe to dismiss, pull to refresh, pinch, a keyboard covering half the screen. A list with more items below the fold. Our benchmark notes put it plainly: "Text can be entered without being saved. A tap can open the wrong item." On mobile, a tool call that succeeds tells you very little. You have to check the change actually stuck.

Why it's worth it anyway

Because that's where people are.

Mobile users averaged about 3.6 hours a day in apps in 2025, 5.3 trillion hours in total across iOS and Google Play. Phones generated 58.96% of worldwide web traffic in September 2026, and that counts only browsing, not time inside apps.

In mobile-first markets the share is higher still: 77.3% in India and 68.7% across Africa. Weixin and WeChat alone had 1.439 billion monthly active accounts as of June 2026. 

A lot of that activity has no API and no web version. Vinted sellers run their shop from the app. Plenty of small businesses run their sales through WhatsApp. Developers who need payout data from a banking app hit the wall we described in How to Automate Apps With No API. If an agent can only use browsers and APIs, it can't reach a large share of the services people use every day.

The desktop is where agents learned to work, the phone is where the work is.

What it takes to make agents work on a phone

We've built Mobilerun around this problem, and four things keep coming up.

Read structure first, pixels second. Mobilerun reads the accessibility tree where it exists and falls back to screenshots where it doesn't. The tree costs fewer tokens than a stream of screenshots, and the agent gets real element labels instead of a guess about coordinates.

Use real devices when the app checks: Emulators are fine for testing your own app, but for apps that run integrity checks, you need actual hardware. Mobilerun runs real Android phones and real iPhones in the cloud. Physical devices come with eSIM and GPS and receive SMS codes directly.

Describe the goal, not the taps: You write a task in plain language and the agent works out the steps on the current screen. When the UI moves, the agent adapts instead of breaking. Once a flow is stable, you can replay it as a deterministic script with no model in the loop.

Verify outcomes: Our harness asks the agent to confirm a change before it reports a task as done.

The framework is open source under the MIT license and LLM-agnostic. You can run it on your laptop, connect a phone you already own through the Portal app, and move to the cloud when one device isn't enough.

Conclusion: The hardest environment may be the most important one

Mobile is a difficult environment for computer-use agents because it combines dynamic interfaces, device-level behavior, authentication, and unpredictable real-world conditions.

But those same characteristics make it an important frontier.

A large part of digital life already happens inside mobile applications. If agents are going to move beyond answering questions and begin carrying out useful work, they will need to operate in the environments where that work actually takes place.

The challenge is to move from agents that can perform isolated interactions to systems that can complete entire workflows reliably, repeatedly, and with appropriate oversight.

That requires better models, better feedback loops, and better infrastructure working together.

Also Read:

• Doubao-vs-Mobilerun
• Automate QA without writing Scripts
• Emulators vs Real devices