Automation Insights

Synthetic users vs human testerswhere each one actually wins

Priyanov 10, 20267 min read
Share

People often ask, if an AI agent can use our app like a person, do we still need human testers?"

Short answer is yes. Longer answer: you need far fewer humans doing far less boring work, and you should stop asking them to do the things agents are better at. This post is about drawing that line clearly.

First, what do we mean by "synthetic users"?

The phrase gets used for two different things, so let's split them:

  1. Synthetic research participants: LLMs simulate users and generate responses to interviews, surveys or research questions. This is an emerging area of UX research.
  2. Synthetic users that interact with apps: AI agents receive a goal, such as signing up, adding a payment method or ordering a pizza, and execute the workflow through an actual application interface.

This article focuses on the second category: AI agents that interact with mobile apps like users do.

What synthetic users get right (and where they fall short)

1. Repetition without boredom

A human tester running the same signup flow for 40th time in a single week may get mental fatigue and lack attention. An agent on the other hand, doesn't get tired, doesn't skip the step it "knows" works.

2. Testing across devices

Mobile testing involves more than just checking whether an app works. Teams need to account for different OS versions, screen sizes, manufacturers, languages, regional settings, and network conditions, creating hundreds of possible configurations. AI agents can run the same test workflows across multiple devices, while cloud device infrastructure provides access to real Android and iOS devices without the need to maintain a physical device lab.

3. testing different user scanarios

You can give each run a persona: a new user, a returning user with an expired card, someone with the phone set to German and 200% font size. Because the agent is driven by plain language, a persona is just a paragraph of text.

4. Not breaking when the UI moves

Script-based tests break when a button moves or an ID changes. An agent re-reads the screen each step, so a redesign doesn't automatically mean a red build. MobileRun does this by reading the accessibility tree on every step, and only sends screenshots to the model when you turn vision on.

5. Watching production

After release, the same flows can run on a schedule against the live app, as a kind of synthetic monitoring. If checkout breaks in one region at 2 a.m., you find out from the run log, not from support tickets.

Where human testers win

1. "This feels wrong"

An agent can tell you the delivery fee appears on the final screen. A human tells you it feels like a trick. That's the difference between a passed test and a one-star review.

2. Discovering unexpected behaviour

Traditional test cases cover expected user behaviour, but real users often do the unexpected: misread labels, tap the wrong buttons, switch apps mid-workflow, or abandon tasks halfway through. AI agents can explore these scenarios, but human exploratory testing remains important for spotting and investigating behaviours that weren't anticipated in the test plan.

3. Evaluating real accessibility

AI agents can check accessibility labels, missing descriptions, text scaling, and navigation behaviours. However, automated checks can't fully capture the user experience. Testing with people who regularly use TalkBack or VoiceOver is essential to evaluate whether navigation, reading order, and workflows are genuinely accessible.

4. Making Judgement calls

Not all quality issues are pass-or-fail checks. Agents can flag unclear consent screens, misleading cancellation flows, dark patterns, and potential compliance concerns. But evaluating context, making judgment calls, and taking responsibility for the outcome still requires human oversight.

5. Understanding what users actually experience

You don't need hundreds of participants to uncover usability issues. Even a small, focused group of users can reveal problems that automated testing might miss. Human testing helps identify friction, confusion, and unexpected behaviours that agents may not recognise.

AI agents vs. human testers: Side by side

Testing task

AI agents

Human testers

Regression testing on every build

Strong fit

Repetitive and time-consuming

Device and OS coverage

Strong fit with device infrastructure

Limited by available time and devices

Testing predefined personas

Strong fit with configured environments

Useful for validating real user behaviour

Reproducing reported bugs

Strong fit

Useful for initial investigation

Discovering unexpected behaviour

Useful, but limited

Strong fit

Evaluating user experience

Limited

Strong fit

Accessibility checks

Good for automated checks

Essential for real-world usability

Production workflow monitoring

Strong fit

Not practical continuously

Release sign-off

Provides test evidence

Makes the final judgement

How to combine AI agents and human testers

A practical mobile testing pipeline doesn't replace your QA team. It uses agents for repetitive testing, freeing human testers to focus on exploratory testing, usability, and more complex scenarios. Here's how to structure it.

1. Every new build: AI-powered regression testing

Agents execute predefined workflows across configured devices, OS versions, and app states, covering scenarios like signup, login, navigation, cart management, and checkout. Failed runs generate recordings, screenshots, and step-by-step logs to help teams investigate issues.

2. Regularly: Human exploratory testing

Human testers focus on new features, confusing interactions, unexpected behaviour and usability.

They explore the application without being restricted to predefined test cases.

3. Turn discoveries into new test goals

When a human tester discovers a problem, the team investigates it and creates a reproducible test. For example, a tester might discover that a checkout screen silently preselects a tip. That discovery can become a new test goal.

4. Continuously: Agents repeat validated checks

Once a test goal is established, agents can run it on subsequent builds and across relevant device configurations. The loop is what makes this approach valuable. Humans keep discovering new problems. Agents help ensure that problems already discovered don't quietly return.

What does this look like on real mobile devices?

Mobile testing requires more than application access. Agents need configured devices, OS versions, valid accounts, and the ability to interact with real interfaces. While emulators cover many testing scenarios, they don't reproduce every aspect of physical devices. MobileRun provides infrastructure to run AI agents on real Android and iOS devices, automating workflows across actual mobile environments. With the open-source Mobilerun framework, you can connect an Android device over ADB and give an agent a testing goal in plain language. For example, testing a food delivery app across different user scenarios.

The honest limitations of AI-powered mobile testing

AI agents are useful, but they aren't perfect testers. There are several limitations teams should account for before relying on them.

1. Agents can fail even when the application works

An agent might misinterpret a screen, select the wrong element or fail to complete an otherwise valid workflow. A failed run doesn't automatically mean there's a bug in your application.

This is why recordings, screenshots and step-level logs matter. Teams need enough context to distinguish an application failure from an agent failure.

2. Agents can miss unexpected behaviour

Agents are effective at executing defined goals, but they don't have the same open-ended curiosity or lived experience as human testers. They can explore, but their findings still need human evaluation.

3. Realistic environments are difficult to maintain

A test is only as useful as the environment in which it runs.

Device configuration, account state, permissions, network conditions and application data can all influence the result. Teams need consistent test environments to make their results meaningful.

4. Reliability needs to be measured

Agents still struggle with complex mobile interactions. On MobileWorld, a benchmark designed to evaluate mobile agents on realistic tasks, even the best-performing agents completed only around half of the tasks in the referenced evaluation.

This is a reminder that successful automation on a few simple workflows doesn't necessarily translate into reliable performance across complex applications.

5. Every run has a cost

Agent execution isn't free. Running hundreds of scenarios across multiple devices can increase execution costs, especially when workflows require many model interactions.

Start with critical user journeys and the most relevant device configurations. Expand coverage based on actual findings rather than running every possible combination.

Frequently Asked Questions

Can synthetic users replace user research? 

No. Use agents to check behaviour, and humans to learn what people think and feel.

How many personas should I start with? 

Three to five that map to real segments: brand new, returning, a known edge case account, and one accessibility setting. Add more only when a human finding suggests one.

Do synthetic users work on apps we don't own? 

Yes, an agent drives the UI the same way a person does, so it works on competitor apps or third party apps your flow depends on. Stick to the app's terms of service.

Also read:

Let agents do the repetition, the matrix and the night shift. Let humans do the noticing. If you want to try the agent half on a real phone, you can start at cloud.mobilerun.ai.

Also Read: