# Understanding the Essentials of Mobile Test Automation

## **Preface**

When it comes to test automation in general, it's easy to say that it's complex. If we were to stick to the classic separation of the [Test Pyramid](https://martinfowler.com/articles/practical-test-pyramid.html), we'd end up with:

\- **Unit tests**, which can be implemented for both *backend* and *frontend*.

\- **Integration tests**, which today can mean anything from *API testing*, *microservices* and *third-party service* *integration*, *user authentication* and *authorization*, to *quasi-performance testing*.

\- **End-to-end tests**, which for the vast majority of test automation engineers and software developers would mean web application testing (to name a few major frameworks: [**Playwright**](https://playwright.dev/), [**Cypress**](https://www.cypress.io/), [**Selenium**](https://www.selenium.dev/)).

There are also tools like [**Pact**](https://docs.pact.io/) for *contract testing*, [**Storybook**](https://storybook.js.org/) for *advanced* [visual](https://storybook.js.org/docs/writing-tests/visual-testing) *and/or* [snapshot testing](https://storybook.js.org/docs/writing-tests/snapshot-testing/snapshot-testing) combined with *interactive documentation*, and probably many more that I don't even know about.

However, these tools are well documented and have huge communities around them, and to be completely honest - I don't think any of these tests are *hard* (except maybe the *contract tests*).

When I started working on mobile test automation, I quickly realised that this was not the case. The more I got into mobile software development, the more confusing it became, the mere amount of dependencies and requirements to just get the application running was really bizarre, let alone automating tests on it. The documentation on the various technologies was sometimes not very helpful, and there were no articles on the full spectrum of possible mobile environments.

So let's fix that and talk about mobile test automation, shall we?

## Development Frameworks Overview

First, before we go any further, we need to consider what technologies will be used to build the applications that will be tested with the mobile testing frameworks. The popular frameworks for mobile development are:

| **Name** | **Maintainer** | **Technology** | **Supported Platforms** | **Product Page** |
| --- | --- | --- | --- | --- |
| ***Flutter*** | Google | Dart | Android / iOS / Web / Windows / macOS | [https://flutter.dev/](https://flutter.dev/) |
| ***React Native*** | Meta | JavaScript | Android / iOS / Web | [https://reactnative.dev/](https://reactnative.dev/) |
| ***Ionic*** | Ionic | JavaScript | Android / iOS / Web / Windows / macOS | [https://ionic.io/](https://ionic.io/) |
| ***NativeScript*** | OpenJS Foundation | JavaScript | Android / iOS / Web | [https://nativescript.org/](https://nativescript.org/) |
| ***.NET MAUI*** | Microsoft | C# | Android / iOS / Windows / macOS | [https://dotnet.microsoft.com/en-us/apps/maui](https://dotnet.microsoft.com/en-us/apps/maui) |
| ***Cordova*** | Apache Software Foundation | JavaScript | Android / iOS / Web / Windows / macOS | [https://cordova.apache.org/](https://cordova.apache.org/) |
| ***Xcode*** | Apple | Swift | iOS | [https://developer.apple.com/xcode/](https://developer.apple.com/xcode/) |
| ***Android Studio*** | Google | Kotlin | Android | [https://developer.android.com/studio](https://developer.android.com/studio) |

According to the [2024 Stack Overflow survey](https://survey.stackoverflow.co/2024/technology#1-other-frameworks-and-libraries), ***Flutter*** is the most popular framework for cross-platform development. It's followed by ***React Native***, ***Electron*** (not listed in our table because it only supports desktop app development), ***.NET MAUI*** (and it's older brother ***Xamarin***, which has been replaced by ***MAUI***), ***Ionic***, and ***Cordova*** closing the list. Interestingly, there is no mention of ***NativeScript*** at all.

## **Mobile Test Automation Frameworks Overview**

Below is a general overview of the *test automation frameworks*. There are some other frameworks that I have intentionally left out because they do not seem to be very popular (e.g. [**EarlGray**](https://github.com/google/EarlGrey/tree/earlgrey2), which is Google's framework for iOS automation), and they are most likely wrappers of the technologies listed in the table anyway (e.g. **Selendroid** or any of the cloud mobile test automation providers).

| **Name** | **Maintainer** | **Main Feature(s)** | **Supported Native Platforms** | **Product Page** |
| --- | --- | --- | --- | --- |
| ***Espresso*** | Google | Google’s default testing framework | Android | [https://developer.android.com/training/testing/espresso](https://developer.android.com/training/testing/espresso) |
| ***XCUITest*** | Apple | Apple’s default testing framework | iOS / macOS | [https://developer.apple.com/documentation/xctest/user\_interface\_tests](https://developer.apple.com/documentation/xctest/user_interface_tests) |
| ***Appium*** | OpenJS Foundation/ Open-source ([Apache-2.0 license](https://github.com/appium/appium?tab=Apache-2.0-1-ov-file#Apache-2.0-1-ov-file)). | [Supports all](https://github.com/appium/appium?tab=Apache-2.0-1-ov-file#Apache-2.0-1-ov-file) of the mainstream platforms | Android, iOS, macOS, Windows, Roku, tvOS, Android TV | [https://appium.io/docs/en/latest/](https://appium.io/docs/en/latest/) |
| ***Detox*** | Wix.com / Open-source ([MIT](https://github.com/wix/Detox?tab=MIT-1-ov-file#readme)) | Tightly coupled with the React Native architecture | Android, iOS | [https://wix.github.io/Detox/](https://wix.github.io/Detox/) |
| ***Maestro*** | Mobile.dev **/** Open-source ([Apache 2.0 License](https://github.com/mobile-dev-inc/maestro?tab=Apache-2.0-1-ov-file#readme)) | Due to implementation of `*.yaml` files as its test files format, it’s probably the most effortless mobile test automation framework | Android, iOS | [https://maestro.mobile.dev/](https://maestro.mobile.dev/) |

### **Appium-based Frameworks Overview**

From the above list, **Appium** can be considered as the most comprehensive and therefore the most complicated framework of them all. In short, it consists of four basic elements: **Appium Core**, **Drivers**, **Clients** and **Plugins**.

We won't go into the details of what each element stands for, but to give you a better understanding of what technologies are supported by the **Appium** framework, here is a list of so-called **Clients**:

| **Name** | **Technology** | **Product Page** |
| --- | --- | --- |
| ***WebdriverIO*** | JavaScript (Node.js) | [https://webdriver.io/](https://webdriver.io/) |
| ***Appium Python Client*** | Python | [https://github.com/appium/python-client](https://github.com/appium/python-client) |
| ***Appium Java Client*** | Java | [https://github.com/appium/java-client](https://github.com/appium/java-client) |
| ***AppiumLib*** | Ruby | [https://github.com/appium/ruby\_lib](https://github.com/appium/ruby_lib) |
| ***Appium .NET Client*** | C# | [https://github.com/appium/dotnet-client/](https://github.com/appium/dotnet-client/) |

## **Scenario-Based Guide to Selecting Mobile Testing Frameworks**

When you would have to make one of the key decisions in the context of testing strategy, that is, the choice of a framework for automating mobile end-to-end testing the choice is quite complicated, as it depends on many factors.

Let's do a little thought experiment and try to describe some possible scenarios to help you decide.

***TL;DR:***

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1735644905798/aa9f459f-cd4f-4a6e-b349-0007b1ee027a.jpeg align="center")

* When developers are using native development tools and want to maintain grey/white box end-to-end testing and/or you’re building an MVP: **Espresso (for Android)** / **XCUITest (for iOS)**
    
* If you're developing an MVP and you only need UI testing ([without physical iOS devices](https://maestro.mobile.dev/getting-started/installing-maestro#connecting-to-your-device) and with [limited scripting possibilites](https://maestro.mobile.dev/advanced/javascript)): **Maestro**
    
* If you're developing a cross-platform **React Native** application and want to improve test scripting experience with [Large Language Models](https://wix.github.io/Detox/docs/copilot/testing-with-copilot): **Detox**
    
* For long-term projects that require stable solutions: **Appium-based** platforms
    

### **Scenario 1: Android or iOS Only**

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1735645790059/8b1b2e17-7ab7-4e5b-accc-ae894285d679.jpeg align="center")

The first scenario is an example for likely early stages of development (e.g. seed stage of a startup) when the focus is on only one mobile platform. This means that the natural choice should be either **Espresso** (for Android) or **XCUITest** (for iOS).

However, there may be other things to consider, such as:

1. **Resource-Constrained Environments:**
    

For example, if the application is being developed on a tight schedule and/or budget, and the developers simply do not have the time to maintain the end-to-end test suites, choosing complex **Espresso** or **XCUITest** frameworks can end up being a disaster.

In this type of environment, I would suggest using **Maestro** as it seems to be the most hassle-free and does not require too much time and effort to maintain test suites and configuration.

2. **Future Proofing for Long-Term Projects:**
    

If your project's time and budget constraints are not the things that keep you up at night, then you should probably think a little more about the potential development path.

I want to emphasise that we should ***avoid over-engineering***, but on the other hand, it's also important to choose a framework that we can use in a more mature test environment. By a more mature test environment I mean things like *automated test data generation and cleanup* (which can be achieved with [API-based model](https://engineering.cloudflight.io/test-automation-api-based-model)) or implementation into the existing *CI/CD pipelines*. And of course we want these things to be relatively easy to implement.

In summary, if you are sure that your project is entirely **Android** or **iOS** based, developers are the ones who will write and maintain the test suites, and you do not plan to extend it to the opposite technology, I would suggest using **Espresso** or **XCUITest**, as they will give you the most solutions adapted to your existing codebase. Otherwise, I'd lean toward the **Appium-based** solutions because they're much more extensible and can be used in black-box environments.

### **Scenario 2: Android and iOS**

Most of the time, companies want to target the broader audience, and since **Android** has ***~72%\**** of the market share, which means that **iOS** users have ***~28%\**** of the market share, it means that most companies want to target both platforms.

<div data-node-type="callout">
<div data-node-type="callout-emoji">💡</div>
<div data-node-type="callout-text"><em>*Data as of 31/12/2024 from </em><a target="_self" rel="noopener noreferrer nofollow" href="https://gs.statcounter.com/os-market-share/mobile/worldwide" style="pointer-events: none"><em>https://gs.statcounter.com/os-market-share/mobile/worldwide</em></a></div>
</div>

This scenario, at least from my perspective, is much more complex in terms of things that need to be considered before an informed decision can be made.

1. **Separate development teams for Android and iOS**
    
    ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1735646158995/95c49be3-5345-4c77-9e7e-8f5aeec678c4.jpeg align="center")
    

If your application is developed by separate teams using **native development tools** (i.e. **Android Studio** and/or **Xcode**), you can probably integrate **Espresso** and **XCUITest** to perform end-to-end testing. However, I would like to point out that this may be a subpar solution, as other frameworks allow you to test both platforms using the same test scripts.

So, for the situation mentioned in this sub-scenario, I'd suggest using **Espresso** and **XCUITest** for smaller scale tests (*component testing* if you will) that would be maintained by the developers themselves and integrating broader scoped frameworks for end-to-end testing (i.e. **Appium** or **Maestro**).

2. **Cross-platform development**
    
    ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1735646686785/3264fb84-651b-422a-84b8-292903597ed6.jpeg align="center")
    

As you can see in the table mentioned in the **Development Frameworks Overview** section, there are many possible frameworks to choose from to develop cross-platform applications. Most frameworks support all platforms (mobile, web, and desktop), but when it comes to mobile development, we focus mostly on **Android** and **iOS**.

Unfortunately, in order to figure out which framework we should choose to write and maintain end-to-end mobile tests, we need to proceed with the sub-categorisation of possible needs and settings.

* If you really don't have the time to maintain test suites, and therefore need something simple, almost effortless to write and maintain, I would suggest going with the **Maestro**.
    
* If you are an **SDET** or an **experienced test automation engineer**, you may want to lean more towards **Appium-based** tools, as **Appium** gives you all the benefits of mobile test automation as well as the freedom to configure the entire test automation framework yourself. I'm not saying that **Maestro** can't do this, but please take a look at the **Maestro Afterthoughts** for more insight.
    

3. **Cross-platform development with React Native**
    
    ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1735647313722/f77f7473-4f1c-443e-8dbe-c7e1fd88f245.jpeg align="center")
    

The tool I haven't mentioned yet is **Detox**. This is because **Detox** is specifically designed for grey-box end-to-end testing for **React Native**. I'd recommend giving it a try if you're developing a **React Native** application, especially given the fact that it's the first known framework to incorporate the [Large Language Models](https://wix.github.io/Detox/docs/copilot/testing-with-copilot) to write tests in natural language.

However, if you and your team are developing applications for external customers (so-called *enterprise-grade applications*) that require stable tools, I'd strongly recommend using **Appium-based** tools because they are much better documented and have much larger communities built around them.

### **Scenario 3: Android, iOS Web and More**

The last scenario we'll talk about is native mobile development combined with any other version of the app. This is paradoxically the easiest of all.

**Appium**.

There is simply no other solution that would allow you to test on all of the available platforms.

Of course, if you'd like, you can mix and match **Maestro** for mobile testing and **Playwright** for web testing. Or **Detox** for mobile, **Appium** for smart TV, and **Selenium** for web, if that's what you want. It just doesn't seem like a good idea, especially since some of the **Appium-based** frameworks allow you to create a single test script that would run sequentially on multiple platforms.

For example, **WebdriverIO** allows you to do something on one mobile device, then switch to a separate web application to change/verify values and perform some other actions on another mobile device.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1735648112651/fc0ceb5e-601d-4f59-8957-aba039b30cd7.jpeg align="center")

## Maestro Afterthoughts

I felt that **Maestro** might seem like a great solution in the end, which it can be in some cases, but I also want you to think about possible downsides if you decide to go with it.

**Maestro** gives you the ability to configure the test automation framework (e.g. [run JavaScript code](https://maestro.mobile.dev/advanced/javascript)), but since it's written with a specific architecture in mind, it's more of an add-on to the standard **YAML** flow files than anything else.

Another thing to keep in mind when talking about the **Maestro** is last year's [Cypress intellectual property update case](https://www.reddit.com/r/softwaretesting/comments/17wqis1/why_is_noone_talking_about_this_cypress_blocking/). In a nutshell, the **Cypress** team [removed the ability to use third-party services for dashboards](https://www.cypress.io/blog/update-defense-intellectual-property) that were previously free, essentially forcing companies to use their own cloud platform.

Not only is that platform not free, but it also prevents **Cypress** from being used where contracts with customers specifically block the use of cloud platforms due to GDPR or similar laws. I mention this because **Maestro** has its own cloud platform for running tests called **Robin**, and it's possible that the similar scenario to the **Cypress** update could happen to **Maestro** as well.

## Development for the Apple ecosystem

The last thing I wanted to mention here is that if you want to write tests for the **Apple** ecosystem, namely **iOS**, **iPadOS**, **macOS**, or **tvOS**, you actually need to own a machine based on **macOS** (either a MacBook, Mac Mini/Studio/Pro, or iMac) because you need to have **Xcode** installed.

The reason is simple - it contains the **Simulator** application, which allows you to run your application in the virtual environment and is required to run tests on **real devices**. You'll also need an [Apple Account](https://developer.apple.com/support/compare-memberships/), and if you want to publish your application on the **AppStore**, you'll need to sign up for the paid [Apple Developer Program](https://developer.apple.com/support/compare-memberships/).