Story behind AutomationTestSupervisor — our custom made tool for Android automation tests
AutomationTestSupervisor is a Python tool that is capable of creating, running Android Virtual Devices, building and installing .apk files on devices, running tests and generating logs. It’s strongest point is customizability as it allows you to easily manage which tests should be ran in session, devices to be used, paths on machine to be read and more. It allows to create run profiles which are set once and can be reused after.
AutomationTestSupervisor project repository is available here.
Example Android project using AutomationTestSupervisor is available under this link.
Why we decided to reinvent the wheel?
Over two years ago we started building automation test architecture for Azimo Android app from scratch.
As we had only test code at start, we had to use built in InstrumentationTestRunner commands for running tests. But it’s provided us with not enough information about test session. Because of that we started using 3rd party tools like spoon from Square Engineering with its spoon-gradle-plugin wrapper from Stanfy.
Spoon
Spoon worked for us for over one year but we had many troubles with it. Sometimes when new versions of gradle dependencies appeared we couldn’t adopt them instantly as there were some problems with spoon-gradle-plugin using older version or just not being adjusted to core spoon project changes that were fixing some serious issue. Whenever we had problem with tool (e.g spoon was crashing test session on taking screenshot, couldn’t fetch screenshots from devices of API23+, spoon freezing and pulling screenshots forever when we even didn’t use screenshot feature) we had to ask for help via GitHub issues and test our app manually. We were unable to add new features or change the way log output looks like as it was hardcoded.
Additionally spoon is capable of starting tests only on currently running devices, so AVD launch part is not automated. In order to do it we were forced to create our own scripts like fastlane-plugin-automated-emulator-run that wrapped spoon launch command in code and handled emulator booting before tests.
But that also required usage of another third party libraries like fastlane. Number of libraries we used, dependencies to keep up to update, documentations to read, GitHub repositories to visit, programming languages we used grew and despite of that quality of our test session wasn’t stable enough to be scheduled for night runs.
After a while we reached over 450 end-to-end tests. We had many test packages that we wanted to run separately. Spoon is capable of running only one package at the time. Whenever we wanted to test specific part of our app with a single test package we had to wait for script to launch an emulator from scratch and then for spoon to build .apk, install it, test it and generate logs. Multiply this operation by ~25 test packages. Our test session could last over 5 hours!
It was huge waste of time. Additionally spoon is using basic InstrumentationTestRunner sharding. So if we had 3 AVD launched, then tests from single package were divided unevenly — decided by Google’s built-in algorithm. Usually one shard finished running ahead of others. In result you had 3 AVD launched but only part of them was actually used. The other part had to wait sometimes few minutes until whole package tests will finish running. You could start another test package on AVD which already finished running all the tests from it’s shard.
That’s why we decided to make single scalable tool that will solve all those problems and grant us possibility to adjust everything the way we need and like. We needed to be independent in order to achieve robustness.
We started AutomationTestSupervisor in February 2017. With systematic but small iterations, we finished beta version in July 2017. From that point we’ve replaced all the tools mentioned before. We are making further upgrades and adjustments according to our experiences with it. Much more things could be improved, that’s why we want to open source it and are aiming to work on it together with community.
AutomationTestSupervisor — what does it have to offer?
Tool can be divided into four main modules where each of them has it’s own config manifest file. Modules are integrated which each other and forming linear pipeline with set of steps. By modifying manifest of each module you can adjust behaviour of each step.
Path Module
This module is responsible for storing all machine related paths on which ATS will build it’s files. Depending on your setup ATS can be either another project stored on each PC that you use for testing, submodule of your Android project or even part of it.
If you have already set ANDROID_HOME or ANDROID_SDK_HOME environment variables then all you need is to set path to your folder with .apk files. If you want them to be built by ATS then you need to specify path to your Android project because access to gradle binary file is necessary to trigger build commands of your application.
Lauch Plan Module
This module can be compared to global config.
In this manifest you can decide how many ADB commands can be launched at the same time and specify time interval between them. Weaker machines have problem with handling many ADB commands. Too many calls to ADB at the same time may result in error. Additionally it is possible to pick if AVD should be reused or recreated according to schemas from AVD module. You can decide if you want to reuse existing .apk or build new one. There is possibility to modify how machine should wait for AVD launch - all at once (not recommended) or in sequence. You can also pick launch timeout value or specify list of ignored devices. There is possibility to use screen recording feature. Screenshot feature is not supported, if you want to learn more about it visit our repo.
AVD Module
This module is responsible for AVD creation. It stores AVD models in JSON schemas. You can create as much schemas as you want and bind them to AVD groups.
To create schema you need to fill data according to Google’s documentation. There is possibility to specify absolute path to emulator image if you want to load some previously prepared device state. You can also overwrite config.ini. It is recommended to use config.ini as you are unable to set every parameter of AVD from terminal. From created schemas you can combine sets where you can pick how many instances of each AVD should be launched. ATS will adjust names and pick open ports for you. You can also modify which ports you would like to use.
Usage of this module is optional. ATS is capable of running tests on currently launched devices. The only restriction is that if you decide to use AVD module then real devices will be ignored and all other currently launched AVD will be killed. ADB cannot handle many sessions. Did you try building Android .apk while tests are running? You can’t. The same way ATS needs to take over control of ADB.
Test Module
This module grants you possibility to create sets of test packages but beforehand you need to list up all test packages (the same way you lists activity in your Android app manifest).
It might look like a lot of work but it’s actually not. Remember that you are doing it only once. Fields “test_classes” and “test_cases” are unused. We had in plan to run every test case separately and clear app state every test and those fields were for this purpose. But this feature was provided by Google in form of adb orchiestrator few months ago so there is no need to implement it anymore.
In the end all you need to do is create alias — shortcut of package name and set package name to it. Example of how it could look like can be found here.
Additionally you can specify more test parameters like shards, gradle parameters, gradle task for building application .apk and test .apk. You can hardcode .apk name but it won’t be too practical if it contains versioning. Therefore you can insert part of your .apk name. ATS will search for .apk files containing this name part in “apk_dir” of Path Module. If there is no .apk found (or there is .apk but there is no test .apk built for it) ATS can build one if you have set “project_dir” in Path Module and set “build_apk” option in Launch Module. If there will be more than one .apk matching name part ATS will search for .apk with highest version code.
Launching ATS
If you have properly set your machine then you need one line of code to launch ATS. You have four optional parameters:
- aset (AVD set)
- tset (Test set)
- lplan (Launch plan)
- pset (Path set)
Example how command can look like:
python3 Launcher.py -aset 3AVD-23 -tset function_all_shardI am specifying which tests should be ran and which AVD should be launched. I don’t need to specify other parameters because I am using pre-configured defaults. Order is also not important.
There is much more you can expect
There are much more good points that we cannot fit in this very first post about ATS so we will list them up.
Few additional features of ATS:
- will check if application, which is currently on device. If so then ATS will remove it in order to avoid INSTALL_FAILED_VERSION_DOWNGRADE error.
- will scrap your .apk manifest to find InstrumentationRunner and app packages.
- installs .apk files in parallel to save your time.
- aligns sharding — if you want to run many test packages on many devices you won’t need to wait until whole package finishes, if any device is free then it starts tests from next package. We have speed up our tests from 1,5h to 25 minutes that way!
- it provides you logs from AVD.
- it provides you with LogCat from each test.
- it provides you with logs in form of JSON files.
- it generates whole HTML dashboard displaying stats from test session: test statuses, links to formatted LogCats, link to videos and more.
- will generate log output from every test that has finished so far even if you stop in middle session.
That’s it
We consider ATS as mid-level tool for people that wants more control over their test session. It’s perfect for companies that doesn’t have device farms but needs to automate their testing process.
We are looking forward for any feedback, pull requests, issue requests.
Towards financial services available to all
We’re working throughout the company to create faster, cheaper, and more available financial services all over the world, and here are some of the techniques that we’re utilizing. There’s still a long way ahead of us, and if you’d like to be part of that journey, check out our careers page.
