iOS End-to-End Testing in a Nutshell

Martin Mochetti
Mar 19, 2020 · 10 min read

Recently we had the necessity to start testing our iOS application with integration tests, more known as “end-to-end” tests. So we came with a simple yet beautiful approach.

Introduction

Over the past 11 years we’ve been crafting a majestic Ruby on Rails monolith that serve our purposes. It goes from handling web requests to mobile API requests. A monolith that has an extensive test suite that uses . It took all this time to code not only this monolith, but also this testing framework that allows us to sleep well at night.

On the other hand, our mobile app testing consisted of: a scarce set of unit tests within the same repository, and a separate repository that performed a reduced amount of integration tests using .

This mobile test suite was using our staging servers to run the tests in two clearly separated stages: first create all the background data needed for the tests to run, and then run all tests.

But we needed something better.

Challenges

Having a test suite that uses external servers as the single source of truth introduced multiple drawbacks. The most notable were:

  • Performance. Tests performed multiple API requests against an external server. This made local test executions slow as they needed an active internet connection.
  • Independence. The execution of one scenario was dependent on this exhaustive initial data setup. This setup created all the necessary data for all the available scenarios. Even if you wanted to run one single scenario, you had to wait until all the data for all tests was created. This makes testing development slow.
  • Isolation. We had to hope that the background data was going to be exactly as we wanted it to be. If a prior test failed leaving data in an inconsistent state, there was no guarantee whatsoever that subsequent tests were going to pass. One failing scenario could potentially make the whole suite of tests fail. This makes testing development non trivial as you have to write tests that don’t modify data, therefore losing many use cases, or have to make sure tests run in a “safe” sequential order making them dependent one of the other. If one test fails, don’t run consequent ones. This makes testing troubleshooting slow.
  • Idempotence. Due to the isolation drawback, tests weren’t idempotent either. There was no guarantee that a test would produce the same outcome each time it run. This makes test debugging a lot harder as the same test could pass while working with different data sets.

What we wanted

Having our web testing framework working smoothly, we didn’t want to re-invent the wheel in order to test our mobile applications. In the end, a mobile application is not more than a different UI client on top of the same (or similar) business logic as the web UI. So we wanted to reuse as much as we could.

Here’s a list of things we aimed for:

  1. A mobile developer should be able to add a test and commit them within the same pull request as the code being tested. No separate testing project. This helps in making sure tests are mandatory. One pull request would have everything, therefore being easier to review.
  2. Each test being run should start with a fresh set of data that is only related with that test. This means we could run the same test N times and the outcome would be the same.
  3. We should be able to use the same testing framework developed for our web application (factories, scripts, support files, etc). This way we allow data to be created/destroyed on each test run without duplicating code.
  4. Use appium as a way to interact with a mobile simulator. Appium is a testing driver that does similar things to what selenium does for the web.
  5. Tests should run on our local setup. No external servers whatsoever. They should start a local web server that exposes all the APIs that the mobile app needs, and the mobile app being tested should hit those APIs.

To tackle all these requirements, we came up with a clean setup. It requires 6 steps, but it definitely worths it.

1. Symlinks to the rescue

You might say: “what? symlinks?”. Yes, symlinks. The pure action of creating a “mirror” facing a folder or file in your file system is very much underrated.

We chose this path because:

  • We didn’t want to duplicate the web testing code.
  • We didn’t want to move all the web testing framework to a separate repository, therefore we couldn’t use git submodules.
  • We didn’t want to use git subtrees as they are hard to reason.
  • We wanted this to be as less intrusive as possible for our web project.

So this is what we did:

  1. Created a test/integration folder within the mobile project root folder.
  2. In that folder created a install.sh script within an exe folder with the following code:
install.sh

In this script, we are first mirroring setup sources from our web project by symlinking the Gemfile, Gemfile.lock, .ruby-version and .rspec file.

Then we symlink an integration_helper_core.rb file. This file is similar to the spec_helper.rb suggested by the rspec gem. It basically contains all the setup needed by capybara -screenshots saving, before and after hooks to create/destroy the database, web server setup, etc- that will be shared by both our web and mobile testing projects. This file will live in our spec folder, so you should create this folder before running the install script or you’ll get a “folder/file not found” error.

Finally we symlink the parts of the web testing framework we need: factories & support files.

We also introduced an uninstall.sh script that allows us to remove all these symlinks created:

uninstall.sh

At this point, your folder structure should look like this:

Step 1 folder structure
Step 1 folder structure
Folder structure before installing the test framework

We are now officially ready to install our testing framework. So standing on the test/integration folder and executing exe/install.sh would result in the following files and folders:

Folder structure after installing the test framework

Given that we don’t want to commit all these symlinks to the mobile app repository, the .gitignore file contains the following:

Gemfile
Gemfile.lock
.ruby-version
.rspec
spec/integration_helper_core.rb
spec/factories
spec/support
build

2. Building the test binary

It’s desirable that any mobile project has its own build target for testing. In our case, we have a build target called “Test” that generates a binary with a hardcoded API base url identical to the one exposed by rspec/capybara when starting the web server. So next step you’ll have to do in your iOS app is to create a new build scheme. I’m not going to dig too much into how to configure a build scheme in an iOS app. I’m not a mobile expert and it would make this post beat around the bush.

After having the build scheme and our generation process properly configured, we need a command-line-way to run the build so it can be used by our CI tools before running tests. This is why we added a build.sh file under the exe folder with the following content:

build.sh

Note that our .xcworkspace file is called MobileApp and our build scheme is MobileApp (T). You should replace those two things with your own names.

After running this script by executing exe/build.sh, you should see the generated binary in this path:

test/integration/build/Build/Products/Debug-iphonesimulator/MobileAppT.app

Finally, after running this step, your folder structure should look something like this:

Folder structure after adding the build step

3. Appium setup

In the first step we symlinked the Gemfile of the web project to be used by the mobile testing porject, so you’ll need to add the appium gems in the web project Gemfile:

group :test do
....
# Mobile testing
gem 'site_prism', require: false
gem 'appium_lib', require: false
gem 'appium_capybara', require: false
...
end

SitePrism is used to be able to target UI elements within each of the mobile app screens. It will allow us to fill in a text input, for example.

Note that we’re adding a require: false keyword so that we can require them on demand. We’re only going to use these libraries for our mobile testing purposes, so we don’t want them being included in our web project by accident.

After appium has been added to the web project Gemfile, we will need to create two file: config/appium.txt containing some appium configuration settings, and spec/appium_helper.rb similar to a spec_helper.rb but with basic appium setup.

appium.txt

Note that you’ll have to replace the app variable with the path to your testing binary.

appium_helper.rb

You might have noticed already that we’re using appium with capybara. But in case you didn’t, here’s the project repository.

4. SitePrism setup

An important thing already mentioned is that we are using SitePrism to interact with the elements in our screens. It adds an abstraction layer that is very easy to reason about.

So following some RoR good practices, we added all our screens in spec/screens and added a base_screen.rb class so that we can have a place to write common behavior across all screens. Also, for the sake of this demonstration, we are adding a login_screen.rb screen that will be used down the road.

After adding these two classes, our spec folder looked like this:

Spec folder after adding SitePrism screens

And this is how this two classes looked like:

base_screen.rb

See how beautifully simple is this action-driven way of coding screens. Later, when writing a test, we will be able to do something like login_screen.login_as(user).

Besides adding our screens, we also want to have a helper to be able to easily include them in our tests. So we came up with this screens_helper.rb module that we located under our spec folder:

screens_helper.rb

Note that to make use of this module you’ll need to require this file in the appium_helper.rb.

5. Tests setup

To run our tests we created a small DSL that allows us to have a fast way of defining feature specs and extract some common setup across all tests. It encapsulates the rspec feature function and exposes a global one of our own that introduced handy shortcuts. We placed it in our appium_helper.rb and it looks like this:

feature function exposed in appium_helper.rb

This function is doing some things for us:

  1. Creating an account and a user of our app using FactoryBot. It also allows to pass certain permissions to restrict user access to our app.
  2. Instantiate screens on demand for our tests
  3. If an authenticated option is passed to the feature declaration, we log in the user to the app before running the test.

Note that this function was implemented to fulfill needs. You may want to have your own flavor of it.

6. “Hello world”

Now that we have capybara, rspec, appium, SitePrism and a small DSL to make our life easier, we’re ready to start kicking. So let’s get on with it.

We can start by writing a small proof-of-concept test that validates the logging in flow of our mobile app. We put all our feature specs in a folder called features and our first test looks something like this:

And if you’re curious, our landing screen looks like this:

And that’s it! Our first test is ready to roll out.

The folder structure after following all these steps should look something like this:

folder structure after the whole setup is complete

Now you are able to run this first test by simply doing:

$ rspec spec/features/login_spec.rb

Takeaways

Mobile end-to-end testing is not trivial. The resources out there aren’t plentiful and it requires a deep thinking time. Many mobile projects don’t have integration tests due to this, so there’s a vast field out there to explore.

This project aimed to tackle end-to-end testing without the need of fixtures or mocks, which in many cases are hard to keep in sync with reality. By including what we needed from our web testing project we were able to run this type of tests in our locals without hassle. Yes, it has some hard-requirements like needing to checkout the whole web project -or the API project- before being able to set it up. But in an era where you lift a rock and find a full stack developer, this shouldn’t be a thing oblivious to reality. Many mobile devs are also building the backend APIs these days.

As a final reflection, sometimes you may be ok by using out-of-the-box solutions for your problems. But sometimes you just go in circles trying to integrate them. It’s right then when you have to stop and start over. Start from the green grass. Start by thinking out of the box, analyzing what you have and what needs to be done. If you do this more often you’ll find it rewarding for three very important reasons:

  1. It’ll allow you to tackle your needs and your needs only.
  2. You’ll also make code easier to maintain and understand in the long run.
  3. You’ll learn something new.

Hope this helps you in some way.

The Startup

Get smarter at building your thing. Join The Startup’s +787K followers.

Sign up for Top 10 Stories

By The Startup

Get smarter at building your thing. Subscribe to receive The Startup's top 10 most read stories — delivered straight into your inbox, once a week. Take a look.

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Martin Mochetti

Written by

Technical lead @ CipherHealth LLC. https://cipherhealth.com/

The Startup

Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +787K followers.

Martin Mochetti

Written by

Technical lead @ CipherHealth LLC. https://cipherhealth.com/

The Startup

Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +787K followers.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store