It works on my device
I bet everyone who has been involved in software development projects has heard a developer say “It works on my computer!” after you tell him there is a problem.
In the last years that sentence changed slightly and adapted to the new reality of mobile Apps, so we started hearing our developers annoyingly say “It works on my device!”. But let me start by telling you how we got there.
We have been developing mobile Apps at Truewind for a long time, even before OutSystems came out with the amazing platform 10. Back in the day we developed several enterprise solutions for Windows Mobile devices, some of them still in use a decade later. As technology evolved we developed native Android, iOS and OutSystems Now Apps, mostly in a controlled setting because these enterprise applications targeted a set of homogenous devices.
When we started delivering public mobile Apps we knew we had to broader our tests, so we tested our Apps on different devices: the company’s testing devices (we didn’t have many… an Android phone and a tablet and also an iPhone and an iPad…) and also in the developers own devices. With this, we were able to have some different devices tested and some quality control over what was being delivered.
But we were aware this wasn’t enough, so when the opportunity came with a customer that was was willing to invest in delivering the perfect App we went out and explored the world of mobile Apps testing.
The Challenge
We needed to find a way to test our Apps in multiple devices, brands, operating systems, versions, form factors and so on.
We also needed to be efficient in doing so, we couldn’t go out to a local electronic shop, buy every available device and put together a room full of people doing manual tests in all of the devices.
The Solution
The solution was simple, we needed to use a cloud service to run automated tests on real devices.
After analyzing several of the available cloud services we ended up choosing Visual Studio App Center (formerly known as Xamarin Test Cloud) for several reasons:
- It had the most comprehensive set of devices
- It had the most comprehensive set of features
- We could develop our test scripts in C#, which is perfect since we have this skill in house
But we needed to validate the Visual Studio App Center in a scenario using an OutSystems 10 App, so we made a pilot and tested a real App.
We explored the creation of test scripts manually (writing code) and also automatically, using a feature called Test Recorder that enables you to interact with the mobile App and record your interaction in a script that you can fine tune to your test goals. The Test Record revealed itself to be the best option and enabled us to quickly create the test scripts.
Even though OutSystems Apps have their code automatically generated, we were able to create the test scripts without any problems: to assure your test scripts are future proof just don’t use generated element Ids because they change over time and it will break your tests. Use other approaches like classes to interact with the screen elements.
We were surprised to find that our test scripts would not run in iOS. But worry not: we created an OutSystems native plugin (and made it available in the OutSystems Forge) that fixed it all and everything worked perfectly.
After running the tests we found out that our App was not perfect after all. There were some minor problems in several devices, but with the insights from the test results we could easily spot the problems, fix them and test the App again using the same script as before.
Now our developers can proudly say “It works on all devices”!
