EarlGrey vs KIF — Testing framework
**Disclaimer**
This post talk mostly about EarlGrey and make a briefly comparison with KIF.
A few weeks ago I've met EarlGrey (Yeah! maybe I'm late). EarlGrey is a UI Automation Test Framework developed by Google its current version is 1.14.0 and is very active on GitHub right now.
At a first look I thought that it can be a good replacement for KIF because:
- Synchronization: Earlgrey synchronizes network request, UI, and threads. We don't need to
waitForVieworwaitanymore! Hate waiting. - Visibility Checks: It checks if the element is visible before performing any action.
- User-Like Interaction: Swipes and taps are performed using app-level touch events.
EarlGrey’s UI interactions simulate how a real user would interact with your app’s UI, and help you to find and fix the same bugs that users would encounter in your app.
EarlGrey installation is simple and it can be easily done though cocoapods. We used the version 1.12.1 because we had some troubles with the 1.14.0.
Testing
With this in mind, we decided to compare KIF and EarlGrey in a real project. To do this we rewrote a few tests with the new framework. Our tests used: Xcode 9.3, Quick, Nimble and Nimble-Snapshots.
To select an Element with EarlGrey we can call:
EarlGrey.select(elementWithMatcher: MATCHER).perform(ACTION)
Where MATCHER can be any GREYMatcher like: accessibilityID, accessibilityLabel, buttonTitle and ACTION any GREYAction e.g. taps, swipes, scrolls. For more info look at this cheat sheet.
Advantages
Writing tests using EarlGrey is easy because it is simple, don't need to wait for almost anything.
Due to this, it can be easier for junior developers to start writing tests with EarlGrey and putting wait someplace in your code can be dangerous.
EarlGrey can be a bit more verbose than KIF but nothing that a Wrapper can't handle.
Problems
The first problem we had was when we are using accessibilityLabel, some calls in EarlGrey can't find our element but it was easily solved by adding an accessibilityID to it.
Another problem is that sometimes EarlGrey found multiple elements even though we know that there is only one element. This was a bit trickier but we solved forcing it to select the first element. We know that probably was not the best solution but works at the moment.
Running tests
After all tests have been created using KIF and EarlGrey we decided to run a few times and check execution time (in seconds).
The table above shows the time in seconds that EarlGrey and KIF take to run the entire test suit. We can notice that EarlGrey is 150% slower so we decided to go deeper and check why.
We wrote down the execution time of each test class. On the left, we have the number of tests for each class. Sometimes EarlGrey is too slow but sometimes it was faster than KIF. So we got confused. What is happening? What are making tests too slower?
So we decided to go deeper. And check what actions/events are called in each test.
We can notice that clear text and entry text are faster on EarlGrey than KIF but swipes, taps, and searches take too much time and we asked ourselves why?
The answer we got is taken of EarlGrey features page:
Taps and swipes are performed using app-level touch events, instead of using element-level event handlers. Before every UI interaction, EarlGrey asserts that the elements being interacted with are actually visible (see Visibility Checks) and not just present in the view hierarchy. EarlGrey’s UI interactions simulate how a real user would interact with your app’s UI, and help you to find and fix the same bugs that users would encounter in your app.
while KIF:
KIF attempts to imitate actual user input. Automation is done using tap events wherever possible.
EarlGrey adds another layer of events that trigger handlers and more. So it makes sense takes more time.
We tried to make it run faster by enabling fast animations
GREYTestHelper.enableFastAnimation()
Which really make it run faster but not faster than KIF.
But we asked ourselves if EarlGrey simulates a real user, enabling fast animations still simulates a real user?
Conclusion
Is EarlGrey worse than KIF? The answer is NO! You can't throw away EarlGrey just because it is slower. EarlGrey's purpose is not to be the fastest test framework available.
So which one should I use? That depends do you want a fast test suit? Or take advantage of synchronization, visibility checks, and user-like interaction?
If you want a faster suit use KIF if you want a simple one maybe EarlGrey is the best call.
Ps: If you like this post, share it on Twitter, recommend it on medium, or both =). This really helps me to reach more people. Thanks a lot.
