End-to-End Test Automation: Overcoming Initial Hurdles
Everything you need to know to prepare yourselves for setting up automated end-to-end testing infrastructure: from choosing a testing framework and seeding your test data to dealing with translations and handling emails
Introduction
When it comes to ensuring the quality of a web application in an end-to-end fashion, there are a few ways to do that: Collecting feedback from end-users, manually mass testing the application before/after every release, or the most effective and efficient way of doing it, which is executing automated end-to-end tests before release to make sure the new code is not breaking any of the existing functionalities.
Anyone who works in tech has heard the term Selenium and has a rough idea of what it is for. I’m sure you do too: It’s a tool that enables a piece of code to talk to the browser, access its DOM elements, and help to execute tests. However, when it comes to setting up the infrastructure to have a maintainable setup, there are a few components that usually are not considered at first, and only during the course of increasing your automation coverage, you start to notice them.
What I want to talk about in this article is sharing my learnings from years of building and maintaining testing setups in a variety of IT companies and for different tech stacks to reduce the headache that one might experience going through setting up all these components.
Common problems that I will answer in this article are:
- How to settle on a testing framework
- How to seed and clean up your test data
- How to test application flows that include sending emails
- And finally, how to handle locales and translations
After the first part which is about deciding on a framework, I will continue the rest of the items based on my framework of choice, to have more practical examples in place and play around with pseudo codes rather than theoretical and abstract sets of instructions.
1. Choose a testing framework
There are different factors that can come into play when deciding on your testing framework, factors as:
- The architecture that suits your need
- The support of the development team behind it
- How experienced you are with the tools and programming languages they come in
- Open source vs. proprietary
- And many more
However, not all of these criteria might be important to everyone. I leave this to you to decide on your own what testing frameworks you are going to be comfortable with, but for the sake of this article, I am only going to mention that my testing framework of choice in Cypress.
I have used the Robot framework, WebdrvierIO, TestCafe, and Cypress in the past, and Cypress was the one that I personally was and have been happy with the most. Because of its architecture and the ability to execute tests within the browser instead of needing a driver like Selenium running, and compared to other Selenium-based frameworks such as WebdriverIOs, Cypress seems like one of the most reliable and the least flaky testing frameworks out there. Of course, due to its architecture, Cypress comes with its own limitations, but it is up to you to see if those limitations are show stoppers for you or not.
For the rest of this read, I will be using Cypress as my testing framework, but all these practices are certainly applicable to any testing framework out there.
2. Seed and clean up your test data
In most cases, for your tests to execute successfully, you will need some data in place, and there are two common ways of preparing these test data:
First option: Using static test data
In this approach test data are prepared only once in a testing/staging database and running the test would heavily depend on the existence of that data. This data will and should not change since any modification might result in failures.
A couple of advantages of this approach are, first of all, not needing any data preparation for your test executions, and secondly, and as a consequence, no need for cleanup either.
The disadvantages, however, are much greater: For one, if the data is deleted for any reason, then it means the missing data has to be reinserted otherwise unwanted failures keep coming. Another reason is that in most testing scenarios we do need to modify the test data in the course of the testing scenario, and having static data that are being used by other tests is in direct conflict with that.
For example, let’s say you have a statically added user with a name, email, address, and so on, that will be used by one of the test cases. Then there is another test case which is about editing that user’s email to test the edit functionality. Now if the second test changes the emails, what happens to the first test in future executions?
Second option: Dynamically creating test data for every test
In this approach instead of relying on test data that are added once, every test suite creates its own test data in the beginning and before the actual test starts.
An obvious disadvantage right of the bet is the huge amount of the created test data. This necessarily does not have to be a problem, but I had a bit of a bad experience in the past when our database reached the point of having tens of millions of test data, which greatly affected the performance of our staging database and environment. However, this disadvantage can easily be avoided by cleaning up the test data after the tests are done!
The main advantages of this approach are that each test suite uses its own data and therefore no conflict between tests running together, and tests won’t rely on the current state of the database either if the database tables are truncated, the tests will still be executed without a problem.
When it comes to dynamically seeding your test data, which is in most cases my method of choice, you can do it either via the existing API endpoint, or directly writing to the database. My method of choice is the latter for the reasons that sometimes and for some specific test data, there are no clear-cut endpoints to create those data, which again, might not be the case everywhere. Either way whether using the API of your backend services or running SQL queries on the database, dynamically seeding your test data is what I will be talking about below.
How would this data creation look like and where would it reside? My current approach is to have a service either on a remote Lambda function or as a standalone service, accessible via a secureSECRET_KEY that’s only available to the tests. This service receives a YAML file (or a JSON conversation of it) and returns a JSON object. This JSON object then contains the test data that are necessary for the execution of the test cases, as well as for the clean-up later on.
As an example let’s say you want to write a test case in which ”a user goes to an online car rental app and out of two existing Toyota Corolla car, they should only see the available car and not the one that’s rented out already”. Here is a simple YAML file for this example:
---
template:
car_default: &car_default
manufacturer: 'Toyota'
model: 'Corolla'
year: 1998rows:
car_available:
table: cars
parameters:
<<: *car_default
rented: false
output: [id, uid] car_unavailable:
table: cars
parameters:
<<: *car_default
rented: true
output: [id, uid]
And itsJSON conversion:
{
"template": {
"car_default": {
"manufacturer": "Toyota",
"model": "Corolla",
"year": 1998
}
},
"rows": {
"car_available": {
"table": "cars",
"parameters": {
"manufacturer": "Toyota",
"model": "Corolla",
"year": 1998,
"rented": false
},
"output": [
"id",
"uid"
]
},
"car_unavailable": {
"table": "cars",
"parameters": {
"manufacturer": "Toyota",
"model": "Corolla",
"year": 1998,
"rented": true
},
"output": [
"id",
"uid"
]
}
}
}Then the mentioned service after receiving the test data file goes through it and starts creating the following database rows for two instances of car_available and car_unavailable
// car_available
INSERT INTO cars ('manufacturer', 'model', 'year', 'rented')
VALUES ('Toyota', 'Corolla', 1998, true) RETURNING id, uid// car_unavailable
INSERT INTO cars ('manufacturer', 'model', 'year', 'rented')
VALUES ('Toyota', 'Corolla', 1998, false) RETURNING id, uid
Which conveniently creates two entries in the cars table. Then this service, having the returned values from the query and the tables’ name, construct and create the following JSON object, and returns it to the running test:
{
"car_available":{
"table": "cars",
"id":1373,
"uid":"xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
},
"car_unavailable":{
"table": "cars",
"id":1374,
"uid":"xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
}
}Afterward, the tests can use these data to access these created entities and accomplish different tasks, such as:
- Going directly to a
urlusing the returnedid/uid(i.e: http://www.mycarrental.com/cars/1373) - Check if certain elements or values exist based on the properties of the added entry (
name,title, etc) - And cleaning up the data, using the
tableandidvalue of each returned object.
You might ask why not have the test data written in JSON in the first place, and the answer is in the template part in the YAML file:
template:
car_default: &car_default
manufacturer: 'Toyota'
model: 'Corolla'
year: 1998This way we can create a default template of the columns and values that might be repeated for several objects, and with this syntax of <<: *car_default, we can easily access these common values which makes the test data easier to write and maintain.
In my Cypress setup, I store the seed files under /fixtures directory and by the same name as the corresponding spec file for better readability:
And roughly this is how the test would incorporate the seeding mechanism:
// globally defined data to be used by all tests
let data = {}describe('my test suite', () => {
before( function () => {
// path to your seed file
const seedPath = 'cypress/fixtures/car_rental.yml'; // call your seeding API and assign the result to data
data = seed(seedPath)
} it('my test case', function () => {
// now in your tests you can access the data object, i.e:
cy.visit('rental_cards/' + data.car_available.uid) // or any other usage you see fit
cy.contains(data.car_unavailable.id).should('have.length', 0)
}) after( function () => {
// here you can clean up the data if you like
cleanup(data)
}
}
This approach is very flexible and there is a good deal of possibilities that you can achieve with your YAML test file. For instance, you can create custom conventions and set some unique characters to deliver specific tasks. Below you can see three use cases for these special characters that I tend to define in my test data files to add more flexibility to my test data and make my life easier:
Case 1: Randomisation (a special character I chose: ^)
When I need to have some values unique in every test execution, I’ll put a^ character in my test data followed by a digit that later on and in the seeding service I can replace with a unique randomized string with the length of the digit that comes after while creating the query.
For example, having:
...
name: AutomatedUser_^4
email: automated.^6@domain.com
...in my YAM test data can be translated into:
INSERT INTO ... (..., 'name', 'email', ...)
VALUES (..., 'AutomatedUser_h7m5', automated.y8wb3c@domain.com, ...)This way every time the tests are executed, unique usernames and emails are generated since most authentication services have a unique database constant over a username and email columns.
Case 2: Handling relative dates (a special character I chose: $)
Another trick that I often do is passing timestamps relative to the current time of test execution, in cases where I need to create a database entry with a date column, that needs to have a timestamp either in the past, present, or future, based on the nature of the test case.
Let’s say I want to create two rentals cars, one that has expired 5 days ago and another one expiring in 10 days. Therefore in my YAML file I can have:
...
rows:
car_1:
...
expires_at: $-5d
...
car_2:
...
expires_at: $+10d
...
...Which again while processing the SQL query, I can replace $ and any +/-, digits, and letters s/m/h/d/... that comes after it with dates relative to the moment of test execution.
Here are some other examples of using this format:
$now // current time
$-2m // 2 minutes ago
$+3h // 3 hours from now
$+36h // 1 and a half day from now
$-8d // 8 days agoIn case you’re using NodeJS or just want to see which library I’m using to transform this syntax to relative dates, you can have a look at this node module.
Case 3: Handling foreign keys (a special character I chose: =)
You will definitely come across cases wherein your test file you want to create an entity, like company and then create a user for that company, which essentially means using the ID of the created company as a foreign key for that employee.
To handle foreign key references I use another special symbol, an equal sign:
company_google:
table: companies
parameters:
...
employee_1:
table: users
parameters:
...
company_id: =company_googleWhen I create and execute queries, I keep track of the returned ID of all my previous queries along with the object’s name (like company_google in this example) and while creating new queries, whenever I come across a = symbol, I look up in my previously executed queries, for the object name that comes right after = and use the returned ID of it as a foreign key for my current query.
Cleaning up the test date
Deleting the test data that your tests create is not something that you have to do. Depending on the volume of the generated data and the capacity of your database, it might not even be worth spending time for.
However, for anyone whom this might come in handy for I will briefly explain the way we usually clean up our test data:
Every time a row of test data is created, in the same endpoint, we save the primary key and the Table nameof the created data in a table we call test_data. Then once a week on a Friday night we execute a cron a job that iterates over that table and DELETE CASCADEs all those data.
That’s basically it!
3. Testing emails in user flows
We often come across user flows in which an email is being to the users as a result of an action, a.k.a transactional emails, which are crucial for us to ensure not only these emails are being sent to users, but also the content of the email is fully correct as well. User flows like signup, forgot password, and invitations are some examples.
Of course, one way to test emails are being correctly sent and received is to access the inbox webpage in the tests, look for the right email, open it, and check its content, but this approach comes with plenty of overhead. First of all, this approach is testing the email clients, such as Gmail, which is not in the scope of the test case, and secondly, this practice is too expensive to maintain.
One thing we can do here is use an email testing solution, such as Mailosaur, that enables us to send emails to specific inboxes and using API_KEY access the inbox, and get specific emails via its REST API. With Mailosaur or similar tools, you can have full control over the transactional emails that are sent by your application and received by your test user, by simply looking for the exact email address.
As an example consider this scenario: You want to try the registration endpoint of your application. All you need to have is a unique email address like test+2dgx97d@xxxx.mailosaur.io, use that email address to register, and in your tests wait for an email to appear in your Mailosaur inbox, with the SentTo value matching the unique email address that you have.
2dgx97dcould be a unique hash that you create while running your tests, which will be different and unique in every executionxxxxis your Mailosaur’sSERVER_ID
As an example, here is a part of our current test suite that checks if our application is sending emails to users in case they’ve forgotten their passwords:
it('Should send email when user clicks Reset Password', function() {
...
cy.mailosaurGetMessage(Cypress.env('MAILOSAUR_SERVER_ID'), {
sentTo: data.test_user.email,
subject: loginPO.resetEmailSubject,
}).then(email => { // Checks the subject of the email
expect(email.subject).to.equal(loginPO.resetEmailSubject); // Checks the body of the email
expect(email.text.body).to.contain(loginPO.resetEmailBody); // Checks the link in the email is pointing to the right place
expect(email.html.links[0].href).to.contain('/user/reset-pw/');
...
})
4. Locales and translations
In some tests, you want to check if certain text or messages appear or not, for instance in the case of users entering the wrong email or password:
...
cy.get(#emailInput).type(emailAddress)
cy.get(#passwordInput).type(wrongPassword)
cy.get(#submitButton).click()
cy.contains('Your email or password is wrong.')
...But the problem with this approach is that if someone in the team, like the product owner or the marketing, decides to modify the message Your email or password is wrong, then the automated tests would fail, which is far from ideal since the main focus of our end-to-end tests should be the functionality and not some minor copy changes.
A solution to this problem could be using the message key that’s being mapped to the message value instead of the value itself, so changes will not break the automated tests their goal is to test the functionality of the web app and not minute details like the message.
This can be done in two very simple steps:
Step 1: In most cases, these messages, or commonly referred to as translations, reside on 3rd party services such as Phrase, and while your app is being built, set or sets of translation files are downloaded within the project directory. For instance, the current project that I’m currently working on has its Phrase translation in this path, which is basically JSON files, one for English translations and one for German:
➜ project: ls src/locales/langs
➜ project: de-DE.json en-US.jsonAll these files contain simply key and value pairs, like:
{
...
"login.password.wrong": "Your email or password is wrong.
"login.password.empty": "Please enter your email."
...
}Therefore the first step is as easy as copying these files under cypress/fixtures, which can be added as a step in your CI pipeline and before the test execution step:
cp -a path/to/translations/. cypress/fixturesYou might ask: Why under fixtures directory? And the answer is found in Cypress’ documentation:
Fixtures are used as external pieces of static data that can be used by your tests. Fixture files are located in
cypress/fixturesby default,
This basically means any file JSON, YAML, etc formats that are set in the fixture path can be directly used within the tests.
Step 2: Now all you have to do is reading from those files in your tests by using the cy.fixture() command:
beforeEach(function() {
// name of the fixture file, without the extension (.json)
cy.fixture('en-US').then(translation => {
// this will contain all the key:value pairs
this.translation = translation;
});
});...it('my test case', function() {
// here you can access fixture values
cy.contains(this.translation['login.password.wrong']);
...
}
And that’s it. From now on, if the translations change by a developer, product owner, or anyone else within the team, the tests will not be affected by it!
Conclusion
If you are about to set up an end-to-end testing infrastructure for your web app, you might benefit from having a look at this article. There might be topics that you haven’t taken into account yet, or you have still open problems on your mind that you are not sure how to tackle.
Either way, here I talk about the common problems that I faced when setting up testing infrastructure in multiple startups, and the way that I overcame them, hoping that these past experiences of mine might come in handy for others heading down the same road.
