Automated QA Setup step by step

Part 1 — the seed

Our story starts with bugs (of course) and laziness.

Every time we wanted to deploy a new version of the app we manually updated the database to test certain scenarios, but never cleaned up after ourselves which would sometimes leave our test db in invalid scenarios causing us to spend large amounts of time looking for bugs that weren’t there.

Unit tests were not enough because most of the cases included several components of our app and complex workflows.

Because we relied so heavily on manual testing, and nobody likes manual testing, it had a lot of effects on the team and the product.

We like to have fun at our job. Manual testing is not fun.

So often times we wouldn’t test as thoroughly as we knew we should have or as often as we should have. Of course, this led to bugs appearing, but it also caused us to be very afraid of deploying. We did not deploy frequently and when we did, we were not as confident as we would’ve liked to be. Often times we would deploy code and then check in production to make sure nothing was broken (none of us are free from sin here).

We didn’t have a clear view of how to approach the testing flow so we started with something.

We all got trouble when manually testing because our data was wrong, so we wanted to ensure that we all got the same upright data from scratch.

The first part of our QA repo was the seed, basically a collection of json files that populate our mongo database with fresh and consistent production like data.

# Folder structure
/qa
 - package.json
 - /seed
   - package.json
   - seed.json
   - /seeds
     + users.json
     + organizations.json
     + files.json

The magic of refreshing the db is done by node-mongo-seeds .

This is a really handy npm module that wipes out the database and recreates the collections using json files.

First of all you have to run seed-setup at the root of your project to create a seed.jsonfile. This file will contain the definition of the different environments you’d like to be able to seed. Ours looks like this:

{ 
  "undefined": "localhost/fashioncloud", 
  "localhost": "localhost/fashioncloud", 
  "dev": "USERNAME:PASSWORD@localhost:DB_PORT/DB_NAME", 
  "staging": "USERNAME:PASSWORD@localhost:DB_PORT/STAGING_DB_NAME" 
}

What we have here is the mongo connection string defined for each of our development stage environments. Credentials and db information are not only hidden for security reasons, but also to avoid re-seeding the wrong db unintentionally.

undefined automatically maps to localhost in case there's no NODE_ENV variable set, so the localhost entry is not really needed.

Then we defined some npm scripts that set the NODE_ENV and run the seed script

// package.json
"scripts": {
  "seedLocal": "export NODE_ENV=localhost && seed",
  "seedDev": "export NODE_ENV=dev && seed",
  "seedStaging": "export NODE_ENV=staging && seed"
}

Running npm run seedLocal will wipe out our local database and recreate it with the seed files.

The seed files are just a bunch of json files named as the collections. For example, a seed for the users collection might look like the following

// seeds/users.json
[
 {
   "_id": {
     "$oid": "59315139da4d1e7fb2808b49"
   },
   "email": "qa-test@fashion.cloud",
   "organization": {
     "$oid": "59315132da4d1e7fb2808b50"
   },
   "created": {
     "$date": "2017-06-02T11:51:21.686Z"
   },
   "__v": 0
 },
 {
   "_id": {
     "$oid": "59315139da4d1e7fb2808b50"
   },
   "email": "qa-test+ftc@fashion.cloud",
   "organization": {
     "$oid": "59315139da4d1e7fb2808b49"
   },
   "created": {
     "$date": "2017-06-02T11:51:21.686Z"
   },
   "__v": 0
 }
]

This was for sure a very helpful first step, consistency was not an issue anymore and we could focus on testing that everything was working as expected in the right scenario.

It is now also really easy to experiment on localhost , we can turn on/off flags, change id references and play with the data for a feature or to recreate a specific scenario, and just by the effort of typing npm run seedLocal we get a fresh copy of the database.

Of course our laziness made sure that the seed data was updated with all of our different data conditions because the effort required to reproduce them manually every time is greater than the effort required to add it to the seed.

Still tired of manual testing (we take annoyance seriously here), we decided to introduce e2e tests.

—

Check the second part here!