Going beyond the "100% coverage" dogme: focusing on functional tests

I'm not the only one saying it.

📖 TLDR: This is a REX, not a flame post. We chose integration tests over unit tests for a more efficient and startup-friendly approach. It's not perfect, but it works. We're happy.

A very long time ago I stumbled on this interview of Dan Abramov. I was already using a lot of functional tests on my previous project, and wondered, do I really need anything else?

While there are dozens of “get to 100% coverage” articles, I took the decision to try a new approach: No unit tests!

It worked well for us, but in a recent interview, I was asked "what's your unit test code coverage percentage?", after explaining our strategy, the candidate was very surprised by this concept and we talked about it for a few minutes.

He asked me to write an article about it, so here we are! 😅

Why are we making tests?

Here is my opinion, based on my personal experience:

What's a codebase without tests ?

• A codebase where you don't have any confidence you just broke something with a new piece of code
• A codebase where you don't have any confidence that you fixed an issue
• A codebase where making changes makes you nervous

So to sum-up, a codebase where any change takes a lot of time, where agility is low and where feature ship slowly. Time is the issue at the end.

For me, tests are made to change this, to reach a state where:

• A codebase you have a strong confidence you did not break anything important with a new piece of code
• A codebase where you have a absolute confidence that you fixed an issue
• A codebase where changes flow and feels simple

The main types of tests we are using

Static Tests

You can use toolings to detect basic errors like typos and syntax. It's well documented, not long to set up and catches a lot of small mistakes.

Unit Tests

You verify that functions are doing what they are meant to do, without external influence. Using patterns like dependency injection to avoid testing multiple behaviors at the same time.

Integration / Functional Tests

You verify a whole range of behaviors that fits a real use case. Usually limited to parts of an application / infrastructure.

End to End Tests

You verify that the whole application: database, server and client works, by simulating an user usage. In our case, simulating clicks in a browser.

So why focusing only on integration tests?

TIME ! All our choices have been made to allow the programmer to be efficient.

Refactoring

In a startup, you don't know for how long the code will be in place. Maybe in two months we will trash the whole feature? Maybe we will change its purpose?

We must be nimble, big changes must be possible in the smallest amount of time, without a big loss of quality. We must find a good equation between speed and regressions.

Understanding the use cases

They were useless because they didn’t actually correspond to anything that the user would see. And also, if you read it like three years later, you’re like, okay, I understand what this test is destined for in terms of this module, but why is this the behavior? Like what does it, what is, what is the problem it was trying to solve?

Focusing on user stories help to understand what the test is in charge of. It clearly states what's the usage of the API and why it's being used.

You trade: "The listener must be able to broadcast the signal with the metadata"

For: "The user must be able to subscribe to the service and get a valid token to identify himself after".

Way faster to write

In our case, it's mostly writing a single GraphQL query, and testing the result while the whole chain will be used:

The HTTP server, Express, Graphql, Sequelize, Postgres… everything.

function projects() {
  return `/api/graphql?query=
      query projects { 
        project(order: "reverse:status,id") { 
          id 
          title
        }
      }
      &operationName=projects`
}

it('Cancelled project should not be visible to invited people', async () => {
    const response = await request(mainServer)
      .get(projects())
      .set('accountid', 3)
    expect(response.body.errors).toBeUndefined()
    expect(response.body).toMatchSnapshot()
    expect(response.body.data.project.filter(p => p.id === 5).length).toBe(0)
  })

This also fits well with the "snapshotting" pattern, where you only really check that outputs changes when updating your code. Another big speed win, at the cost of being extra-careful while reviewing the code you push.

Writing faster does not means with less use cases

We have at least one test file per model. All security checks MUST have a test. All server fixes MUST have a test, bugs can happen, never twice. All visual bugs on the critical path MUST be added to the end-to-end test.

In total we have 3827 tests, mostly functional ones.

We also have a few other tests

Let be clear: Unit Tests are not "bad", there are maybe just not what you need at a specific time of your project.

We still have less than 10 unit tests, for pure functions with critical business rules.

Around 20% of our test files are end-to-end tests. Even if we write them with puppeteer, which is very stable and easy to work with, it's still very slow to debug and a hassle to run. So we limit them to critical paths only.

How to make it work?

Running a whole database while testing your server code obligates you to:

🎯 Have a way to initiate quickly a database with new tests data (called seeds). Here we are using Sequelize:

yarn db-reset-datastudio 
  && ./node_modules/.bin/sequelize db:migrate --config config/sequelizeConfig.js 
  && ./node_modules/.bin/sequelize db:seed:all --config config/sequelizeConfig.js 
  && node ./server/scripts/refreshSequences.js
  && yarn db-refresh-views

🎯 Have a way to "reset" the database quickly.

Here's our script:

 await sequelize.query(`SELECT public.tool_truncate_schema_tables('public')`)
 await sequelize.query(
    `SELECT public.tool_truncate_schema_tables('teamplanning')`
  )
 await sequelize.query(
    `SELECT public.tool_truncate_schema_tables('statistics')`
  )
 await sequelize.query('SELECT public.tool_reset_all_sequences()')

Here's our script to truncate the table, it quicker than deleting rows:

--
-- Name: tool_truncate_schema_tables(text); Type: FUNCTION; Schema: public; Owner: teamstarter
--
      
CREATE FUNCTION public.tool_truncate_schema_tables(schema_name text) RETURNS void
    LANGUAGE plpgsql
    AS $$
        DECLARE
            query TEXT;
        BEGIN
            SELECT format('TRUNCATE %s CASCADE;', string_agg(quote_ident(schema_name) || '.' || quote_ident(table_name), ', '))
            INTO query
            FROM information_schema.tables
            WHERE table_schema=schema_name AND table_type='BASE TABLE';
            EXECUTE query;
        END;
    $$;

Yet it requires to reset the sequences after bulk-inserting the seeds:

CREATE FUNCTION public.tool_reset_all_sequences_to_max_values() RETURNS void
    LANGUAGE plpgsql
    AS $$
    DECLARE
      rec RECORD;
      query TEXT;
      queryRestart TEXT;
    BEGIN
      query := 'SELECT PGT.schemaname, S.relname, C.attname, T.relname as "tRelname" FROM pg_class AS S,  pg_depend AS D,     pg_class AS T,     pg_attribute AS C,     pg_tables AS PGT WHERE S.relkind = ''S''  AND S.oid = D.objid  AND D.refobjid = T.oid  AND D.refobjid = C.attrelid  AND D.refobjsubid = C.attnum  AND T.relname = PGT.tablename ORDER BY S.relname;';
      FOR rec IN EXECUTE query
      LOOP
        queryRestart := 'SELECT SETVAL(' ||
                       quote_literal(quote_ident(rec.schemaname) || '.' || quote_ident(rec.relname)) ||
                       ', COALESCE(MAX(' ||quote_ident(rec.attname)|| '::int), 1) ) FROM ' ||
                       quote_ident(rec.schemaname)|| '.'||quote_ident(rec."tRelname")|| ';';
        RAISE NOTICE 'Will execute (%)', queryRestart;
        EXECUTE queryRestart;
      END LOOP;
    END;
    $$;

🎯 Have an optimised database to improve performances, for example here's our docker file dedicated to Postgres:

version: '3'
services:
  postgres:
    image: postgres:11.13
    ports:
      - '${PGPORT}:${PGPORT}'
    environment:
      - POSTGRES_USER=${PGUSER}
      - POSTGRES_PASSWORD=${PGPASSWORD}
      - POSTGRES_DB=${PGDATABASE}
      - POSTGRES_PORT=${PGPORT}
    networks:
      - teamstarter
    command:
      [
        'postgres',
        '-c',
        'fsync=off',
        '-c',
        'synchronous_commit=off',
        '-c',
        'full_page_writes=off',
      ]
networks:
  teamstarter:
    driver: bridge

🎯 Have the tests run sequentially

Sadly you cannot spin a single database per test. So you have to call the tests with:

node ./node_modules/.bin/jest server/tests --runInBand --ci

Hopefully you can still run multiple test workers in parallel in most CIs !

Are we happy?

Mostly, but we have some subject to improve:

• Better seeds discipline, the quality of the use cases tested directly depends on how covering are the seeds.
• Improving Postgresql performance, it should be possible to monkey-patch Sequelize to transform all "CREATE TABLE" instructions and add the "unlogged" keyword.
• Benchmarking a switch to Playwright. Some of the Puppeteer's authors now work on it, and it covers more browsers. Should be even more reliable.
• We have too few functional tests on the Frontend, mostly end-to-end tests. I would invest more in our Storybook, in order to have a better coverage of our design system.

A typical test, straight to the point.

And you ?

• How are you testing your code application?
• What framework are you using?
• What's your feedback and feelings about your current setup?

Living in Paris and looking for a job or an Internship? https://job.teamstarter.co/

Edited :
- Added a bit more details about run time and how many tests we have.
- The article’s title was changed, as “Unit Tests Are Dead, At Least for Us” was met with backlash and people criticized the piece without actually reading it.