Unit Testing and Continuous Integration (CI) using Github Actions

A simple python based tutorial to demonstrate unit testing and continuous integration (CI) pipeline.

Photo by Sigmund on Unsplash

Testing in Python is a huge topic and maybe a little complex for the beginners, but it doesn’t need to be hard. We implement automated testing for majority of our data science and machine learning projects at TheLorry.

Before we get into the more exciting stuff, let us review a few basic terminologies related to testing and quality assurance in the world of software engineering.

Manual Testing vs. Automated Testing

When you start to write your code, you already do the manual testing every time you run your code to see the results. You explore your code to see if the code gives the desired result as intended. This is also called exploratory testing. Exploratory testing is a form of testing that is done without a plan. In an exploratory test, you’re just exploring the application as you are writing your code.

Manual testing is always required, but automating it helps speed up processes in your bigger or longer-term projects. Manual Testing of all workflows, all fields, all negative scenarios is time and money consuming. If there are repetitive tests to run, why not automate them?

Automated testing is the execution of your test plan by a script instead of a human. It does not require Human intervention and we can run automated test unattended. Automation gives the tester a more efficient way to test new features and bug fixes. Instead of continuous manual testing, we should try to develop the habit of automating the tests.

Just like majority of the programming languages, Python already comes with tools and libraries to help you create automated tests for your application.

Unit Tests vs. Integration Tests

source: http://dinda-dinho.blogspot.com/2014/11/perbedaan-unit-test-integration-test.html

Unit testing
It is a software testing method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures are tested to determine whether they are fit for use.

Integration testing
It is the phase in software testing in which individual software modules are combined and tested as a group. Integration testing is conducted to evaluate the compliance of a system or component with specified functional requirements.

Unit testing and Integration testing has a different level in term of doing testing (see image above to know the level of testing and its relations to design phases)

This tutorial will show you how to create unit testing using the ‘unittest’ framework. Next, we continue with Github action for Continuous Integration (CI) as integrated testing. Okay enough talking, let’s get started with the code:

“Talk is cheap. Show me the CODE!” Linus Torvalds.

Prepare your project (1.a)

First, we need to do a few things to prepare for the project. We assume that you have already installed python, and it is working well in your machine. If not, then please follow this tutorial first before continuing. For complete example code, the code can be downloaded or cloned from the github repo shown here.

Open your terminal (command prompt in windows), change your current working directory to the location you want to create the project’s folder. To create a project’s folder, you can use this command:

mkdir mypython
cd mypython

Nice! Now you are in the working directory.

For this tutorial we assume you use a macOS, if you use Windows or Linux, please refer to the tutorial for their environments. The commands maybe slighlty different than those shown here.

First, you need to install virtualenv python’s package:

pip install vitrualenv

Next, you need to create a virtual environment and then activate it using this command:

vitrualenv env
source env/bin/activate
(env) $

and then, let’s install flask in a new active virtual environment

pip install flask

Now, let's create a package called app that will host the application. Make sure you are in the mypython directory, and then run the following command

(env) $ mkdir app

The file __init__.py for the app package is going to contain the following code:

from flask import Flask
app = Flask(__name__)
from app import routes

The script above creates the application object as an instance of the class Flask imported from the flask package. The __name__ variable passed to the Flask class is a Python predefined variable, which is set to the name of the module in which it is used. The application then imports the routes module, which doesn't exist yet. Let’s create it in location app/routes.py

from app import app

@app.route('/')
@app.route('/index')
def index():
    return "Hello, World!"

In this example, there are two decorators, which associate the URLs / and /index to this function. This means that when a web browser requests either of these two URLs, Flask will invoke this function and pass the return value of it back to the browser as a response.

To complete the application, you need to have a Python script at the top-level that defines the Flask application instance. Let’s call this script mypython.py, and define it as a single line that imports the application instance in mypython folder:

from app import app

To make sure that you are doing everything correctly, below, you can see a diagram of the project structure so far:

mypython/
  env/
  app/
    __init__.py
    routes.py
  mypython.py

Before running it, though, Flask needs to be told how to import it by setting the FLASK_APP environment variable and then lets us run the app:

(env) $ export FLASK_APP=mypython.py
(env) $ flask run
* Serving Flask app "mypython.py"
 * Environment: production
   WARNING: This is a development server. Do not use it in a \ production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

Let’s type the IP 127.0.0.1 and host number 5000 in your browser, and the browser will show the result similar to this:

Congratulation, you have finished the first step to create a simple flask application.

Before we continue our “testing” application, let us install another python package called python-dotenv, which can automatically run the environment variable when the flask is running.

(env) $ pip install python-dotenv

then you can write the environment variable name and value in a .flaskenv file in the top-level directory of the project. To do that, create a new file called .flaskenv and add the following line to it:

FLASK_APP=mypython.py

Add function to be tested (1.b)

Ok, let’s us add a new route and a new function in our routes.py.

@app.route('/')
@app.route('/index')
def index():
    return "Hello, World!"
@app.route('/add')
def add():
    data = request.args.get('data', None)
    _list = list(map(int, data.split(',')))
    
    total = sum(_list)
    return 'Result= ' + str(total)
def sum(arg):
    try: 
        total = 0
        for val in arg:
            total += val
    except Exception:
        return "Error occured!", 500
return total

We add a new function called “add” which can be accessed from the route “/add”. It will have a variable argument called data. Then we convert into a list of integers and call a function called sum() to count the summation of that data. In sum() function, we add an error handler to tackle the condition when the data can not be processed, and the function will return an error message and status code 500.

Please run back your code by typing the command flask run:

(env) $ flask run
* Serving Flask app "mypython.py"
 * Environment: production
   WARNING: This is a development server. Do not use it in a \ production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

In the browser, you can call that route and supply the variable data with a few integer numbers like this:

127.0.0.1:5000/add?data=1,2,3,4,5

The result will show:

Write the unit tests (2.a). Finally!

Before you dive into writing tests, you’ll want first to make a couple of decisions:

1. What do you want to test?
2. Are you writing a unit test or an integration test?

Then the structure of a test should loosely follow this workflow:

1. Create your inputs
2. Execute the function being tested and then capturing the result
3. Compare the result with the expected result

For this application, you’re testing sum() function. There are many behaviors in sum() you could check, such as:

• Can it sum a list of whole numbers (integers)?
• Can it sum a tuple or set?
• Can it sum a list of floats?
• What happens when you provide it with a bad value, such as a single integer or a string?
• What happens when one of the values is negative?

The most simple test would be a list of integers. Create a file in a folder called tests, tests/sum_test.py with the following Python code:

import unittest
from routes import sum

class TestSum(unittest.TestCase):
    def test_list_int(self):
        data = [1, 2, 3, 4, 5]
        result = sum(data)
        self.assertEqual(result, 15)

The code is a unittest class which has a function to testing sum function with a list of integer data, and we expected the result of sum()is 15.
Ok, let’s us run our unit testing

(env) $ python -m unittest tests/sum_test.py
.
------------------------------------------------------------------
Ran 1 test in 0.000s
OK

The result shows that sum() gives us the correct result, the same as we expected is 15.

Let’s add some other testing code

import unittest
from routes import sum

class TestSum(unittest.TestCase):
    def test_list_int(self):
        data = [1, 2, 3, 4, 5]
        result = sum(data)
        self.assertEqual(result, 15)
    def test_lisf_float(self):
        data = [1.2, 2.4, 2.7, 0.5, 1.8]
        result = sum(data)
        self.assertEqual(result, 8.6)
    def test_list_with_negative_value(self):
        data = [1, 2, 3, 4, -5]
        result = sum(data)
        self.assertEqual(result, 5)
    def test_with_tupple(self):
        data = (1, 2, 3, 4, 5)
        result = sum(data)
        self.assertEqual(result, 15)
    def test_fail_with_string(self):
        data = [1, 2, '3', '4', '5']
        result = sum(data)
        self.assertEqual(result[0], "Internal Server Error")
        self.assertEqual(result[1], 500)
    def test_fail_with_single_value(self):
        data = 1
        result = sum(data)
        self.assertEqual(result[0], "Internal Server Error")
        self.assertEqual(result[1], 500)

and then, let’s rerun our unit testing.

(env) $ python -m unittest tests/sum_test.py
.
------------------------------------------------------------------
Ran 6 tests in 0.000s
OK

You can add a file called tests/BaseCase.py to put a process before and after you do the testing. This file with a specific function will execute every time you run the unit testing file.

Import unittest
from app import app
class BaseCase(unittest.TestCase):
def setUp(self):
    # this statement will be executed before testing
    self.app = app.test_client()
    # self.db = db.get_db() # get db by example
    
def tearDown(self):
    # this statement will be executed after testing
    # Delete Database collections after the test is complete
    # for collection in self.db.list_collection_names()
    # self.db.drop_collection(collection)

and then, change in your unit testing code as follows. You can see that we are using/extending to the BaseCase here.

import unittest
from routes import sum

class TestSum(BaseCase):
    def test_list_int(self):
        data = [1, 2, 3, 4, 5]
        result = sum(data)
        self.assertEqual(result, 15)

Using Github Action’s for our Continuous Integration (CI) workflow (2.b)

Github Actions enables you to create custom software development lifecycle workflows directly in your Github repository. These workflows are made out of different tasks so-called actions that can be run automatically on certain events.

This enables you to include Continuous Integration (CI) and continuous deployment (CD) capabilities and many other features directly in your repository.

Github provides multiple templates for all kinds of CI (Continuous Integration) configurations which make it extremely easy to get started. You can also create your own templates which you can then publish as an Action on the Github Marketplace.

Actions are completely free for every open-source repository and include 2000 free build minutes per month for all your private repositories which is comparable with most CI/CD free plans.

Before we start with Github Actions, first, we must add a .yml file at the location .github/workflows/main.yml.

name: Continuous Integration
on: [push]
jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [3.6, 3.7, 3.8]
    steps:
      - uses: actions/checkout@v2
      - name: Set up Python all python version
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}
          architecture: x64
     - name: Install dependencies
       run: pip install -r requirements.txt 
     - name: Run Test
       run: python -m unittest discover tests

we can now create a file requirements.txt in the main directory of our project.

flask
python-dotenv

Ok, let’s create our first CI action.

First, please sign into your Github account. Create one if you don’t have it yet by click here. Create one repository called mypython, as shown in this picture.

We’ll get the instruction to create a new repository or push an existing repository. Since we already have our code, the ci workflow should run when push our code to the repository.

Ok, next, we need to create a local repository from our code. We assume you already install git on your local computer. If not, please follows instruction in this tutorial.

To create a new repo for our code, make sure you are in the mypython directory and then run the following command:

git init
git add .
git commit -m 'Initial commit'

next, follows instruction from Github to push an existing repository.

git remote add origin [YOUR OWN REPOSITORY]
git branch -M main
git push -u origin main

Every time you push your code into Github, the workflows will be executed and the testing code will be running and checked. If any fail for this process, you will get an email to your registered email in Github.

Summary

Automated testing is, well, automated. This differs from manual testing where a human being is responsible for single-handedly testing the functionality of the software in the way a user would. Because automated testing is done through an automation tool, less time is needed in exploratory tests and more time is needed in maintaining test scripts while increasing overall test coverage. Implementing unit testing gives us the confidence that our code will always be tested before it gets to production.

In this article, we discussed the basic terms in software testing. We discussed the differences between manual and automatic testing, the difference between unit testing and integration testing, and then implemented a basic sample flask application for sample tests. The tutorial continues with a discussion about how we create simple unit testing. We improved our unit testing with multiple scenario that may happen when you code. Finally we created a repository on Github and created a Github Actions workflows to execute the testing plan (as integration testing) every time the code is pushed to that repository.

We hope, this article will show you how easy it is to implement an automated testing CI pipeline and encourage you to implement and learn more about these concepts.

References

[1] https://realpython.com/python-testing/

[2] https://www.onpathtesting.com/blog/manual-testing-vs-automation-testing

[3] http://dinda-dinho.blogspot.com/2014/11/perbedaan-unit-test-integration-test.html

[4] Kolawa, Adam; Huizinga, Dorota (2007). Automated Defect Prevention: Best Practices in Software Management. Wiley-IEEE Computer Society Press. p. 75. ISBN 978–0–470–04212–0.

[5] ISO/IEC/IEEE International Standard — Systems and software engineering. ISO/IEC/IEEE 24765:2010(E). 2010. pp. Vol., no., pp.1–418, 15 Dec. 2010.

[6] https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world