How to Make your Code Shine with GitLab CI Pipelines

Published in

Semantix

7 min readMay 26, 2021

A brief introduction to some tools for a cleaner Python code by applying isort, Black, Flake8, and Pylint automatically using GitLab CI Pipelines.

How to Make your Code Shine with GitLab CI Pipelines — Photo by Pankaj Patel on Unsplash

In this post you will find:

An introduction to code styling issues,
Python code style tools used,
GitLab CI Pipeline, and
BONUS: Configure virtual environments for development dependencies.

Code styling issues

Frequently, we work on projects of any size that involve the collaboration of several developers, and the implementation of the same code style and good practices ends up being left in the background, perhaps due to a lack of knowledge and/or experience, and mostly lack of time.

Thus, when the project starts to grow, inevitable, problems can also grow, and maintaining code quality standards such as code compliance standards, e.g., PEP8, ends up being a great challenge.

Sometimes during the code review, comments like…

Could you add a new line at the end of the code? Could you remove the unused variable? or, Could you format or include a docstring in the function?

… It can end up diverting your attention from what matters the most that are what your code is supposed to do. This kind of situation distracts you from taking an in-depth review of the main functionality of the project’s code.

Fortunately, some technologies can help us to automate formatting and styling tasks to keep our codes within the standards. We can use some tools with specific purposes to check our codes and automate those processes.

Next, some of the tools that are useful to standardize and maintain good Python programming practices and how they can be implemented in automated processes will be presented.

Python Tools

We can use a simple deck of programs to get our code styling done. We can use isort for sorting the library imports (yes, imports have a suggested order), we can check the existence of undesired artifacts using Flake8 and Pylint, and keep the code within the same style using Black.

Those tools can be configured to be PEP8 compliant. PEP8 — Python Enhancement Proposal, is a style guide that provides guidelines and best practices suggestions on how to write Python code.

isort

isort is a tool that sorts imports alphabetically and automatically separated into three main sections: built-ins, third-party and local libraries. This tool can be quite useful to organize all the imports in our codes.

Black

Black is a tool that formats the code. Although this still is a beta product, it has been widely adopted in several projects.

It has to be said, Black doesn’t format comments or docstrings, it only formats the Python code. To get your comments formatted, you can use other extensions/plugins like Rewrap for VS Code or include a guide in Editor: Rulers.
Change the line length to 79, which is the PEP8 standard used by Flake8.

VS Code > Settings > Editor: Rulers > Edit in settings.json

Flake8

Flake8 is a tool that checks if your code complies with the PEP8 style, programming errors such as libraries or unused variables, and the complexity of the implemented functions.

Pylint

Pylint is a tool that checks for bugs, helps enforce a coding standard, and offers simple refactoring suggestions. At the end give a score of your code!

Now, it is time to automate the whole process. For this, we can create a pipeline in GitLab CI (Continuous Integration) that runs from top to bottom: isort to sort the imports, then Black to format the code, then Flake8 to check the style of the code, and finally Pylint to detect possible errors in the code.

We can create automatic actions triggered by repository events, i.e., we can run our styling pipeline when a commit is pushed to the repo, or a new merge request is opened. Here, we are going to configure the styling checking to be executed when a new merge request is opened.

As consequence, at the end of the pipeline, if every check passes, the reviewer spends less time formatting the code and can focus on code logic, implementation, and testing of each code, and finally, approve the Merge Request!

GitLab CI Pipeline

GitLab CI/CD is a tool built into GitLab for software development through continuous methodologies such as Continuous Integration (CI), Continuous Delivery (CD), and Continuous Deployment (CD).

Continuous Integration (CI)

This methodology works by pushing small code chunks to your application’s codebase, and to every push, run a pipeline of scripts to build, test, and validate the code changes before merging them into the main branch.

1. Create the `.gitlab-ci.yml`

We can create the .gitlab-ci.yml file to configure our pipeline. By default, Gitlab looks for it in the project’s root directory.

The .gitlab-ci.yml file below, describes the sequence of steps for standardizing and maintain good Python programming practices.

.gitlab-ci.yml extended version. — **.gitlab-ci.yml** extended version.

And as a complement, if you want to centralize the project tools configuration you can include a pyproject.toml file. This is a configuration file defined in PEP 518 and expanded in PEP 621. It is designed to store build system requirements and any tool configuration for your Python project.

2. Configure the CI pipeline for Merge Request

In GitLab go to Settings> General> Merge Request and Activate Pipelines must succeed.

3. Create a Merge Request and test!

If your code looks something like this, it may not pass the tests…

Example linear_regresion.py with some linting errors.

… Maybe you have already noticed some errors, for example, unorganized imports, code lines width greater than 79, some variables and imports declared but unused. Let’s try it with our CI!

and.. yes, we can’t do Merge in the repository, because the code has some formatting and style errors that we must fix.

If you want to see the errors in more detail, you can read a detailed report in CI/CD>Pipelines in the tab Failed Jobs.

Failed Jobs for Flake8 in linear_regresion.py file.

Ok, now that we know what are the code errors, we must update the script. It looks something like this now…

Example linear_regresion.py with correct formatting and style.

So, if we do a git push with the updated code, the pipeline will be executed again and we will have the following result.

The code in each step of the pipeline passed. It has the imports organized, it has a format with line length 79, the name of the variables is in the pattern used and they are all used, so we can finally evaluate the logic of the code, implementation, and testing of each code, and approve and Merge Request!

You can also use a more condensed version of the .gitlab-ci.yml file. The difference is in the time that each one ends up consuming. The extended version gives a more detailed report of possible errors but depending on the number of files to review it can end up consuming more time.

Finally, a good practice that can be implemented at the beginning of any project is to configure a virtual environment keeping the development dependencies such as isort, Black, Flake8, and Pylint separated in a single file.

Configure Environment

Create the conda environment

(base)$: conda create --name myenv

or create the conda environment from an environment.yml file

(base)$: conda env create -f environment.yml

Install development dependencies environment-dev.yml

(base)$: conda env update -n myenv -f environment-dev.yml

Activate the environment

(base)$: conda activate myenv
(myenv) $:

and we are done! go ahead!

Gitlab repository

Look at the complete codes of the GitLab pipeline setup.

GitHub - mafda/python_best_practices: How to Make your Code Shine with isort, Black, Flake8, and…

How to Make your Code Shine with isort, Black, Flake8, and Pylint. Guidelines and best practices suggestions on how to…

github.com

fernanda rodríguez / gitlab-ci-pipeline