Getting Started with CircleCI: A Beginner’s Guide to Continuous Integration and Deployment

(Pt. 1)

CircleCI is a tool used to make building, testing, and deploying software easier by using intelligent automation.

To enhance your understanding of CircleCI, it is important to know what CI/CD is. CI/CD(Continuous Integration/Continuous Deployment) ensures that quality code is shipped out.

Continuous Integration refers to the practice where code is added or merged to a particular branch of a repository that is shared amongst developers and engineers. Features are built, committed, and integrated into a branch, which triggers automated tests and builds each time there is a commit to the branch. This also makes it easy to check failed build logs, troubleshoot, revert to the last successful build, or recommit to solving the failed builds.

Continuous Deployment refers to the practice where quality code is released due to chosen development branches or environments; for example, a TEST, STAGING, and PROD environment or branch (env). This approach helps refine code and detect bugs or abnormalities before they get to users or the target audience.

Image from Edureka on the CI-CD process

Adding CircleCI to your Workflow

To add CircleCI to your workflow, you will need to have a code repository on a VCS(Version Control System) like Github which has to provide authorization for CircleCI to connect to as a project on circleci.com.
As a result, every time you make a change to your code, add the change, commit and push, it triggers automated tests which run each job in a separate container or virtual machine(VM).

By default, CircleCI sends messages indicating the SUCCESS or FAILED state of a job and code coverage after the tests are complete. You can also integrate apps like Slack and IRC to receive real-time alerts(you can find more of these in the orb registry). Orbs are reusable packages of parameterizable configuration that can be used in any project.

You can also deploy code to environments like:

• AWS S3 buckets
• AWS EC2 Container Service (ECS)
• Heroku
• Azure Container Registry and so on.

Why should you use CircleCI?

Honestly, there are other tools which could be used for the CI/CD process, however, CircleCI builds are fast and can be optimized to increase speed through running pipelines that have caching by reusing information from the jobs which have been previously run and also defined resource classes.

Aside from that, you can :

• Use orbs which are reusable packages which can be integrated with third-party apps like AWS and Slack.
• Using ARM executors by specifying a Linux virtual machine image that includes arm resources, and then specifying an Arm resource class.
• Use pre-built Docker images in a variety of languages.
• Use the CLI to access advanced tools locally. CircleCI CLI tool helps you debug and validate your CI config, run jobs locally, query CircleCI’s API and so on.
• SSH into any job to debug your build issues as ssh details are provided at the end of every build so you can access the environment and execute the failed build.

and others.

Now, before we have a look at a sample config.yml file, let’s have a look at CircleCI concepts because they are super important in grasping how everything comes together.

CircleCI Concepts

CircleCI concepts help you understand how pipelines are managed. You will often come across the keywords below when writing your config.yml file.

• Concurrency:
  Generally, concurrency refers to performing multiple computations at the same time. In the CircleCI world, concurrency is making use of multiple containers to run multiple jobs at the same time. For each resource you use within CircleCI, there are soft limits set for concurrency thus if you experience any lag when running your jobs, it’s highly likely that you are hitting those limits. These limits can be modified according to your needs.
• Configuration:
  CircleCI employs the Code as Configuration approach and Configuration simply refers to how things are put together to execute or run in a certain way. The config can be modified to suit your project needs.
• Contexts:
  With Contexts, we can secure and share environment variables(envvars) across projects. The environment variables are defined as name/value pairs and are supplied at runtime. You can also specify the context key in your workflow section of your .circleci/config.yml to provide access to the envvars related to that context.
• Data Persistence:
  The word persistence when it relates to data often has to do with the availability of data, manipulation, storing or retrieving data in databases or non-volatile storage systems. With CircleCI, data persistence refers to the movement of data between jobs to accelerate the speed of jobs. With certain programming languages, APIs(Application Programming Interfaces) manage the persistence of data. CircleCI employs 3 ways of persisting data: Artifacts, Caches and Workspaces.
  * Artifacts :
  Artifacts persist data after a workflow is completed and may be used for longer-term storage of the outputs of your build process.
  * Caches:
  Caching is the process where copies of data are stored for quick access or retrieval.
  A cache stores a file or directory of files such as dependencies or source code in object storage. To speed up the build, each job may contain special steps for caching dependencies from previous jobs.
  * Workspaces:
  Workspaces are a workflow-aware storage mechanism. A workspace stores data unique to the job, which may be passed along to other downstream jobs(in the same workflow). Each workflow has a temporary workspace associated with it.
• Docker Layer Caching(DLC):
  DLC caches the individual layers of Docker images built during your CircleCI jobs. Any unchanged layers are used on subsequent runs, rather than rebuilding the image each time.
  In the image below, you realize that the configuration specified, docker_layer_caching is true, meaning that every layer built in a previous job will be accessible in the Remote Docker Environment thus the image layer is saved for the build job.

• Dynamic Configuration:
  Dynamic configuration allows you to trigger builds from specific parts(for example, failed parts or a particular job) of your project, rather than rebuilding everything each time you run. To use the CircleCIs dynamic configuration feature:
  First, you will need to enable that feature under the Advanced section, found in Project Settings.
  Next, you need to add setup: true your config. I believe this has similarities with the setup annotation in Java.
  The other thing will be to use the continuation orb so that you can use it as an executor.
• Execution Environments:
  In the previous concept, I mentioned an executor. In the image above, the executor is docker which spins up a virtual Ubuntu image. An executor can be a Docker container, or a virtual machine running Linux, Windows, or macOS.
• Images:
  An image is a packaged system that includes instructions for creating a running container. In a .circleci/config.yml file, you will find the primary container is defined as the first image. This is where commands are executed for jobs, using the Docker or machine executor.
• Jobs:
  Jobs are essential to the structure of your configuration file. They define the steps which run commands/scripts as required. Each job must have a unique name and must declare an executor that is either docker, machine, windows, or macos.
• Orbs:
  Orbs are reusable parts of code that help automate repeated processes, accelerate project setup, and make it easy to integrate with third-party tools like Slack and Opsgenie.
• Parallelism:
  Tests are very necessary for ensuring code quality even though jobs are configured to. However, a lot of tests are known to slow down the running of jobs on a single machine.
  With Parallelism, you can spread your tests across a specified number of separate executors; this can be referred to as test-splitting.
• Pipelines:
  A CircleCI refers to the processes executed when triggered on a project.
  In CircleCI version 2.0 upwards, the entirety of the config.yml file is basically the pipeline. This coordinates the jobs you specify in your config.yml file.
• Projects:
  A CircleCI project refers to the code repository in your VCS(like Gitlab or Github).
  You can set it up by clicking Set Up project and choosing the branch on which the config.yml resides on.

You can also follow the project to get access to the pipelines or subscribe to get email notifications.

• Resource Classes:
  A resource class provides a way of specifying available computing resources like CPU and RAM for jobs. This is similar to specifying your instance types in AWS.
• Steps:
  Steps are executable commands which are required to complete your job.
  An example is the run step which specifies custom commands to be run and the checkout step which checks out the source code for a job over SSH.
• User Types:
  In CircleCI, there are different user types defined with permissions which are derived from the associated VCS.
  Github: Owner
  Gitlab: Admin
  Bitbucket: Admin
• Workflows:
  Workflows automate the execution of jobs and run in the order which is defined. You can run jobs concurrently, sequentially, on a schedule, or with a manual gate using an approval job.

How to Connect to CircleCI

For this part,

• You need to have a VCS like Github or Gitlab account.
• You need to sign up and connect to the CircleCI web app.

After that, you will see a Dashboard with pipelines run and a Projects tab for the projects listed in your repository.

When you connect your VCS to CircleCI and authorize, it creates a deployment key on GitHub for you. However, you can go ahead to create a user key which is a user-specific SSH key-pair and add it to your Project Settings.

To generate a user-specific SSH key-pair:

On Mac :

  ssh-keygen -t ed25519 -C "your_email@example.com"

On Windows:

You can use Putty.

CircleCI adopts the configuration as code concept where configuration is specified within a config.yml file. A config.yml file is where the whole process from build to deployment is orchestrated. This file is created in a folder called .circleci which should be at the root of your repository.

Now, let’s create a sample config.yml file. You can also download a sample project here on my Github, clone or fork it to your repo. I have added comments on each line which highlight the aforementioned CircleCI concepts.

version: 2.1                                                 
                                  
jobs:                                                        
  hello-job:
    docker:                         
      - image: cimg/node:lts          
          username: mydockerhub-user                      
          password: $DOCKERHUB_PASSWORD                  
    steps:
      - checkout                                           
      - run: echo "Hello World. I am a successful job."      

workflows:                                                 
  sample-workflow:                                         
    jobs:
      - hello-job  

In the next article, we will use the sample config file to run our first green build and check out CircleCI’s CLI.

Let me know if you have questions, corrections, contributions or comments.

Till then,

Stay safe :)

References:

• CircleCI docs
• Photo credit: Photo by Vector Logo Zone, Edureka