CI/CD Pipeline with Jenkins, Docker, K8s, AWS EC2, and GitHub
Automate your application with AWS, EC2, and the rest of alphabet tech soup, so you can blame the robots when it breaks.
Introduction
I am sure you have heard these fancy words like CI/CD, DevOps, development practices, automatization, etc. Those are must-have tools in modern software development. These automatization practices were so good that they spread in all IT spheres including AI, specifically machine learning development, so familiarity with them is necessary in the tech world. Hence, this series of practical articles, which starts with a simple CI/CD pipeline and ends with a suffisticated MLOps pipeline. Namely, it consists of:
- This one: theory + continuous integration part (GitHub+AWS EC2+Jenkins+Docker)
- Continuous deployment part (creating a K8s cluster and deployment there)
- Setting up a private docker registry
- MLFlow: how to automate your machine learning experiment
- MLOps: modifying the classical CI/CD pipeline for machine learning
This article consists of shallow theory (what is CI/CD, what tools we will use), practical section(Part 1: setting AWS EC2 for Jenkins and downloading Docker there,, Part 2: setting GitHub webhook + creating all the necessary files), and possible troubleshooting. At the end of the day, our pipeline should look like this:
