GitLab CI/CD for Django on AWS.
Setting up a local Gitlab repository on the same machine as a runner for deployment on AWS EB.
A tutorial for a specific case of using Gitlab on Docker to set up a local Gitlab repository on an Ubuntu machine. This machine can also be used for CI/CD purposes by running a Gitlab Runner, which starts a local staging server running Django and deploys the project to AWS when specified.
Setting up a GitLab repository
To set up your own GitLab repository on Ubuntu there are a few options. You can install it locally or using a Docker image. We will do it using Docker as it is more structured and optimal.
Firstly as already mentioned we must install Docker so we can run an image containing everything GitLab needs. If the command docker does not work please refer here for more details on how to install Docker.
Running a GitLab image
As Docker is installed we must run the Docker image that contains everything a GitLab server needs to function. The following command worked for me:
sudo docker run --detach --hostname <insert your ip> --publish 443:443 --publish 80:80 --publish 23:23 --name gitlab --restart always --volume /srv/gitlab/config:/etc/gitlab --volume /srv/gitlab/logs:/var/log/gitlab --volume /srv/gitlab/data:/var/opt/gitlab gitlab/gitlab-ce:latest
This sets up a GitLab Docker container that runs on a specified hostname and ports. It also restarts it when we reboot the machine. More details about the command can be found here.
With this command, you can check your running Docker containers and see if the GitLab image was correctly set up.
Very important here is that all the parameters are set correctly and by understanding them as etc. an incorrectly set hostname can produce errors later with not finding the host.
If everything is set up correctly we will find a version of GitLab on a specified IP inside the network that can access it. You should see a login page like:
Congrats! Now you have your own GitLab instance!
Here you have to Register for a new account as this is separate from the official GitLab.com instance. If we would like to access this GitLab instance from outside the local network some port forwarding must be configured and preferably a static IP obtained and combined with a DNS service.
Setting up Continous Integration and Continous Deployment
GitLab is a great tool for version control, but lately, it also excels in DevOps tasks as CI/CD. It combines the two tasks in one platform and makes it easy to implement both.
Set up your own Runner
A GitLab Runner is a separate service usually ran on top of Docker Container that performs commands which are defined in a .gitlab-ci.yml file in your repository.
Let’s first mention that we do not have to set-up our own GitLab Runner for running tasks and can opt for a shared Runner which is done by default. However, this comes with a cost as you are not able to fully use it always and have to wait for others to finish using it or you can literally pay to use it.
How to install a runner on an Ubuntu machine is best described in the official documentation.
After we have created a Runner we must register it for our GitLab project to use. This is again best described in the official documentation. When following commands in the docs it is important to use a Docker executer as that is what we will use later for execution. When adding tags to the Runner make sure they are in sync with your projects tags as the worker might not work otherwise and it is best to not assign them at all in that case.
If Registration is successful we must firstly Disable Shared Runners for a specific Project:
Now, if everything was successful, we can see a working Runner and enable it:
Congrats now you have a Runner to run your deployment scripts!
POSSIBLE PROBLEMS WITH DOCKER NOT CONNECTING
When I finished setting up runners I came across a problem where I get this error when trying to use docker-compose in a staging script:
I checked and I already install docker-compose in one command before so it should work right? NO! The solution is in modifying config.toml file that configures Runners and I found the solution here:
I'm trying to get the docker in docker build to work. But every time I end up with `Cannot connect to the Docker…gitlab.com
Prepare project for CI/CD
Creating a .gitlab-ci.yml in your projects root directory is the next thing to do. This file is very important as it defines the behaviour of the project when new code gets pushed to it. How to set it up for a Python application (etc. Django) is described in the documentation. Also, we must take care of requirements.txt file as al the packages defined there get installed on our Runner every time.
Documentation for GitLab Community Edition, GitLab Enterprise Edition, Omnibus GitLab, and GitLab Runner.docs.gitlab.com
Local staging server and AWS deployment
My requirements were to have a local test server for testing a platform everytime new code gets pushed and deploying the code to AWS Elastic Beanstalk only when the code is pushed to a certain deploy branch. This was not described in official documentation so I had to adapt. Used deployment files that are needed by Docker and GitLab are contained in this repository:
To achieve local staging server running runserver command simply on the manage.py file was more difficult than expected as a Runner is not meant to run a server. Thus we have to run Docker inside Docker Runner and we need some additional files describing this Container. This part was adapted using this tutorial.
Adding Dockerfile to the project directory is the first step and describes how to set up a Docker Container and what ports to open for the server to work. The following code was used for my case:
RUN mkdir /spinnerStage
ADD . /spinnerStage
RUN pip install -r requirements.txt
Now we need to define the commands that are run when composing a Docker container that will serve a Django server on port 8000. This is done by adding a file docker-compose.yml to the project directory. Example of mine:
command: python manage.py runserver 0.0.0.0:8000
Firstly we need files for deploying to AWS that describe what to connect to. This is described here in 2., 3. and 4. step so that eb deploy command works on the Runner. Secret variables for deployment must be specifically provided to a runner by GitLab and set specifically for each project in Settings:
Finally, after setting things up it is very important to have the final .gitlab-ci.yml file correctly set as it defines most things. The final version of mine: stages:
- echo “Deploying the app”
- pip install docker-compose
- docker-compose build
- docker-compose up -d
- mkdir ~/.aws/
- touch ~/.aws/credentials
- pip install -r deployReq.txt
- printf “[eb-cli]\naws_access_key_id = %s\naws_secret_access_key = %s\n” “$AWS_ACCESS_KEY_ID” “$AWS_SECRET_ACCESS_KEY” >> ~/.aws/credentials
- touch ~/.aws/config
- printf “[profile eb-cli]\nregion=ap-southeast-1\noutput=json” >> ~/.aws/config
- export PATH=~/.local/bin:$PATH
- eb deploy
A result of this configuration is that every push on the master branch gets compiled on a runner server in a Docker container which serves a Django server on port 8000 for testing. More importantly, when we are sure that our code is as bugless as possible, we can push the code to the branch named deployed and then the code gets deployed to AWS automatically.
Comment: This article is under construction and will improve with time thus corrections are welcome. For any questions contact the Author and refer to a file GitLab Insights.