Docker Orchestration and Continous Development
What is Docker?
Docker is a container platform that is useful for creating and deploying applications. We are using docker for deploying our front-end service, unlike back-end service which uses AWS serverless for deployment.
Docker and Gitlab Interaction in short
In short, Docker has a registry that contains Docker image and receives the image from gitlab. Docker user can run the image via portainer, in which the image is then rendered to become one of more instances of a container.
Docker Image
Docker image is a compressed file used to execute code in a Docker Container. in this case, the Docker image is our project files and can be created according to the implementation of the Dockerfile. When the Docker user runs the Docker image, it will become one or more instances of that container, as mentioned earlier.
This is what Dockerfile looks like. First, we use alpine, which is a lightweight, linux distribution. Currently, we are going to use Docker for front-end,which uses ReactJS, thus we are going to use npm install--silent and npm install react-scripts@3.0.1 -g--silent to install dependency. running the ENV PATH and COPY beforehand is essential so that the package.json can be accessed by the npm.We also need to run EXPOSE 80 to explicitly tell Docker which port we are going to assign(in this case, port:80).
Then, we are running npm run build to finally compile the front-end resources(.html, .css, .js). After building, we change our environment from node to nginx, since we dont need node environment for deployment. we are using COPY--from=build to copy to the correct folder. The next 2 command lines, rm and COPY is used to change the nginx configuration file. After calling EXPOSE 80 , we call CMD to tell docker to run the command when running the image
There are a lot of things that occur in .gitlab-ci.yml file, but we can split them into three: development, staging and production. These three can be seen as the stages of our project, which depends on how ready is our branch is. In the yml file, the difference is the endpoint of the docker build, which can be seen above. Some of the codes that is written there is the implementation on how to run Docker in fasilkom’s Docker.
The important portion of the codes is located on the script, which includes pull, build and push. docker pull pulls the latest image from the Docker platform, if exists (using ||true). docker build recreates the Docker image and docker push push the image back to the platform.
Our use of Nginx
Nginx is open source server, commonly used as a reverse proxy and load balancer to manage incoming traffic and distribute them to slower upstream server. With its goal to create the fastest web server around, it is an excellent choice for our front end deployment. After building using npm, these front end files become static files, in which we use Nginx to deploy them. In Moodah POS case, it is used to route request and response to or from the backend server(s).
DIND (Docker in Docker)
We implement Dicker with Docker so that the container(chuck of files), will not run on its own Docker daemon, but connects to the Docker daemon of the host system. This means that the Docker CLI in the container and host system will connect to the same, one Docker Daemon. The following codes are responsible for creating Docker in Docker service:
services:
- docker:19.03.0-dindContinous Development: Pipeline Configuration with gitlab-ci.yml
To create a continuous development, Git provides a way with a YAML file called gitlab-ci.yml . This file should be placed within each project on the root, and will defines the structure and order of the pipelines.
With gitlab-ci.yml, Git will know what to execute, and the steps that is needed before finally deploying to the main server. This includes having tests that can check whether the application met the criteria or not.
To illustrate, the image above shows three checklists, which indicates that the stages have passed. The two right checklists shows that the tests and release stage have passed, which also passes the pipeline as a whole. If there is an “x” mark, the pipeline will not be accepted and your server will not deploy that specific branch, which saves your running application from crashing. With gitlab-ci.yml, you can modify these if you want to create more constraints before accepting certain pipelines.
Our gitlab-ci.yml file has stages that will seperate the other statements that s called jobs. These stages are test, deploy backend, release. From the YAML file, the 5 jobs below the stages have different stage category, in which will be used to differentiate on which should go first. Jobs that are in the same stage will run in parallel, and will the next jobs will wait until the test stage has finished.
Test
During the test stage, the test frontend and test backend will work in parallel. They will use alpine as their image node, for javascript application and lightweight images. The script will move to two different respective folders using cd and install the required packages that is inside each folder using npm install . After installing needed dependencies, the jobs run npm run test that runs the tests file in each folder that expects the codes to meet the given criteria. The coverage will then be extracted, checking for percentage of criteria met, using regex command coverage: /All files[^|]*\|[^|]*\s+([\d\.]+)/
For test frontend , npm run gql-gen will be executed before test so that the GraphQL can be executed from the frontend ReactJS. On the other npm run lint is executed on test backend because we established a styling check for the GraphQL functions.
Deploy Backend
Specific to backend AWS lambda server, we are using serverless library. Thus, we need to install the additional dependencies via npm install -g serverless . STAGE inside the variables are used to be placed in the command $STAGE in serverless deploy --stage $STAGE --verbose , in which option --verbose is used to show all stack events during deployment, and display any stack output. This is also useful in catching errors when running the serverless deploy command.
Above is an illustration of the environment key used to classify jobs in the Gitlab repository. This is shown under the Environment section. Below the environment key, there is only key that restricts the running process of this job under specific changes. In our case, the deploy prod backend job will only be run when there is a change in backend/serverless.yml under master branch. This specific YAML file is used for base server configuration in the GraphQL server, in which the end point of that server needs to be catch by the frontend. On normal terms, the GraphQL server tend to be configured manually at the serverless website, whenever needed changes are dealt separately from the frontend architecture.
Release
The last 2 jobs, release prod frontend and update prod backend functions are fallen into release stage, in which these 2 architecture will be deployed in two different containers. The frontend will have a docker image, which will be deployed in the Fasilkom’s docker registry. On the other hand, backend architecture will be released to AWS lambda server with serverless architecture.
For frontend release, the tags, services and variables key are configured to handle the DIND problems that stops working after the 12.1.0 update. All specified keys are added to choose which runners to run the job, defines another docker image that will be run on the job and is linked to the defined image in the image key. A set of commands from before_script are run before the script key to set the proxy server form Fasilkom docker side.
The script key in release prod frontend will be run afterwards. The first command form the script is docker pull,which pull the latest image from the docker platform and pulls the latest image from the Docker platform, if exists (using ||true). docker build recreates the Docker image, and can use the cache option, which can benefit cache made by the docker pull and specifications that’s inside Dockerfile. $CI_REGISTRY_IMAGE is a variable that is stored inside the Environment Variables under CI/CD Git Settings. Such useful features can be used to store important variables that are marked private, so that it will not be seen by unauthorized people. docker push will push the image to the specified address in the Fasilkom’s docker platform.
update prod backend functions from the backend server will add or update functions in the AWS lamba server. Similar to deploy prod backend , it uses serverless library and alpine image as well. However, update prod backend functions uses different deploy commands. Instead of using serverless deploy stage that changes the whole url endpoint, it uses serverless deploy function which will only change the function inside the endpoint. Thus, the stage will not change and frontend architecture will not need to check through new endpoints anymore.
There are 3 serverless deploy function that will serve functions under different url endpoints: graphql,playground,graphql-introspect . All of these 3 points are made for different uses. For example, playground url will be used for developers to check what functions are available, and their interactions with the backend Rest API endpoints. Other urls will also be used when developers want to use the GraphQL functions, under specified Secret Key.
