DevOps Tools

When we talk about DevOps tools, the abundance of options can be a bit overwhelming, sparking some concern. However, don’t worry! In this article, I’ll ease your worries and explain the purpose of these tools with a simple story.

Let’s dive into our story!

The members of New World Company, filled with brilliant ideas, are gearing up to step into the digital realm to market their innovations. To achieve this, they not only need a perfect application but also must provide uninterrupted services. Additionally, as the company is in constant competition with market rivals, investing in innovation is crucial.

We are confident about creating this application. What’s the first thing we need to do? Of course, write code!

The code we’ve written is now running on our computer and is accessible through the HTTP localhost IP address on port 8000. However, it’s currently not shared with the world. Even if you find a way to share it from your computer, once you shut it down, no one can access it.

In this case, we haven’t met our client’s need for continuous accessibility. To address this, the application must exist in a system that never shuts down. This system could be a physical server in a data center or a virtual machine (VM) in the cloud.

Let’s copy our code and deploy it to the server…

Copying the code doesn’t necessarily mean it will run on the server. To be able to run the application, you need to configure the system. For example, if the application is written in Python, Java, or another programming language, you must have those programming languages or runtimes on the server. If the application uses libraries or packages, you should configure the server with the exact versions of those libraries and packages.

After configuring our system, we can then move our developments to this environment. We can call this environment our “DEVELOPMENT” environment.

Our application is now operational in the Development environment. The server in our Development environment has an IP, so it’s accessible through this IP address. I believe it’s time to have a server that can be accessed worldwide. Now, we can open it up to the entire world. However, we don’t want our clients to access it via an IP address. For this purpose, we purchase a domain name and link it to our server. We can now refer to the environment where our application resides as the “PRODUCTION” environment.

Everything is ready, and now we are ready to share with the world! The service offered by the members of New World Company has been highly praised by people! Our customer base is growing every day… We have an application with thousands of users! Our application needs to continuously evolve to meet user needs and accommodate new ideas!

We are starting to review our current workflow for the Development environment. The Integrated Development Environment (IDE) installed on the developer’s computer is essentially creating a text file. The code in text format is not sufficient to be run as an application by end-users, depending on the programming language of the software. For text files, tools like MAVEN or GRADLE, suitable for compiling, will compile your project based on system specifications and make it executable.

For such a large project, having just one developer is impossible! To prevent confusion when working on the same code as a development team, GIT is used. GIT assists all developers in working simultaneously on the same application and collaborating effectively. GIT is installed on all Developer PCs. Having GIT installed allows the centralized code developed collaboratively to be pulled to the Developer PC through a central HUB. A developer can make the necessary changes to the code and then push it back to the central HUB.

GIT is a fundamental technology that provides version control for code and enables collaboration among multiple developers. GitHub is a web-based interface built on GIT, offering many capabilities such as pushing/pulling code and defining different accesses for various users. Its counterparts include GitLab and Bitbucket.

Our team can now collaborate on developing shared code. They can build the code on their own computers and deploy it to the production environment.

Developers need to get the latest version of the code from their teammates. Instead of building it on their local machines, they can build it in a pre-production environment. We can now refer to the environment where our application will be built as the “BUILD” environment.

Furthermore, we don’t want to deploy a built code directly to the production environment. We’ll need an environment where we can test the code without showing potential errors to end-users. Additionally, we should conduct tests using tools like JMeter to assess the performance of our application. We can now call the environment where our application will be tested the “TEST” environment.

Our environments seem ready for code development and deployment. The developer works on their code in their environment and can push it to GitHub. After that, they can copy the code to an environment where it can be built and made executable. Once the executable code is moved to the test environment, we can conduct tests. If there are no errors during testing, we can deploy our code to the production environment, accessible to users.

This workflow is currently happening through manual transfers, and the application’s accessibility may be interrupted due to the slow nature of these manual processes. The most critical issue is the errors occurring during the deployment phase.

If we categorize deployment errors broadly, we can identify:

1. Version Incompatibilities: Incompatibility between a new version and an old version can arise due to changes in database schemas or incompatibility with old APIs.
2. Dependency Issues: Incomplete or incorrect versions of external dependencies can lead to deployment errors.
3. Configuration Errors: Incorrect configuration files or missing settings in the live environment can cause issues during the deployment process.
4. Server Problems: Hardware or software issues on servers can result in errors during deployment.
5. Network Issues: Connection disruptions, network configuration errors, or traffic congestion can cause problems during deployment.

These errors vary based on the organization’s development, testing, and deployment process quality, automation level, team experience, and the technology used. An effective CI/CD (Continuous Integration/Continuous Deployment) process, utilizing regular tests and automated deployment procedures, can minimize deployment errors.

At this point, our New World application can benefit from tools like Jenkins, GitHub Actions, or GitLab CI/CD to automate manual tasks and expedite the process through a pipeline. CI involves developers frequently merging and testing their code, covering Version Control, Automated Build, and Automated Test steps.

On the other hand, CD ensures the automatic deployment of software to the live environment after testing and approval. CD includes steps like Artifact Management, Deployment Automation, and Release Orchestration.

For developments made based on the application technology we will use, consistency in dependencies and libraries should be maintained across all environments. In this case, if a new package is to be used, we would need to manually install it on all environments.

Skipping the correct configuration of one of these packages in the right version can lead to the software not functioning properly and result in unexpected outcomes on different systems.

Right at this point, container technology comes into play. Containers allow you to package the application and dependencies into an image that can run on any system without concerns.

One of the technologies that work with containers is Docker. You can create a Dockerfile that specifies the dependencies for developers, and this Dockerfile can be used during configuration to build an image. With a simple command, you can then run the created image. This way, Docker facilitates the packaging and running of applications seamlessly across various systems, eliminating concerns about dependencies.

Now that we can work with containers, we can create various containers on each server group. One of the key tasks of containers is to provide isolation. This means that applications and processes can run independently of each other, limiting the interaction of components within a container with the system outside. Containers use isolation mechanisms integrated into the operating system, isolating the environment in which an application or service runs from other applications and system resources. As a result, each container has its own independent working environment and does not interfere with other containers or system resources.

In a scenario where our user count is increasing, we want to increase the number of servers in the production environment, considering redundancy. Since we are familiar with container technology, we can run containers on each server group.

Here, depending on the load imbalance for certain server groups, we might want to increase the number of containers. Or, we may want a container to automatically restart when there’s an issue.

Kubernetes is a popular technology in container orchestration. It helps determine how containers will be deployed and monitors the continuous availability status. Kubernetes can automatically scale containers and the underlying infrastructure based on need, helping manage resources on servers to ensure the most effective utilization. It provides a solution for scenarios where we want to dynamically adjust the number of containers or automatically recover containers in case of issues.

Let’s summarize the technologies used in the described process:

1. Developers code on their local computers and upload their code to a Git system.
2. Developers send their code to a centralized repository like GitHub via Git.
3. Whenever there’s an update in the repository, a Continuous Integration (CI) tool like Jenkins is automatically triggered. Jenkins pulls the updated code from the repository and performs configured build processes.
4. Tools like Maven manage dependencies and resolve necessary dependencies for the project. A Docker image is created as a result of the build process.
5. Docker creates a container image containing the environment necessary for the application to run.
6. Jenkins deploys the created Docker image to the test environment, automatically running application tests. Performance tests can be conducted using tools like JMeter.
7. After successful tests, Jenkins deploys the container to the specified environment on Kubernetes.
8. Kubernetes manages the application with the specified number of running containers.

As the user count of the application developed by New World increases, it will demand more resources. Therefore, infrastructure management is of great importance. Each new server must have a compatible structure with other servers in the environment. The correct version of the operating system and effective resource management are critical in this process.

Additionally, components such as storage units added to the server, as well as elements like Docker runtime or necessary Kubernetes packages, need to be preconfigured correctly.

This process is a significant challenge, requiring infrastructure to be automatically and consistently configured rather than manually clicking through the interface of the cloud platform at each server provision stage. The manual configuration process can be time-consuming and may lead to errors in the infrastructure, requiring the recreation of all servers.

Here, Infrastructure as Code (IaC) technology using Terraform can be employed. Terraform automates the provisioning and configuration of servers on any cloud platform, ensuring that configured servers are always in the same state.

By creating a manifest file on Terraform, we can easily manage our entire infrastructure. Moreover, with the ability for quick configuration when adding a new server, Terraform can prevent issues caused by misconfigurations or human errors. We can achieve this by writing code in a file, as shown.

We activate Terraform by controlling these configured files in a separate server group.

Now, after provisioning servers, their configuration can be automated using tools like Ansible. While Terraform is primarily an infrastructure provisioning tool, Ansible is an automation tool that assists in configuration after the infrastructure is provided. These tools have the capability to automate both infrastructure provisioning and software configuration on that infrastructure. Terraform is often used for infrastructure provisioning and configuration processes on the provided infrastructure, while Ansible is used for subsequent configuration activities such as software installation and configuration on these servers.

Ansible uses code to configure servers. This code is referred to as “Ansible playbook,” and these codes are placed in a source code repository on GitHub.

The large-scale system of New World Company is now in place! To ensure the continuous sustainability of this system, we need metrics. These metrics should be provided to measure end-user satisfaction and take preventive measures against potential anomalies in the system. For this purpose, monitoring technologies are essential.

Prometheus collects information or metrics from different servers and stores them in a central repository. Since merely collecting metrics would pose challenges in our analysis, we’d like to visualize these metrics using technologies like Grafana.

Grafana, by making the data collected by Prometheus meaningful, helps transform them into graphs and charts, facilitating better insights and understanding of the system’s performance.

The developers working on the New World application begin to write code to bring their innovative ideas to life. The code is compiled to support a robust infrastructure. The compiled code undergoes rigorous testing processes. Once the application successfully passes the tests, it becomes ready to be offered to users. As the user count increases, the application is expanded, and new features are added. We continue to work on improving its performance and efficiency continually. We monitor our application consistently, closely tracking metrics such as server performance, error rates, and user interactions. Based on the feedback we receive from users, we continue to iterate and enhance the application.

“Our goal at New World is to provide you with the best experience and continually improve our application. Every feedback from you helps us achieve this objective. Thank you, and we are grateful for your continued usage!”

Yes, our story has come to an end. I hope I was able to provide you with useful information about some popular tools in the DevOps world.

Here are brief descriptions of the referenced tools in the story:

• Git: A distributed version control system used for tracking and collaborating on code.
• GitHub: A Git-based platform hosting code repositories and facilitating collaboration.
• Jenkins: An open-source automation tool providing continuous integration and continuous deployment.
• Maven: A project management and comprehension tool for Java projects, managing dependencies and compiling projects.
• Docker: A lightweight container technology platform used to run applications in isolated environments.
• JMeter: An open-source load testing tool used for performance testing.
• Kubernetes: An open-source platform providing container orchestration for automatic deployment and management of applications.
• Terraform: An infrastructure-as-code tool managing server resources across different cloud platforms.
• Ansible: An automation tool used for configuration management, application deployment, and task automation.
• Prometheus: An open-source tool used for system and service monitoring, collecting and storing metrics.
• Grafana: An analysis and visualization platform used to visualize data from Prometheus and other databases.

We’ve reached the end of our article. Until next time!

To reach me: Linkedin