Lessons Learned from Managing Docker Services by Using Rancher (Version 1.x)
Overview
Moving from traditional deployment to container-based deployment required a lot of steps. First, we had to understand how container-based deployment such as using Docker work. Second, we had to know how could we orchestrate them in single place easily. In that case, two of my colleagues were choosing container management for Docker implementation at Sensehealth B.V. then end up with Rancher v1.
Rancher is made by Rancher Labs. It has container orchestration called Cattle Engine. Rancher consists of one server node and multiple agent nodes. They are communicating mostly via HTTP or HTTPS. Rancher has a really nice admin page that help us a lot while we are working for day to day operation on container-based deployment.
In this article, I want to share my personal experience while I am working as an infrastructure engineer at Sensehealth B.V. which works with multiple deployments from developer team such as from frontend, backend and data team.
After more than two years, I have a good overview for using Rancher v1 for container management. Here we go!.
Terminology
Before continuing to another section, here are some importants terminology that should you know first:
- Rancher Server: a node that control over multiple agent nodes
- Rancher Agent: a node that run and control the deployment also communicate to the server node to confirm the deployment
- Environments: A group of multiple stacks that server for its own purposes (e.g.: Staging, Production, Sandbox, etc.)
- Stacks: a group which hold multiple services
- Services: Run the Docker image as a service. It will run the image over multiple agent nodes.
- Cattle: Orchestration engine that powered Rancher v1
A. Operational
A.1. Manage multiple services with Rancher Stack
Let say that you have a collection of services that run on same project. You have backend service, frontend service, worker service, scheduled job service, push notification service and so on.
Grouping those services in single place might let those services more organized. Rather than run the service ungrouped. You are also able to apply a task upon those services such as stop or restart those services together. On the other hand, you could see the status of your services under that stack and apply same action to those particular services such as upgrading, stopping or restarting.
A.2. Easily scale our service
I only need to choose one of services that required to be scaled up. Then, I could increase or even decrease the amount of containers that I want. It will be deployed over all rancher agents or add new container in same rancher agent. Depending to the specified setting for that service.
A.3 Manage multiple stacks with Rancher Environment
Separating stacks from different context gives the team the ability manage them easily. Moreover, our team could limit someone from accessing some environments which are not allowed to get into it.
It’s not only stop there, you may have different orchestration engines or specific ACL rules for users who are going to access that limited enviroments.
A.4. Providing useful template for Etcd deployment from Community Catalogue
When I wanted to deploy Etcd for configuration management, Its really tricky if I have to deploy it manually. Then, Ricky suggested me to check catalog from Rancher where the community provides various templates. It will be deployed to Rancher easily.
So I choosed Etcd template from that catalog then I have it as a stack within particular environment. I don’t have to spend time a lot to deploy it. Moreover, it could be linked to another services that required Etcd as it’s configuratio management.
A.5. Monitor Rancher Agent by using Rancher Exporter
It’s not officialy Rancher template, but we can integrate it to our Rancher environment easily. We have to make sure our rancher agents are always available every second. Once there is an incident, I could get the alert from Prometheus and receive it from my OpsGenie app.
At least it could help me to know about the status of every rancher agent by using Rancher Exporter without frequent monitoring.
A.6. Manage Docker Registries
Currently, we only have DockerHub as our Docker registry. But I could say we could have more than one Docker registry in the future and add it to our Rancher environment.
A.7 Mark the Service Deployment by Scheduling
This is the thing that might be crucial. You can deploy specific containers in particular rancher-agents. Might be some containers will not be deployed on rancher-agents with high server specification.
Ricky used this technique to deploy chat service (based on Ejabberd) in rancher agent with more high specification. So other containers will not be deployed there. The resource in that rancher agent will be consume by chat service only.
B. Security
B.1. Manage ACL for infrastructure team
Not every environment should be accessible to everyone. I can limit someone who is granted to access “Infrastructure” environment. Of course, only infrastructure team which are able to access that environment. The other Rancher user wouldn’t be allowed to interact with infrastructure team related services.
B.2. Integrated SSL Certificate management
Most of our services required SSL certificate. That’s why we couldn’t locate it in every rancher agents. Therefore, we have to manage it in centralized way so we only need to load it in built-in load balancer and attached it to particular subdomain or domain name for particular services.
I could also check the expiration of some SSL certificates through this page.
B.3. Manage secrets
Secrets are kind of variables that should be kept from someone. It might be credentials or environment variable setting. You can manage those secrets in Rancher for some services who need it. On the other hand, we could reuse some secrets for multiple services.
C. Integration
C.1. Adding new Rancher Agent to existing cluster easily
I have tried to new host in some environments. To add new rancher agent, you only need to run rancher agent service within new node. Then, you can click “Add Host” in “Hosts” menu at Rancher Admin.
You will be asked to run Curl operation for joining new rancher agent to existing rancher cluster. Under a minute, it will be running with other rancher agents.
C.2. Trigger service deployment with Rancher Webhook
In some cases, Rancher could perform auto deployment for specific tag if the latest image for that tag is published. So for example, in the end of backend release process, new image for backend service will be created. Then, Rancher will receive an event from Docker Hub that will ask Rancher to pull new image for that specific tags. Usually, within a second, backend service will be upgraded to that latest image.
C.3. Built-in Load Balancer management with HAProxy for Exposing new Service
One of key features of Rancher is load balancer. They provided us HAProxy which has a built-in rule editor that could help the infrastructure team to manage service rules.
Usually, I worked for exposing some services to public internet or enable some restrictions to some services such as allowing some IP Addresses to access confidential services.
We could create HAProxy rule by defining the protocol, request endpoint, port on host, targeted service, service port and label for that rule.
We could also attach any SSL certificate to particular service who need HTTP connection.
C.4. Integrate existing non-container services as External Service
I think this is one of key features in Rancher. Some services need to access databases. We could build the database within Rancher cluster but it might be less effective since database service might overwhelm other services.
That’s why if we have existing databases or plan to run database in separate place, “External Service” is the answer.
We could point that external service to some accessible IP Addresses from our legacy databases then it would be linked to services who required those databases. Then, external databases might communicate with services in Rancher cluster.
C.5 Send Docker logs to ELK stack using Gelf Driver
This is the part i really enjoy. There are around 50 services that send their Docker log to ELK stack by bridging it using Gelf Driver. So our team could reduce the amount of disk usage within rancher agents and let ELK Stack manage those Docker log.
Moreover, I could perform error tracing for some services in integrated dashboard by using Kibana. With the power of Elasticsearch query, searching for errors in specific service or among services, it’s more handy than opening service page and looking for error in its log browser.
Conclusion
Working with Rancher v1 consistenly since 2018 gave me a lot of overview about managing Docker services by using container orchestration (Cattle) and container management (Rancher v1 itself).
It should be enough as first step to know about the main task of container orchestration and container management. If I should move to another container orchestration such as Apache Mesos, Kubernetes, Nomad or OpenShift, I might not be surprised because I have those common experiences with Rancher v1.
On the other hand, there are a lot of container managements in the market such as Apache Marathon, AWS Fargate and Kubernetes-like in some cloud providers. Rancher alone release Rancher v2 as the successor of Rancher v1 but it built for Kubernetes alone.
With this experiences, it gave me a lot of knowledge that I didn’t expected before. Thanks infrastructure team!
Ricky Hariady who introduced me to Rancher Cluster and infrastructure team who are always helping me work with Rancher for infrastructure-related tasks.
