Manage Docker Containers using CoreOS — Part 1
If you are reading this post, I am sure that you have already heard of Docker. I have written a very basic introduction of Docker here. I have also written a basic docker command line and docker file tutorial.
Before I wrote this post, I was pretty convinced with the idea of docker containers but I was still trying to figure out answers of following questions.
- What is the best way to manage docker containers?
- What is the best way for service discovery in an environment where all the components are running inside separate docker containers?
CoreOS seems to have answers of all my questions and i decided to try it out.
CoreOS 101
CoreOS is a Linux based OS that’s designed to be a modern, minimal base to build your platform. CoreOS is designed for security, consistency, and reliability. Instead of installing packages via yum or apt, CoreOS uses Linux containers to manage your services at a higher level of abstraction. A single service’s code and all dependencies are packaged within a container that can be run on one or many CoreOS machines. Linux containers provide similar benefits as complete virtual machines, but focused on applications instead of entire virtualized hosts. Since containers don’t run their own Linux kernel or require a hypervisor, they have almost no performance overhead. The lack of overhead allows you to gain density which means fewer machines to operate and a lower compute spend.
The main building block of CoreOS is docker, a Linux container engine, where your applications and code run. Docker is pre-installed on each CoreOS machine.
CoreOS uses etcd, systemd and fleet along with docker to bring everything together. All these services are pre-installed on a CoreOS machine.
etcd 101
etcd is an open-source distributed key value store that provides the backbone of CoreOS clusters. It is very similar to zookeeper and can be used for service discovery, configuration management or any other distributed key value situation. We will be using it for service discovery and configuration management in part 2 of this article.
systemd 101
systemd is a system and service manager for Linux. A very good tutorial about systemd is available here. We will use systemd to manage services on CoreOS machine including start/stop of docker containers in part 2 of this article. We will never be interacting directly with systemd, we will always use fleet (introduced below).
fleet 101
You can think of fleet as an extension of systemd that operates at the cluster level instead of the machine level. Systemd is a single machine init system; fleet is a cluster init system. Fleet works by ingesting systemd unit files with a few extra attributes that control how your jobs are dispersed in the cluster. We will be using it to query our CoreOS cluster later in this article.
CoreOS cluster
I would like to talk about CoreOS cluster management before we start installing CoreOS. CoreOS claims to be clustered by default. What does it mean and what is the importance of it?
As shown in diagram above (picture taken from CoreOS website), you can connect your CoreOS machines together to form a cluster. Just assume this is your DEV environment which consists of 5 machines (core-host1 to core-host5). All these machines are managed by fleet (most probably running on a separate machine). You are free to run any number of docker containers on these machine easily using fleet.
How does CoreOS machines form a cluster:
CoreOS machines form a cluster using discovery API (explained in detail here.) The discovery API communicates with a separate etcd cluster to store and retrieve the list of peers. CoreOS provides https://discovery.etcd.io as a free service, but you can and should run your own etcd cluster for this purpose.
This brings up another question, If you are launching your CoreOS machine in Cloud, how do you pass the discovery URL so that the machine you are launching joins the cluster? coreos-cloudinit is one of the answer. coreos-cloudinit can also be used for other purposes, please go through the documentation.
Let’s start 3 CoreOS machines using the discovery service provided by CoreOS and make sure they form a cluster.
Step 1: Launch CoreOS cluster
We will be using Vagrant to build 3 CoreOS virtual machines. Vagrant file and instructions are available in this github repository. If you follow the instructions, you should have 3 CoreOS VMs on your machine and these virtual machines form a CoreOS cluster.
Step 2: Validate cluster configuration : There are multiple ways to validate cluster configuration.
Using Discovery API: Hit Discovery URL that you configured in user-data file in Step 1 and you should see the output with your CoreOS nodes details.
{“action”:”get”,”node”:{“key”:”/_etcd/registry/c2285a3c6fe10cae6045c77b1b2ec428",”dir”:true,”nodes”:[{“key”:”/_etcd/registry/c2285a3c6fe10cae6045c77b1b2ec428/ce3902061308464fad51003aa001cfed”,”value”:”http://172.17.8.101:7001","expiration":"2014-05-21T19:32:20.679317136Z","ttl":604669,"modifiedIndex":17534674,"createdIndex":17534674},{"key":"/_etcd/registry/c2285a3c6fe10cae6045c77b1b2ec428/067358b60dd54957afda42a1246a7cce","value":"http://172.17.8.102:7001","expiration":"2014-05-21T19:32:42.672832076Z","ttl":604691,"modifiedIndex":17534773,"createdIndex":17534773},{"key":"/_etcd/registry/c2285a3c6fe10cae6045c77b1b2ec428/96f4580bde7c4e4ba0ccef89a2f078ad","value":"http://172.17.8.103:7001","expiration":"2014-05-21T19:33:05.246183702Z","ttl":604714,"modifiedIndex":17534869,"createdIndex":17534869}],"modifiedIndex":17526118,"createdIndex":17526118}}Using fleetctl from one of the cluster machines: Connect to one of the cluster machines and run fleetctl list-machines
[mohit:~/github/coreos-vagrant] master ± vagrant ssh core-01
CoreOS (alpha)
core@core-01 ~ $ fleetctl list-machines
MACHINE IP METADATA
ce390206… 172.17.8.101 -
067358b6… 172.17.8.102 -
96f4580b… 172.17.8.103 -Using fleetctl from external host: I prefer to run fleetctl from a separate machine, If you are in the same situation you will have to download fleet binaries from https://github.com/coreos/fleet/releases, please make sure you download the version that matches the version of fleet running on the CoreOS cluster. Apart from binary on this separate machine, fleetctl requires 2 more things
- SSH access for a user to at least one host in the cluster
- ssh-agent running on a user’s machine with the necessary identity
In order to satisfy #1 and #2 above, i added following to .ssh/config on the separate machine where I am running fleetctl. Vagrant insecure_private_key is available in ~/.vagrant.d, which i have copied to shared directory /vagrant.
Host core-01
User core
HostName 172.17.8.101
IdentityFile /vagrant/insecure_private_keyOnce the fleetctl binary is installed and communication is established, run following command
[vagrant@vagrant-centos65 ~]$ export FLEETCTL_TUNNEL=172.17.8.101
[vagrant@vagrant-centos65 ~]$ ssh-add /vagrant/insecure_private_key
Identity added: /vagrant/insecure_private_key (/vagrant/insecure_private_key)
[vagrant@vagrant-centos65 ~]$ fleetctl list-machines
MACHINE IP METADATA
ce390206… 172.17.8.101 -
067358b6… 172.17.8.102 -
96f4580b… 172.17.8.103 -At this point of time, we have a CoreOS cluster up and running with 3 machines and each machine has following services installed or running.
- docker
- etcd
- systemctl
- fleet
Another very important thing which I have not mentioned yet is that etcd running on our cluster machines forms a etcd cluster. This etcd cluster is a cluster for your environment and is different from etcd cluster that we used for discovery. We will be using this cluster for service discovery and configuration management in part 2 of this article. Let’s make sure etcd cluster is working
On Machine core-01, add a key-value pair in etcd
core@core-01 ~ $ curl -L http://127.0.0.1:4001/v2/keys/mykey -XPUT -d value=”this is awesome”
{“action”:”set”,”node”:{“key”:”/mykey”,”value”:”this is awesome”,”modifiedIndex”:1554,”createdIndex”:1554}}Retrieve it on Machine core-01
core@core-01 ~ $ curl -L http://127.0.0.1:4001/v2/keys/mykey
{“action”:”get”,”node”:{“key”:”/mykey”,”value”:”this is awesome”,”modifiedIndex”:1554,”createdIndex”:1554}}Let’s retrieve it on Machine core-02core@core-02 ~ $ curl -L http://127.0.0.1:4001/v2/keys/mykey
{“action”:”get”,”node”:{“key”:”/mykey”,”value”:”this is awesome”,”modifiedIndex”:1554,”createdIndex”:1554}}
Awesome…
In this post, I have explained the basic concepts of CoreOS and how it works. It took me quite some time to figure out all this. This is kind of one time setup and you probably don’t care much about this ☺. You really want to know how can you now use this cluster to deploy and manage your docker containers. I will be covering that in Part 2 of this article.
