docker and Kubernetes The beginner's introduction

Published in

Data Engineering and Linux

10 min readAug 30, 2018

Time has come when any big data developer must be concerned with containers and management of containers. Everything is moving towards decoupled development for higher scalability and maintainability. so it is going to be a De facto standard to understand and get accustomed with these technologies.

Docker the introduction :

Docker revolutionized the industry standard of how visualization can be handled for lightweight and easy development.

For the basics, I personally recommend to read and understand How Hypervisor Works and get to know how docker manages to deal with this and do the magic of containerizing anything and everything.

Some reads will help :

Linux namespace.
Linux Containers (LXC)
Types of Hypervisors (Type 1 and 2) FYI just attaching a basic image

4. Docker architecture Vs Virtualization architecture:

Here we can clearly see that docker simply removes the workload of a foreign operating system which in reality makes them light and less memory and process hungry without compromising the environment isolation.

Now the question arises :

How we can communicate in between containers?

How we can manage containers and use them when they are needed so that we can Pause them when not in use?

The answer is either Docker Swarm or Kubernetes

It’s not exactly easy to compare Kubernetes with Docker Swarm as Docker Swarm is a simple and easy solution to work with whereas Kubernetes is targeted those who require full fledge support with higher complexity. Docker Swarm is preferred in environments where simplicity and fast development is favored. Whereas Kubernetes is suitable for environments where medium to large clusters are running complex applications.

As for our study, I am going with Kubernetes.

Kubernetes Introduction:

Developed by Google, Kubernetes work as a cluster orchestration and management tool.

For learning purpose Kubernetes comes with a solution for Single Node cluster i.e. minikube

Hands on instructions

Docker

Installing Docker On Ubuntu

Source Here

Docker release: Docker CE

First, make update and upgrade of your packages

$ sudo apt-get update

$sudo apt-get upgrade

Now,

Install packages to allow apt to use a repository over HTTPS:

$ sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common

Add Docker’s official GPG key:

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Verify that you now have the key with the fingerprint9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88, by searching for the last 8 characters of the fingerprint.

$ sudo apt-key fingerprint 0EBFCD88pub   4096R/0EBFCD88 2017-02-22
      Key fingerprint = 9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88
uid                  Docker Release (CE deb) <docker@docker.com>
sub   4096R/F273FCD8 2017-02-22

Add the Docker repository to APT sources:

$ sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

Next, update the package database with the Docker packages from the newly added repo:

$ sudo apt-get update

ake sure you are about to install from the Docker repo instead of the default Ubuntu 16.04 repo:

$ apt-cache policy docker-ce

Now install Docker:

$ sudo apt-get install -y docker-ce

Docker should now be installed, the daemon started, and the process enabled to start on boot. Check that it’s running:

$ sudo service docker status

Configuring Environment to easily work with docker

Add the docker group if it doesn’t already exist:

$ sudo groupadd docker

Add the connected user “$USER” to the docker group. Change the user name to match your preferred user if you do not want to use your current user:

$ sudo gpasswd -a $USER docker

Now you can easily do docker commands from your own user

Just do a logout and login and you will not need sudo before every docker command

Docker Commands

Start Docker Service:

$ sudo service docker start

View available images:

$docker images

View Running containers

$docker ps

View all docker containers (running & stopped)
$docker ps -a

pulling an image from docker hub

$docker pull <imagename>

e.g. $docker pull nginx

Running docker via port 80 :

$docker run — name test-nginx -p 80:80 nginx

attaching a file to nginx html directory and running in nginx server:

$docker run — name test-nginx -p 80:80 -d -v /home/thirdeye/html:/usr/share/nginx/html nginx

accessing nginx :

$docker run -it nginx /bin/sh

Stopping Docker container:
$ docker stop contailner_id(find it from # docker ps)

FYI: to identify not running(created) containers:
$docker ps -a

Deleting docker container
$ docker rm comtainer_id

Deleting docker image
$ docker rmi image_id

Create your Own docker Image:

Creating a custom docker image for docker is a necessary skill because most of the cases you have to create your own Docker image and deploy it on the cluster:

In docker, we have to create a file named Dockerfile where we have to put all the info and steps to create a new image.

basically, we first pull a base image and add our required packages and files to that and then create the custom image.

Here is an example

Here i am just deploying a python flask app in docker image:

Contents Of Dockerfile

FROM ubuntu:latest
MAINTAINER Arani Mukherjee
RUN apt-get update -y
RUN apt-get install -y python-pip python-dev build-essential
COPY . /flask-app
WORKDIR /flask-app
RUN pip install -r requirements.txt
EXPOSE 5000
ENTRYPOINT [“python”]
CMD [“flask_docker.py”]

Contents of requirements.txt file

Just mention python package names

flask
redis

Contents Of flask_docker.py

from flask import Flask
app = Flask(__name__)

@app.route(‘/’)
def hello_world():
return ‘Flask Dockerized’

if __name__ == ‘__main__’:
app.run(debug=True,host=’0.0.0.0')

Now We have to build the docker image from the Dockerfile

$ cd directory/where/the/Dockerfile/is/kept

$ docker build -t “my_flask_image:Dockerfile” .

It will eventually pull the image and also download all required packages and add to the base image and finally create the custom image

Result:

To deploy this image in kubernetes there are 2 ways:
1 . Push the image in dockerhub or any private repository
2. Somehow make kubernetes use the local images for deployment
Now the easy one is pusing the image in DockerHub or any public repository
But in for ease of use you can use this way:
We have to somehow point a local docker build be automatically done in minikube space:
Step : 1
$ eval $(minikube docker-env)
window 1
$ minikube ssh
$ docker images
window 2
$ docker images
check the o/p of window 1 and window 2 are same or not if yes then the work is done:
Now any docker build will automatically create docker image under minikube space

Minikube ( Kubernetes Local single node cluster instance )

Installation:(Acording to 2nd AUG, 2018)

STEP 1: Install and configure docker

This is my docker info:::::::

$ sudo docker version
[sudo] password for user:
Client:
Version: 18.06.0-ce
API version: 1.38
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:11:02 2018
OS/Arch: linux/amd64
Experimental: false

Server:
Engine:
Version: 18.06.0-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:09:05 2018
OS/Arch: linux/amd64
Experimental: false

Machine Info :

Os: ubuntu 16.04LTS , RAM:8GB, Processor: intel core i3

Installing kubectl

URL :: https://kubernetes.io/docs/tasks/tools/install-kubectl/
First and foremost :
Stop swap memory location because Kubernates requires swap to be disabled
$ sudo swapoff -a
Make a restart of your machine:
$ sudo init 6
Now install kubectl:::::
$ sudo apt-get update && sudo apt-get install -y apt-transport-https
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
$ sudo touch /etc/apt/sources.list.d/kubernetes.list
$ echo “deb http://apt.kubernetes.io/ kubernetes-xenial main” | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
$ sudo apt-get update
$ sudo apt-get install -y kubectl
Check if kubectl is installed or not:::::
$ kubectl cluster-info
Kubernetes master is running at https://192.168.99.100:8443
KubeDNS is running at https://192.168.99.100:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use ‘kubectl cluster-info dump’.

Install Hypervisor(KVM)

KVM is what i used, one can configure hypervisor using VirtualBOX(oracle)
$ sudo apt-get install qemu-kvm libvirt-bin virtinst bridge-utils cpu-checker
$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

Installing Minikube

URL:: https://github.com/kubernetes/minikube/releases
$ curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.28.2/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

Basic Get To know your Minikube commands:

Start docker ::
$ sudo service docker start
$ sudo service docker status
● docker.service — Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2018–08–02 10:37:19 IST; 1h 4min ago
Docs: https://docs.docker.com
Main PID: 1981 (dockerd)
Tasks: 32
Memory: 102.0M
CPU: 13.320s
CGroup: /system.slice/docker.service
├─1981 /usr/bin/dockerd -H fd://
└─2045 docker-containerd — config /var/run/docker/containerd/containerd.toml
Start minikube ::
$ minikube start
$ minikube status
minikube: Running
cluster: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100
Get minikube dashboard::
$ minikube dashboard
get info regarding nodes present under your minikube::
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready master 1h v1.10.0
get info regarding pods present under your minikube::
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-6bfb654d7c-fc9xn 1/1 Running 0 1h
(by default on your first run there will be no pods)
Check available namespaces::
$ kubectl get namespaces
NAME STATUS AGE
default Active 1h
kube-public Active 1h
kube-system Active 1h
Check pods based on namespaces ::
$ kubectl get pods — all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default nginx-6bfb654d7c-fc9xn 1/1 Running 0 1h
kube-system default-http-backend-59868b7dd6-f7hms 1/1 Running 0 1h
kube-system etcd-minikube 1/1 Running 0 1h
kube-system kube-addon-manager-minikube 1/1 Running 0 1h
kube-system kube-apiserver-minikube 1/1 Running 0 1h
kube-system kube-controller-manager-minikube 1/1 Running 0 1h
kube-system kube-dns-86f4d74b45-rrbsn 3/3 Running 0 1h
kube-system kube-proxy-49wzf 1/1 Running 0 1h
kube-system kube-scheduler-minikube 1/1 Running 0 1h
kube-system kubernetes-dashboard-5498ccf677-gxd6p 1/1 Running 0 1h
kube-system nginx-ingress-controller-5984b97644-qr64t 1/1 Running 0 1h
kube-system storage-provisioner 1/1 Running 0 1h
Check availavle services ::
<There are basically 2 commands to check services and last one for checking based on specific namespace>
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 1h
nginx NodePort 10.96.115.213 <none> 80:30557/TCP 1h
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 1h
nginx NodePort 10.96.115.213 <none> 80:30557/TCP 1h
$ kubectl get services -n default
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 1h
nginx NodePort 10.96.115.213 <none> 80:30557/TCP 1h
( -n : namespace name , On first run there should not be any services except kubernates running)
Check availavle deployments ::
$ kubectl get deployments
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
nginx 1 1 1 1 1h
(For the first run there will be no deployments)
Delete deployment or service ::
$ kubectl delete deployment,service <deployment_name/service_name>
Check availave ingress::
$ kubectl get ingress
No resources found.
$ kubectl get ing
No resources found.
(ing/ingress does mean the same)
Delete ingress::
$ kubectl delete ingress <ingress_name>

Back in flask Deployment of my custom container

Now we have to deploy the flask_app in Kubernetes:

for that, I need two yml file

Deployment.yml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: flask
spec:
replicas: 1
template:
metadata:
labels:
app: flask
spec:
containers:
— name: flask
image: arani007/my-flask-image
ports:
— containerPort: 80
command: [“python”,”/flask-app/flask_docker.py”]
imagePullPolicy: Always