Creating a kubernetes cluster from scratch in just few minutes

By Arbaaz Khan

Deploying a kubernetes cluster used to be a tedious job when we started working on it. We used to manually set it up which took a lot of time. The chances of messing up the set-up was high as there was always a chance of skipping steps or firing up wrong commands.

However as the time passed there came a number of tools those are having the capability to get K8S clusters up and running with very little manual intervention. Tools like KOPS ( Kubernetes Operations [ https://github.com/kubernetes/kops]), KUBEADM ([ https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/]), KUBESPRAY, MINIKUBE, BOOTKUBE etc. are few among them.

Ok! Enough of talking… Let’s get our hands dirty by building a K8S cluster from scratch. You may refer to my github repository where I have explained the steps to set up a K8S cluster [ https://github.com/arbaaz-khan/Kubernetes-Cluster-Creation]. If you wish not to waste even a split second and jump right into it then follow along with me.

The entire process mentioned here can be automated so that the chances of any error can be reduced.

WHAT EXACTLY ARE WE GOING TO DO ?

We are going to deploy a Kubernetes Cluster with a master and two Linux worker nodes. For this we need the following:

1. Cloud platform: AWS(an AWS account is required)
2. Tool to create the cluster: KOPs
3. Tool to access the cluster: Kubectl
4. Place to store the cluster state(KOPs needs it): S3 bucket

HOW TO SETUP AWS ACCOUNT ? (skip it if you already have set it up)

Here we are going to set up AWS account and configure it.

1. Go to AWS website and create an AWS account. We may choose free Tier one or paid one, choice is yours. (AWS may charge you for using resources beyond what comes with free tier)
2. Set up IAM user. This is a must in order to make any changes to your AWS account like creating/updating resources. You may visit this web page [ https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html ] to setup an IAM user or follow along to create it.

2.1. Go to AWS management console.

2.2. Click on the IAM link under Security, Identity, & Compliance.

2.3. Click on Users under IAM Resources.

2.4. Click on Add User button.

2.5. Enter the User name.

2.6. Tick on Programmatic access on Access type.

2.7. Click on Next: Permission button.

2.8. Click Attach existing policies directly.

2.9. Add the following permissions.

• AmazonEC2FullAccess
• AmazonS3FullAccess
• AmazonVPCFullAccess
• AmazonRoute53FullAccess
• IAMFullAccess

2.10. Click on Next:Tags It’s optional(You may skip it).

2.11. Click on Next:Review.

2.12. Click on create Create user button.

Note down or download the Access ID and secret access key, both are required during AWS CLI configuration.

HOW TO CONFIGURE AWS CLI ? (skip it if you already have configured AWS CLI)

Here we are going to configure AWS CLI on our local machine.(The following instructions assume that you have a linux machine)

Run these following commands:

sudo apt-get update
sudo apt install awscli
aws configure
AWS Access Key ID: <Enter your access KEY>
AWS Secret Access Key: <Enter your Screte Access Key>
Default region name: <Enter the zone in which you want to create your resources>
Default output format: <Enter json/text, if left blank then json is assumed>

To check whether it’s configured correctly:

aws sts get-caller-identity 

It returns details about the IAM identity whose credentials are used to call the API, confirming that our AWS CLI has been configured correctly.

WHERE WILL KOPs STORE THE CLUSTER STATE ?

Here we will be creating a S3 bucket that KOPs will use to store the cluster state.

1. Go to AWS Management Console
2. Click on the S3 link under Storage.
3. Click on Create bucket button.
4. Enter the bucket name. For example, kops-state-store,cluster-state etc.
5. Ensure that the zone is the same in which we are going to create our cluster.
6. Click on Create .
7. Great! We have created the S3 bucket.

WAIT! WHERE ARE THE TOOLS FOR K8S ??

Here we are going to download the tools required to create the cluster.

1. Install Kubectl binary.

sudo apt-get update && sudo apt-get install -y apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo “deb https://apt.kubernetes.io/ kubernetes-xenial main” | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl

Kubectl is a command line tool for kubernetes that can be used to talk with the kubernetes API server. It requires to be configured correctly with the K8S API server. (Don’t worry KOPs will take care of the configuration).

Check whether kubectl has been installed correctly:

kubectl -version

2. Install KOPs binary.

curl -LO https://github.com/kubernetes/kops/releases/download/$(curl -s https://api.github.com/repos/kubernetes/kops/releases/latest | grep tag_name | cut -d ‘“‘ -f 4)/kops-linux-amd64
chmod +x kops-linux-amd64
sudo mv kops-linux-amd64 /usr/local/bin/kops

Test whether KOPs has been installed correctly:

kops version

3. Awesome!! Finally we have installed KOPs .

WHEN ARE WE ACTUALLY CREATING THE CLUSTER ?

Now we are going to create K8s cluster by using KOPS. But before that there are a few things that we should know:

• We are going to create a cluster with 1 master and 2 workers.
• All the nodes are Linux based machines.
• We will be using Gossip DNS (This removes all those requirements of setting up private/public hosted zones and setting up the NS records).
• We don’t have to do much for using Gossip DNS, just by naming our cluster with a name ending with .k8s.local we do all the necessary things.

1. Setup the required ENV variable.

export KOPS_STATE_STORE=s3://<s3bucketname>

For Example:

export KOPS_STATE_STORE=s3://cluster-state-store

2. Generate the ssh key pair if not already present.

ssh-keygen 
Enter file in which to save the key (/home/username/.ssh/id_rsa): <ENTER>
Enter passphrase (empty for no passphrase): <ENTER>
Enter same passphrase again: <ENTER>

KOPs will use this key pair to ensure the we can ssh the nodes using this key. KOPs will copy the ssh public key to all the nodes to achieve this.

3. Generate the cluster configuration by replacing the specific values.

sudo kops create cluster \
-  cloud aws \
- ssh-public-key /home/<machine’s username>/.ssh/id_rsa.pub \
- master-size <size of master> \
- node-size <size of worker nodes> \
- networking <networking to be used> \
- state s3://<S3 bucket name> \
- zones <zone name> \
- node-count=<number of worker nodes> \
- name <name of the cluster>

For example:

sudo kops create cluster \
- cloud aws \
- ssh-public-key /home/ubuntu/.ssh/id_rsa.pub \
- master-size t2.medium \
- node-size t2.medium \
- networking calico \
- state s3://cluster-state-store \
- zones eu-west-2a \
- node-count=2 \
- name cluster.k8s.local

We may remove the networking flag completely, which will mean that cluster will be created with kubenet for networking.

4. Check whether configuration has been successful.

kops get cluster

The above command will show the name of the cluster.

5. Editing the cluster.

kops edit cluster <cluster-name> 

The above command will show the cluster configurations, if we wish, we can change the configurations. For this we need to edit the file and then save it. Here we are proceeding with the default configuration.

6. Deploying the cluster.

To deploy the cluster run the following command:

kops update cluster <cluster-name> -yes

The -yes flag ensures that the cluster is deployed.

7. Checking for successful cluster creation.

kops validate cluster

The above command will show all the nodes and a message that whether the validation has been successful or not. This will take 2–3 minutes(It may vary depending on the size of the cluster). Once the validation is successful, we can communicate with database using kubectl.

8. Communicating with the K8S API server.

kubectl get nodes -o wide

This should talk to the kubernetes API server and return the list of nodes present in the cluster that we have created.

9. SSH to the master node.

Now, we can ssh into the master node and create/delete resources from there.

ssh -i ~/.ssh/id_rsa ubuntu@<public ip of the master>

We can get the public ip of the master from the command in step 9, under the External IP section.

Hurray!!! We have the setup ready. Now feel free to play around with it.