Kubernetes adventures — Creating a Kubernetes cluster

Kubernetes is one of the hottest topics in technology today and is becoming the standard when it comes to orchestrating the containers. My goal with this series is to document the necessary steps to create and administer a kubernetes cluster, and also record my studies for the Certified Kubernetes Administrator.

The first thing that we will need is, of course, a cluster. There are many ways to create a k8s cluster, you can install manually (as taught on Kubernetes the hard way), use a tool such as kops or kubeadm or use a managed solution such as EKS and GKE. For this series I used Kops and created the cluster on AWS, to achieve high availability I created the masters and worker nodes on 3 availability zones (AZs), with 6 subnets (3 private subnets and 3 public subnets, with each pair of private and public subnets in an AZ).

Kops is capable to create everything that you need, including VPCs and subnets but I prefer to create the basic infrastructure by myself, because of this I need to change some parameter before creating the cluster. So let’s begin our adventure.


To create the cluster we will need a vpc (as I mentioned before kops can create it if you want) and an s3 bucket, where kops will store its configuration. After creating the network infrastructure and the bucket you will need to install kops on your machine, to install on Linux you just need to use these commands:

curl -LO https://github.com/kubernetes/kops/releases/download/$(curl -s https://api.github.com/repos/kubernetes/kops/releases/latest | grep tag_name | cut -d ‘“‘ -f 4)/kops-linux-amd64
chmod +x kops-linux-amd64
sudo mv kops-linux-amd64 /usr/local/bin/kops

you also will need to install kubectl

After that you just need to execute:

kops create cluster — node-count <number of nodes> — vpc=<id of your vpc > — zones <the AZs for the worker nodes> \
— master-zones <AZs for the master nodes> — node-size <worker’s instance type>\
— master-size <master’s instance type> — networking <type of networking> <FQDN for the cluster>

On the network parameter you can choose between calico, flannel, and others, I prefer to use calico, more information about kops parameters can be found on the documentation

Now the files that kops will use to create our cluster are created but I want to make some tweaks to better fit in my architecture, for example, I want to choose which subnets the cluster will use, so I need to edit the configurations, I can easily do that with the command

kops edit cluster — name <cluster’s fqdn>

After typing the command you will be able to edit the file, an example can be found here

I recommend that you enable the etcd backup and enable TLS on ETCD:

etcdClusters:
- backups:
backupStore: s3://<bucket-name>/<cluster name>/backups/etcd/main
etcdMembers:
- instanceGroup: master-us-east-1a
name: a
- instanceGroup: master-us-east-1b
name: b
- instanceGroup: master-us-east-1c
name: c
name: main
enableEtcdTLS: true
version: 3.2.24
- etcdMembers:
- instanceGroup: master-us-east-1a
name: a
- instanceGroup: master-us-east-1b
name: b
- instanceGroup: master-us-east-1c
name: c
name: events
enableEtcdTLS: true
version: 3.2.24

I used a vpc that I had created earlier so I also need to change some parameters on the network configuration, to make kops use the subnets that I already created:

subnets:
- cidr: 172.31.160.0/19
name: us-east-1c
type: Private
zone: us-east-1c
egress: nat-xxxxx
id: subnet-xxxx

I want to use a nat gateway to enable communication between the private subnets and the internet, that’s the reason I put the egress parameter pointing at my nat gateway, the “id” parameter is also very important, this is the parameter that tells kops to use a subnet that already exists, without that kops will try to create a new subnet and the creation of the cluster will fail.

With everything configured the way we want, it’s time to start the cluster

kops update cluster --name xxxxx.k8s.local
I0210 19:00:16.062489 18902 apply_cluster.go:542] Gossip DNS: skipping DNS validation
W0210 19:00:16.088911 18902 firewall.go:250] Opening etcd port on masters for access from the nodes, for calico. This is unsafe in untrusted environments.
I0210 19:00:16.095090 18902 executor.go:103] Tasks: 0 done / 93 total; 34 can run
I0210 19:00:17.218664 18902 executor.go:103] Tasks: 34 done / 93 total; 31 can run
I0210 19:00:18.963261 18902 executor.go:103] Tasks: 65 done / 93 total; 20 can run
I0210 19:00:22.310295 18902 executor.go:103] Tasks: 85 done / 93 total; 5 can run
W0210 19:00:22.452896 18902 keypair.go:140] Task did not have an address: *awstasks.LoadBalancer {"Name":"api.xxxxx.k8s.local","Lifecycle":"Sync","LoadBalancerName":"api-xxxxx-k8s-local-sedars","DNSName":null,"HostedZoneId":null,"Subnets":[{"Name":"kubernetes-vpc-public.us-east-1c.xxxxx.k8s.local","ShortName":"kubernetes-vpc-public.us-east-1c","Lifecycle":"Sync","ID":"subnet-099337ed0af13c5da","VPC":{"Name":"xxxxx.k8s.local","Lifecycle":"Sync","ID":"vpc-0d7d1b3d8bfcac423","CIDR":"172.31.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true,"Tags":null},"AvailabilityZone":"us-east-1c","CIDR":"172.31.64.0/19","Shared":true,"Tags":{"SubnetType":"Utility","kubernetes.io/cluster/xxxxx.k8s.local":"shared","kubernetes.io/role/elb":"1"}},{"Name":"kubernetes-vpc-public.us-east-1b.xxxxx.k8s.local","ShortName":"kubernetes-vpc-public.us-east-1b","Lifecycle":"Sync","ID":"subnet-xxxxx","VPC":{"Name":"xxxxx.k8s.local","Lifecycle":"Sync","ID":"vpc-0d7d1b3d8bfcac423","CIDR":"172.31.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true,"Tags":null},"AvailabilityZone":"us-east-1b","CIDR":"172.31.32.0/19","Shared":true,"Tags":{"SubnetType":"Utility","kubernetes.io/cluster/xxxxx.k8s.local":"shared","kubernetes.io/role/elb":"1"}},{"Name":"kubernetes-vpc-public.us-east-1a.xxxxx.k8s.local","ShortName":"kubernetes-vpc-public.us-east-1a","Lifecycle":"Sync","ID":"subnet-xxxxx","VPC":{"Name":"xxxxx.k8s.local","Lifecycle":"Sync","ID":"vpc-0d7d1b3d8bfcac423","CIDR":"172.31.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true,"Tags":null},"AvailabilityZone":"us-east-1a","CIDR":"172.31.96.0/19","Shared":true,"Tags":{"SubnetType":"Utility","kubernetes.io/cluster/xxxxx.k8s.local":"shared","kubernetes.io/role/elb":"1"}}],"SecurityGroups":[{"Name":"api-elb.xxxxx.k8s.local","Lifecycle":"Sync","ID":"sg-0b467c8018765faa4","Description":"Security group for api ELB","VPC":{"Name":"xxxxx.k8s.local","Lifecycle":"Sync","ID":"vpc-0d7d1b3d8bfcac423","CIDR":"172.31.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":true,"Shared":true,"Tags":null},"RemoveExtraRules":["port=443"],"Shared":null,"Tags":{"KubernetesCluster":"xxxxx.k8s.local","Name":"api-elb.xxxxx.k8s.local","kubernetes.io/cluster/xxxxx.k8s.local":"owned"}}],"Listeners":{"443":{"InstancePort":443,"SSLCertificateID":""}},"Scheme":null,"HealthCheck":{"Target":"SSL:443","HealthyThreshold":2,"UnhealthyThreshold":2,"Interval":10,"Timeout":5},"AccessLog":null,"ConnectionDraining":null,"ConnectionSettings":{"IdleTimeout":300},"CrossZoneLoadBalancing":null,"SSLCertificateID":""}
I0210 19:00:23.045627 18902 executor.go:103] Tasks: 90 done / 93 total; 3 can run
I0210 19:00:23.749210 18902 executor.go:103] Tasks: 93 done / 93 total; 0 can run
Will create resources:
AutoscalingGroup/master-us-east-1a.masters.xxxxx.k8s.local
MinSize 1
MaxSize 1
Subnets [name:us-east-1a.xxxxx.k8s.local id:subnet-066bf8452726af037]
Tags {k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup: master-us-east-1a, k8s.io/role/master: 1, Name: master-us-east-1a.masters.xxxxx.k8s.local, KubernetesCluster: xxxxx.k8s.local}
Granularity 1Minute
Metrics [GroupDesiredCapacity, GroupInServiceInstances, GroupMaxSize, GroupMinSize, GroupPendingInstances, GroupStandbyInstances, GroupTerminatingInstances, GroupTotalInstances]
LaunchConfiguration name:master-us-east-1a.masters.xxxxx.k8s.local id:master-us-east-1a.masters.xxxxx.k8s.local-20190210185210
SuspendProcesses []
AutoscalingGroup/master-us-east-1b.masters.xxxxx.k8s.local
MinSize 1
MaxSize 1
Subnets [name:us-east-1b.xxxxx.k8s.local id:subnet-0615a4129f1a72b5e]
Tags {k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup: master-us-east-1b, k8s.io/role/master: 1, Name: master-us-east-1b.masters.xxxxx.k8s.local, KubernetesCluster: xxxxx.k8s.local}
Granularity 1Minute
Metrics [GroupDesiredCapacity, GroupInServiceInstances, GroupMaxSize, GroupMinSize, GroupPendingInstances, GroupStandbyInstances, GroupTerminatingInstances, GroupTotalInstances]
LaunchConfiguration name:master-us-east-1b.masters.xxxxx.k8s.local id:master-us-east-1b.masters.xxxxx.k8s.local-20190210185210
SuspendProcesses []
AutoscalingGroup/master-us-east-1c.masters.xxxxx.k8s.local
MinSize 1
MaxSize 1
Subnets [name:us-east-1c.xxxxx.k8s.local id:subnet-04409c178f2af6b1f]
Tags {Name: master-us-east-1c.masters.xxxxx.k8s.local, KubernetesCluster: xxxxx.k8s.local, k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup: master-us-east-1c, k8s.io/role/master: 1}
Granularity 1Minute
Metrics [GroupDesiredCapacity, GroupInServiceInstances, GroupMaxSize, GroupMinSize, GroupPendingInstances, GroupStandbyInstances, GroupTerminatingInstances, GroupTotalInstances]
LaunchConfiguration name:master-us-east-1c.masters.xxxxx.k8s.local id:master-us-east-1c.masters.xxxxx.k8s.local-20190210185210
SuspendProcesses []
AutoscalingGroup/nodes.xxxxx.k8s.local
MinSize 3
MaxSize 3
Subnets [name:us-east-1a.xxxxx.k8s.local id:subnet-066bf8452726af037, name:us-east-1b.xxxxx.k8s.local id:subnet-0615a4129f1a72b5e, name:us-east-1c.xxxxx.k8s.local id:subnet-04409c178f2af6b1f]
Tags {k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup: nodes, k8s.io/role/node: 1, Name: nodes.xxxxx.k8s.local, KubernetesCluster: xxxxx.k8s.local}
Granularity 1Minute
Metrics [GroupDesiredCapacity, GroupInServiceInstances, GroupMaxSize, GroupMinSize, GroupPendingInstances, GroupStandbyInstances, GroupTerminatingInstances, GroupTotalInstances]
LaunchConfiguration name:nodes.xxxxx.k8s.local id:nodes.xxxxx.k8s.local-20190210185210
SuspendProcesses []
Keypair/master
AlternateNames [100.64.0.1, 127.0.0.1, api.internal.xxxxx.k8s.local, api.xxxxx.k8s.local, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local]
Signer name:ca id:cn=kubernetes
Subject cn=kubernetes-master
Type server
Format v1alpha2
LoadBalancer/api.xxxxx.k8s.local
LoadBalancerName api-xxxxx-k8s-local-sedars
Subnets [name:kubernetes-vpc-public.us-east-1c.xxxxx.k8s.local id:subnet-099337ed0af13c5da, name:kubernetes-vpc-public.us-east-1b.xxxxx.k8s.local id:subnet-xxxxx, name:kubernetes-vpc-public.us-east-1a.xxxxx.k8s.local id:subnet-xxxxx]
SecurityGroups [name:api-elb.xxxxx.k8s.local id:sg-0b467c8018765faa4]
Listeners {443: {"InstancePort":443,"SSLCertificateID":""}}
HealthCheck {"Target":"SSL:443","HealthyThreshold":2,"UnhealthyThreshold":2,"Interval":10,"Timeout":5}
ConnectionSettings {"IdleTimeout":300}
SSLCertificateID
LoadBalancerAttachment/api-master-us-east-1a
LoadBalancer name:api.xxxxx.k8s.local id:api.xxxxx.k8s.local
AutoscalingGroup name:master-us-east-1a.masters.xxxxx.k8s.local id:master-us-east-1a.masters.xxxxx.k8s.local
LoadBalancerAttachment/api-master-us-east-1b
LoadBalancer name:api.xxxxx.k8s.local id:api.xxxxx.k8s.local
AutoscalingGroup name:master-us-east-1b.masters.xxxxx.k8s.local id:master-us-east-1b.masters.xxxxx.k8s.local
LoadBalancerAttachment/api-master-us-east-1c
LoadBalancer name:api.xxxxx.k8s.local id:api.xxxxx.k8s.local
AutoscalingGroup name:master-us-east-1c.masters.xxxxx.k8s.local id:master-us-east-1c.masters.xxxxx.k8s.local
Will modify resources:
Subnet/kubernetes-vpc-public.us-east-1a.xxxxx.k8s.local
Tags {} -> {kubernetes.io/cluster/xxxxx.k8s.local: shared, SubnetType: Utility, kubernetes.io/role/elb: 1}
Subnet/kubernetes-vpc-public.us-east-1b.xxxxx.k8s.local
Tags {} -> {kubernetes.io/cluster/xxxxx.k8s.local: shared, SubnetType: Utility, kubernetes.io/role/elb: 1}
Subnet/kubernetes-vpc-public.us-east-1c.xxxxx.k8s.local
Tags {} -> {kubernetes.io/cluster/xxxxx.k8s.local: shared, SubnetType: Utility, kubernetes.io/role/elb: 1}
Subnet/us-east-1a.xxxxx.k8s.local
Tags {} -> {kubernetes.io/role/internal-elb: 1, kubernetes.io/cluster/xxxxx.k8s.local: shared, SubnetType: Private}
Subnet/us-east-1b.xxxxx.k8s.local
Tags {} -> {SubnetType: Private, kubernetes.io/role/internal-elb: 1, kubernetes.io/cluster/xxxxx.k8s.local: shared}
Subnet/us-east-1c.xxxxx.k8s.local
Tags {} -> {kubernetes.io/cluster/xxxxx.k8s.local: shared, SubnetType: Private, kubernetes.io/role/internal-elb: 1}
Must specify --yes to apply changes

if everything is configured as you want, execute

kops update cluster --yes --name <cluster's name>
I0210 19:00:32.573286   19219 apply_cluster.go:542] Gossip DNS: skipping DNS validation
W0210 19:00:32.601802 19219 firewall.go:250] Opening etcd port on masters for access from the nodes, for calico. This is unsafe in untrusted environments.
I0210 19:00:35.430228 19219 executor.go:103] Tasks: 0 done / 93 total; 34 can run
I0210 19:00:37.654805 19219 executor.go:103] Tasks: 34 done / 93 total; 31 can run
I0210 19:00:40.625748 19219 executor.go:103] Tasks: 65 done / 93 total; 20 can run
I0210 19:00:49.353037 19219 executor.go:103] Tasks: 85 done / 93 total; 5 can run
I0210 19:00:51.668019 19219 vfs_castore.go:736] Issuing new certificate: "master"
I0210 19:00:52.880639 19219 executor.go:103] Tasks: 90 done / 93 total; 3 can run
I0210 19:00:54.201305 19219 executor.go:103] Tasks: 93 done / 93 total; 0 can run
I0210 19:00:55.226297 19219 update_cluster.go:290] Exporting kubecfg for cluster
kops has set your kubectl context to lakatos.k8s.local
Cluster is starting.  It should be ready in a few minutes.
Suggestions:
* validate cluster: kops validate cluster
* list nodes: kubectl get nodes --show-labels
* ssh to the master: ssh -i ~/.ssh/id_rsa admin@api.lakatos.k8s.local
* the admin user is specific to Debian. If not using Debian please use the appropriate user based on your OS.
* read about installing addons at: https://github.com/kubernetes/kops/blob/master/docs/addons.md.

You can use the command

kops validate cluster

to check if the cluster is healthy, you should see something like that

kops validate cluster
Using cluster from kubectl context: xxxx.k8s.local
Validating cluster xxxx.k8s.local
INSTANCE GROUPS
NAME ROLE MACHINETYPE MIN MAX SUBNETS
master-us-east-1a Master t2.small 1 1 us-east-1a
master-us-east-1b Master t2.small 1 1 us-east-1b
master-us-east-1c Master t2.small 1 1 us-east-1c
nodes Node t2.small 3 3 us-east-1a,us-east-1b,us-east-1c
NODE STATUS
NAME ROLE READY
ip-172-31-122-68.ec2.internal node True
ip-172-31-123-160.ec2.internal master True
ip-172-31-139-192.ec2.internal master True
ip-172-31-143-51.ec2.internal node True
ip-172-31-168-159.ec2.internal node True
ip-172-31-188-133.ec2.internal master True
Your cluster xxxx.k8s.local is ready

Now you have a fully functional kubernetes cluster with HA, ready to serve production traffic.


I hope that you liked this really long post and I wait to see you again in the next adventure. Please leave your comments.