How I ran an inexpensive ‘sandbox’ multi-node Kubernetes cluster on AWS using EC2 Spot Instances for 1¢ per hour!
Overview
While preparing for my CKA & CKAD certification exams, I wanted to build a solution to create ephemeral, ultra low-cost ‘sandbox’ Kubernetes clusters that could be spun-up and dismantled quickly. I did not want to use a solution like Minikube locally since that would only let me create single-node clusters. I required access to a true multi-node cluster environment to practice topics like Kubernetes cluster upgrades & maintenance.
AWS Spot Instances
To keep costs down, I looked at AWS Spot Instances to build my ‘sandbox’ cluster. EC2 Spot Instances are unused EC2 instances which are sometimes available at up to 90% discount compared to On-Demand EC2 instances. The downside of Spot Instances is AWS can terminate Spot Instances with little to no warning. Since resiliency was not a primary driver, I was not very concerned if the Spot instances in my cluster got terminated suddenly; as long I had a way to spin up another one quickly without too much effort. Hence, I resorted to CloudFormation to create a re-usable script that I could run and provision an entire cluster within couple of minutes.
The CloudFormation script k8s-spot.yaml
The entire CloudFormation script is shared at https://github.com/dhiman-halder/k8s-aws-spot
https://raw.githubusercontent.com/dhiman-halder/k8s-aws-spot/master/k8s-spot.yaml
The CloudFormation Script provisions the following AWS resources in us-east-1:
- A VPC with 1 public subnet, an Internet Gateway, a Public Route and a Security Group
- 1 t3a.small EC2 Spot Instance for master node (2 vCPU, 2 GiB memory) running Ubuntu 16.04. While the current On-Demand pricing is $0.0188 per hour, the Spot price is just $0.0056 at the moment- which is close to 70% discount compared to On-Demand price!
- 1 t3a.micro EC2 Spot Instance for worker node (2 vCPU, 1 GiB memory) running Ubuntu 16.04. The current On-Demand price is $0.0094 per hour and the Spot price is $0.0028- again a 70% discount!
- Each EC2 Instance is paired with 1 8GB SSD EBS volume. The per hour cost for each volume is $0.0011
- IAM role for the Spot Fleet
The Cluster is bootstrapped using Kubeadm through the UserData script as part of the EC2 provisioning process. The shell script does the following:
On the Master node:
- Set the hostname to master
- Turn-off SWAP
- Install and run Docker as the Container Runtime Engine
- Install Kubeadm, Kubelet & Kubectl tools
- Start the Kubelet service
- Use Kubeadm to initialize the cluster on the master
- Setup KubeConfig on the master
- Install Calico as the pod network add-on
- Create a join.txt file on the user’s root with a Kubeadm command to let the worker node join the Kubernetes cluster
On the Worker node:
- Set the hostname to node01
- Turn-off SWAP
- Install and run Docker as the Container Runtime Engine
- Install Kubeadm, Kubelet & Kubectl tools
- Start the Kubelet service
The UserData scripts follows the Kubernetes official guide as documented here:
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
Run the CloudFormation Script
You can run the CloudFormation script either using the AWS CLI or the AWS Management console. You need to supply an EC2 KeyPair for SSH access into the EC2 Instances.
Manual Step
Once the EC2 instances are provisioned and running:
- SSH into the master node & copy the Kubeadm join command from within the join.txt file on the user’s root
- SSH into the worker node and execute the copied command to join the worker node to the cluster.
Testing
- From the master node, run kubectl get nodes -this should list the 2 nodes and they should be in Ready state!
- Run kubectl run nginx --image=nginx --restart=Never followed by kubectl get pods to list the newly created nginx pod
Clean-up!
Delete the created CloudFormation stack to tear down the cluster!
Conclusion
Ignoring any data transfer cost, the current cost of running the 2 EC2 Spot Instances along with their EBS volumes is just $0.0056+$0.0028+$0.0011+$0.0011=$0.0106 which is ~1¢ per hour. Essentially, for $1, the environment could be run for ~4 days! This is of-course assuming the average spot price during the period do not deviate substantially.
Thanks for Reading!
Found this article useful? Hit that clap button. Really like it? Hold the clap, give it two, or fifty!