Kube-up: I know what you did on AWS!
Kubernetes is notoriously known for its troublesome configuration steps, and to ease that pain it is shipped with “one ring”: kube-up.sh. After we set the environment variables properly (mentioned in environment variables section), and called the script, this is what we come up with:
Unfortunately this magic handles a bit more than just providing some API. At this point, instead of explaining these created endpoints, we are going under the hood of the kube.sh execution steps. This is what it does specifically:
1- Upload installation files to S3
2- Create IAM roles
3- Create a key pair and publish to AWS
4- Create VPC
5- Create Subnet
6- Create Internet Gateway
7- Create Routing Table
8- Create Security Groups
9- Create and attach persistent volume to master
10- Create master instance
11- Create node instances
Upload Files to S3
Two different tarballs exist under $KUBE_ROOT/server directory. As the initial step, these are uploaded to an AWS S3 bucket with name kubernetes-staging-[MD5], or the name can be customized with the $AWS_S3_BUCKET env var.
- kubernetes-salt.tar.gz: Comprised of saltstack configuration files.
- kubernetes-server-linux-amd64.tar.gz: Includes binaries that will be installed to Kubernetes cluster instances. You can find the details of these binaries here.
Create IAM Roles
AWS Identity and Access Management (IAM) enables you to securely control access to AWS services and resources for your users.
At this step, kubernetes-master and kubernetes-minion IAM roles are created to be used consecutively by the master and node users in the cluster.
kubernetes-master role grants full access to its users for elastic load balancing, EC2 instances and required S3 buckets.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["ec2:*"],
"Resource": ["*"]
},
{
"Effect": "Allow",
"Action": ["elasticloadbalancing:*"],
"Resource": ["*"]
},
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::kubernetes-*"
]
}
]
}kubernetes-minion role grants attach/detach volumes permission and full access for S3 required resources.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::kubernetes-*"
]
},
{
"Effect": "Allow",
"Action": "ec2:Describe*",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "ec2:AttachVolume",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "ec2:DetachVolume",
"Resource": "*"
}
]
}Create a key pair and publish to AWS
For accessing Kubernetes cluster via SSH, we need to generate a public/private key pair, and the public key needs to be defined in AWS.
By default kube-up script looks for the existence of ~/.ssh/kube_aws_rsa key pair. If it does not exist, a fresh key is created by ssh-keygen, and the public key is published to AWS as kubernetes-[key finger print]. If you want to utilize one of your existing key pairs, you can export its path to the $AWS_SSH_KEY env var.
Create VPC
Amazon Virtual Private Cloud (Amazon VPC) lets you provision a logically isolated section of the Amazon Web Services (AWS) Cloud where you can launch AWS resources in a virtual network that you define.
Kube-up creates a VPC named kubernetes-vpc with a /16 bit CIDR IP block. If you already have a VPC on AWS with the same name, the scripts fetch its id, and uses it for the further steps.
By default VPC CIDR block is 172.20.0.0/16, and, as of the writing of this article, it is not configurable (unless you change the hardcoded value). Beside this, its DNS resolution and hostname options are enabled, and tags are set as ‘Name=kubernetes-vps, KubernetesCluster=kubernetes’. These tags are later used for filtering the result set while executing the script.
Note: Even if it is not mentioned in upcoming sections, all the network elements of AWS are described with KubernetesCluster=kubernetes tag.
Create Subnet
A subnet is a range of IP addresses in your VPC. You can launch AWS resources into a subnet that you select.
The script creates a subnet with CIDR block 172.20.0.0/24 within the given available zone in the $KUBE_AWS_ZONE env var (by default it is us-west-2a).
Note: Within the script, it is possible to use an already-created VPC and subnet by utilizing consecutively $VPC_ID and $SUBNET_ID env vars. On the other hand since the IP block assignment is hardcoded, this is not a fully supported feature currently.
Create Internet Gateway
An Internet gateway is a horizontally scaled, redundant, and highly available VPC component that allows communication between instances in your VPC and the Internet.
At this step, an Internet Gateway is created and is attached to the existing VPC for providing Internet access.
Create a Route Table
A route table contains a set of rules, called routes, that are used to determine where network traffic is directed.
Script creates a route table for the VPC, and it is associated with the subnet created in the previous step. In addition, a new route is added to route table for Internet gateway for the address 0.0.0.0/0. By doing so, all external IP addresses are exposed within the VPC.
Create Security Groups
A security group acts as a virtual firewall to control the traffic for its associated instances.
In this step, necessary security groups are created for master, and minions with names respectively kubernetes-master-kubernetes and kubernetes-minion-kubernetes.
After security groups are created, necessary inbound traffic access permissions are added. For allowing network access between the instances within the cluster, all ports of all protocols are opened in kubernetes-master and kubernetes-minion security groups. In addition, port 22 is opened in both groups for allowing SSH connections. As an extra permission, port 443 is opened for master instance, for allowing access to API endpoints. You can see all allowed inbound traffic below:
Create and attach persistent volume to master
An Amazon EBS volume is a durable, block-level storage device that you can attach to a single EC2 instance.
The master instance needs persistent storage for keeping etcd and various other data. For storing data, depending on $MASTER_DISK_TYPE and $MASTER_DISK_SIZE, the EBS volume is created. By default, type is gp2 (general purpose ssd) and disk size is 20GB. Availability zone of the volume is same with existing subnet.
Creating Master Instance
Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud.
After we are done with all network configurations, now it is time to proceed with master EC2 instance creation.
Image IDs are hardcoded within the scripts and depending on selected OS distribution, related image id is selected. You can optionally choose between CoreOS, Debian Jessie/Wheezy, Ubuntu Trusty Tahr/Vivid Vervet operating systems by utilizing $KUBE_OS_DISTRIBUTION env var with the following values: coreos, jessie, wheezy, trusty, vivid. By default it is set as vivid.
Instance type is also configurable via the $MASTER_SIZE env var which is by default t2.micro. Beside instance type, all other fields like IAM role, security groups etc. depend on newly created configurations seen in previous steps. For bootstrapping the instance, a master-start.sh temporary script is created. We have created a copy of this file in this gist repo.
After the master instance is created, our already-created persistent disk volume is attached to master, and an IP range (10.246.0.0/24) is added to route table. This IP is hardcoded within the script.
Create Node Instances
Unlike the master instance, instead of directly creating node instances, we are going to need the assistance of two other services: Launch Configuration and Auto Scaling Group.
A launch configuration is a template for the EC2 instances launched into an Auto Scaling group.
We need to create a Launch Configuration with a required set of instance settings. While creating it, the only optional part is instance type, which is defined with $MINION_SIZE env var and set as t2.micro by default. Beside this, security group, IAM role, key name and image id options are assigned in the creation command. Like the master-start.sh script used in bootstrapping master, the minion-user-data script is used for instance provisioning. You can find a related gist here. Finally, a launch configuration is created with the name: kubernetes-minion-group.
Warning: I am not sure if this was only an issue of mine, but I won’t suggest t2.micro minion instances even for testing purposes. After I built my cluster, t2.micro nodes started dropping into not ready state. Probably, they were running out of memory, and this was breaking the connection between kubelet and master. I suggest using at least t2.small instances.
Auto Scaling helps you maintain application availability and allows you to scale your Amazon EC2 capacity up or down automatically according to conditions you define.
When we are done with launch configuration, we need to define an auto scaling group for scaling up and down instance count in our cluster. By default min and max counts are identical, which is 2 and set by the $MINION_SIZE env var. All instances are created within the same availability zone with our existing subnet.
After this step is completed, all minion and master server availabilities are checked with SSH connections. If everything went well, we should see the “Kubernetes cluster is running” message on the terminal. When an error occurs in one of the steps, it is better to run kube-down.sh script for reverting the changes.
kube-down.sh
When you are done with your cluster, or things get mad, you can run the kube-down.sh script for reverting all created instances and network settings. Specifically, the following gets deleted:
1- Deleting auto-scaling group: kubernetes-minion-group
2- Deleting auto-scaling launch configuration: kubernetes-minion-group
3- Deleting instances in VPC: vpc-12312312
4- Deleting VPC: vpc-12312312
5- Deleting security groups
And these are the leftovers for us to remove:
1- Key pair
2- IAM roles: kubernetes-master, kubernetes-minion
3- Kubernetes master persistent volume: kubernetes-master-pd
Environment Variables
There are a bunch of environment variables, which was a bit confusing when we encountered them in config-default.sh. Here is a list of some opt-in variables:
KUBE_AWS_ZONE: Subnet Availability zone (default: us-west-2a)
MASTER_SIZE: instance type of master (default: t2.micro)
MINION_SIZE: instance type of minion (default: t2.micro)
NUM_MINIONS: minion count set by auto scaling group (default: 4)
AWS_S3_REGION: S3 bucket region (default: us-east-1)
AWS_SSH_KEY: key used for connecting to cluster instances (default: ~/.ssh/kube_aws_rsa)
MASTER_DISK_TYPE: disk type used by master persistent storage (default: gp2)
MASTER_DISK_SIZE: disk size used by master persistent storage (default: 20)
KUBE_ENABLE_CLUSTER_MONITORING: enable/disable monitoring for cluster (options: none, influxdb) (default: influxdb)
KUBE_ENABLE_NODE_LOGGING: enable/disable logging for node (options: true, false) (default: true)
KUBE_LOGGING_DESTINATION: set logging stack (options: elasticsearch, gcp) (default: elasticsearch)
KUBE_ENABLE_CLUSTER_LOGGING: enable/disable logging for cluster (options: true, false) (default: true)
KUBE_OS_DISTRIBUTION: distribution used for cluster instances. (options: coreos, jessie, trusty, vivid, wheezy) (default: vivid)
KUBE_MINION_IMAGE: image id to be used by minion instances. When not set, already defined image id is selected depending on os distribution.
Some Tips
— You can find basic authentication credentials, which is needed for accessing some of the endpoints, in ~/.kube/config under users->username, password
— For SSHing into machines, if you did not explicitly manipulate it, a key is generated as ~/.ssh/aws_kube_rsa
— t2.micro instances can be problematic for minions. Unless you disable monitoring and logging, it is better to choose at least a t2.small instance.
Conclusion
As we have tested so far, Kubernetes AWS integration is quite impressive. Beside these network related settings that are generated by kube-up script, we have also tested service creations and elastic load balancer creation corresponding to each service which again worked like a charm. At this point, we should say that we did not run any complex tests against sticky sessions. We also didn’t try to run a cluster in an already existing VPC etc. The only problem we encountered was trying to manually build a cluster, but after a while we gave up and went with kube-up script. Probably, we are going to post another article about our manual Kubernetes cluster installation.
As the one last thing, a shout-out to @jkautz and @cihangir for their effort on editing this article. We would really appreciate further comments, additions and correction requests.
Note: All excerpts are taken from AWS documentations.
