A Tale of an Elastic Kubernetes Service Setup
Setting up a Kubernetes cluster using AWS and the challenges hit and how to get it working.
Kubernetes
Overall I won’t cover Kubernetes as there are plenty of detailed write ups on what it is and how it works. Search Google to find out more about it, if you aren’t familiar. What this will focus on, is how to get Kubernetes stood up in Amazon Web Services (AWS). I wrote this up when I was taking a class to help others that were struggling with the setup of AWS Elastic Kubernetes Services (EKS). The course and other documentation seemed to make it look easy but every attempted I made, there was a different issue. I had issues with security (can’t access the cluster), node group issues (not able to connect), kubectl issues (not authorized). It was a nightmare, but as with any great set of failures there was a lot of learning. Hopefully this will help someone else in the future. Also keep in mind I was learning while doing this so I’m sure there are areas that can be improved so feel free to let me know.
Command Line Overload
I really like Kubernetes and the command line interface (kubectl) is a nice tool for working with Kubernetes. Standing up Kubernetes cluster is an entirely different story and I’ve always had assistance in setting one up in the past. With my current work, I was under the impression that using AWS simplified this setup and I found that if you’re an expert on everything AWS, then maybe, but for a newbie to AWS, it is difficult.
First there are many command line tools in play: eksctl; aws; kubectl; Not sure why, but each seems to server a different purpose and all are needed to fully do the job. Then there are all the different things in AWS but we’ll start with the Command Line Interfaces (CLIs).
AWS CLI
The AWS Command Line Interface (aws cli) tool is designed to provide a way to work with AWS services on the command line. This is to allow scripting and automation in interacting with AWS. It is currently at version 2 which if you’re going to start out you should begin with this version. The install instructions are pretty easy. I used Ansible to download the package (https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip), unpack it somewhere then run the install script that was in the package. After finishing the install, you can run aws to act on your AWS account.
EKSCTL
eksctl is a command line tool to work with Amazon’s EKS and was developed by Weaveworks. It is used to create and manage an AWS EKS cluster. It touts being able to create a cluster in minutes which I found is mostly true. Luckily installing it was very simple. Again I used Ansible which downloaded the latest version, unpacked it in a directory (/usr/local/bin/eksctl) and change permissions to make it executable.
Kubectl
This is the Kubernetes command line tool built for managing Kubernetes. This means it can be used to deploy, remove, manage, debug and so on. However, it requires a functional Kubernetes environment to have something to interact with.
Luckily this was even easier to setup with Ansible. My Ansible role simply download the kubectl and placed it in the /usr/local/bin directory with execute permissions.
Starting out
I’m still not fully clear where to begin at this point. You can setup a Cluster (Kubernetes instance) via the management console or through the aws CLI. Since I’ve tried the console multiple times without any luck I decided to move over to the CLI.
With all my attempts I did figure out that there are a few necessities before creating the cluster. The basics are:
iam —AWS Account Management — It all starts with a new user and a set of roles.
vpc — Virtual Private Cloud — Create a network domain for all this to reside in.
eks — Elastic Kubernetes Service — Create the cluster and finally node groups.
kubectl — Kubernetes CLI — Finally lets deploy something to the cluster.
IAM
This is one of the parts that threw me the most but it seems that you have to create a dedicated user to own and manage the cluster. I originally attempted to do this using my root account because I was only playing around but I couldn’t get everything to work with kubectl after creating the cluster. I finally gave up and created a user solely for this purpose.
I am still not fully understanding why a user must be created to specifically manage the cluster. It makes sense that the same user must be used to create the cluster and perform the work on it but why it has to be a separate user is not well documented. Also with the use case of creating a cluster for a team, and then wanting to hand it over to someone else for maintenance would not allow a new user to be in place. Just seems like I’m missing something here.
It is now finally time to create a cluster user. I did this using the aws CLI through an Ansible playbook. This was to allow it to be automated in the future. The playbook first checks if the defined user exists and if not it will then create a user. This was done using the aws CLI and the iam subcommand. It is pretty straight forward with a get-user and a create-user call.
Here is a copy of the Ansible playbook (sorry I never finalized the Ansible Role setup to publish it to GitHub). The variables in this are:
- eksuser — The user name to be created.
- eksuser_group — The group to bind the eksuser to. Must be a valid group in IAM.
- eksuser_path — The path of the user. Typically this is ‘/’ or whatever else you decide but it must end with a ‘/’.
# User Creation work
- name: check if user exists
shell: "aws iam get-user --user-name {{ eksuser }}"
register: eks_user_exists_output
ignore_errors: yes - name: create user
shell: "aws iam create-user --path {{ eksuser_path }} --user-name {{ eksuser }}"
when: eks_user_exists_output.stderr.find('NoSuchEntity') != -1
register: eks_new_user_result- name: add user to group
shell: "aws iam add-user-to-group --group-name {{ eksuser_group }} --user-name {{ eksuser }}"
when: eks_user_exists_output.stderr.find('NoSuchEntity') != -1- name: generate access key
shell: "aws iam create-access-key --user-name {{ eksuser }}"
when: eks_user_exists_output.stderr.find('NoSuchEntity') != -1
register: eks_user_access_key_output
The next step, we will need 2 roles created. One to manage the cluster and the other to manage the node group. I found this on Juan Delgado’s blog post. There is a lot of good information in here on creating a cluster so its worth the read if you want more details.
Creating the roles was impossible with the aws CLI because you need to provide a trust policy in json format and it doesn’t provide a method to give an arn . I tried many things to make this work but it seems like this is a flaw in the design or intentional to prevent hacking and gaining elevated rights. Maybe this will be possible at some point but for now I created these manually through the AWS console.
To create the cluster role, log into the AWS console and perform the following:
- Head to IAM by using the the search box
- Select Roles from the IAM submenu
- Choose Create Role
- Make sure selected Type is set to AWS service
- Find EKS in the big list and click it
- Select EKS — Cluster in the list that appears
- Click Next: Permissions
- On this step, confirm that the policy listed is AmazonEKSClusterPolicy
- Click Next: Tags
- Click Next: Review
- Enter the role name and a good description
- Click Create Role
Next step is to create the second role to manage the Node Group. It is mostly the same but the role will be different. Since you just did this above, you should already be on the Roles page.
- Choose Create Role
- Make sure selected Type is set to AWS service
- Find EC2 in the big list and click it
- Select EC2 in the list that appears
- Click Next: Permissions
- On this step, use the filter to find each of the following named items and select them: AmazonEC2ContainerRegistryReadOnly; AmazonEKSWorkerNodePolicy; AmazonEKS_CNI_Policy
- Click Next: Tags
- Click Next: Review
- Enter the role name and a good description
- Click Create Role
This should complete the user and role creation needed for the EKS cluster. Now we’ll need to establish a network configuration using VPC.
VPC
The Virtual Private Cloud (VPC) is a concept that I was completely lost (probably because I’m not a network guy), until I went through this exercise. I started reading through a lot of the Amazon material on VPC to try and get a better understanding. I do have to say this article did help a fair amount. I would not say I fully understand it all, but since this was an area that was greatly glossed over in the cluster setup it was the part I felt I was missing the most.
The main thing to know about a VPC, is that it is there to create a IP address range for you. They call it a Classless Inter-Domain Routing (CIDR) block. This is just a fancy term for a range of IPs. It uses a form of the base IP / a decimal number. There is a Wikipedia article on it if you care. I really didn’t, but it is somewhat important.
Creating a VPC was really easy, since all you have to do is provide the CIDR. As best I can tell, what you choose doesn’t matter as long as it is a valid IP address. Creating this with the aws CLI was cake. Running with the subcommand ec2 create-vpc and providing the cidr-block as a parameter. I also decided to add the tenancy value as default for completeness but I think that is the default if not provided.
Again this is the Ansible playbook part used for this. The variable in this are:
- eksvpc_cidr_block — The CIDR block to create.
- eksvpc_tenancy — The tenancy to use. I set it to ‘default’.
# Setup VPC- name: find vpc
shell: "aws ec2 describe-vpcs --filters Name=cidr,Values={{ eksvpc_cidr_block }}"
register: eks_vpc_exists_output
ignore_errors: yes- set_fact:
vpcs: "{{ eks_vpc_exists_output.stdout | from_json | json_query('Vpcs') | length }}"- name: create vpc
shell: "aws ec2 create-vpc --cidr-block {{ eksvpc_cidr_block }} --instance-tenancy {{ eksvpc_tenancy }}"
register: eks_vpc_creation_output
It is finally time to work on creating the Kubernetes Cluster.
EKS
After having the VPC, Roles and User in place, we can now work on creating the cluster. In this, we’ll use the eksctl to create the cluster. Since my goal was to use Ansible to create this I found some really good examples on setting this up. Dale-c-anderson published a GitHub repository with examples and this file provided the basis of what I used.
Since this playbook had the eksctl setup I lifted the check, create, kube config and oidc portion to add to my setup. After running it, it ran for a while (17m 11.893s) and I finally had a working cluster that I could use kubectl with.
I ended up using this multiple times to recreate the cluster and in doing so, found that the eksctl actually ends up creating a lot for you. It use Cloud Formation. It really did simplify the whole setup. I’m still not sure why the course I’m taking didn’t start with this.
Kubectl finally
Now that there is a working Kubernetes cluster we can start to really use Kubernetes. kubectl was used to run some basic tests like getting services (kubectl get svc), applying a deployment (kubectl apply -f deployment.yml) and removing it (kubectl delete -f deployment.yml). All this seems to work straight out of the box after using eksctl. I have to give it to the team that put eksctl together because they really did simplify the process. I wish I started with it, because the console path wasted a lot of my time.
I used this setup to pass my course project. Hopefully these notes will help someone else who has to muddle through this.
Future Plan
Originally when I started this I planned to make it an Ansible playbook to allow this to be used later for potentially a cluster I want to use for a personal project. I ended up trying to do this using Ansible Galaxy which I also wanted to learn. After doing this I found both topics are a little more complex so I know my galaxy project isn’t close to ready. Hopefully as I learn more I’ll bring it back to life and publish it.
If you have any suggestion or comments on areas I may have gotten wrong feel free to reach out and provide some feedback.
I love receiving comments and suggestions and if you like this please remember to clap and follow me.