Nodeless Kubernetes on EKS

Kubernetes has become the de facto container orchestration framework. It still requires, however, capacity planning and quite some effort for provisioning, configuring, finetuning and maintaining the cluster. Nodeless or serverless systems, like AWS Fargate, Azure ACI, or Elotl Milpa, on the other hand, will let you focus on designing and building your applications instead of managing the underlying infrastructure: there is no infrastructure you need to provision upfront.

There is also a sweet spot between the two, where the power of Kubernetes is combined with a nodeless approach, offering great scalability and promising less infrastructure management overhead. This post focuses on how to set up a nodeless Kubernetes cluster on AWS via replacing the container runtime with a nodeless engine on EKS.

Milpa

Milpa is our cloud native “serverless” product that uses the same declarative manifest-based approach for deploying applications as Kubernetes. It is serverless in the sense that there are no servers or nodes to manage: Milpa schedules applications to run on cloud instances in a public cloud, such as AWS or Azure. Users can use their regular container images, and Milpa will take care of finding or starting a cloud instance that matches the resource requirements of the application.

The Kubernetes CRI

Milpa can also act as a container runtime for Kubernetes via the Container Runtime Interface (CRI). The CRI is the interface between Kubernetes and the engine that creates and stops containers to run applications (by default, Docker). Using Milpa as the runtime makes it possible to schedule unmodified Kubernetes workloads (pods, deployments, etc) in a cloud-native manner. When launching a pod in Kubernetes, Milpa will provision a right-sized cloud compute instance for your pod, and terminate the instance when Kubernetes stops the pod (under the hood, there is a shim service running between the Kubelet and Milpa, called Kiyot, which translates CRI requests and responses back and forth).

Setup

You will need:

• An AWS account that can create and manage EKS clusters.
• A license for Milpa. Get one for free here.
• Terraform for provisioning the EKS cluster. Get it here.
• Kubectl for interacting with Kubernetes. Install a compatible version from here. Kubectl supports one release of version skew, and here we use Kubernetes 1.10, so you will need kubectl 1.10 or 1.11. Version 1.11.0 is available here.
• The following command line tools installed: aws, jq (used when tearing down the cluster). They are available in most Linux distributions.

Note: creating and running an EKS cluster on AWS will cost you money.

Clone the repo with the example Terraform configuration:

$ git clone https://github.com/elotl/eks-milpa; cd eks-milpa/terraform

This directory contains an example Terraform configuration that creates an EKS cluster, provisions a worker node, installs Milpa on the node, and configures the node to use Milpa as its container runtime.

First, check “providers.tf”. This file configures how Terraform can access your AWS account. Set “region”, “access_key” and “secret_key”, or make sure the corresponding environment variables are set:

$ cat providers.tf
provider "aws" {
# You can also use environment variables instead of setting these variables:
# $ export AWS_ACCESS_KEY_ID="anaccesskey"
# $ export AWS_SECRET_ACCESS_KEY="asecretkey"
# $ export AWS_DEFAULT_REGION="us-east-1"
# See https://www.terraform.io/docs/providers/aws/ for more information.
#  region = "us-east-1"
#  access_key = ""
#  secret_key = ""
}

There is an “env-example.tfvars” file that contains all the tunable parameters. Copy it as “env.tfvars”, and fill in all the variables in it:

$ cp env-example.tfvars env.tfvars
$ vi env.tfvars
[...]
$ cat env.tfvars
# The name of the EKS cluster.
cluster-name = "eks-milpa-test"
# The access keys Milpa will use.
aws-access-key-id = ""
aws-secret-access-key = ""
# The SSH key name on EC2 for accessing the worker instance.
ssh-key-name = ""
# License information for Milpa.
license-key = ""
license-id = ""
license-username = ""
license-password = ""

Fill in all parameters. The AWS keys will be used by Milpa to create instances on AWS. You can obtain a free community edition license for Milpa here if you don’t have one already.

If this is the first time you use this Terraform configuration, initialize it:

$ terraform init
Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "aws" (2.2.0)...
The following providers do not have any version constraints in configuration,
so the latest version was installed.
To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.
* provider.aws: version = "~> 2.2"
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Create your cluster

Now you are ready to create the cluster. You can check first what resources Terraform will create:

$ terraform plan -var-file env.tfvars
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.aws_region.current: Refreshing state...
data.aws_availability_zones.available: Refreshing state...
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
[...]
Plan: 26 to add, 0 to change, 0 to destroy.
------------------------------------------------------------------------
Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

You can go ahead and create the cluster:

$ terraform apply -var-file env.tfvars
data.aws_availability_zones.available: Refreshing state...
data.aws_region.current: Refreshing state...
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
 + create
 <= read (data resources)
Terraform will perform the following actions:
[...]
Plan: 26 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
 Terraform will perform the actions described above.
 Only 'yes' will be accepted to approve.
Enter a value:

Type “yes”, and Terraform will start creating your cluster resources, which will take a few minutes. Once finished, you will see some output in green:

Apply complete! Resources: 26 added, 0 changed, 0 destroyed.
Outputs:
config_map_aws_auth =
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::689494258501:role/terraform-eks-demo-node
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
kubeconfig =
apiVersion: v1
clusters:
- cluster:
    server: https://1F1C16AF21741CCC70F4199D8CA8AEB5.sk1.us-east-1.eks.amazonaws.com
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRFNU1ETXlOVEl3TkRBMU1Gb1hEVEk1TURNeU1qSXdOREExTUZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTjBTCms4djV1emd1NkFEcnM3SjFKamI3UEdqNkFkcGV2Z01rT2lVc1dnSTJRVFJ1SDM1cnltTE1GWkFuNXZWZ2NqMFAKY2ZJUlFCd2JoeVJxRCt2N3B5SUpRQ3Q2cDNtWWdxY2ZEZ29XelRvTDJiQnJWZ3JINU1nd3BMRVNsUTNOeWR5Ywo1Q1JORzh6SmIzSXJUcTBOaDZoOGttdjdpZTNlbTg5RndQZ2tCMXE2TVN6UUNBTzhIb3VWRjBKaUZNV01uaHAyCk41VzBkcCtwZVduMmxJczlQSXFZOURiN2E0RDVxdzkvajRtbTlFdWtMejNkRzZMSmFlMjBFbXlPQ0Y5UlBMeDcKVS81N3F2RHJpQitwdENvak1hRFNTMG9xNXRZUGh2VlZ2bDIvTlpXMUZOQlJKbnc3bkdtRkZBeTFyTXE0RDhkWgpkOEE2dnpWUVJLMTV0M3p0ZDZzQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFMRVhSR1pPL2w3L1h3d0lWQ0c5S1FqR2dXeTgKUzIvZForeFhNVk9mcUNieVN3a2xLU3AzaWVWeEIzMzlxR3l6WExDbWMxaGxuWDl3OTVNS202QTd2d0k1aG9FZgpsdXBTRW9vVjdVT3FEZUF1UXhYN2JVa1prVXpQMVYzS2RnRkpmYVhBUW9jY3NVL1pqSGozQnlORWVDY2cwUDk5CitrQ2ZWSlRlQ2daVWY4bzlyQktVWFNlNnhoMllrM0p3bUlQemlvNXdiRHp3LzJtUHVGNnNyK2ZNVVIwUW9aT3oKVmJiR0NvL3laQjJ2ZFVZTElPS3dWK3cwMEdRY0RVZ2dadjZqUUF3VDZnR1JJSWc5d1JUTTlxTDIyQTMxb0llcAo3djlaM1oxZVJhcm9QQjBua2hvSlJZMUFwVnE1bW1WcERuZGdaaTF3c1Z0Q0Jwc01lakR4TmI4bjRZaz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: aws
  name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: aws-iam-authenticator
      args:
        - "token"
        - "-i"
        - "vilmos-eks-test"
worker-ips = [
    52.91.7.207
]

Save the items “config_map_aws_auth” as “config_map_aws_auth.yaml”, and “kubeconfig” as “kubeconfig”, and set the environment variable “KUBECONFIG=$(pwd)/kubeconfig” to have kubectl use it (you can also save “kubeconfig” in “~/.kube/config”, but you might already have existing Kubernetes clusters configured in that file; see here for more information on how to configure the clusters kubectl interacts with):

$ vi config_map_aws_auth.yaml # Save config_map_aws_auth.
$ vi kubeconfig # Save kubeconfig.
$ export KUBECONFIG="$(pwd)/kubeconfig"

You will need “aws-iam-authenticator” to be installed on your system to be able to authenticate against EKS. Installation instructions are here.

If you check the EKS web console, you will see that your new cluster is now ready:

EKS console

Next, configure your new EKS cluster to allow the worker node to join the cluster:

$ kubectl apply -f config_map_aws_auth.yaml # Saved from the output of the "terraform apply" run above.
configmap "aws-auth" created

Allow Milpa to talk to the API server:

$ kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous
clusterrolebinding.rbac.authorization.k8s.io "cluster-system-anonymous" created

Next, delete the “aws-node” daemonset. This starts a CNI plugin on workers, which is unnecessary with Milpa. Remember, pods run in cloud instances, and they get an IP address from their VPC.

$ kubectl -n kube-system get ds
NAME         DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
aws-node     0         0         0         0            0           <none>          1m
kube-proxy   0         0         0         0            0           <none>          1m
$ kubectl -n kube-system delete daemonset aws-node
daemonset.extensions "aws-node" deleted

Kube-proxy also runs in a cloud instance. Milpa will set up routes so that traffic from pods going to cluster VIPs will be routed via this instance, but we will need to enable masquerading in kube-proxy, so that network traffic coming back from services will also pass through the kube-proxy instance (otherwise applications in pods would see a non-cluster IP responding to their requests, sent to the service’s cluster VIP).

$ kubectl -n kube-system get daemonset -oyaml kube-proxy > /tmp/kube-proxy-ds.yaml
$ sed -i 's,- kube-proxy --resource-container="",- kube-proxy --masquerade-all --resource-container="",g' /tmp/kube-proxy-ds.yaml
$ kubectl apply -f /tmp/kube-proxy-ds.yaml
daemonset.extensions "kube-proxy" configured

Now kube-proxy and kube-dns should start up and your cluster should be operational (this might take a minute):

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                        READY     STATUS    RESTARTS   AGE
kube-system   kube-dns-6f455bb957-t42mt   3/3       Running   0          4m
kube-system   kube-proxy-db4rx            1/1       Running   0          4m

If you check out the EC2 console, you will see that the kube-system pods are running as EC2 instances:

EC2 console

You can poke around, deploy your applications and test that everything works. We also have some documentation on how to run example Kubernetes applications here. You usual Kubernetes workloads should work as is, without any modifications necessary. There are some differences when using Milpa as the container runtime, though, summarized here.

Scaling out a deployment

To see how your pods get scheduled to run in cloud instances, let’s create a deployment and scale it out.

Create a deployment:

$ kubectl run nginx --image=nginx
deployment.apps "nginx" created

This deployment will start one pod:

$ kubectl get pods
NAME                        READY     STATUS    RESTARTS   AGE
nginx-65899c769f-ml4jn      1/1       Running   0          1m

Scale out this deployment:

$ kubectl scale deployment nginx --replicas=5
deployment.extensions "nginx" scaled

You can try to scale it out to use as many replicas as you like, but keep in mind that our Community Edition license only allows 20 pods to be running at a time.

If you now go the EC2 console, you will see that Milpa created cloud instances for each replica:

Pods from the deployment run in cloud instances

Kubernetes labels also get propagated as instance tags, making it easier to find your pods from the EC2 console:

Kubernetes labels get propagated as instance tags

Teardown

Once you’ve finished, it’s easy to remove all resources in the cloud:

$ for ns in $(kubectl get namespaces | tail -n+2 | awk '{print $1}'); do
    kubectl delete --all pods --namespace=$ns
    kubectl delete --all deployments --namespace=$ns
    kubectl delete --all services --namespace=$ns
    kubectl delete --all daemonsets --namespace=$ns
done
pod "nginx-65899c769f-4lv8r" deleted
pod "nginx-65899c769f-4p2gd" deleted
pod "nginx-65899c769f-56jt6" deleted
[...]

Finally, remove all resources created via terraform:

$ terraform destroy -var-file env.tfvars