If Amazon offered a managed Kubernetes
Disclaimer: I know nothing more then the general public, and I am an Kubernetes user on AWS.
Walking back from lunch today, I was having a conversation with my coworker, Ethan Rogers, about what it would be like if AWS offered a managed Kubernetes service. This is part thought experiment, part wish list of what the possible integrations could be.
Managed control-plane
If AWS had a managed control plane that you could just “create a control plane” for, and attach nodes to, it would be phenomenal. Selection of single node and multi-az would take away the headache of managing the components and just let you use it. A bonus would also be moving all the certificate management of etcd and the apiserver off to AWS. While managing the Kubernetes control plane allows for a high degree of configurability, most often I only need one or two of the defaults changed on any given component (etcd, apiserver, controller-manager, scheduler). If AWS offered a “parameter group” similar to how RDS and Elasticache, I could use the defaults, but modify flags that I needed to without having to choose every flag.
Authentication & Authorization
The story on custom authentication in Kubernetes is that there are the existing auth-n mechanisms, namely X509, static tokens, basic auth, service account (JWT), OIDC, and webhook, and keystone (doc link), there is not much in the documentation about setting up your own provider. Digging into the kube-apiserver source, there is an interface for authenticating a user’s request.
While AWS could implement a custom authorizer, it would be really great to allow human users to authenticate via IAM or IAM cross-account roles, and use Kubernetes own RBAC system for the auth-z on top of it. I’d hate to have to manage user access to Kubernetes components via IAM. I’d love to tell developers to use IAM access keys and a cross account role in their ~/.kube/config .
Node bootstrapping
Kubernetes has a specific permission for creating a certificate signing request, so that new nodes can join the cluster only after approval. AWS could manage the TLS bootstrapping (and later approval) process by attaching an IAM role to a node that only allows for making the initial CSR. Ideally, creating a node could be as configurable as it is with the ECS agent, just install a Kubelet on your OS of choice, attach an IAM role that enables bootstrapping, and you’re good to go.
Networking
This is one of the larger pain points of operating a cluster on AWS. If you need network ingress firewall based on namespaces, you’re probably using Calico + Flannel (AKA Canal), or Weave. At Skuid we use Canal, and it works for our needs, but for a larger cluster it can become inefficient to wrap every packet. I’m not entirely sure what AWS’s solution here would be, but they have lots of options.
Logging
We currently pipe our logs off to Cloudwatch logs, and then to AWS managed Elasticsearch (read more in my other post), but again this is another area AWS has multiple good options.
Secrets
There is currently a feature roadmap in draft for secrets in Kubernetes, but it would be great if AWS could get their voice in the conversation.
EC2 Container Registry
Kubernetes already can pull ECR images, but my biggest pain with ECR is that it is a regionalized service. This is not so much a Kubernetes issue, but if you’re using Kubernetes in multiple regions, this becomes a major pain. For us it was enough of a pain we switched to use Quay.io and haven’t looked back.
3rd Party Controllers
This is an area there is already a lot of open source work being done, but AWS could really take these types of integration to the next level. CoreOS has an ALB Ingress controller, the External-DNS controller can manage Route53 entries based on annotations. Another one could be to provision ACM Certificates for ELB’s based on an annotation set.
There are surely other areas that AWS could make running a Kubernetes cluster more pleasant, but I think this is a good place to start.
