Kubernetes: EKS, Calico and custom Admission Webhooks

5 min readOct 1, 2021

Problem

When using EKS with only Calico CNI the Kubernetes API server on the control plane (managed by AWS) cannot reach webhooks that use a service pointing to pods on Calico pod network.
The visible error is a timeout when you perform an operation that triggers the webhook, but it may include:

First try, without “hostNetwork: true”. It could not resolve the service DNS name:

Error from server (InternalError): Internal error occurred: failed calling webhook "myservice.mynamespace.svc": Post "https://myservice.mynamespace.svc:443/mutate?timeout=30s": Address is not allowed

Trying with an Ingress (ALB DNS) pointing to the service:

Error from server (InternalError): Internal error occurred: failed calling webhook "myservice.mynamespace.svc": Post "https://myingress.myhost.internal/mutate?timeout=30s": dial tcp: lookup myingress.myhost.com on 127.0.0.1:53: no such host

Service type NodePort did not work either (as recommended in Kubernetes docs) because it could not reach the Endpoint IP addresses in Calico, it resolves the Service DNS name though:

Error from server (InternalError): Internal error occurred: failed calling webhook "myservice.mynamespace.svc": Post "https://myservice.mynamespace.svc:443/mutate?timeout=30s": Address is not allowed

The problem looks like this:

Related issues:

Context

Why Calico CNI?

We use Calico CNI to implement an overlay network for our pods. It is a virtual network that do not consume VPC IP addresses as AWS VPC CNI does.

Known limitations of using Calico on EKS

From Calico EKS configuration page (https://docs.projectcalico.org/getting-started/kubernetes/managed-public-cloud/eks#install-eks-with-calico-networking):

Note: Calico networking cannot currently be installed on the EKS control plane nodes. As a result the control plane nodes will not be able to initiate network connections to Calico pods. (This is a general limitation of EKS’s custom networking support, not specific to Calico.) As a workaround, trusted pods that require control plane nodes to connect to them, such as those implementing admission controller webhooks, can include hostNetwork:true in their pod spec. See the Kuberentes API pod spec definition for more information on this setting.

So there is a limitation to how EKS works with custom pod networks as Calico, but it works well with AWS VPC CNI.

How webhooks are used in Kubernetes?

Kubernetes is an extensible platform, so it is easy to plug stuff in. One of the things you can customize is the flow for new requests in the API by setting Admission controllers.

Admission controllers come into main flavors: ValidatingAdmissionWebhooks and MutatingAdmissionWebhooks.

What are webhooks?

A WebHook is an HTTP callback: an HTTP POST that occurs when something happens; a simple event-notification via HTTP POST. A web application implementing WebHooks will POST a message to a URL when certain things happen.“,
From https://kubernetes.io/docs/reference/access-authn-authz/webhook/

From the blog post (https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/):

…the respective action is obtained from a REST endpoint (webhook) of a service running inside the cluster. This approach decouples the admission controller logic from the Kubernetes API server, thus allowing users to implement custom logic to be executed whenever resources are created, updated, or deleted in a Kubernetes cluster.”Many tools that you can install on Kubernetes will implement webhooks to modify (Mutating Webhook) or validate (Validating Webhook) resources, usually custom resources.

Solutions

1. Set pod “hostNetwork: true”

A possible workaround is setting “hostNetwork: true” to the webhook pod so it exposes a port in every Node. This way the Kubernetes service pointing to the pod would have <nodeIP:hostPort> in the endpoints.

As the node IP is part of the VPC the EKS Master can reach it out successfully.
In my case that was not possible because other pods were already using the same hostPort I needed and unfortunately the container was not configurable to set another port. Note that you have containerPort and hostPort in the pod/spec/container/ports but containerPort must match hostPort. If HostNetwork is specified in the pod spec, hostPort must match containerPort.

2. Why not both? CNI Genie.

Another possible solution to run both AWS VPC CNI and Calico CNI is CNI Genie, so you can operate AWS VPC CNI alongside Calico vxlan.
I did not go this path as it would require changes to networking in the cluster and more tests. (https://github.com/aws/amazon-vpc-cni-k8s/issues/176#issuecomment-521046590).

Sounds like an interesting solution.

3. Ambassador pattern: Use a NGINX with hostNetwork to proxy traffic to webhook Calico pod

This solution worked for me without changes to current networking.
The idea is to have another pod running Nginx configured to stream requests to the webhook service/pod running on Calico network. This Nginx pod uses hostNetwork (see solution 1) but the difference is I can expose any hostPort I want and in my scenario I had the limitation of not changing the port the container listens.

This solution is similar to the Ambassador pattern (A container that proxy the network connection to the main container) but in this case it is a “sidecar pod” because hostNetwork is defined at pod spec level, not container level.

Example of Ambassador pod pattern

Source code:

GitHub - denstorti/kubernetes-webhook-example

Contribute to denstorti/kubernetes-webhook-example development by creating an account on GitHub.

github.com

Based on this example:

The important pattern of this solution is:

Nginx just streams to the webhook pod
In the AdmissionWebhooks instead of pointing to the webhook service, you point to the nginx proxy service (and nginx proxy will stream to the webhook service).

Nginx configuration example:

If you got to this point, thanks a lot :)

If you have any other ideas or enhancements please post a comment.