Open in app

Sign in

Write

Sign in

Amazon EKS, setup external DNS with OIDC provider and kube2iam

Marcin Cuber
The Startup
Published in
5 min readOct 24, 2019

Implementation details for external DNS using two methods; OIDC provider and kube2iam.

Overview

In this story I am going to concentrate on configuration of External DNS using OpenID Connect provider (IAM Role for service accounts) and kube2iam. ExternalDNS makes Kubernetes resources discoverable via public DNS servers. Like KubeDNS, it retrieves a list of resources (Services, Ingresses, etc.) from the Kubernetes API to determine a desired list of DNS records. Unlike KubeDNS, however, it’s not a DNS server itself, but merely configures other DNS providers accordingly such as AWS Route53. In a broader sense, ExternalDNS allows you to control DNS records dynamically via Kubernetes resources in a DNS provider-agnostic way.

As we are only going to concentrate on External DNS, ensure that you have implemented OIDC provider and kube2iam using the following stories:

The above two stories go through the steps on how to make both solutions work and provide necessary information to understand what they are meant to do.

At the end of this story you should understand how to configure external DNS and have two working examples of two different setups.

Assumptions

As we are not going to setup kube2iam or OIDC. I am going to assume that we have two IAM Roles.

  • kube2iam IAM role arn- arn:aws:iam::123456789012:role/kube2iam-external-dns
  • OIDC IAM role arn- arn:aws:iam::123456789012:role/oidc-external-dns

Note that both IAM roles above have exactly the permissions which are required by external DNS. And the permissions policy is as follows:

{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Effect": "Allow",
     "Action": [
       "route53:ChangeResourceRecordSets"
     ],
     "Resource": [
       "arn:aws:route53:::hostedzone/*"
     ]
   },
   {
     "Effect": "Allow",
     "Action": [
       "route53:ListHostedZones",
       "route53:ListResourceRecordSets"
     ],
     "Resource": [
       "*"
     ]
   }
 ]
}

Lastly, we need to assume that trusted relationship policy is correct on both IAM roles. They will be different because kube2iam doesn’t work the same way as OIDC provider.

kube2iam trusted relationship policy

IAM Role: arn:aws:iam::123456789012:role/kube2iam-external-dns

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/kubernetes-worker-role" # This is the IAM Role attached to your EKS worker nodes
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

OIDC trusted relationship policy

IAM Role: arn:aws:iam::123456789012:role/oidc-external-dns

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/server.example.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "server.example.com:sub": "system:serviceaccount:default:external-dns"
        }
      }
    }
  ]
}

In the above examples, the OIDC arns and other values are just example arns. If you want to know exactly why and how to setup your trusted relationship policies please refer to the following stories:

External DNS setup and implementation for Route53

Now that we have configured IAM roles which will be used by our external DNS deployment, we can put together Kubernetes yaml templates. Again, we are going to have two sections one for kube2iam and one for OIDC provider.

External DNS config using OIDC provider

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: external-dns
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: registry.opensource.zalan.do/teapot/external-dns:v0.5.17
        args:
        - --source=service
        - --source=ingress
        # - --domain-filter=example.text # will make ExternalDNS see only the hosted zones matching provided domain, omit to process all available hosted zones
        - --provider=aws
        - --policy=upsert-only # would prevent ExternalDNS from deleting any records, omit to enable full synchronization
        - --aws-zone-type=public # only look at public hosted zones (valid values are public, private or no value for both)
        - --registry=txt
        - --txt-owner-id=ceng-eks-test
      securityContext:
        fsGroup: 65534
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
  namespace: kube-system
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/oidc-external-dns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-dns
rules:
- apiGroups:
    - ""
  resources:
    - services
    - pods
    - nodes
  verbs:
    - get
    - watch
    - list
- apiGroups:
    - extensions
  resources:
    - ingresses
  verbs:
    - get
    - watch
    - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: kube-system

The above template will deploy fully working external DNS which is making use of OIDC provider configured to work with your EKS cluster. The important parts of the template are:

securityContext:
        fsGroup: 65534

and

annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/oidc-external-dns

Both blocks must be specified in order for IAM Role to be used by your service account.

External DNS config using Kube2iam

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: external-dns
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: external-dns
      annotations:
        iam.amazonaws.com/role: arn:aws:iam::123456789012:role/kube2iam-external-dns
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: registry.opensource.zalan.do/teapot/external-dns:v0.5.17
        args:
        - --source=service
        - --source=ingress
        # - --domain-filter=example.text # will make ExternalDNS see only the hosted zones matching provided domain, omit to process all available hosted zones
        - --provider=aws
        - --policy=upsert-only # would prevent ExternalDNS from deleting any records, omit to enable full synchronization
        - --aws-zone-type=public # only look at public hosted zones (valid values are public, private or no value for both)
        - --registry=txt
        - --txt-owner-id=ceng-eks-test
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-dns
rules:
- apiGroups:
    - ""
  resources:
    - services
    - pods
    - nodes
  verbs:
    - get
    - watch
    - list
- apiGroups:
    - extensions
  resources:
    - ingresses
  verbs:
    - get
    - watch
    - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: kube-system

Important block for kube2iam is the deployment’s annotation:

annotations:
        iam.amazonaws.com/role: arn:aws:iam::123456789012:role/kube2iam-external-dns

You need to ensure that this role arn is used with the correct annotation that kube2iam makes use of.

Conclusion

I have provided full implementations for external-dns that work with both kube2iam and OIDC provider. Using OIDC is the new way of using IAM Roles for pods and it is there to replace kube2iam and kiam.

I also committed external DNS templates to github repository which you can find under https://github.com/marcincuber/eks/tree/master/k8s_templates/external-dns

More about external DNS can be found in the official repository https://github.com/kubernetes-incubator/external-dns

In collaboration with NewRelic a technical blog on EKS and my work can be found -> https://blog.newrelic.com/product-news/news-uk-content-capabilities-amazon-eks-new-relic/

AWS
Kubernetes
Amazon Eks
External Dns
Oidc

--

--

Marcin Cuber
The Startup

Written by Marcin Cuber

4.3K Followers
Writer for

Principal Cloud Engineer, AWS Community Builder and Solutions Architect

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams