Install Ingress-Nginx and ExternalDNS with Pulumi on GKE Autopilot

5 min readJul 31, 2021

This is the second part of articles to use Pulumi as infrastructure as Code (IaC) to create and maintain the infrastructure and deploy applications on Kubernetes cluster on the Google Cloud Platform GKE Autopilot and pipelines to automated everything with GitLab Pipelines. You can check the first article here!

We will see how to install ingress-nginx, external-dns, CloudFlare to configure domains and create a Ingress for our application guestbo

Ingress

Kubernetes Ingress exposes HTTP/HTTPS, SSL/TLS from outside the cluster to services. We can use traffic routes, load balancers, access control, websockets, etc. All those things with one integration with service load balancer.

ExternalDNS

ExternalDNS allows you to control DNS records dynamically via Kubernetes resources in a DNS provider-agnostic way.

This means when you install a new service in the cluster, this will configure automatically the DNS for the new service, adding the new entry A in the DNS for this service.

We will use Cloudflare to control our DNS, Cloudflare provides much more than manager DNS, they provide free SSL certificates, ant-DDOS, CDN, and so on, all those features in the free tier.

Configuration for Ingress-Nginx

So first we will configure Ingress-Nginx with helm chart and Pulumi.

Our project structure will be like this (please check the first article to have context):

├── Pulumi.dev.yaml
├── Pulumi.yaml
├── gcp
│   └── gke
│       └── index.ts
├── index.ts
├── k8s
│   ├── external-dns
│   │   ├── index.ts
│   │   └── namespace.ts
│   ├── guestbook
│   │   ├── config.ts
│   │   ├── index.ts
│   │   ├── namespace.ts
│   │   └── redis.ts
│   └── ingress-nginx
│       ├── intex.ts
│       └── namespace.ts
├── package-lock.json
├── package.json
└── tsconfig.json

The only variable we will use for helm chart is the version of helm chart, for that we just need add this information on the Pulumi.dev.yaml or using the cli for that:

pulumi config -s dev set ingressNginxChartVersion 3.34.0

The files of folder k8s/ingress-nginx

Configuration for ExternalDNS

Configure CloudFlare

First, you need to create an account in CloudFlare for your domain and create the token on Cloudflare, it's pretty straightforward.
Goes to My Profile -> Api Tokens click to create token button select to Edit zone DNS template and that it!

Handle with secrets on Pulumi

Now we use another interesting feature on Pulumi, the Secrets, how we can handle that. In the docs of Pulumi you have the entire information: https://www.pulumi.com/docs/intro/concepts/secrets/

For our project we just create a new secret:

pulumi config -s dev set --secret externalDnsCloudFlareToken {YOUR_TOKEN_HERE}

P.S (In the future article we will see how to use Secrets Manager on GCP with Pulumi).

And let's edit our Pulumi.dev.yaml file to adding more variables for external-dns service (domainFilters and version), the final version of yaml is:

config:
  gcp:project: {CHANGE_FOR_YOUR_GCP_PROJECT_NAME}
  infrastructure:externalDns:
    domainFilters:
      - pulumi-gke.com // change for your domain
    version: 5.1.4
  infrastructure:externalDnsCloudFlareToken:
    secure: {SECRET_BY_PULUMI}
  infrastructure:gke-min-version: "1.19"
  infrastructure:guessbook:
    replicas: 1
  infrastructure:ingressNginxChartVersion: 3.34.0

The files of folder k8s/external-dns (to more details of helm chart values, you can check here: https://github.com/bitnami/charts/tree/master/bitnami/external-dns)

The last step is to change your file /index.ts to get references of those 2 new applications.

export * from './gcp/gke';
export * from './k8s/guessbook';
require('./k8s/ingress-nginx');
require('./k8s/external-dns');

And run the up!

pulumi up -s dev

Configure the Ingress for Guestbook application

Now we just need to configure the ingress to expose your guestbook application by the domain.

We will create a new entry in your Pulumi.dev.yaml file for the domain, that way each stack that represent each environment could have the different domains.

pulumi config -s dev set --path  guestbook.host guestbook.pulumi-gke.com

Change the k8s/guestbook/config.ts to identify this new config value host

import * as pulumi from '@pulumi/pulumi';

interface GuestBookConfiguration {
    replicas: number;
    host: string;
}

const config = new pulumi.Config();
const guestbookConfig: GuestBookConfiguration = config.requireObject('guestbook');

export { config, guestbookConfig };

Modify the service to remove the type, we don't need anymore the load-balance service, this will handle by ingress-nginx that contain the service load-balance.
When we try just to remove type or change the type of service for ClusterIP we got a error from Pulumi.

the Kubernetes API server reported that "guestbook/guestbook-p3bune9t" failed to fully initialize or become live: Service "guestbook-p3bune9t" is invalid: spec.ports[0].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'

To solve it, firs we just comment the service on the file k8s/guestbook/index.ts and run pulumi up -s dev

// const service = deployment.createService({
//     ports: [
//         {
//             port: 80,
//         },
//     ],
//     type: k8sx.types.ServiceType.LoadBalancer,
// });

// export const response = {
//     ip: service.endpoint
// };

Now we are able to recreate service without issues.

And create the new Ingress for the guestbook application, the final version of file k8s/guestbook/index.ts

And again the command pulumi up -s dev

And now finally we can check the logs on the external-dns

kubectl logs -n external-dns deployment/external-dns

You see something like that:

time="2021-07-31T17:45:53Z" level=info msg="Changing record." action=CREATE record=guestbook.pulumi-gke.com ttl=1 type=A zone=e2018573b2f2b81180a455b0a969784f
time="2021-07-31T17:45:53Z" level=info msg="Changing record." action=CREATE record=guestbook.pulumi-gke.com ttl=1 type=TXT zone=e2018573b2f2b81180a455b0a969784f

And you can check on the web console on CloudFlare the entries were created.

And access the domain to check guestbook application is working, the best thing we do not need configure SSL, CloudFlare does it for us!

Conclusion

Using Pulumi as infrastructure as code we can control better what the dependencies we need in our cluster, ingress-nginx and external-dns are a perfect combination to automated the process to adding entries on DNS.

You can check all files in our repository: https://gitlab.com/felipe.girotti/gke-pulumi-pipelines/