Manage AWS Global Accelerator from Kubernetes resources

Today, there are many ways to expose services on a Kubernetes cluster through Load Balancer on AWS. For example, the famous one is AWS Load Balancer Controller.

Welcome

However, what should we do when you want to create a Global Accelerator in front of the Load Balancer? I have developed an OSS to solve it.

GitHub - h3poteto/aws-global-accelerator-controller: A Kubernetes controller for Global…

How to use

This controller can manage AWS Global Accelerator and Route53 records from Service and Ingress resources. And it detects annotations from Service and Ingress to control AWS resources, so you only need to add a few annotations.

• When you add aws-global-accelerator-controller.h3poteto.dev/global-accelerator-managed: "yes"as an annotation, a Global Accelerator will be created and the related LoadBalancer will be added to an Endpoint Group.
• When you add aws-global-accelerator-controller.h3poteto.dev/route53-hostname: your.hostnameas an annotation, route53 records associated with the Global Accelerator will be created.

Use it with Service type Load Balancer

Please create a Service type: LoadBalancer, and specify the above annotations.

apiVersion: v1
kind: Service
metadata:
  annotations:
    aws-global-accelerator-controller.h3poteto.dev/global-accelerator-managed: "yes"
    aws-global-accelerator-controller.h3poteto.dev/route53-hostname: "foo.h3poteto-test.dev"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
  name: h3poteto-test
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 80
    - name: https
      port: 443
      protocol: TCP
      targetPort: 443
  selector:
    app: h3poteto
  sessionAffinity: None
  type: LoadBalancer

After you apply this resource,

• A Network Load Balancer will be created. You can confirm it in the Service.

status:
  loadBalancer:
  ingress:
    - hostname: your-lb-name.elb.ap-northeast-1.amazonaws.com

• A Global Accelerator will be created, and the Load Balancer will be added as an Endpoint Group
• foo.h3poteto-test.dev record will be created

Use it with AWS Load Balancer Controller

AWS Load Balancer Controller has a plan to support Global Accelerator.

[Feature Request] Global LoadBalancer support · Issue #1572 ·…

But the controller I built is unrelated to this issue.

When you use Ingress, the way is very similar.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: h3poteto-test
  namespace: default
  annotations:
    aws-global-accelerator-controller.h3poteto.dev/global-accelerator-managed: "yes"
    aws-global-accelerator-controller.h3poteto.dev/route53-hostname: "foo.h3poteto-test.dev,bar.h3poteto-test.dev"
    alb.ingress.kubernetes.io/scheme: internet-facing
spec:
  ingressClassName: alb
  rules:
  - http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: h3poteto-test
            port:
            number: 80

You can specify multiple hostnames in the route53-hostname annotation.

Use it with ingress-nginx

First, please add aws-global-accelerator-controller.h3poteto.dev/global-accelerator-managed annotation to the Service of ingress-nginx. If you are using helm, you can add it using the following command:

$ helm install ingress-nginx ingress-nginx/ingress-nginx \
--set controller.service.annotations."aws-global-accelerator-controller\.h3poteto.dev/global-accelerator-managed"="yes"

Next, please apply your ingress resource.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: h3poteto-test
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginx
    aws-global-accelerator-controller.h3poteto.dev/route53-hostname: "foo.h3poteto-test.dev,bar.h3poteto-test.dev"
spec:
  rules:
  - http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: h3poteto-test
            port:
              number: 80

How it works

This controller is monitoring Service and Ingress resources and checks annotations when it is changed. And it creates Global Accelerators if needed. However, this controller waits until the Load Balancer becomes active. After the Load Balancer becomes active, it will create a new Global Accelerator. If there is no related Load Balancer, it will not create any Global Accelerators even if you specify the annotation.

Furthermore, it creates Route53 records after Global Accelerator is created. Also at this time, if there is no Global Accelerator, it will not create any Route53 records even if you specify the annotation.

How to detect the created Accelerator

Of course, Kubernetes Service does not have a field to store created Global Accelerator name, so the controller can not save it to the Service resource.

The controller saves the Service or Ingress resource name in Global Accelerator’s tag. Based on the information, it is determined whether or not a Global Accelerator for that Service was created.

How to detect the created Route53 record

In route53, this controller adopted the same method as external-dns. That means it stores the Service or Ingress resource name in a TXT record.

Future work

I’ve created some issues like these.

• Binding existing Global Accelerator
  This is similar to the Target Group Binding of AWS Load Balancer Controller. I want to add a Load Balancer as an EndpointGroup to the existing Global Accelerator.
• Support endpoint weight
  If the above feature is implemented, I want to specify endpoint weight for each Service or Ingress.

If you have some requests, feel free to create issues or pull requests. I would be happy to review your PR.

Epilogue

I’m working at oVice as a part-time developer, and I developed this controller for oVice. Because oVice’s Kubernetes clusters are using ingress-nginx and we wanted to add Global Accelerator for our services.

However, oVice decided to stop using ingress-nginx and we will use ALB and TargetGroup Binding of AWS Load Balancer Controller. Our application needs to track client IP addresses, but Global Accelerator with NLB can not preserve client IP addresses. On the other hand, Global Accelerator with ALB can preserve it. So we decided to change the Load Balancer.

Generally, when using TargetGroup Binding, ALB and TargetGroup are managed by another method, for example, Terraform. In that case, Global Accelerator and Route53 records should also be managed by Terraform.

So, we create ALB, TargetGroup, Global Accelerator, and Route53 records with Terraform. As a result, unfortunately, we don’t need to use this controller in oVice……

That’s why this repository has become my hobby OSS. oVice will not support this, but I will continue to develop.

Thank you.