Switching from an NGINX Ingress Regional Load Balancer to a Global HTTPS Load Balancer on GCP

If you’re a GitLab user who has established a cloud environment on Google Cloud Platform (GCP) through GitLab’s cluster-management project, you may have encountered some missing pieces in your setup. In this guide, we’ll focus on addressing one of these gaps: the implementation of a Global HTTPS Load Balancer as an alternative to a Regional HTTPS Load Balancer. This modification is essential for using Cloud Armor within your Cloud Project.

Problem Description

GitLab’s cluster-management project offers a convenient way to swiftly set up various components for your Kubernetes cluster, such as cert-manager or ingress definitions. However, I encountered a challenge when attempting to integrate Google Cloud Armor into our Google Cloud Platform (GCP) environment. To use Cloud Armor, you must have a Global HTTP(s) Load Balancer in place. In the case of the GitLab cluster-management project, the automatically generated ingress serves as a load balancer. Google Cloud provides three types of load balancers:

• Internal: Designed for internal traffic within your network.
• Regional: For external traffic with backend endpoints located in a single region.
• Global: For external traffic with backend endpoints distributed across multiple regions.

The issue at hand is that, Google restricts the use of Cloud Armor exclusively to Global Load Balancers. The load balancer automatically created by GitLab is of the regional type, and there’s no straightforward way to switch it to a global configuration. Consequently, we find ourselves in a situation where we must take matters into our own hands to resolve this challenge effectively.

Step-by-Step Guide

The following sections are describing on which configuration we needed to change throughout our whole system.

Changing the Ingress Configuration of the cluster-management project

The first step in addressing this challenge is to modify the values.yaml file, which contains the ingress configuration. We need to make the following adjustments:

controller:
  stats:
    enabled: true
  podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "10254"
  service:
      type: ClusterIP
      annotations:
        cloud.google.com/neg: '{"exposed_ports": {"80":{"name": "ingress-nginx-80-neg"}}}'

With these settings, we override the ingress type to ClusterIP (it was previously set to LoadBalancer by default). Additionally, we annotate the ingress with a Network Endpoint Group (NEG), which will act as the backend for the new Global Load Balancer we need to create.

After making these changes, trigger the pipeline to ensure that the modifications are deployed within your cluster. As a result, the ingress should now be deployed with the type ClusterIP.

Adjustments on the Google Cloud Platform Side

Begin by connecting to your Google Cloud Platform (GCP) project and ensure that you are working within the correct context if you manage multiple contexts. You can verify your current context using the command kubectl config get-contexts, which displays the active context with a "star" marker.

Next, set up some essential global variables. These variables can be defined as follows:

ZONE=europe-west6-c
CLUSTER_NAME=my-gke-cluster
HEALTH_CHECK_NAME=nginx-ingress-controller-health-check
NETWORK_NAME=my-review-default-vpc
NETWORK_TAGS=$(gcloud compute instances describe \
    $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') \
    --zone=$ZONE --format="value(tags.items[0])")

It’s important to note that most, if not all, of the subsequent configurations can likely be performed via the GCP web interface. However, for the sake of this guide, we will mostly stick to the command-line approach.

Creating a Firewall Rule

In order to enable our new Global Load Balancer to access and communicate with our cluster, we need to create a firewall rule. This rule allows access for specific IP addresses, which are crucial for health checks and represent the IP addresses of the Google Front End (GFE) that connects to the backend. Without these IP addresses, the health checks for the Network Endpoint Group (NEG), which serves as the backend of the Load Balancer, will not succeed.

To create the necessary firewall rule, execute the following command:

gcloud compute firewall-rules create ${CLUSTER_NAME}-allow-tcp-loadbalancer \
    --allow tcp:80 \
    --source-ranges 130.211.0.0/22,35.191.0.0/16 \
    --target-tags $NETWORK_TAGS \
    --network $NETWORK_NAME

Creating the Health Check

To ensure the upcoming Backend Service monitors the health of our system effectively, we must create a health check. Execute the following command to create the health check:

gcloud compute backend-services create ${CLUSTER_NAME}-backend-service \
    --load-balancing-scheme=EXTERNAL \
    --protocol=HTTP \
    --port-name=http \
    --health-checks=${CLUSTER_NAME}-nginx-health-check \
    --global

This command establishes a health check that will assess the well-being of our system, providing vital insights to the Backend Service.

Associating the Ingress (NEG) with the Backend Service

The Backend Service of the forthcoming Load Balancer will now have the ingress, which has been transformed into a Network Endpoint Group (NEG), attached to it. Execute the following command to make this association:

gcloud compute backend-services add-backend ${CLUSTER_NAME}-backend-service \
  --network-endpoint-group=ingress-nginx-80-neg \
  --network-endpoint-group-zone=$ZONE \
  --balancing-mode=RATE \
  --capacity-scaler=1.0 \
  --max-rate-per-endpoint=100 \
  --global

This command links the Network Endpoint Group (NEG) to the Backend Service, ensuring that the Load Balancer can effectively route traffic to the ingress.

Creating the New Global HTTPS Load Balancer

To create the new load balancer, use the following command. Please exercise patience, as this process may take a few minutes to complete or become visible in the Google Cloud Platform (GCP) frontend.

gcloud compute url-maps create ${CLUSTER_NAME}-loadbalancer \
    --default-service ${CLUSTER_NAME}-backend-service

Generating a Certificate

Self-Managed Certificate

Since we’re using an HTTPS Load Balancer, it’s essential to have a certificate in place. I’ve created a self-managed certificate using OpenSSL and uploaded the certificate file content to Google Cloud Platform (GCP). Alternatively, you can achieve this through the GCP console:

gcloud compute ssl-certificates create $CERTIFICATE_NAME \
    --certificate=my-cert.pem \
    --private-key=my-cert-key.pem \
    --global

Please do not use this on production since the certificate will not be valid and the connection will not be secured. Refer to one of the upcoming blogpost on how to handle certification on the loadbalancer.

Google-Managed Certificates

If you want to have a google managed certificate, please refer to their documentation: https://cloud.google.com/certificate-manager/docs/deploy-google-managed-dns-auth

The documentation also provides guidance on obtaining certificates for wildcard domains. In our system, we utilized wildcard domains, and here are some additional details about this approach.

Adding of the acme-challenge via frontend

When adding the acme-challenge to the DNS-Zone via the frontend or Cloud Console, you'll come across a field labeled "canonical name." In this field, you should input the value found in the "data" field from the created DNS Authorization.

Exercise Patience

The creation of the DNS Authorization should occur relatively quickly. However, when it comes to generating the certificate itself, be prepared for it to take some time, potentially up to 2 hours. To monitor the progress, periodically refresh the status. You’ll want to ensure it transitions from “provisioning” to “Authorized”.

In my experience, it took approximately 5 minutes for the applied certificate mapping to the target proxy of the load balancer to become visible. Once this happened, I was able to confirm the presence of the new certificate in the system .

Modifying the Load Balancer in the GCP Console

The load balancer should now be visible in the Google Cloud Platform (GCP) console. To configure it, select the load balancer and click on the “Edit” button. Most of the settings should already be configured, with the exception of the frontend configuration. Perform the following configurations:

• Select “Add Frontend IP and Port.”
• Give it a name and select “HTTPS.”
• If you reserved a static IP address, use it in the “IP address” field; otherwise, use “Ephemeral.”
• Choose the previously created certificate (you can also use a Google-managed one).
• If you have a static IP address, you can enable HTTP to HTTPS redirect. This will create a new “load balancer” or “mapping” without a backend. It’s important to note that this is essentially a forwarding rule.
• Save the load balancer configuration.
• Verify that everything is healthy and operational.
• Congratulations, the load balancer is now up and running! 😃

Additional Information on How This Works

How Is the NEG Updated?

The Network Endpoint Group (NEG) and Load Balancer are designed to stay in sync with the deployment of the ingress. If any changes occur, whether it’s scaling up the ingress or other modifications, the NEG is notified and updates itself accordingly. This synchronization is achieved through the default configuration in the ingress deployment, which includes the --publish-service flag.

Firewall Rule Specific Network Tag

The firewall rule for the connection to the backend is specific to a particular network tag, which typically resembles a node label. For instance, it might appear as “gke-staging-456b4340-node.”

Importantly, this network tag is applied to every compute instance within the cluster. This setup ensures that the health checks remain effective, even as new nodes are added or existing ones undergo changes. It provides a robust and adaptable mechanism for maintaining connectivity and security between the load balancer and the backend.