10 Tips when Building New Kubernetes Cluster on GCP, Part 1
This article is more relevant by 2019, some of the tips are no longer valid today.Building on Kubernetes is not difficult, but going through wrong direction may sometimes cost you precious time. Some may even require you to scrap your existing cluster and start over.
1. Use private IP
Private IP is the easiest way to connect to Cloud SQL like Postgres or MySQL. However, it’s not enabled by default, and you can not enable it after creating your cluster. So, it’s the very first tip to make sure we do the below extra steps to enable the private IP when creating the cluster.
Enable Private IP involves ticking on a toggle VPC native. Follow the 2 steps on the below.
Your GKE application should now be able to connect to Cloud SQL using database url like the below
postgres://db_user:db_pass@10.122.1.1:5432/dbnameActually a lot more requirements are happening under the hood, see Private IP document. But you need to tick this option to get yourself started, and the rest should not create must trouble. Luckily this VPC-native will soon become GKE default and we no longer need to recreate the cluster just to connect to the database. (happened to me ~~~)
Note, if you’re literally creating new Kubernetes cluster, make sure you also read the next tip before clicking on the Create button.
2. Use managed certificate
If your service need to serve http, you probably want to have https certificate. (You should, as with most other web services, https is the default now.) However, setting up certbot in GKE requires multiple steps, and Google do provide a ready-to-use solution with no additional cost, why not let Google do the heavy lifting ?
Managed Certificate is currently in beta, but is stable enough to get the job done. You need to upgrade your kubernetes version to at least 1.12.6-gke.7. Unlike the VPC-native, the upgrade can be performed on existing cluster. But selecting the right version upfront will save you several minutes of waiting.
Create cert.yml similar to the below:
apiVersion: networking.gke.io/v1beta1
kind: ManagedCertificate
metadata:
name: my-cert-name
spec:
domains:
- www.example.twHowever, using managed certificate has the following prerequisites
- You need to own the domain name
- You need to have a static IP on GCP first
- You need to point your DNS to GCP static IP before provision the cert
- Your DNS is better with DNSSEC
The first two are documented, but I find either rule 3 or 4 also play the role. When I created my certificates, some of them just refused to be created no matter how long I waited (days). In the end I migrated my DNS to Google’s Cloud DNS, delete the failed to be created certificates and recreate them. The certificate was provisioned in something like an hour.
I was very hesitated to delete the failed certificate provisioning, for I have waited a very long time. But re-creating prove to be very effective. When I was on other DNS provider, pointing to GCP static IP is helpful, eventually. But using Google’s Cloud DNS is a lot faster.
I’m not sure, are they biased? Or more likely because of the lack of DNSSEC in my previous provider.
3. Add proper health check
Having setting up your deployments, services and ingress. You think you’re ready to launch. But out of sudden, your web service was rejecting connection with 502 server error.
Please describe your ingress: kubectl describe ingress/network. Looking for the line under Application > backends: (The describe results appear to be unordered, look carefully)
backends: {"k8s-be-00000--12345678":"HEALTHY","k8s-be-11111--12345678":"UNHEALTHY"}Note the UNHEALTHY in the above status ? Your service was not considered healthy by ingress. You should use port forward to test your service. But there are times, your service is actually serving, just it was considered unhealthy.
For example, your API service may not have responded to GET / request, which is the default place ingress will be checking for healthiness. This happens, after all the main purpose of API service is to respond to /api. You should alter your application.yml to include readiness probe and liveness probe.
livenessProbe:
httpGet:
path: /ping
port: 4000
successThreshold: 1
failureThreshold: 3
initialDelaySeconds: 15
periodSeconds: 60
readinessProbe:
httpGet:
path: /ping
port: 4000
successThreshold: 1
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 60You may have noticed, the health check doesn’t have to be GET /. So long as you provided a readiness probe, health check will perform accordingly.
PS. I’d suggest you try very hard to avoid the UNHEALTHY state, because it takes time to come back to HEALTHY.
4. Use port forward to diagnose
I believe this is a commonly known trick, but it’s so useful I think I should have mentioned it. We will have a whole lot of reasons wanting to diagnose our services:
- Use a hidden service (redis for example)
- Diagnose a specific pod
Port forward like the name suggest, will forward a local port to the specified remote port. It can work on service or pod, depending on your need.
kubectl port-forward pods my-pod-name-is-here 4001:4000The above bind local port 4001 to port 4000 of your pod my-pod-name-is-here. You may point your browser location to http://localhost:4001/ now to give it a try.
Directly logging on to your pod is also very useful, use the below if you have bash installed on your image:
kubectl exec -it my-pod-id bash5. Plan the namespace
K8s has a very nice feature, namespace. This allows you to segment your services very cleanly. Services with the same name do not collide with each other if they belong to different namespaces. This is useful if you have multiple projects, or different environments like staging and production.
However, namespace does not come without cost. Here, the cost literally means money.
Say you have the following typical structure:
deployment ←→ service ←→ ingressBy segmenting the elements into 2 namespaces, you thereafter have to pay for one ingress per namespace, because ingress can only connect to service on the same namespace. That will be, 2 ingress in total to be paid for. And ingress, in my opinion, is kind of expansive comparing with other resources on GCP. So, plan your namespace wisely.
I personally would suggest avoiding namespace on small or single project cluster. Only use namespace if you have multiple projects running.
Contact Me
Did you learn something new? If so please:
↓ clap 👏 button below️ so more people can see this.
