Manage certificates automatically for IBM Cloud Private with Cert-Manager and Istio Citadel

A certificate, also known as a digital certificate or identity certificate, is an electronic document — it is mainly used to identify the owner of the certificate. For example, for TLS, the client identifies the server with the server’s certificate; for mutual TLS, both client and server need to identify each other with their certificates. In a traditional way, the certificate management (requesting, issuing, renewing, and revoking) is either manual or coupled with a service. We want to find a way that can manage the certificates automatically and decouple the certificate management works with service — both Cert-Manager and Istio Citadel can manage the certificate automatically for IBM Cloud Private.

Cert-Manager

Cert-Manager is a Kubernetes add-on. It can issue, renew and revoke a certificate from various issuing sources automatically. The issuing sources include:

• ACME Issuer: Cert-Manager obtains certificates from an ACME server (e.g., Let’s Encrypt) using ACME protocol. The ACME protocol supports two challenge mechanisms (DNS-01 or HTTP-01), which are used to validate a public domain.
• CA Issuer: Cert-Manager issues certificates using a signing key pair stored in a Kubernetes Secret resource.
• Vault Issuer: Cert-Manager issues certificates using Hashicorp Vault.
• Self-signed: Cert-Manager issues self signed certificates.

Cert-Manager vs. Istio Citadel

Citadel is a component of Istio — it automatically manages certificates for Istio ingress proxy, egress proxy, and envoy proxy. For example, in Kubernates, it watches the Kubernetes apiserver, creates a SPIFFE certificate and key pair for each of the existing and new service accounts and stores them as Kubernetes secrets. It also watches the lifetime of each certificate and automatically rotates the certificates by rewriting the Kubernetes secrets. When a pod is created, Kubernetes mounts the certificate and key pair to the pod according to its service account via the Kubernetes secret volume.

Compared with Cert-Manager, the certificates that are managed by Istio Citadel are signed by Citadel itself and these certificates are mainly used by the service mesh. Cert-Manager can obtain certificates from a 3rd part certification authority (e.g. CA Let’s Encrypt); these certificates are trusted by many root programs like Microsoft, Google, Apple, Mozilla, etc. Cert-Manager can issue these certificates for public services and these services are available publicly via the ingress controller’s external IP address.

Issue a certificate to Istio IngressGateway

By default, IBM Cloud Private has a ClusterIssuer icp-ca-issuer that holds an ICP self-signed certificate and key pair; the certificate and key pair are stored as secret cluster-ca-cert under the kube-system namespace. In this case, we will use icp-ca-issuer to issue the certificate to one Istio IngressGateway.

Firstly, you need to deploy the httpbin service to Istio, then define a Certificate httpbin-example-com. It will issue a new certificate that is signed by IBM Cloud Private’s self-signed certificate, the certificate is stored as secret istio-ingressgateway-certs under the istio-system namespace, and the secret will be mounted and available to the Istio gateway automatically. See the Certificate section below:

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: httpbin-example-com
  namespace: istio-system
spec:
  # The secret must be called istio-ingressgateway-certs 
  # in the istio-system namespace, or it will not be 
  # mounted and available to the Istio gateway.
  secretName: istio-ingressgateway-certs
  issuerRef:
    name: icp-ca-issuer
    kind: ClusterIssuer
  # Certificate SANs (Subject Alternative Names)
  commonName: httpbin.example.com
  dnsNames:
  - httpbin.example.com

Then we define a Gateway with a server section for port 443:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin-gateway
spec:
  selector:
    istio: ingressgateway # use istio default ingress gateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "httpbin.example.com"
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      # The location of the certificate and the private
      # key must be /etc/istio/ingressgateway-certs,
      # or the gateway will fail to load them.
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key
    hosts:
    - "httpbin.example.com"

Finally, we configure routes for traffic entering via the Gateway:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
spec:
  hosts:
  - "httpbin.example.com"
  gateways:
  - httpbin-gateway
  http:
  - match:
    - uri:
        prefix: /status
    - uri:
        prefix: /delay
    route:
    - destination:
        port:
          number: 8000
        host: httpbin

You can use curl to test your HTTPS service. For example:

CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.clusterIP}')
curl -v -HHost:httpbin.example.com \
--resolve httpbin.example.com:443:$CLUSTER_IP \
--insecure https://httpbin.example.com:433/status/418

Conclusion and what’s next

In IBM Cloud Private, you can use ClusterIssuer icp-ca-issuer to issue a certificate to Istio IngressGateway via Cert-Manager. Or you can also use Cert-Manager to obtain certificate from a 3rd part CA and issue the certificate to a public ingress.

• To better understand how IBM Cloud Private integrates with your environment, see the Private cloud reference architecture.
• Want to try IBM Cloud Private locally on your computer? Download IBM Cloud Private Community Edition. It’s free!
• Want guidance from IBM Cloud Garage experts? Check out the IBM Cloud Private service offering.