How To: Implement AWS KMS as Root CA using SkyScanner for Kubernetes

Published in

Israeli Tech Radar

6 min readDec 15, 2021

Use case

If you are looking to implement an external Certificate Authority for your Kubernetes cluster, and your Cloud Provider is AWS, you should consider using AWS Key Management Service.

In order to integrate AWS KMS to our Kubernetes Cluster, we will use SkyScanner KMS-Issuer. KMS issuer is a cert-manager Certificate Request controller that uses AWS KMS to sign the certificate request.

Last but not least, we will showcase a dome of how to integrate AWS KMS as Root CA for Istio using Istio-CSR.

versions

SkyScanner kms-issuer : v1.0.0
Cert-Manager : v1.5.* , v1.4.*
istio: 1.11.*
istio-csr: 0.3.0

What will be discussed:

What is AWS Key Management Service
Benefits of using AWS KMS
Applying AWS KMS and KMS-Controller
How To: Set KMS-Issuer as Root CA for Istio using Istio-CSR Integrated with KMS-Issuer

What is AWS Key Management Service

AWS Key Management Service (KMS) makes it easy for you to create and manage cryptographic keys and control their use across a wide range of AWS services and in your applications.

KMS works both with a symmetric key and an asymmetric key and supports asymmetric keys that represent a mathematically related RSA or elliptic curve (ECC) public and private key pair. You can download the public key for distribution and use outside of AWS. You can create asymmetric KMS keys for encryption and decryption, or signing and verification, but not both.

Benefits of using AWS KMS Asymmetric Key

https://aws.amazon.com/kms/features/#Overview

Centralized Key Management: AWS KMS provides you with centralized control over the lifecycle and permissions of your keys. You can create new keys whenever you wish, and you can control who can manage keys separately from who can use them.
AWS Service Integration: The service is integrated with other AWS services making it easy to encrypt data you store in these services and control access to the keys that decrypt it.
Secure: AWS KMS is designed so that no one, including AWS employees, can retrieve your plaintext keys from the service. The service uses hardware security modules (HSMs) that have been validated under FIPS 140–2, or are in the process of being validated, to protect the confidentiality and integrity of your keys. Your plaintext keys are never written to disk and only ever used in volatile memory of the HSMs for the time needed to perform your requested cryptographic operation.
Low Costs: $0.03 per 10,000 requests involving RSA 2048 keys. for more information regarding pricing: https://aws.amazon.com/kms/pricing/

Applying AWS KMS and KMS-Controller

When deploying KMS-Issuer v1.0.0 manually make sure to fix the known issues described later on.

STEP 1: Install Cert-Manager

When installing cert-manager, ensure the installCRDs is set to ‘true’.
Installing Cert-Manager is a prerequisite for using KMS-Issuer.

helm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--version v1.5.0 \
--set installCRDs=true

STEP 2: Fork Skyscanner/kms-issuer git repository

STEP 3: Install KMS-Issuer CRD’s

kubectl apply -k config/crd

STEP 4: Create AWS-KMS asymmetric key with RSA_2048

Copy the KMS ARN for the use of AWS Policy.

Note: KMS key can be created from the Kubernetes Cluster, as described: https://github.com/Skyscanner/kms-issuer.
In order to do that more permissions needs to be added to the Assume Role’s Policy.

STEP 5: Create AWS Policy and Role with Web Identity

The Role Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowUseOfTheKey",
            "Effect": "Allow",
            "Action": [
                "kms:DescribeKey",
                "kms:GetPublicKey",
                "kms:Sign",
                "kms:Verify"
            ],
            "Resource": "<KMS_KEY_ARN>"
        },
        {
            "Sid": "AllowAttachmentOfPersistentResources",
            "Effect": "Allow",
            "Action": [
                "kms:CreateGrant",
                "kms:ListGrants",
                "kms:RevokeGrant"
            ],
            "Resource": "<KMS_KEY_ARN>",
            "Condition": {
                "Bool": {
                    "kms:GrantIsForAWSResource": "true"
                }
            }
        }
    ]
}

STEP 6: Deploy KMS-Controller

Note: before running the deployment file make sure to fix the known issues for kms-issuer v1.0.0

Fix known issues for kms-issuer v1.0.0

Wrong image: the deployment.yaml stated that the image is skyscanner/kms-issuer:dev, but when deploying we got an error “the image was not found”.
The correct image can be found in Skyscanner/Kms-issuer repository.
Correct image:

ghcr.io/skyscanner/kms-issuer:sha-3ee48e5

2. Kubernetes Service Account RBAC: was missing permission to the resource ‘leases’. Therefore, we added permissions to this resource. The permission was added to the ‘Role’ — ‘kms-issuer-leader-election-role’:

- apiGroups:
  - "coordination.k8s.io"
  resources:
  - leases
  verbs:
  - "*"

Both fixes can be found in the:

deploy/kubernetes/kms-issuer.yaml

After fixing the known issue, you can run the deployment file

kubectl apply -f deploy/kubernetes/kms-issuer.yaml

STEP 7: Add Assume Role ARN to Kubernetes Service Account

apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
  namespace: kms-issuer-system
  annotations:
    eks.amazonaws.com/role-arn: <ARN_ROLE>

STEP 8: Deploy issuer

KMS-Issuer should run on the same namespace of your application.

apiVersion: cert-manager.skyscanner.net/v1alpha1
kind: KMSIssuer
metadata:
  name: kms-issuer
  namespace: <APP_NAMESPACE>
spec:
  keyId: alias/<example_kms_key> # same name as AWS kms key
  commonName: My Root CA # The common name for the root certificate
  duration: 87600h # 10 years

STEP 9: Test that the certificate was issued:

kubectl get kmsissuer kms-issuer -n <NAMESPACE> -o json | jq -r ".status.certificate" |  base64 --decode  | openssl x509 -noout -text

Applying with Terraform

Example for Terraform deployment is in the following git repository:

kms-issuer/chart at main · guysaarTikal/kms-issuer

KMS issuer is a cert-manager Certificate Request controller that uses AWS KMS to sign the certificate request. …

github.com

How To: Set KMS-Issuer as Root CA for Istio using Istio-CSR Integrated with KMS-Issuer

Deploying istio-csr

Set istio-csr to sign the certificate using the kms-issuer that we deployed earlier:

helm upgrade -i -n cert-manager cert-manager-istio-csr jetstack/cert-manager-istio-csr --set "app.certmanager.issuer.name=kms-issuer" --set "app.logLevel=5" --set "app.certmanager.issuer.kind=KMSIssuer" --set "app.certmanager.issuer.group=cert-manager.skyscanner.net"

istio-csr will create a secret named ‘istiod-tls’ that holds the tls.crt and tls.key. It will also create a configmap named ‘istio-ca-root-cert’ that holed the root-ca.pem.
We will map them later on when deploying istio.

Deploy Istio-Operator

For more info regarding Istio-Operator installation: https://istio.io/latest/docs/setup/install/operator/

istioctl operator init

Deploy Istiod integrated with Istio-CSR

Configure IstioOperator Deployment for istiod

Deploy Istio profile: minimal
Configure External CA address for workloads
Disable istiod as the CA Server
Provide TLS certs for istiod from cert-manager
map configmap ‘istio-ca-root-cert’ and secret ‘istiod-tls’.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istio-controlplane
  namespace: istio-system
spec:
  profile: "minimal"
  values:
    global:
      caAddress: cert-manager-istio-csr.cert-manager.svc:443
  components:
    pilot:
      k8s:
        env:
          # Disable istiod CA Sever functionality
        - name: ENABLE_CA_SERVER
          value: "false"
        overlays:
        - apiVersion: apps/v1
          kind: Deployment
          name: istiod
          patches:
            # Mount istiod serving and webhook certificate from Secret mount
          - path: spec.template.spec.containers.[name:discovery].args[7]
            value: "--tlsCertFile=/etc/cert-manager/tls/tls.crt"
          - path: spec.template.spec.containers.[name:discovery].args[8]
            value: "--tlsKeyFile=/etc/cert-manager/tls/tls.key"
          - path: spec.template.spec.containers.[name:discovery].args[9]
            value: "--caCertFile=/etc/cert-manager/ca/root-cert.pem"
          - path: spec.template.spec.containers.[name:discovery].volumeMounts[6]
            value:
              name: cert-manager
              mountPath: "/etc/cert-manager/tls"
              readOnly: true
          - path: spec.template.spec.containers.[name:discovery].volumeMounts[7]
            value:
              name: ca-root-cert
              mountPath: "/etc/cert-manager/ca"
              readOnly: true
          - path: spec.template.spec.volumes[6]
            value:
              name: cert-manager
              secret:
                secretName: istiod-tls
          - path: spec.template.spec.volumes[7]
            value:
              name: ca-root-cert
              configMap:
                defaultMode: 420
                name: istio-ca-root-cert

Deploy Istio-Ingress

Configure IstioOperator Deployment for Istio-Ingress

It is recommended by Istio to deploy Istio-Ingress on a different namespace than istiod.

https://istio.io/latest/docs/setup/additional-setup/gateway/
As a security best practice, it is recommended to deploy the gateway in a different namespace from the control plane.

Set pilot Cert Provider: istiod

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istio-ingress
  namespace: istio-system
spec:
  profile: empty # Do not install CRDs or the control plane
  components:
    ingressGateways:
    - name: ingressgateway
      namespace: istio-ingress
      enabled: true
      label:
        # Set a unique label for the gateway. This is required to ensure Gateways
        # can select this workload
        istio: ingressgateway
  values:
    gateways:
      istio-ingressgateway:
        # Enable gateway injection
        injectionTemplate: gateway
    global:
      pilotCertProvider: istiod