Spinnaker Security — SSO and SSL Setup

This is the second of the three articles on deploying Spinnaker onto the Tencent Cloud Kubernetes cluster.

In the last article, we walked through how to deploy Spinnaker into a Tencent Kubernetes cluster on the Tencent Cloud platform. Spinnaker doesn’t have a dedicated user management system and cannot provide an API access token. Its security access is controlled by the SSO. This documentation describes how to enable two types of SSO on Spinnaker, SAML and x509, and SSL to ensure that users can securely access the Spinnaker.

Overall architecture of SSO and SSL system

To enable SSL and SSO on Spinnaker, it’s recommended to build the system using the following architecture, where UI users have SAML to conduct the SSO verification and API users have x509 verification. Of course, there should be a variety of ways of building this system so that both UI users and API users can access Spinnaker securely.

Before we start, please remember NOT to apply the change in place if you are setting up a Spinnaker for a new environment. You should copy out the configuration to your specific working environment. The folder that this document directs you to is for the dev environment.

After you apply a new configuration, verify it in incognito mode to make sure the new feature works.

Setup SSL on Spinnaker

To implement the architecture above, the first thing you need to do is to turn on SSL. If you have followed the doc — Spinnaker Installation, you should have two K8s services, `spin-deck-private` and `spin-gate-private`, running on port 9000 and 8084 separately. Now it’s the time to have a K8s ingress to be the interface with users and have SSL on it.

• Add SSL cert to your k8s secrets

Go to our repo folder `$path-to-spinnaker/k8-spinnaker-service/` and copy out the resource block `ssl-secret on main.tf` to add the SSL Cert to your K8s cluster. You may want to change the `ssl_cert_id` value.

• Create an ingress to connect to Spinnaker deck and gate

Go to folder `$path-to-spinnaker/k8-spinnaker-service/` and copy out the resource block `spinnaker-ingress` on `main.tf` to create an ingress on your K8s cluster. Change the configuration accordingly based on your needs. Typically, you would need to change:

• the host value for two forwarding rules
• the sercet_name value to the SSL cert you stored on K8s

• Add the DNS records

Go to folder `$path-to-spinnaker/k8-spinnaker-service/` and copy out the resource block `spinnaker-dns` and `gate-dns` on `main.tf` to add the domain to your root DNS record.

After you apply the configuration, you should see the records you just added on your DNS provider.

Checkpoint

Before you move to next stage, check the following two things and make sure you have everything set up correctly:

a) Check if your K8s ingress is forwarding traffic to your Spinnaker service. You can check this from your ingress configuration and you should also find your LB is healthy.

b) Go to your browser and type in your Spinnaker deck URL. You should be able to access Spinnaker with https protocol.

Setup SAML on Spinnaker

Now let’s set up SAML SSO on Spinnaker that only gives specific users UI access to the Spinnaker. If you are not your IDP admin, you would need to have your IDP admin with you to set it up together.

• Configure an application in your IDP

You or your IDP admin should follow the standard procedure of adding a new application. Check the following things while you set up the application:

The specific login url should following the following format.

https://<spinnaker-UI-deck-url>/saml/SSO.

Specify a unique entity ID (we’ll use spinnaker.test here as an example).

a) The only special setting is put value `memberOf` under `Group Attribute Statement`, and if you have a filter of this value, you should put *.

b) Add the users to your new Spinnaker Application

• Enable the SAML SSO on Spinnaker

a) The first thing you need to do is have the SAML metadata ready. It should be something similar to what pasted below:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<md:EntityDescriptor 
    xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata"    
    entityID="https://accounts.google.com/o/saml2?idpid=SomeValueHere" 
    validUntil="2021-05-16T15:17:27.000Z">
  <md:IDPSSODescriptor 
      WantAuthnRequestsSigned="false" 
      protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol">
    <md:KeyDescriptor use="signing">
      <ds:KeyInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
        <ds:X509Data>
          <ds:X509Certificate>
MIIDdDCCAlygAwIBAgIGAVS/Sw5yMA0GCSqGSIb3DQEBCwUAMHsxFDASBgNVBAoTC0dvb2dsZSBJ
bmMuMRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MQ8wDQYDVQQDEwZHb29nbGUxGDAWBgNVBAsTD0dv
b2dsZSBGb3IgV29yazELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWEwHhcNMTYwNTE3
MTUxNzI3WhcNMjEwNTE2MTUxNzI3WjB7MRQwEgYDVQQKEwtHb29nbGUgSW5jLjEWMBQGA1UEBxMN
TW91bnRhaW4gVmlldzEPMA0GA1UEAxMGR29vZ2xlMRgwFgYDVQQLEw9Hb29nbGUgRm9yIFdvcmsx
CzAJBgNVBAYTblVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMIIBIjANBgkqhkiG9w0BAQEF46OCAQ8A
MIIBCgKCAQEA4JsnpS0ZBzb7DtlU7Zop7l+Kgr7NzusKWcEC6MOsFa4Dlt7jxv4ScKZ/61M5WKxd
5YX0ol1rPokpNztj+Zk7OXrG8lDic0DpeDutc9pcq0+9/NYFF7WR7TDjh4B7Txnq7SerSB78fT8d
4rK7Bd+cu/cBIyAAyZ5tLeLbmTnHAk093Y9vF3mdWQnfAhx4ldOfstF6G/d2ev7I5xjSKzQuH6Ew
3bb3HLcM4uEVevOfNAlh1KoV4vQr+qzbc9UEFcPRwzuTwGa6QjfspWW7NgXKbHHC+X6a+gqJrke/
6l2VvHaQBJ7oIyt4PCdel2cnUkvuxvzHPYedh1AgrIiSP1brSQIDAQABMA0GCSqGSI34DQEBCwUA
A4IBAQCPqMAIau+pRDs2NZG1nGfyEMDfs0qop6FBa/wTNis75tLvay9MUlxXkTxm9aVxgggjEyc6
XtDjpV0onrH0jBnSc+vRI1GFQ48EO3owy3uBIeR1aMy13ZwAA+KVizeoOrXBJbvIUZHo0yfKRzIu
gtM58j58BdAFeYo+X9ds/ysvZ8FIGTLqMl/A3oO/yBNDjXR9Izoqgm7RX0JJXGL9Y1AgmEjxyqo9
MhxZAGxOHm9HZWWfVMcoe8p62mRJ2zf4lkNPBnDHrQ8MDPSsXewAuiSnRBDLxhdBgyThT/KW7Q06
rGa6Dp0rntKWzZE3hGQS0AdsnuFY/OXbmkNG9WUrUg5x
          </ds:X509Certificate>
        </ds:X509Data>
      </ds:KeyInfo>
    </md:KeyDescriptor>
    <md:NameIDFormat>urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress</md:NameIDFormat>
    <md:SingleSignOnService 
        Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect" 
        Location="https://accounts.google.com/o/saml2/idp?idpid=SomeValueHere"/>
    <md:SingleSignOnService 
        Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" 
        Location="https://accounts.google.com/o/saml2/idp?idpid=SomeValueHere"/>
  </md:IDPSSODescriptor>
</md:EntityDescriptor>

b) Generate a keystore and key in a new Java Keystore with a strong password:

keytool -genkey -v -keystore saml.jks -alias saml -keyalg RSA -keysize 2048 -validity 10000

c) Execute the following halyard commands to turn on SAML on Spinnaker:

KEYSTORE_PATH= # /path/to/keystore.jks
KEYSTORE_PASSWORD=hunter2
METADATA_PATH= # /path/to/metadata.xml
SERVICE_ADDR_URL=https://localhost:8084
ISSUER_ID=spinnaker.test

hal config security authn saml edit \

 --keystore $KEYSTORE_PATH \
 --keystore-alias saml \
 --keystore-password $KEYSTORE_PASSWORD \
 --metadata $METADATA_PATH \
 --issuer-id $ISSUER_ID \
 --service-address-url $SERVICE_ADDR_URL

hal config security authn saml enable

d) Apply the changes:

hal deploy apply

Checkpoint

At this point, you should have to verify your identity before you can access Spinnaker. You may see similar login window as below:

After you verify your identity, you should be able to use Spinnaker as before. If you have many users and you don’t want them to access everything. Then you need to set up different service accounts to control the access. For details, you can follow the doc (WIP) here.

Setup x509 on Spinnaker

Right now UI users can access the Spinnaker without issue, but you may find you cannot access the Spinnaker from CLI and API anymore. To solve this issue, you need to set up an extra k8s service and x509 verification method that is dedicated for this type of access, because you cannot verify your identity when you try to do an API call. It routes you to the ID verification page above, where you can’t easily verify it through CLI.

This section of the guidance mainly combines the docs Set up Self-Signed SSL, x.509 Client Certificates and Expose Spinnaker API Endpoint. You can also use them as a reference when doing the settings.

• Generate key and self-signed certificate and server certificate for Gate

We will use `openssl` to generate a Certificate Authority (CA) key and server certificates. These instructions create a self-signed CA.

a) Create the CA key. This command below references the passphrase environment variable used to encrypt the key `ca.key`.

openssl genrsa \
 -des3 \
 -out ca.key \
 4096

b) Self-sign the CA certificate. This command below references the pass phrase environment variable used to decrypt the key ca.key.

openssl req \
 -new \
 -x509 \
 -days 1000 \
 -key ca.key \
 -out ca.crt

c) Create a server key for Gate. This command below references the passphrase environment variable used to encrypt the key gate.key. Keep this file safe!

openssl genrsa \
 -des3 \
 -out gate.key \
 4096

d) Generate a certificate signing request for Gate. Specify localhost or Gate’s eventual fully-qualified domain name (FQDN) as the Common Name (CN).

openssl req \
 -new \
 -key gate.key \
 -out gate.csr

e) Use the CA to sign the server’s request and create the Gate server certificate (in pem format).

openssl x509 \
 -sha256 \
 -req \
 -days 1000 \
 -in gate.csr \
 -CA ca.crt \
 -CAkey ca.key \
 -CAcreateserial \
 -out gate.crt

f) Convert the pem format Gate server certificate into a PKCS12 (p12) file, which is importable into a Java Keystore (JKS).

openssl pkcs12 \
 -export \
 -clcerts \
 -in gate.crt \
 -inkey gate.key \
 -out gate.p12 \
 -name gate

g) Create a new Java Keystore (JKS) containing your p12-formatted Gate server certificate.

keytool -importkeystore \
 -srckeystore gate.p12 \
 -srcstoretype pkcs12 \
 -srcalias gate \
 -destkeystore gate.jks \
 -destalias gate \
 -deststoretype pkcs12

h) Import the CA certificate into the Java Keystore.

keytool -importcert \
 -keystore gate.jks \
 -alias ca \
 -file ca.crt

Checkpoint

Verify the Java Keystore contains the correct contents . It should contain two entries:

• `gate` as a `PrivateKeyEntry`
• `ca` as a `trustedCertEntry`

You can use the following command to do the verification.

keytool \
 -list \
 -keystore gate.jks

• Generate key and server certificate for client

a) Create the client key. Keep this file safe!

openssl genrsa \
 -des3 \
 -out client.key \
 4096

b) Generate a certificate signing request for the server. Ensure the Common Name is set to a non-empty value.

openssl req \
 -nodes 
 -newkey rsa:2048 \
 -keyout key.out \
 -out client.csr \
 -sha256

c) Use the CA to sign the server’s request.

openssl x509 \
 -req \
 -days 1000 \
 -in client.csr \
 -CA ca.crt \
 -CAkey ca.key \
 -CAcreateserial \
 -out client.crt \
 -sha256

• Create Additional Gate Service

Go to `path-to-spinnaker/k8-spinnaker-service/` and copy out the resource block spinn-api on main.tf to create an additional Spinnaker Gate k8s service. Change the configuration accordingly based on your needs. Typically, you would need to change the subnetid for the additional gate service.

• Reroute Spinnaker Gate Service Traffic

a) Add an extra port mapping policy to your spin-gate service.

b) Go to CLB to edit gate’s backend protocol.

c) Switch backend protocol to HTTPS, and click Next until you see Save and save the change.

Alternatively, for step 2 and step 3, you can also configure it through Terraform. The following steps describe how to do it with Terraform.

a) Add extra annotation to your `spinnaker-ingress` resource. The value of `ingress.cloud.tencent.com/tke-service-config` should align with the resource you create in the next step.

b) Use the resource block `spinnaker_ingress_config` to configure the LB you created for Spinnaker ingress. You may want to change some value of the ingress yaml file indicated in this resource block.

After you apply the above changes, you should see the same result if you do it through the console.

• Configure Additional SSL for Gate and turn on x509

a) Upload the above certificates and keys to your Halyard container. We use Halyard to set up SSL for Gate.

KEYSTORE_PATH= # /path/to/gate.jks

hal config security api ssl edit \
 --key-alias gate \
 --keystore ${KEYSTORE_PATH} \
 --keystore-password \
 --keystore-type jks \
 --truststore ${KEYSTORE_PATH} \
 --truststore-password \
 --truststore-type jks \
 --client-auth WANT

hal config security api ssl enable

b) Create a file called `gate-local.yml` inside `~/home/spinnaker/.hal/default/profile`, and add an extra port for Gate API endpoint.

default.apiPort: 8085

c) Use Halyard to turn on x509.

hal config security authn x509 enable

d) Apply the changes

hal deploy apply

• (Optional) Configure the DNS for API service

If you want to use the domain name instead of the IP address to access Spinnaker from the API, you can set up a DNS record for it. Just simply go to your DNS provider, and add an A record to point the domain name to your API service IP address.

Final Checkpoint

After doing all the steps above, UI users should be able to log in Spinnaker as usual, and API users can also access Spinnaker with the correct certificate. If you run command:

curl https://spinnaker-api-client-dev-usw1.uncappedtech.com/applications -v -k --cert client.crt --key key.out

You should see result similar to:

*   Trying 172.***.***.***:443...
* Connected to ***.*****.com (172.***.***.***) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):

* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=CA; O=***; CN=***.***.com
*  start date: Apr 25 04:22:35 2023 GMT
*  expire date: Aug  7 04:22:35 2026 GMT
*  issuer: ***
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
> GET /applications HTTP/1.1
> Host: ***
> User-Agent: curl/7.77.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: POST, GET, OPTIONS, DELETE, PUT, PATCH
< Access-Control-Max-Age: 3600
< Access-Control-Allow-Headers: x-requested-with, content-type, authorization, X-RateLimit-App, X-Spinnaker-Priority
< Access-Control-Expose-Headers: X-AUTH-REDIRECT-URL
< Vary: origin,access-control-request-method,access-control-request-headers,accept-encoding
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=31536000 ; includeSubDomains
< X-SPINNAKER-REQUEST-ID: e451cbe0-a0d7-4f2f-8a29-935b99502e78
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< Content-Type: application/json;charset=UTF-8
< Content-Length: 3703
< Date: Mon, 08 May 2023 23:10:02 GMT
< 

* Connection #0 to host ***.com left intact
[{"name":"***","email":"***"}]%

Congratulations and good job! If you have a 200 for the above command and you have no problem accessing Spinnaker from the UI, that means you successfully set up the architecture I described in the beginning of the documentation.

Author: Minze Tao

Minze Tao is a Solution Architect and Product Operation Engineer at Tencent America. He has profound experiences with Tencent Cloud products and architecture. He has dedicated his latest effort in supporting Tencent Cloud’s gaming customers on multi-cloud architecture.