Trillian on Google Cloud Platform

As my better-educated colleague — Annie — pointed out yesterday, “Trillian” is named after a character in “The Hitchhiker’s Guide to the Galaxy” this is (a) excellent; (b) not Greek which is a refreshing change having been submerged in Kubernetes. The eponymous project is a Google OSS contribution: Trillian.

This post is a quick-hit walking through all the hard-work that’s been done by this team in developing Trillian and in making it straightforward to deploy to Kubernetes.

Please see the README on the Trillian GitHub here for the source of these instructions.

Setup

  1. Create a Google Cloud Platform project.
  2. Enable “container” and “spanner” APIs
  3. Request additional Quota for IPs in your preferred zone (you need ≥12)
  4. Create a single-zone (for now) Kubernetes cluster.
  5. Download Trillian
PROJECT=[[YOUR-PROJECT]]
BILLING=[[YOUR-BILLING]]
gcloud projects create ${PROJECT}
gcloud beta billing projects link $PROJECT \
--billing-account=$BILLING
for SERVICE in container spanner
do
gcloud services enable ${SERVICE}.googleapis.com \
--project=${PROJECT}
done

For this project (${PROJECT}), you will need ≥12 IP addresses as 12 machines (4+2+4+2) are created by the Kubernetes cluster. The default Quota is 8 and so you may need to request more Quota. You may do so through the Cloud Console:

https://console.cloud.corp.google.com/iam-admin/quotas?project=${PROJECT}&service=compute.googleapis.com&metric=In-use%20IP%20addresses&location=${ZONE}

Cloud Console: Default Zonal IP Quota is 8

Select the Quota that represents the “In-use IP Address” for the zone you wish to use, click the “Edit Quotas” and request ≥12 IPs. Await the email confirming the Quota has been approved.

NB Do not proceed until you receive confirmation of the Quota increase because the scripts will fail without sufficient Quota.

Here’s a screenshot *after* the Quota was approved and *after* I’d run ./create.sh as shown in the next section:

Cloud Console: Increased Zonal IP Quota now 11

Trillian

You will need a local copy of Trillian in order for the Deployment scripts to build the containers that will be used in the Kubernetes Deployment:

WORKING_DIR=[[YOUR-WORKING-DIRECTORY]]
cd ${WORKING_DIR}
mkdir go
export GOPATH=$PWD/go
export PATH=$PATH:$GOPATH/bin
go get github.com/google/trillian
cd go/src/github.com/google/trillian/examples/deployment/kubernetes

Please edit the config.sh file to correctly reflect your preferences:

export PROJECT_NAME=[[YOUR-PROJECT]]
export CLUSTER_NAME=trillian-01
export REGION=[[YOUR-PREFERRED-ZONE]] # This needs suff Quota
export ZONE=${REGION}-b
export CONFIGMAP=trillian-cloudspanner.yaml

We are ready to begin the 2 phase deployment.

First, run:

./create.sh

This provisions the Google Cloud Platform resources: Kubernetes cluster and a Cloud Spanner instance.

Here’s the Kubernetes cluster:

Kubernetes Engine Cluster w/ 4 Node Pools

If the cluster does not provision with 4 Node Pools as shown above, you should revisit the instructions. The deployment will not succeed without this configuration.

A soupcon of Cloud Spanner:

https://console.cloud.google.com/spanner/instances/trillian-spanner/databases?project=${PROJECT}

Cloud Spanner

And, a couple of container images:

https://console.cloud.google.com/gcr/images/${PROJECT}/

Container Registry

If everything appears to have completed successfully, you may proceed to the deployment phase with:

./deploy.sh

This script builds the container images and pushes them to Google Container Registry. Then it creates Kubernetes Deployments that leverage these images.

If this script succeeds, you will be presented with an enumeration of all the Kubernetes resources on the command-line (not reproduced here) and, you may confirm the Deployments (is proceeding | have completed) using Cloud Console:

All Workloads: etcd and Trillian’s Log Server and Log Signer

And:

Workload: trillian-logserver-deployment

And:

Workload: trillian-logsigner-deployment

This brings you to the step in the GitHub README “Next steps”. We’ll want to do something useful with our Trillian deployment to Kubernetes (backed by Cloud Spanner).

You may not yet be using Trillian’s minimal container images but — hopefully — you will soon be using them. When you are, you can’t ssh into a trillian-log-server Pod and run curl commands because the Pods have no shell and don’t include curl. Instead, you can port-forward to test the Trillian API

Trillian API

Assuming you have Trillian deployed, you should have 4 trillian-log-server and 2 trillian-log-signer Pods running. The follow command will grab the 0th trillian-log-server Pod name for us and plonk it in ${LOG_POD}:

LOG_POD=$(\
kubectl get pods \
--selector=io.kompose.service=trillian-log \
--output=jsonpath='{.items[0].metadata.name}')

Knowing that these Pods all expose port 8091 as an HTTP endpoint, we can then setup port-forwarding from your local machine to the Pod. For convenience, we’ll use the same port on either end:

kubectl port-forward ${LOG_POD} 8091:8091
Forwarding from 127.0.0.1:8091 -> 8091
Forwarding from [::1]:8091 -> 8091
Handling connection for 8091
Handling connection for 8091
Handling connection for 8091

From another shell, you should then be able to issue the command cited in the Trillian tutorial:

HOST=localhost
PORT=8091
RESPONSE=$(curl \
--silent \
--request POST \
http://${HOST}:${PORT}/v1beta1/trees \
--data '{ "tree":{ "tree_state":"ACTIVE", "tree_type":"LOG", "hash_strategy":"RFC6962_SHA256", "signature_algorithm":"ECDSA", "max_root_duration":"0", "hash_algorithm":"SHA256" }, "key_spec":{ "ecdsa_params":{ "curve":"P256" } } }')
TREE_ID=$(echo ${RESPONSE} | jq --raw-output .tree_id)
echo ${RESPONSE} \
| jq .

And, all being well, you should receive something of the form:

{
"tree_id": "1234567890123456789",
"tree_state": "ACTIVE",
"tree_type": "LOG",
"hash_strategy": "RFC6962_SHA256",
"hash_algorithm": "SHA256",
"signature_algorithm": "ECDSA",
"storage_settings": {
"@type": "type.googleapis.com/spannerpb.LogStorageConfig",
"num_unseq_buckets": "4",
"num_merkle_buckets": "16"
},
"public_key": {
"der": "[[REDACTED]]"
},
"max_root_duration": "0s",
"create_time": "2018-00-00T00:00:00.000000000Z",
"update_time": "2018-00-00T00:00:00.000000000Z"
}

And, then you may:

curl \
--silent \
--request POST \
http://${HOST}:${PORT}/v1beta1/logs/${TREE_ID}:init \
| jq .

And you should receive something of the form:

{
"created": {
"timestamp_nanos": "1527642133340965256",
"root_hash": "[[REDACTED]]",
"key_hint": "[[REDACTED]]",
"log_root": "[[REDACTED]]",
"log_root_signature": "[[REDACTED]]"
}
}

I’ll update this post next week with something for us to use it for. One of the members of the Trillian team has a nifty example :-)

2018–06–01: Update

I was focused elsewhere this week and didn’t get to build a app on Trillian. However, I did gloop around with the API and discovered the next useful method.

Following on from above… with ${TREE_ID} and having called ${TREE_ID}:init:

VALUE="[[SOME-STRING]]"
EXTRA="[[SOME-STRING]]"
EVALUE=$(echo ${VALUE} | base64)
EEXTRA=$(echo ${EXTRA} | base64)
curl \
--silent \
--request POST \
http://${HOST}:${PORT}/v1beta1/logs/${TREE_ID}/leaves \
--data "{ \"leaf\": { \"leaf_value\": \"${EVALUE}\", \"extra_data\": \"${EEXTRA}\" } }" \
| jq .

Or for some bashity to get a random strings of 256 characters...

EVALUE=$(\
cat /dev/urandom \
| tr --complement --delete 'a-zA-Z0-9 ' \
| fold --width=256 \
| head --lines=1 \
| base64 --wrap=0\
)

More soon!

Conclusion

Trillian is an interesting solution from Google. This post has (thus far) shown how you may easily deploy Trillian to a Kubernetes cluster. Now we just need an example application!