Daz Wilkin
Jul 5, 2018 · 8 min read

2019–05–08 Update: The code in Ben’s repo for Registers is out of date. Please use the copy of his solution in https://github.com/google/trillian-examples

2018–12–11 Update: Thanks to my colleague — Paul — for his helpful feedback on this post that is now incorporated.

I wrote a story recently explaining how to deploy Google’s Trillian project to GCP. At the time, I said I’d write-up an example for Trillian too. Google’s Ben Laurie has developed an elegant example using the UK Government’s Registers in conjunction with Trillian.

This story is a run-through of the code in Ben’s repo to show you how to work through it. It is written with Ben’s permission. Although this tutorial uses Google Cloud SQL, any MySQL instance will work. Trillian and Ben’s example code runs on your local workstation and is not deployed to GCP.

Setup

Environment:

BILLING=[[YOUR-BILLING]]
PROJECT=[[YOUR-PROJECT]]
REGION=[[YOUR-REGION]]
INSTANCE=[[YOUR-INSTANCE]]

Then, if needed, create a GCP project and enable billing. You’ll need billing enabled as we’ll be using Cloud SQL:

gcloud projects create ${PROJECT}
gcloud beta billing projects link ${PROJECT} \
--billing-account=${BILLING}

Create a SQL instance (${INSTANCE}) and ensure it’s root password is unset. Don’t worry, we’re going to use Cloud SQL Proxy to access the instance so your instance remains secure:

gcloud sql instances create ${INSTANCE} \
--database-version=MYSQL_5_7 \
--tier=db-g1-small \
--region=${REGION} \
--project=${PROJECT}
gcloud sql users set-password root \
--instance=${INSTANCE} \
--host=% \
--password='' \
--project=${PROJECT}

Update 2018–11–19: Duplicate the Test user for all hosts (%) to avoid localhost issues when using Cloud SQL

gcloud sql users create test \
--instance=${INSTANCE} \
--host=% \
--password='zaphod' \
--project=${PROJECT}

Download the Cloud SQL Proxy and leave it running in a separate shell:

wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxychmod +x cloud_sql_proxy./cloud_sql_proxy \
-instances=${PROJECT}:${REGION}:${INSTANCE}=tcp:3306

You can test the Proxy with by connecting through it from another terminal session:

mysql \
--host=127.0.0.1 \
--port=3306 \
--user=root \
--password=''

NB For this to work, use 127.0.0.1 not localhost.

Trillian

Establish a Golang workspace and grab Trillian:

mkdir go
export GOPATH=$PWD/go
export PATH=$PATH:$GOPATH/bin
go get github.com/google/trilliancd ${GOPATH}/src/github.com/google/trillian/
go get -t -u -v ./...

Reset the Cloud SQL database:

MYSQL_HOST=127.0.0.1 ./scripts/resetdb.sh

Update 2018–11–19: DB_HOST is now MYSQL_HOST.

Optional: if you’d like, you may run Trillian’s integration tests:

MYSQL_HOST=127.0.0.1 ./integration/integration_test.sh

Example

Then clone Ben’s examples:

mkdir -p ${GOPATH}/src/github.com/benlaurie && \
cd ${GOPATH}/src/github.com/benlaurie
git clone https://github.com/benlaurie/trillian-examples
cd trillian-examples
tree
.
├── AUTHORS
├── CONTRIBUTING.md
├── CONTRIBUTORS
├── etherslurp
├── LICENSE
└── README.md

There are 7 steps to Ben’s example and each is represented in a distinct branch of his repo.

1. Getting the data

git checkout register-1
go get ./...
git branchmaster
* register-1
tree
.
├── AUTHORS
├── CONTRIBUTING.md
├── CONTRIBUTORS
├── etherslurp
├── LICENSE
├── README.md
├── registers
└── scripts

NB The branch includes registers and scripts.

Then, you should be able to successfully:

cd registers
go run dump/main.go

And you’ll start to see the Registers site data being dumped as JSON:

2018/07/04 11:28:12 &register.Register{baseURL:"https://register.register.gov.uk/", info:map[string]interface {}{"total-records":44, "total-entries":66, "register-record":map[string]interface {}{"phase":"beta", "register":"register", "fields":[]interface {}{"register", "text", "registry", "phase", "copyright", "fields"}, "registry":"cabinet-office", "text":"Registers maintained by the UK government"}, "custodian":"Arnau Siches", "last-updated":"2018-07-03T09:56:50Z", "domain":"register.gov.uk"}}
2018/07/04 11:28:13 map[string]interface {}{"index-entry-number":"1", "entry-number":"1", "entry-timestamp":"2016-08-04T14:45:41Z", "key":"register", "item-hash":[]interface {}{"sha-256:5fa9c4b569a71542c9c791203d54b6b94c125f189a9c529c63466a0b34cbe38c"}} sha-256:5fa9c4b569a71542c9c791203d54b6b94c125f189a9c529c63466a0b34cbe38c map[string]interface {}{"fields":[]interface {}{"register", "text", "registry", "phase", "copyright", "fields"}, "register":"register", "phase":"beta", "registry":"cabinet-office", "text":"Registers maintained by HM Government"}
...

You may interrupt the dump.

NB If you are wondering which Register is being used, it’s the Register of Registers. You may preview this data at: https://www.registers.service.gov.uk/registers/register

Each branch has a revised TUTORIAL.md file that summarizes what’s being done. I don’t wish to duplicate Ben’s content but, to save you moving back-and-forth, I’ll add summaries as we go.

2. Add Data to a Log

git checkout register-2
go get ./...
git branch master
register-1
* register-2

This step add Registers data to a Trillian Log.

NB In the ensuing steps, you’ll want to be able to run multiple shells concurrently to monitor each of the various processes’ output.

Ensure your Cloud SQL Proxy is running.

Let’s also start a Trillian Log Server:

$GOPATH/bin/trillian_log_server --logtostderr

You should output similar to:

Using MySQL QuotaManager
RPC server starting on localhost:8090
HTTP server starting on localhost:8091
Deleted tree GC started

The next step generates a new Log and returns the Log’s ID. So, we can run this in the same (!) session:

LOGID=$($GOPATH/bin/createtree --admin_server=localhost:8090) && \
echo ${LOGID}

Which will output the unique ID of the Log, something of the form:

1234567890123456789

Then run the following command:

go run dump/main.go -log_id=${LOGID}

You should see output similar to step 1 as each record is pulled from Registers and added to the Trillian Log.

This time, please let the command run to completion. It should complete with output similar to:

2018/07/04 15:49:04 New entries: 66
2018/07/04 15:49:04 Duplicate entries: 0

3. Extract Log Entries

git checkout register-3
Switched to a new branch 'register-3'
git branch
master
register-1
register-2
* register-3

We’ll start by running a Trillian Log Signer. Open another shell and:

$GOPATH/bin/trillian_log_signer \
--logtostderr \
--force_master \
--http_endpoint=localhost:8092 \
--rpc_endpoint=localhost:8093 \
--batch_size=1000 \
--sequencer_guard_window=0 \
--sequencer_interval=200ms

NB I’ve added a different port (8093) for rpc_endpoint to avoid a collision with the log server’s rpc_endpoint.

You should see output of the form:

**** Log Signer Starting ****
**** Acting as master for all logs ****
Using MySQL QuotaManager
Creating HTTP server starting on localhost:8092
Log operation manager starting
HTTP server starting
creating mastership tracker for [3558678532662980721 5598166607403133250]
create master election goroutine for 3558678532662980721
create master election goroutine for 5598166607403133250
now acting as master for 0 / 2, master for:
...
Group run completed in 0.20 seconds: 2 succeeded, 0 failed, 0 items processed
...

We can then run Ben’s Log extractor:

go run extract/main.go --log_id=${LOGID}

Yes, so far, we’ve pulled the Registers’ entries, added them to a Trillian Log and then pulled them back out again.

BUT the intent here is to provide a pattern for adding and getting Trillian Log Data. And, because it’s a Trillian Log, the data is persisted in a Merkle Tree (itself backed by Cloud SQL). The Merkle Tree provides us with assurances about the integrity of the data that we’re adding.

4. Create a Trillian Map

Trillian supports 2 modes. The first, we’ve used which is its append-only Log. The second is a key-value Map. In this step, we’re going to iterate over the Registers data that’s now in a Trillian Log and add each record to a Trillian Map.

git checkout register-4
Switched to a new branch 'register-4'
git branch
master
register-1
register-2
register-3
* register-4

We need a Trillian Map server (and we’ll continue to use the Log server). So, open another shell and:

$GOPATH/bin/trillian_map_server \
--logtostderr \
--rpc_endpoint=localhost:8095 \
--http_endpoint=localhost:8096

NB You need not specify the — http-endpoint flag but doing so, avoids an error when the Map server attempts to open an HTTP endpoint on the same, default port (8091) that’s being used by the Log server.

The output is similar to that from the Log server:

Using MySQL QuotaManager
RPC server starting on localhost:8095
HTTP server starting on localhost:8096
Deleted tree GC started

In a similar manner to how we created a Log for the Log server in step #2, we’ll create a Map for the Map server:

MAPID=$(\
$GOPATH/bin/createtree \
--admin_server=localhost:8095 \
--tree_type=MAP \
--hash_strategy=TEST_MAP_HASHER) && \
echo ${MAPID}

And you’ll receive another unique ID. This one for the Map.

Then we’ll run the code that iterates over the log, extracts entries and creates map entries:

go run mapper/main.go \
--log_id=${LOGID} \
--map_id=${MAPID}

You should see all….66 entries (end of step #2) added to the Map viz:

2018/07/04 16:35:40 k: country ts: 2016-08-04 14:45:41 +0000 UTC
2018/07/04 16:35:40 key=country leaf=
evicting country -> &{map[index-entry-number:2 item-hash:[sha-256:610bde42d3ae2ed3dd829263fe461542742a10ca33865d96d31ae043b242c300] key:country entry-number:2 entry-timestamp:2016-08-04T14:45:41Z] [map[fields:[country name official-name citizen-names start-date end-date] phase:beta register:country registry:foreign-commonwealth-office text:British English-language names and descriptive terms for countries]]}

5. Extract Map Entries

git checkout register-5
Switched to a new branch 'register-5'
git branch
master
register-1
register-2
register-3
register-4
* register-5

Then, by eye-balling the keys created by the ingestion in step #4 above (E.g. “country” as shown), you can run extract map for specific keys:

go run extractmap/main.go \
--map_id=${MAPID} \
country
country
{"Entry":{"entry-number":"49","entry-timestamp":"2018-06-07T13:43:14Z","index-entry-number":"49","item-hash":["sha-256:7a0afa860ea3fd1f92c9c3d96a5df26cd9345360adaf6fc8d6ac1e0a282d372f"],"key":"country"},"Items":[{"fields":["country","name","official-name","citizen-names","start-date","end-date"],"phase":"beta","register":"country","registry":"foreign-commonwealth-office","text":"British English names and descriptive terms for countries as recognised by the UK government"}]}
go run extractmap/main.go \
--map_id=${MAPID} \
territory
territory
{"Entry":{"entry-number":"56","entry-timestamp":"2018-06-07T13:46:32Z","index-entry-number":"56","item-hash":["sha-256:e5a90c513739ee7636987485e3b69ec721b3860868a3c6487367930299c6593d"],"key":"territory"},"Items":[{"fields":["territory","name","official-name","start-date","end-date"],"phase":"beta","register":"territory","registry":"foreign-commonwealth-office","text":"British English names and descriptive terms for political, administrative and geographical entities that are not recognised as countries by the UK government"}]}

6. Refactor

In step #7, we’ll put a web server in front of the Trillian Map server. The web server needs to be able to grab ranges of keys, so this step refactors the extractmap from the step #5 to make this possible:

git checkout register-3
Switched to a new branch 'register-3'
git branch
master
register-1
register-2
register-3
register-4
register-5
* register-6

For simplicity, I’m going to diverge from Ben’s instructions and, rather than delete and create a new tree, we’re just going to create a new tree:

MAPID=$(\
$GOPATH/bin/createtree \
--admin_server=localhost:8095 \
--tree_type=MAP \
--hash_strategy=TEST_MAP_HASHER) && \
echo ${MAPID}

NB If you wish to retain access to the first tree, retain a copy of ${MAPID} before overwriting its value or use a different variable.

Then rerun the Log →Map mapper:

go run mapper/main.go \
--log_id=${LOGID} \
--map_id=${MAPID}

We can then extract the entire Map:

go run extractmap/main.go --map_id=${MAPID}

And lastly…

7. Web server

git checkout register-7
git branch
master
register-1
register-2
register-3
register-4
register-5
register-6
* register-7

Then:

go run webserver/main.go --map_id=${MAPID}

And you should be able to browse /records.json on that host’s port 8080:

localhost:8080/records.json

Conclusion

Hopefully this quick-hit story shows you how to navigate an excellent example Trillian app and understand Trillian better through the process.

Next step…. Build our own app!

Tear-down

Assuming you want to revert everything, kill each of the processes logging out to the various sessions that you created.

You may terminate the Cloud SQL Proxy too.

You may delete the Cloud SQL Database only. If you want to delete the Cloud SQL Instance, the database will be deleted along with it.

NB These operations are irrevocable:

gcloud sql databases delete test \
--instance=${INSTANCE} \
--project=${PROJECT}
gcloud sql instances delete ${INSTANCE} \
--project=${PROJECT}

If you wish to delete the GCP Project:

gcloud projects delete ${PROJECT}

That’s all!

Google Cloud Platform - Community

A collection of technical articles published or curated by Google Cloud Platform Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.

 by the author.

Daz Wilkin

Written by

Google Cloud Platform - Community

A collection of technical articles published or curated by Google Cloud Platform Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade