Google Trillian for Noobs (1)

The Missing Manuals Series

Google Trillian provides a tamper-proof append-only log (and upcoming log-backed map). It is an open-source project and is the foundation of Certificate Transparency, Key Transparency, the Go team’s Go Module Mirror (proposal) and a few not-yet-public applications.

[Google’s] Engineers are excellent at building wonderful technologies but often deficient when it comes to explaining to us mortals how to use them! In a short series of posts, I’m going to document my experience in prototyping an Trillian application (Trillianeers call these “personalities”) for three reasons:

• Clarify my understanding through an explanation
• Help others understand Trillian
• Solicit feedback from the Trillian team to improve my understanding

Full-dislosure: I’m a Googler but not part of the Trillian team. I’m endeavoring to fill the role of an external developer to help the Trillian team understand this experience and improve its documentation, samples etc. I’ve tried to keep the majority of my communications with the Trillian team in the open: on the team’s public Google Group and using GitHub issues. As a prospective customer of the team, I’d expect you’d receive even better attention than me ;-)

Outline

1. In this post, we’ll get booted into Trillian and append data to a log
2. In the next post, we’ll build a basic gRPC client-server personality
3. Perhaps rounding out the sample; a Rust client; let’s see…

Getting Started

https://github.com/google/trillian

The Trillian team provides documentation starting with the README on the repo. If that content is the level you need to get started, I recommend you stop reading this and continue there.

If — like me — your head explodes, read on…

We will use 4 distinct components:

1. A database (MySQL|MariaDB) to persist the logs
2. A Trillian Log Server with which we interact
3. A Trillian Log Signer
4. Our Trillian sample application aka “personality”

Database

Let’s create a Docker Volume to persist the database:

docker volume create trillian-data

Run your preferred MySQL database:

docker run \
--name=database \
--env=MYSQL_ALLOW_EMPTY_PASSWORD=yes \
--mount=source=trillian-data,target=/var/lib/mysql \
--publish=3306:3306 mariadb:10.4

From a second shell, connect to the database engine, create a database test with a user test with a password zaphod.

NB Trillian is named after a character in the Hitchhikers’ Guide to the Galaxy. zaphod is the name of another character.

docker run \
--interactive --tty \
--net=host \
mariadb:10.4 \
  sh -c 'exec mysql --host=127.0.0.1 --port=3306'

Then:

drop database if exists test;
create database test;
create user if not exists test@'%' identified by 'zaphod';
grant all on test.* to test@'%';

Then:

docker run \
--interactive --tty \
--net=host \
--volume=${PWD}/trillian.sql:/trillian.sql \
  mariadb:10.4 sh -c 'exec mysql --host=127.0.0.1 --port=3306 --user=test --password=zaphod test < /trillian.sql'

Here’s trillian.sql:

To confirm everything’s restored correctly, re-enter the MySQL client and:

MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| test               |
+--------------------+
4 rows in set (0.001 sec)
MariaDB [(none)]> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [test]> show tables;
+-------------------+
| Tables_in_test    |
+-------------------+
| LeafData          |
| MapHead           |
| MapLeaf           |
| SequencedLeafData |
| Subtree           |
| TreeControl       |
| TreeHead          |
| Trees             |
| Unsequenced       |
+-------------------+
9 rows in set (0.001 sec)

Trillian servers

We’re going to use Docker Compose as possibly the simplest way to run the cohort of containers we’ll need: the database, the Trillian log server and the Trillian log signer. The docker-compose.yml below will configure these 3 services for us:

docker-compose --file=docker-compose.yml up

NB adminer is included purely to confirm that the database is establish correctly. If you browse localhost:8080 after starting the services, you should be able list the test database tables:

NB Once the Docker Compose services are running, you can check the state of all services with docker-compose ps and the logs of individual services using docker-compose logs... and e.g. trillian-log-server.

The Trillian services expose Prometheus metric endpoints. The log server’s http endpoint is :8091 and the log signer’s is on :8092. So, e.g.:

http://localhost:8091/metrics

NB The log server and the log signer connect to the database service. They do not connect to one another.

A Basic Personality

The Trillian services support gRPC. We’ll use Trillian’s basic SDK which is auto-generated from the services’ protobufs to connect to the log server and create Leaves in it. Yes, we’re creating Leaves in a Log. I assume because we’re really creating Leaves in a Merkle Tree.

Here’s a simple(st?) Trillian Log Server client:

NB There are more file in the client that just this main.go. Please clone the repo for the entirety: https://github.com/DazWilkin/simple-trillian-log-1

See below for configuration and running steps.

All this code does is connects to the Trillian Log Server that we started previously. It connects to a specific Log that you’ll create in the next step. We then create an arbitrary Thing type that will form leaf values (in the log’s Merkle tree) and utilize the ability to associate (metadata) with each leaf value using a second, arbitrary type Extra that will form extra data.

Because each leaf is a unique hash, each leaf value must be unique. To achieve these, we simply prefix the Thing string value (creatively) "Thing" with the current datetime. Well, unique by the second so don’t try running it too quickly :-)

We add this tuple of the leaf value and extra data to the log.

Then, we retrieve it by computing the hash of the leaf value and querying the log for it.

Before we can run the sample, we need to create a Log in the Trillian Log Server:

go get -u github.com/google/trillian/cmd/createtree
RPCS=8090
LOGID=$(\
  go run github.com/google/trillian/cmd/createtree \
  --admin_server=:${RPCS} \
) && echo ${LOGID}

Then… drumroll please:

GO111MODULES=on \
go get github.com/DazWilkin/simple-trillian-log-1
GO111MODULES=on \
go run github.com/DazWilkin/simple-trillian-log-1 \
--tlog_endpoint=:8090 \
--tlog_id=${LOGID}

And you should see:

[main] Establishing connection w/ Trillian Log Server [:8090]
[main] Creating new Trillian Log Client
[main] Creating Server using LogID [2709711571613439438]
[server] Creating
[main] Creating a 'Thing' and something 'Extra'
[thing:new] Creating: [2019-06-28T11:56:13-07:00] Thing
[extra] Creating
[main] Submitting it for inclusion in the Trillian Log
[server:put] Entered
[thing:marshal] Marshaling: [2019-06-28T11:56:13-07:00] Thing
[extra:marshal] Entered
[server:put] ok
[main:put] ok
[main] Retrieving it from the Trillian Log
[server:get] Entered
[thing:marshal] Marshaling: [2019-06-28T11:56:13-07:00] Thing
[server:get] hash: 02a1829d...
[server:get] 0: [2019-06-28T11:56:13-07:00] Thing
[main:get] ok

Teardown

The client puts-gets and then terminates.

You may terminate the Docker Compose services by CTRL-C’ing the process. Alternatively, you may docker-compose down and, if you wish docker-compose rm from within the directory where you created the docker-compose.yml file.

If you wish to delete the Docker volume containing the Trillian Log database, you may docker volume delete trillian-data.

Conclusion

I hope this story helps others overcome any reticence in using Google Trillian formed from “Where do I start?”. As I mentioned above, the intent is to help you get started and to help me confirm my understanding of Trillian.

In a follow-up story, I plan to enhance this simplest starting point by wrapping the client in a gRPC server and using more complex Thing as leaf values.