Deploying Highly Available Immutable Audit Logs, Part One

Here at FutureTech Industries, we build a variety of high-availability, planet-scale systems to support our clients’ needs. Audit logs are a crucial topic that arises frequently out of sheer necessity to track access to sensitive data, and that means we need a robust solution right off the bat. Reviewing the open and closed source options available, we found Factom to be a powerful candidate. Backed by blockchain and anchored at a stable price to write entries (unlike Ethereum), what could be a better fit than a hashed, immutable log quorumed out the wazoo?

How It Works

Factom is a Blockchain as a Service that provides the ability to write immutable logs. This fits a variety of our use cases, especially in regards to auditing access to personally identifiable information. Running a Factom deployment requires a few pieces of its infrastructure as well as a cryptocurrency to make transactions against it. The design involves converting Factoids (coins) to Entry Credits (ECs), which can be spent on writing Entries (logs) into a Chain (stream of logs)!

factomd is the bridge daemon that talks to the Factom network, since the blockchain is decentralized. It's akin to an API endpoint for writing non-blockchain software, except it runs in the same network as the software communicating with it. factom-walletd handles wallet-related behaviors, and provides support for writing Chains and Entries to factomd by connecting to factom-walletd. This is a good reference for setting up Factom on one's server/home computer in a non-HA setup.

However, we’re setting this up for a production environment, which means we need at least two copies of everything, so we’re resilient to failures and capable of doing maintenance/testing without taking down our bridge into the network. Our favorite way to orchestrate that is via Kubernetes.

The Setup

For newcomers to Kubernetes, this is a good primer on the available workloads. Unfortunately there’s no index for the Controllers page, but there’s a menu on the left to browse through the different workloads under Workloads > Controllers, and we’ll link to the relevant documentation throughout this article. The components are as such:

factom-bridge StatefulSet: Manages instances of factomd
factom-wallet StatefulSet: Manages instances of factom-walletd
factom-bridge Service: Exposes factomd within our cluster
factom-wallet Service: Exposes factom-walletd within our cluster

StatefulSets are a workload that allows for easy management of persistent disks and more predictable scalability. Kubernetes Deployment workloads would work just as well, but the team over at Factom advises us to be careful with scaling too quickly, and StatefulSets handle several related concerns for us.

Assembling the Prerequisites

Kubernetes requires a Docker image for deployment of any application. To that end, we have constructed a custom image which contains Factom’s components. The Ubuntu-based Dockerfile installs dependencies with Apt, then downloads Factom from the GitHub Factom distribution repo using wget/dpkg.

In order to deploy, the image needs to be available to the cluster via a registry. We provide our image via Docker Hub with the tag futuretechindustries/factom-ha:0.0.1. You can alternatively build your image off Factom's official image.

Persisting Data from factomd

Kubernetes provides a workload called a StatefulSet which exists to provide applications guaranteed startup ordering and assignment of persistent volumes to pods. Other Kubernetes workloads can assign persistent volumes, but will not auto-provision and manage them appropriately. If managed with a Deployment, for example, restarts will lose all data. Since Factom is a blockchain, and blockchains such as Factom frequently have long boot times due to verification and resync, keeping disks associated with pods is important.

Rolling Out The Stack

In order to allow applications to receive traffic from external clients, Kubernetes provides Services. A Service uses a pod selector, a construct similar to a simple database query, to find Factom’s pods and place itself in front as a load balancer. Coupled with healthchecks, Kubernetes automatically handles connecting clients to known-good pods. A service also provisions either an internal or external IP, depending on its configuration. The Kubernetes configuration of a cluster defines how it’ll provision the related load balancer entities.

Helm is a Kubernetes package manager, similar to apt for clusters. We provide a helm chart which can be used to install our configuration. Our default configuration is production ready for our use case. To install it, run the following commands:

$ helm repo add fti 'https://futuretechindustriesllc.github.io/charts/'
$ helm repo update
$ helm install --namespace factom-ha --name factom-ha fti/factom-ha

The namespace and name in the above command can be changed, however it’ll require modification of upcoming commands. We’ll cover automatic wallet setup and synchronization in upcoming parts of this series; this article covers manual configuration.

Verifying The Deployment

To verify the deployment, we use kubectl port-forward to forward the port running Factom's Control Panel. Run the following command and browse to localhost:8090:

$ kubectl port-forward factom-bridge-0 8090

Our deployment of Factom doesn’t expose the Control Panel via the service for the sake of security. It is a Kubernetes best practice to use kubectl port-forward when possible, especially for control planes.

Finally, set up your wallet. The current version of the chart doesn’t support more than one running walletd instance. Part 2 of this series will provide an updated chart and instructions, but for now use Factom's official instructions. To get a shell in the running walletd, run:

$ kubectl exec -itn factom-ha factom-wallet-0 /bin/bash

Part Two of this series is coming soon, which will go over automating wallet/identity setup, optimizing Factom synchronization, and scaling!

If you find an issue with this configuration, please open an issue with our charts repository. Special thanks to Ian Seyer and Brian Deery from Factom Inc for helping us get everything up and going so quickly!
Co-authored by Austin Hicks, Madison Steiner, and the rest of the wonderful team at FutureTech Industries. Thanks for making this possible.