Deploy Scalable Tezos Nodes in the Cloud
Updated April 13, 2023
Hosting any kind of service on the internet can be hard to get right. With the advent of cloud computing, container orchestration, and high speed networks we can protect against many different system failure modes. With the addition of an “infrastructure as code” (IaC) approach we can iterate on our solutions to make sure we are always codifying and deploying based on best practices. We have applied these techniques to serve up a highly available Tezos RPC endpoint.
This tutorial will guide you in deploying your own Tezos rolling nodes on AWS cloud’s managed Kubernetes offering, EKS. The tool you will be using to do so is a Pulumi IaC Typescript node module. You will also expose your nodes’ RPC endpoint to the internet via a load balancer. The resulting infrastructure is highly available, scalable, customizable, and supports rolling upgrades. You can scale the number of Tezos nodes as desired. You can find the code for this tutorial on Github here.
Prerequisites
Our tutorial as well as the referenced Pulumi documentation go through the installation and configuration of the items in this list:
Note that as the time of this writing these are the latest versions of Pulumi CLI and K8s used in the tutorial:
Pulumi CLI: v3.61.0
K8s: 1.25Getting Started
We will be creating a Pulumi project which will contain the source code for our program deploying Tezos nodes. You can learn more about general Pulumi concepts here.
Follow along the Pulumi tutorial to create an AWS account if you don’t already have one, and install Pulumi. Your AWS IAM user must have the ability to create a VPC and EKS cluster:
Continue along to create your Pulumi project:
For our tutorial we will run the following, similar to what the Pulumi documentation shows:
mkdir tezos-aws && cd tezos-aws
pulumi new aws-typescriptAfter running the above commands, which will create the Pulumi project as well as ask you to log in to the Pulumi service, review the next page to get an understanding of the files Pulumi generated:
The Code
We will provide here in the tutorial some gists. The full code can be found on github.
Install the tezos-pulumi node module. This node module provides functionality to create both AWS and Kubernetes resources for your Tezos deployment. It utilizes Oxhead Alpha’s tezos-k8s Helm chart project to spin up a Tezos node in k8s.
npm install @oxheadalpha/tezos-pulumi --save-exactNote at the time of this writing the latest npm packages in the project are:
"dependencies": {
"@oxheadalpha/tezos-pulumi": "1.1.1",
"@pulumi/aws": "5.34.0",
"@pulumi/awsx": "1.0.2",
"@pulumi/pulumi": "3.61.0"
}If when going through this tutorial you run into Typescript issues, a difference in dependency versions may be the cause.
We provide you here with commented files containing the code to deploy your cluster:
values.yamlfile that specifies what we’d like the tezos-k8s Helm chart to deploy inside k8s. You may see here for the full tezos-k8s values spec but what we provide here is sufficient for the tutorial.index.tsFile which will deploy the Tezos infrastructure. It is very customizable as we can easily write IaC in languages like Typescript.ebsCsi.tsFile that creates the IAM role and policy for the EBS CSI Driver K8s addon.
Create a values.yaml file with the below yaml:
# Define the types Tezos of nodes we want to deploy in our cluster
nodes:
rolling-node:
storage_size: 100Gi
# We create a new k8s storage class in the Pulumi index.ts file that allows
# us to expand an EBS volume's size. The default gp2 storage class created
# by EKS does not allow for volume expansion. We tell our `rolling-nodes` to
# use this storage class.
storageClassName: "gp3"
# Run the Octez implementation of Tezos node.
runs:
- octez_node
# Create 2 Tezos rolling nodes that will be distributed across the 2 cluster
# EC2 nodes we will be deploying.
instances:
- config:
shell:
history_mode: rolling
- config:
shell:
history_mode: rolling
# Have the nodes download and use a tarball of a mainnet rolling node
# filesystem. This allows the nodes to sync much faster with the head of the
# chain than if they started from the genesis block. By default, tezos-k8s will
# download and unpack a Tezos native snapshot. A tarball is a LZ4-compressed
# filesystem tar of a node's data directory. It is faster to download and
# bootstrap the node. https://xtz-shots.io/ is the default source of the tar.
prefer_tarballs: truePaste the following code in the index.ts file generated by Pulumi, overwriting the Pulumi default generated code when the project was created. Read the comments to get an understanding of what is happening.
import * as aws from "@pulumi/aws"
import * as awsx from "@pulumi/awsx"
import * as eks from "@pulumi/eks"
import * as k8s from "@pulumi/kubernetes"
import * as pulumi from "@pulumi/pulumi"
import * as tezos from "@oxheadalpha/tezos-pulumi"
import createEbsCsiRole from "./ebsCsi"
/** https://www.pulumi.com/docs/intro/concepts/project/ */
const project = pulumi.getProject()
/** https://www.pulumi.com/docs/intro/concepts/stack/ */
const stack = pulumi.getStack()
const projectStack = `${project}-${stack}`
/** Create a vpc to deploy your k8s cluster into. The vpc will use the
* first 2 availability zones in the region. Public and private subnets
* will be created in each zone. Private, for cluster nodes, and public
* for internet-facing load balancers.
*/
const vpc = new awsx.ec2.Vpc(
projectStack,
{
numberOfAvailabilityZones: 2,
subnetSpecs: [
// Tag subnets for specific load-balancer usage.
// Any non-null tag value is valid.
// See:
// - https://docs.aws.amazon.com/eks/latest/userguide/network_reqs.html
// - https://github.com/pulumi/pulumi-eks/issues/196
// - https://github.com/pulumi/pulumi-eks/issues/415
{ type: "Public", tags: { "kubernetes.io/role/elb": "1" } },
{ type: "Private", tags: { "kubernetes.io/role/internal-elb": "1" } },
],
},
{
// Inform pulumi to ignore tag changes to the VPCs or subnets, so that
// tags auto-added by AWS EKS do not get removed during future
// refreshes and updates, as they are added outside of pulumi's management
// and would be removed otherwise.
// See: https://github.com/pulumi/pulumi-eks/issues/271#issuecomment-548452554
transformations: [
(args: any) => {
if (["aws:ec2/vpc:Vpc", "aws:ec2/subnet:Subnet"].includes(args.type)) {
return {
props: args.props,
opts: pulumi.mergeOptions(args.opts, { ignoreChanges: ["tags"] }),
}
}
return
},
],
}
)
/** Stack outputs: https://www.pulumi.com/learn/building-with-pulumi/stack-outputs/ */
export const vpcId = vpc.vpcId
export const vpcPublicSubnetIds = vpc.publicSubnetIds
export const vpcPrivateSubnetIds = vpc.privateSubnetIds
/** Create the EKS cluster. The cluster will be created in the new vpc. The
* autoscaling group will spin up 2 cluster nodes (EC2 instances) where they
* will be distributed across our 2 private subnets. Each subnet is in 1 of 2
* vpc zones.
*/
const cluster = new eks.Cluster(projectStack, {
version: "1.25",
createOidcProvider: true,
vpcId,
publicSubnetIds: vpc.publicSubnetIds,
privateSubnetIds: vpc.privateSubnetIds,
// At time of writing we found this instance type to be adequate
instanceType: "t3.large",
// Set `minSize` and `desiredCapacity` to 0 if you ever want to pause your
// cluster's workload.
minSize: 2,
desiredCapacity: 2,
})
/** Stack outputs: https://www.pulumi.com/learn/building-with-pulumi/stack-outputs/ */
export const clusterName = cluster.eksCluster.name
export const clusterId = cluster.eksCluster.id
export const clusterVersion = cluster.eksCluster.version
export const clusterStatus = cluster.eksCluster.status
export const kubeconfig = pulumi.secret(cluster.kubeconfig)
export const clusterOidcArn = cluster.core.oidcProvider!.arn
export const clusterOidcUrl = cluster.core.oidcProvider!.url
/** https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html */
const csiRole = createEbsCsiRole({ clusterOidcArn, clusterOidcUrl })
const ebsCsiDriverAddon = new aws.eks.Addon(
"ebs-csi-driver",
{
clusterName: clusterName,
addonName: "aws-ebs-csi-driver",
serviceAccountRoleArn: csiRole.arn,
},
{ parent: cluster }
)
/**
* The default gp2 storage class on EKS doesn't allow for volumes to be
* expanded. Create a storage class here that allows for expansion. Also use the
* gp3 type.
*
* https://aws.amazon.com/blogs/storage/migrate-your-amazon-ebs-volumes-from-gp2-to-gp3-and-save-up-to-20-on-costs/
*
* https://www.jeffgeerling.com/blog/2019/expanding-k8s-pvs-eks-on-aws
*/
const gp3ExpansionStorageClass = new k8s.storage.v1.StorageClass(
"gp3",
{
metadata: {
name: "gp3",
},
allowVolumeExpansion: true,
provisioner: "ebs.csi.aws.com",
volumeBindingMode: "WaitForFirstConsumer",
reclaimPolicy: "Delete",
parameters: {
type: "gp3",
fsType: "ext4",
},
},
{ provider: cluster.provider, parent: cluster }
)
/** We will use the cluster instance role as the default role to attach policies
* to. In our tutorial, the only policy will be the alb controller policy. */
const clusterInstanceRoles = cluster.instanceRoles.apply((roles) => roles)
const defaultIamRole = clusterInstanceRoles[0]
/**
* Deploy the AWS loadbalancer controller to manage the creation of the load
* balancers that expose your Tezos node. An application load balancer will be
* created for the RPC ingress. The IAM policy created for the controller is
* attached to the default cluster node role.
*
* https://github.com/kubernetes-sigs/aws-load-balancer-controller
*/
const albController = new tezos.aws.AlbIngressController(
{
clusterName: cluster.eksCluster.name,
iamRole: defaultIamRole,
},
{ provider: cluster.provider, parent: cluster }
)
const namespace = "mainnet"
/** Create the k8s namespace to deploy resources into */
const mainnetNamespace = new k8s.core.v1.Namespace(
namespace,
{ metadata: { name: namespace } },
{ provider: cluster.provider, parent: cluster }
)
/** Deploy the tezos-k8s Helm chart into the mainnet namespace. This will create
* the Tezos rolling node amongst other things. */
const helmChart = new tezos.TezosK8sHelmChart(
`${namespace}-tezos-aws`,
{
namespace,
// The path to a Helm values.yaml file
valuesFiles: "./values.yaml",
// The latest tezos-k8s version as of the time of this writing.
version: "6.19.0",
},
{
provider: cluster.provider,
parent: mainnetNamespace,
}
)
/** Create the RPC ingress to expose your node's RPC endpoint. The alb
* controller will create an application load balancer. */
export const rpcIngress = new tezos.aws.RpcIngress(
`${namespace}-rpc-ingress`,
{ metadata: { name: `${namespace}-rpc-ingress`, namespace } },
{
provider: cluster.provider,
dependsOn: albController.chart.ready,
parent: mainnetNamespace,
}
).ingress.status.loadBalancer.ingress[0].hostnameFinally, create the ebsCsi.ts file.
import * as aws from "@pulumi/aws"
/** https://docs.aws.amazon.com/eks/latest/userguide/csi-iam-role.html */
const createEbsCsiRole = ({
clusterOidcUrl,
clusterOidcArn,
namespace = "kube-system",
}: any) => {
const serviceAccountName = "ebs-csi-controller-sa"
const csiRole = clusterOidcUrl?.apply(
(url: string) =>
new aws.iam.Role("ebs-csi", {
assumeRolePolicy: {
Version: "2012-10-17",
Statement: [
{
Effect: "Allow",
Principal: {
Federated: clusterOidcArn,
},
Action: "sts:AssumeRoleWithWebIdentity",
Condition: {
StringEquals: {
[`${url}:sub`]: `system:serviceaccount:${namespace}:${serviceAccountName}`,
[`${url}:aud`]: "sts.amazonaws.com",
},
},
},
],
},
})
)
new aws.iam.RolePolicyAttachment("ebs-csi-pa", {
role: csiRole,
policyArn: "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy",
})
return csiRole
}
export default createEbsCsiRoleDeploy
Time to deploy our Tezos infrastructure!
Run pulumi up. Pulumi will first run pulumi preview which previews the changes Pulumi will make. Your output should look similar to the following screenshot:
Pulumi produces a graph consisting of all the resources that will be created. Some are AWS objects deployed by the AWS provider, some are Kubernetes objects deployed by the Kubernetes provider. Notice the AWS VPC for example. Pulumi will automatically create all the necessary resources that go along with it, like subnets and nat gateways. The EKS cluster will contain 2 EC2 nodes as specified in our code. Notice similarly the k8s TezosK8sHelmChart resource. It will create a rolling-node statefulset amongst other things. Once you are done with the tutorial, you can tear down all of these resources as explained below
For the sake of this tutorial you don’t need to be familiar with all of these resources, in terms of what they do and where they can be found in the AWS console or inside the k8s cluster. Still, it is important to keep in mind that what we are doing isn’t magic, and knowledge of AWS, k8s, and Pulumi is important to maintain and develop your infra.
Getting back to your terminal, you may select the details option to get a more in depth view of what Pulumi will do when it deploys.
Only after selecting yes will Pulumi do the actual deployment. Initial deployment (that creates a VPC and EKS cluster) can typically take 15–20 minutes or so to complete.
Pulumi will display a url that you can navigate to view the status of your deployment on the Pulumi web console. This is in addition to watching it in your terminal:
A successful deployment will finish with output that looks similar to this:
but the output is still similar.You can see in index.ts that these output values are exported.
Connecting to your Cluster
Install k9s: https://github.com/derailed/k9s
This is a wonderful CLI tool we can use to connect to our cluster and run commands against it. Please see the documentation there for how to navigate within the cluster.
Next we need a kubeconfig file which allows k9s to connect to our cluster’s api server and view our k8s resources.
# Get the cluster's kubeconfig and save it to a file
pulumi stack output kubeconfig --show-secrets --json > kubeconfig.json# Set the KUBECONFIG environment variable to point k9s to our kubeconfig.
export KUBECONFIG=./kubeconfig.json# Let's get into our cluster!
k9s
Once we are in, type: : pods and then hit 0 which will display all of the pods in all namespaces running in the cluster. Feel free to explore.
Lets navigate to our rolling-node-0 (in red). This is the first Tezos node of our rolling-node statefulset.
We can see the containers that run in the rolling-node-0 pod. Let’s go and check out the octez-node container that hasn’t synced with the head of the blockchain yet, and hence this container and pod are marked as unready to receive RPC queries.
We can see the logs of the Tezos node!
Go ahead and explore the logs for any other container in the pod. To exit the node logs, hit the escape button.
Next, let’s view our ingress that has an associated AWS load balancer (created by the ALB ingress controller resource), allowing us to query our Tezos nodes’ RPC endpoint. Type : ing to navigate there.
Notice the Address column. That is the url of the load balancer. Let’s curl it to get Mainnet’s chain id:
$ curl http://<ADDRESS>/chains/main/chain_id
"NetXdQprcVkpaWU"You can also run this to get the url:
pulumi stack output rpcIngressNOTE: If all of your nodes are still syncing, the ingress won’t be able to query them and hence you will not get a response. You may need to wait a few minutes for your nodes to fully sync. You can keep an eye on them with k9s.
Tearing Down your Deployment
tl;dr
Tear down your cluster in 2 stages. First, comment out / delete the rpcIngress resource from the code. Then run pulumi up. Next do the same thing for the remaining code.
Explanation
Notice in index.ts that the rpcIngress sets its dependsOn property to albController.chart.ready. What this does is set an explicit dependency of the rpcIngress component on the albController. We do this because Pulumi has no way of knowing that the ingress is dependent on the controller to have load balancers created. The controller is a k8s resource that creates load balancers outside of Pulumi’s purview.
Therefore, we want the ingress to wait for the controller to be fully ready. With the way Pulumi currently works and if an ingress doesn’t wait, Pulumi may error during the deployment due to the possibility of the controller not being ready while the ingress tries to resolve.
On deletion of the cluster, the first stage will tear down the ingress. This allows the controller to delete all of the backing AWS resources of the ingress, like the load balancer. The second stage will tear down the rest of the cluster and remaining Pulumi resources such as the VPC. If everything was torn down in a single stage, some AWS resources might be orphaned from your deployment stack. This is because a race condition might take place between the controller being deleted and the controller trying to delete the AWS resources.
(There are other ingress controllers such NGINX, that work differently than the ALB controller. Using them would avoid issues like needing to explicitly wait for them and having a 2 stage teardown. We haven’t yet implemented them though. Nevertheless an important point is made clear. That one should keep in mind that sometimes resources are external to the infrastructure itself, or there is an implied dependency due to an ordering or eventual consistency requirement. See dependsOn)
Notes
- This tutorials setup is more complex than just spinning up an EC2 instance and installing Tezos node, but there are many benefits. The setup’s resulting infrastructure is highly available, scalable, customizable, and supports rolling upgrades. This is due to the power of Kubernetes and Pulumi.
Kubernetes abstracts away many layers such as storage, networking, scaling, and load balancing. It monitors and manages the health of cluster nodes, pods, and containers. You can install directly into your cluster additional software such as Prometheus and Grafana.
Pulumi gives us the power to build, deploy, and manage everything using developer friendly code. - Using this infrastructure implementation is a financial commitment. Please be aware of costs.
- If you would like to keep your infrastructure up but want to pause your workload, you may choose to scale down your cluster nodes to 0. This will save on EC2 costs.
- Increase the number of cluster nodes and/or Tezos nodes if you have scaling issues. You can also modify the cluster node instance type. Updates will happen in a rolling fashion.
- This RPC endpoint is an AWS generated URL. It is possible to deploy the RPC behind a domain name. This would require buying a domain on AWS or importing your domain from another source into Route53. You would then need to issue certificates using AWS Cert Manager. This can be done using Pulumi components to configure the certs. You can also use a k8s controller like cert-manager. K8s external-dns controller can be used to automatically create Route53 records. All of this is out of scope of this tutorial.
- Some updates to statefulsets will cause Pulumi to do a delete-replace operation, which deletes the statefulset and hence all of its pods simultaneously. There will be no pods to receive traffic until the statefulset is recreated and the nodes fully sync. As opposed to an update operation where the pods will update in a rolling fashion, starting with the highest ordinal pod. An example of a case where a delete-replace would take place would be when modifying the storage size of the
volumeClaimTemplatesof a statefulset. (See this article about expanding volume sizes of pods.) Being that you can’t update this property on a k8s statefulset, Pulumi is forced to delete the statefulset and recreate it. A case where an update would take place would be where a Docker image is modified, which results in a rolling update.
For a delete-replace operation, we recommend that you orphan the pods from the statefulset manually using kubectl before running Pulumi. This way the statefulset will be deleted but the pods will still exist. Then Pulumi will recreate the statefulset and the pods will be associated with it. The command would look like this:kubectl delete statefulset rolling-node --cascade orphan --namespace mainnet.
