HCP Boundary Multi-Hop Deployment with Terraform
Boundary is HashiCorp’s newest product, which enters the privileged access management (PAM) arena for cloud-native, dynamic workloads. As a HashiCorp Solutions Engineer, I have seen a growing call and interest in Boundary — how it can help address current customer problems and also how it integrates with the rest of the HashiCorp stack.
From 0.12, Boundary solved the problem where inbound network traffic is prohibited into private networks. The notion of multi-hop sessions, which allows for multiple PKI workers to be daisy chained together through reverse-proxy connections, solves this. With workers residing in a private network, we can therefore ensure that resources do not have to be exposed publicly.
In this blog I will walkthrough a multi-hop deployment with ingress and egress workers to facilitate secure remote access to a resource in a private network, all driven through Terraform.
This method is not an official HashiCorp-recommended best practice. It’s just one example of how it can be achieved. The code for the deployment can be found here.
This blog assumes basic knowledge of Boundary and its constructs. If not, the following link provides all relevant information pertaining to Boundary: https://developer.hashicorp.com/boundary
HCP Boundary multi-hop workers
The introduction of multi-hop brought the notion of “upstream” and “downstream” workers. The HCP Boundary (HCPb) controllers reside at the top of the multi-hop chain, with additional workers underneath forming that downstream chain. Furthermore, each worker can have filters applied to specify which workers are to handle the initial connection prior to a session being established, and which workers are actually used to access targets — these are defined as being either “ingress” or “egress”.
This feature is important when we want to have control around inbound connectivity. One doesn’t want to have to expose target resources in a private network by adding a public IP address in order to facilitate an SSH connection, as an example. With multi-hop, we can achieve daisy-chaining through reverse proxy connections. That results in an egress Boundary worker residing in a private subnet, only allowing inbound TCP 9202 from its upstream worker, for communication purposes between upstream workers and controllers. Resources within this private subnet remain private and rightly so.
Currently, multi-hop workers are only supported as PKI workers, which authenticate to Boundary using a certificate-based method. This differs from the KMS workers, which use a shared KMS to authenticate with the controllers.
Deployed architecture
Before walking through some key areas to focus on in a HCPb multi-hop worker deployment, it’s useful to visualise what we will be creating.
We have two VPCs deployed, and our target resides in a private 192.168.x.x subnet with an associated Boundary egress worker, which is assigned an IP address out of the 192.168.x.x pool. A Transit Gateway facilitates the communication between the two VPCs, with a static default route pointing towards the private 172.x.x.x subnet. When the user issues a connection to the target, it will flow via the Internet Gateway, NAT gateway, ingress worker, Transit Gateway and finally egress worker which is associated with the target.
Registering workers
So let’s look into the configuration of the two self-managed workers in the example above. There are two methods to register self-managed Boundary workers and bring them into the cluster; worker-led authorisation and controller-led authorisation.
Worker-led authorisation
During the installation of the boundary-worker binary, the worker will create an authorisation token. It will print this out during the installation, and it is also written to a file called auth_request_token. This can be copied and then pasted into the Boundary controller, as shown in the below workflow diagram.
From an operational perspective, there is still manual effort here — having to copy the token and then paste it into the Boundary controller. This is why I prefer the Controller-led workflow with Terraform.
Controller-led workflow
With this, the Boundary controller will generate a single-use token, which the self-managed workers will use for authorisation. Combining this workflow with Terraform means that a self-managed worker can be created and registered with the controller without any human intervention. We’ll now dive deeper into how we can achieve this.
Boundary & Terraform
As with all of the HashiCorp products, everything can be created and managed via Terraform — Boundary is no different and has an official provider to consume. I am not going to cover the general AWS resource creation, networking, or routing, as this should be familiar to most, but the worker configuration is where we shall focus.
Authorisation methods
Returning back to the authorisation methods, the below Terraform resource is what’s required for a controller-led authorisation workflow. The worker_generated_auth_token should be left blank here. Upon doing so, it will result in the controller generating a token.
resource "boundary_worker" "ingress_pki_worker" {
scope_id = "global"
name = "bounday-ingress-pki-worker"
worker_generated_auth_token = ""
}That’s step 1 accomplished! We now know we are going to get a generated token to pass to our self-managed workers. The next step is to take that and actually give it to the worker during its installation process.
Ingress & egress PKI worker HCL — Controller token
Each worker requires an .hcl file to provide the configuration. It is within this file that we need to pass the one-time generated controller token. When this configuration file is executed, the self-managed worker will be automatically registered within the Boundary controller. It is irrelevant whether the worker is ingress or egress, each one requires a token. However, there are aspects of the configuration that do differ between the two types of workers and we will go through the differences later.
First, I used a combination of Terraform locals and cloud-init to obtain the desired result. The snippet of code below illustrates the process of passing the controller generated token into the worker configuration file. For brevity there are other parameters removed, but I have attached the link to the GitHub repository in this blog, which contains the holistic code deployment.
worker {
controller_generated_activation_token = "${boundary_worker.ingress_pki_worker.controller_generated_activation_token}"
}The boundary_worker resource earlier in this blog has a list of read-only attributes that you can reference. One of these is controller_generated_activation_token and is what we can reference to pass the token into the configuration file. That’s the next part taken care of — we have added the configuration to pass the controller generated activation token into our worker configuration file. As this worker is the ingress and the one that has ‘line of sight’ to the Boundary controllers, we now need to ensure we specify the Boundary cluster ID.
Ingress PKI worker HCL — Cluster ID
The Boundary cluster ID forms part of the URL to sign into Boundary. From an HCPb perspective, the URL resembles something like:
https://123456ab-78c9-12de-f345-678g912h3i4j.boundary.hashicorp.cloud
However, we are only interested in the UUID part of the URL: 123456ab-78c9–12de-f345–678g912h3i4j. This is all we need for our ingress worker. Unlike the previous configuration to define the token, the below code does not sit in a dedicated stanza:
hcp_boundary_cluster_id = "${split(".", split("//", var.boundary_addr)[1])[0]}"The split() function produces a list by dividing a given string at all occurrences of a given separator. From here we specify two values.
[1] ensures that we skip the https:// and look at the rest of the string as we specified a split after “//”. The following [0] references the UUID part of the string as we are also wanting to split using “.”, which gives us what we’re interested in. A great way to ensure you have the desired result of a split() is to test using the Terraform console command.
Egress PKI worker HCL — Cluster ID
I mentioned previously that there are slight nuances between ingress and egress worker configuration files — cluster ID being a difference. As the egress worker is downstream from the ingress, we are daisy-chaining off from the ingress. Therefore, we do not need to specify the cluster ID again here. Instead, we include an initial_upstreams = [] construct to make reference to the upstream worker.
Our ingress worker sitting in our public VPC has both a public IP address and a private IP address. It is the private address that must be referenced as the initial upstreams.
It is also worth noting that within the worker stanza, for both ingress and egress configuration files, there is a public_addr construct, which is fine for our ingress worker as that has a public IP. For our egress worker, we don’t have a public IP, so we can just specify our private 192.168.x.x address here. We also must remember to include :9202 as that’s the TCP port the workers use to communicate. An example of an egress worker configuration would be as follows:
worker {
public_addr = "192.168.0.8:9202"
initial_upstreams = ["172.31.32.93:9202"]
}Now we have got to this point we have some of the important ingress and egress worker configuration complete. We can move onto the last important piece before we’re ready to deploy and connect.
Worker tags
Tags are used to define the type of worker and to make it distinguishable amongst other workers. You have the ability within Boundary to specify which workers you wish to connect to. An example being you have two different subnets — one subnet contains databases and the other, servers. An egress worker is deployed in each and when you create your Boundary target(s) you want to ensure you’re using the correct worker to reach the intended target(s). In a multi-hop deployment you need to specify both ingress and egress worker filters in your Boundary configuration, to ensure the end-to-end connectivity, with the correct workers, is in place.
Again, in our worker stanza within the worker configuration file, you add the tags as follows for an ingress worker (you can choose your own tags to reference the workers as you wish):
worker {
tags {
type = ["sm-ingress-upstream-worker1", "upstream"]
}
}You do the same for the egress worker:
worker {
tags {
type = ["sm-egress-downstream-worker1", "downstream"]
}
}This means we can now use those tags when we define our targets in Terraform:
resource "boundary_target" "aws_linux_private" {
type = "tcp"
name = "aws-private-linux"
description = "AWS Linux Private Target"
egress_worker_filter = " \"sm-egress-downstream-worker1\" in \"/tags/type\" "
ingress_worker_filter = " \"sm-ingress-upstream-worker1\" in \"/tags/type\" "
scope_id = boundary_scope.project.id
session_connection_limit = -1
default_port = 22
host_source_ids = [
boundary_host_set_static.aws-linux-machines.id
]
}Time to connect!
The final piece is now to actually connect to the target! In future blogs I will document using the Boundary credentials store natively and with HashiCorp Vault, for brokered and injected credentials. For this example I just have my .pem file locally that I’m going to reference. It is worth noting that the -target-id of the target we want to get to, can be easily obtained from the Boundary client.
boundary connect ssh -target-id=ttcp_0KX9RI7w87 -- -l ec2-user -i ec2-key.pemNow when we connect via Boundary to the AWS Linux target in the private VPC, Boundary knows which ingress and egress workers to use as a result of the tags, and we establish connectivity!
The authenticity of host 'hst_z19d5ymgvd ([127.0.0.1]:53846)' can't be established.
ED25519 key fingerprint is SHA256:F6zpFsQd6k47Ta3ZdJRvz4GIuVt09HIGZbHy/OittEk.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'hst_z19d5ymgvd' (ED25519) to the list of known hosts.
__| __|_ )
_| ( / Amazon Linux 2 AMI
___|\___|___|
https://aws.amazon.com/amazon-linux-2/
[ec2-user@ip-192-168-0-8 ~]$ Summary
By removing the requirement to have dedicated VPNs and the operational overhead of managing and maintaining firewall rules for access to targets, Boundary provides a much safer solution. As we do not have to give users complete access to the network anymore, we’re also reducing our attack surface, to give us better security posture.
With Boundary and the integration with Terraform, not only have you got a safer workflow, but also a more consistent one. To get started with Boundary and discover other powerful integrations, such as leveraging Vault to provide just in time, dynamic, ephemeral credentials, check out the documentation here.
To try this solution for yourself, the GitHub repo can be found here. Enjoy!
