Boundary is an identity-aware proxy solution that provides a simple, secure way to access hosts and critical systems on your network and Vault is HashiCorp’s comprehensive secrets management solution. The typical workflow would be like:

• The Boundary user, through CLI/Desktop app, authenticates to Boundary Control Plane using an identity provider.
• Once authenticated, based on their role in the organization, they are shown certain targets which they can connect to.
• Once they connect to a target, the Boundary Control Plane gets the required dynamic credentials from Hashicorp Vault.
• These dynamic credentials are either brokered or injected in Boundary workers and a proxy connection is made to the target.

One of the key advantages of this approach is that you will not be specifically bridging the client onto the private network, so, we stay in line with sort of a zero-trust network philosophy where we’re not bridging users onto this private network.

However, as with any technology, adopting and scaling Boundary across large or complex environments comes with its own set of challenges.

Let’s take a quick look at some of these challenges. So, during adoption phase, you’ll notice that the credentials are still exposed to the users in case they’re using credential brokering feature or they still end up leveraging static SSH keys from Vault’s KV secret engine. Whereas during scaling phase, you often struggle with accountability and traceability issues in case of shared-user Linux accounts. Also, you might see some resource sprawling in case of multi-user Linux accounts. And then we can’t forget about automating these Boundary and host configurations when you’re talking about thousands of machines which you need to manage.

Challenge 1 : Credentials still exposed to the users

Organizations often using Boundary OSS rely on credential brokering. However, this approach lacks the ability to fully hide credentials from users. When a user attempts to connect to a target, Boundary’s control plane retrieves the necessary credential from Vault. Unfortunately, this credential is then visible to the user, allowing them to copy and use it to connect to the target. Even if the credential is dynamically generated, it is still exposed to the user, which poses potential security risks.

This is where credential injection comes into play. As part of this feature, when a user attempts to connect to a target, Boundary’s control plane retrieves the necessary credential from Vault. But this time, instead of exposing credentials to the user, the Boundary control plane directly injects the credential to the worker and the session is established. This gives a passwordless experience to the user. This is an Enterprise/HCP only feature.

Challenge 2 : Using static SSH keys from Vault’s KV secret engine

We can solve this issue with SSH certificates. With SSH certificates, the user’s authenticity is based on possession of a certificate signed by the trusted certificate authority (CA). Instead of having to provision and deprovision keys across your infrastructure, SSH certificates allow you to specify how long a certificate is valid for or which users the certificate holder can login as. In this workflow:

• Boundary generates SSH key-pair and sends to Vault
• Vault signs the certificate and sends back to Boundary control plane
• Control plane injects credentials to worker

Demo for passwordless authentication

Please follow the demo in below talk which I presented during HashiConf 2024 in Boston.

From an organization’s perspective, there may be good reasons for sharing usernames on the hosts, such as making it easier to configure unix permissions, configuring common reusable scripts, resource limits or maybe simplifying the overall provisioning of the hosts.

• Different Boundary users are authenticating using one single shared-user.
• A single credential library is created in a credential store with shared user as username. This causes accountability and traceability issues.

To solve this, you can enable SSH Recording feature.

• With SSH Session Recording feature, every session recording will be associated with a user.
• Create a storage bucket with a given provider and then enable session recording at target level.

Challenge 4 : Resource sprawling issues with multi-user Linux accounts

I have created a dedicated blog on this issue

Challenge 5 : Struggling with automating configurations?

• With credential injection, you need to add public key of CA (Hashicorp Vault) in every host/target.
• Creating Boundary projects, targets, credential stores and libraries can be difficult while scaling.

To solve this:

• Use HashiCorp Packer to create golden images with public key of Vault (as a CA), embedded in the image.
• Use HashiCorp Terraform’s Boundary provider to configure all resources within Boundary

Conclusion

So, if we have to summarize this blog, in order to introduce a robust solution for dynamically creating and managing SSH certificates at scale, you need to:

• Enable SSH Certificates in HashiCorp Vault
• Use HashiCorp Packer to embed the public key of CA in the golden image to spin up the SSH machines.
• Use HashiCorp Terraform to automate all Boundary configurations.
• Enable dynamic credential templatiing for credential libraries.
• Introdduce SSH credential injection.
• Enable SSH recording for all SSH targets.

Originally published at https://japneetsahni.com on December 1, 2024.

Background

• Is your organisation using Hashicorp Boundary as a PAM (privileged access management) solution for thousands of hosts residing in private network?
• Are you using Hashicorp Vault for generating dynamic secrets for multiple users and hosts?
• Have you faced a challenge of maintaining user specific targets and credential libraries in Boundary and eventually ended up in a resource sprawl?

In this article, I am going to highlight the solution using dynamic credential templating in Hashicorp Boundary which will help in avoiding resource sprawl.

Problem Statement

• In Boundary, a credential store is a resource that can retrieve, store, and potentially generate credentials (like Hashicorp Vault).
• Credential Store will contain credential libraries pointing to specific paths within Vault.
• These credential libraries are then mapped to Boundary targets which allows a Boundary user to connect to a host residing in private network. Now, if Vault is making use of a secret engine where we have defined user-specific roles like SSH-OTP (for linux servers) or LDAP (for Windows servers), in Boundary, we end up creating user-specific credential libraries pointing to user-specific Vault paths as shown below. This leads to resource sprawl within Boundary, resulting in hundreds to thousands of individual credential libraries at scale.

In Boundary 0.12, support for credential templating within credential libraries was added. This allows Boundary administrators to configure one target with one credential library that generates per-user credentials. hence, you don’t need to maintain a target for each user as shown below. The paths in these credential libraries can be mapped to Boundary user’s or account’s information as highlighted here. The user’s/account’s information is dynamically populated while accessing credentials.

Code Snippet & Snapshots (Before vs After)

Before : Using Static Credential Libraries and Targets

# Static Linux Host Set
resource "boundary_host_set_static" "linux_servers_ssh" {
  type            = "static"
  name            = "linux_servers_ssh"
  description     = "Host set for linux servers"
  host_catalog_id = boundary_host_catalog_static.linux_servers_catalog.id
  host_ids        = [for host in boundary_host_static.linux_servers_host : host.id]
}

# User specific credential libraries pointing to user-specific Vault paths.
resource "boundary_credential_library_vault" "cl_vault_ssh" {
  name                = "cl-ssh-${each.key}"
  for_each            = var.linux_users
  credential_store_id = boundary_credential_store_vault.cs_vault.id
  path                = "ssh/creds/${each.key}"
  http_method         = "POST"
  http_request_body   = <<EOT
{
  "ip": "1.2.3.4"
}
EOT
}

# 1:1 mapping between user-specific targets and user-specific credential libraries
resource "boundary_target" "linux_servers_target" {
  for_each     = var.linux_users
  type         = "tcp"
  name         = "linux-target-${each.key}"
  description  = "Linux Server Target"
  scope_id     = boundary_scope.project.id
  default_port = "22"

  host_source_ids = [
    boundary_host_set_static.linux_servers_ssh.id
  ]
  brokered_credential_source_ids = [
    boundary_credential_library_vault.cl_vault_ssh["${each.key}"].id
  ]
}

After : Using dynamic credential libraries and targets

# Credential Store for Vault
resource "boundary_credential_store_vault" "cs_vault" {
  name            = "cs-pam-vault"
  description     = "Vault for PAM"
  address         = var.vault_address
  token           = var.vault_token
  scope_id        = boundary_scope.project.id
}

# Dynamic Credential Library for Vault SSH-OTP paths
resource "boundary_credential_library_vault" "cl_vault_dynamic_ssh" {
  name                = "cl-dynamic-ssh"
  credential_store_id = boundary_credential_store_vault.cs_vault.id
  path                = "ssh/creds/{{ .User.Name }}"
  http_method         = "POST"
  http_request_body   = <<EOT
{
  "ip": "1.2.3.4"
}
EOT
}

# Dynamic Host Set for every team
resource "boundary_host_set_plugin" "dynamic_host_set_team" {
  name            = "dynamic-host-set-${var.team}"
  host_catalog_id = var.host_catalog_id
  attributes_json = jsonencode({
    "filter" = "tagName eq 'team' and tagValue eq '${var.team_tag}'",
  })
}

# Dynamic Target for every team
resource "boundary_target" "dynamic_target_team" {
  type                           = "tcp"
  name                           = "dynamic-target-${var.team}"
  description                    = "Dynamic Target for team - ${var.team}"
  scope_id                       = var.project_id
  default_port                   = 22
  host_source_ids                = [boundary_host_set_plugin.dynamic_host_set_team.id]
  brokered_credential_source_ids = boundary_credential_library_vault.cl_vault_dynamic_ssh.id
}

Conclusion

Due to dynamic credential templating, you can very easily create managed groups in Boundary and assign team-specific targets mapped to dynamic host catalogs and single dynamic credential library path using user’s information as shown in the above code snippet.

For more insights and to get notifications on upcoming content, follow me on LinkedIn and Medium

Originally published at https://japneetsahni.com on February 16, 2023.

Background

• Is your organisation using HCP Boundary as a PAM (privileged access management) solution for hosts residing in private network across multiple cloud providers?
• Does your organisation has strict networking constraints like resources residing in private network having only outbound access?
• Does your organisation require to use your own Boundary workers instead of HCP workers?

In this article, I am going to highlight and demo the solution using multi-hop sessions with HCP Boundary and how you can access resources in private networks across Azure and AWS.

What we’ll be seeing in this article/demo?

• Working with self managed PKI ingress workers (Installation & Registration)
• Creating worker-aware targets
• Connecting to targets using Boundary Desktop Application
• Challenges with ingress workers
• Introducing multi-hop sessions with HCP Boundary
• Working demo of multi-hop sessions

Working with self managed workers

Self-managed workers allow Boundary users to securely connect to private resources without exposing their organizations’ networks to the public. Self-managed workers use public key infrastructure (PKI) for authentication. PKI workers authenticate to Boundary using a certificate-based method

Networking requirements

• Outbound access to HCP Boundary Control Plane (worker -> HCP control plane)
• Outbound access to the private target (worker -> host)
• Inbound access from user trying to establish the session (user -> worker on port 9202)

Installation of Ingress worker

Assuming you have a running HCP Boundary cluster, follow below steps:

• Create Azure VM and AWS EC2 linux machines with just private IP (no public IPs associated).
• Create Azure and AWS Ingress workers in the respective cloud providers using below steps:
• Install boundary-worker utility on both workers

mkdir /home/japneet/boundary/ && cd /home/japneet/boundary/ 
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add 
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main" 
sudo apt-get update && sudo apt-get install boundary-worker-hcp -y-

• Add pki-worker.hcl on both workers at path /home/username/boundary. Replace value of hcp_boundary_cluster_id with actual value.

./boundary-worker server -config= "/home/japneet/boundary/pki-worker.hcl"

Registration of Ingress Worker

• Go to HCP Boundary cluster URL and authenticate with admin credentials.
• Once logged in, navigate to the Workers page.
• Click New and scroll down to the bottom of the New PKI Worker page and paste the Worker Auth Registration Request key you copied earlier.
• Click Register Worker.
• Follow the above steps for next Ingress worker.

Awesome ! You now have two self-managed PKI Ingress workers (one for Azure and one for AWS) registered in HCP Boundary cluster. But how do you tell which ingress worker to use while connecting Azure or AWS target? Let’s see this now.

Creating worker-aware targets

• Open the HCP Boundary Admin Console UI and create a organisation called “multi-hop” followed by a project called “multi-hop”.
• Navigate to the targets page within the multi-hop project.
• Create 2 targets (one for Azure and one for AWS) with following details:

— Target name (aws-target and azure-target)

— Target address (private IP of AWS EC2 and Azure VM respectively)

— Default port 22

— Toggle the Ingress worker filter switch to add an ingress worker filter that searches for workers that have the following tag:

Connecting to targets using Boundary Desktop Application

With the ingress filters assigned above, any connections to target will be forced to proxy through the assigned ingress worker.

• The end user authenticates to HCP Boundary cluster using Boundary Desktop application.
• The user navigates to Targets section and will the authorized targets based on his roles and permissions.
• The user connects to the aws/azure target using “Connect” button.
• A proxy URL is published which user uses to establish a session

ssh username@127.0.0.1 -p <port mentioned in proxy url>

Challenges with Ingress workers

• The Boundary ingress workers have to be accessed by user/clients which means either those users need to have some VPN access or the ingress worker is publicly accessible using a public IP. Some organisations may not allow this.
• Other organisations might give access to boundary ingress workers residing in public network but resources are in private network with only outbound access, hence boundary ingress worker won’t be able to proxy the connection.

Managing multi-hop sessions with HCP Boundary

In order to mitigate the above networking challenges with some organisations, 0.12 release of HCP Boundary introduced a feature of multi-hop sessions wherein now you can configure different types of Boundary workers based on their networking requirements and eventually create a chain of workers through reverse-proxy connections. This creates multiple “hops” in the chain from a worker to a controller.

• The new type of worker which is introduced as part of this feature is called an “Egress” worker.
• This worker has network access into the hosts and connects to an existing upstream/ingress worker which is accessible to the clients.
• The “Egress” worker can reside in a private network with only outbound access and inaccessible to the clients.

Installation of Egress Worker

• Create 2 Egress workers (one in each cloud) with private IP’s and only outbound access.
• Install same boundary-worker utility similar to Ingress worker.
• Add pki-worker.hcl on both workers at path /home/username/boundary. Note initial_upstreams parameter which tells which ingress worker to connect to.

./boundary-worker server -config= "/home/japneet/boundary/pki-worker.hcl"

Registration of Egress Worker

Follow same steps as ingress worker for registering the egress worker.

Modifying worker-aware targets

• Open the Boundary Admin Console UI navigate to the targets page within the “multi-hop” org.
• Click on the aws-target/azure-target. Scroll to the bottom the the target details page and click Edit Form.
• Toggle the Egress worker filter switch to add an egress worker filter that searches for workers that have the following tag:

Working Demo

Conclusion

Awesome ! You have now connected to a target residing in a private network with only outbound access by creating a chain of “ingress” and “egress” workers and establishing a “hop” in the session.

For more insights and to get notifications on upcoming content, follow me on LinkedIn and Medium

Originally published at https://japneetsahni.com on May 15, 2023.

In this context, I’ll discuss how HashiCorp Boundary and Vault come to the rescue and how their integration helps us achieve the core fundamental of Zero Trust Security: “Trust Nothing. Authenticate and Authorize Everything.

Traditional workflow for privileged access management (PAM)

Let’s say we have to access a private resource, it can be a database or a virtual machine and these machines reside on private datacenter on-premises or any private network of a cloud service provider.

Now a user, let’s take an example of a database administrator who wants to access the database in this private network, he is ideally not on the private network, they’re either working in their office or they’re working in their home.

• VPN/Bastion Hosts: Now, if this user has to connect themselves to the private network, first of all, the organisations set up a VPN gateway or a bastion host. In case of VPN, they might get an IP address that puts them on the network, or in the case of bastion host, they need to get their SSH keys configured.
• Firewalls: Once, we are inside the private network, we can literally access everything. So in practice, what we probably also do is, have some form of a firewall setup that’s going to restrict traffic. So, in our case, we’re going to create a firewall that says, ok, traffic can go from our bastion host or from our VPN to this database residing in the private network.
• Application Credentials: And finally, once the user connects all the way through to the database, they also need a username and password to connect to it.

If we consider this workflow, you will notice that there’s a bunch of information the user needed.

1. Firstly, the user needed to have VPN or SSH credentials.
2. Secondly, they needed to know the IP or the host name to connect to.
3. Then finally, they actually needed the database credentials to get all the way through to that system.

Challenges with traditional workflow

So, let’s see what are the challenges with this traditional workflow.

• Onboarding/Off-boarding: The very first one being the onboarding and off-boarding of users. I don’t want to distribute VPN credentials and SSH keys to everyone who joins the company or everyone who leaves the company. This would definitely become a burden, especially if we consider best practices like key rotation on a regular interval.
• Increasing the attack surface: Secondly, once the user has VPN access or access to Bastion Host, he/she literally has a wider network access thus increasing the attack surface. As seen in last slide, we will try to bring firewalls in order to mitigate this issue.
• Managing IP’s in dynamic environment: Now this firewall setup is fine unless and until we have small number of static hosts where IP’s aren’t changing much but this wouldn’t scale when we have a highly dynamic infrastructure especially that auto-scales. Also, nowadays we are moving to ephemeral platforms like Kubernetes, where if a node dies, applications might be migrated to a different node. We’re also adopting more serverless things where you really don’t think about a host or a node or an IP anymore. It’s really about this logical service that you’re trying to route to. With all of these, we’re really moving away from relatively static infrastructure to much more dynamic infrastructure.
• Static Credentials: Lastly, if I have to expose database credentials directly to my user, what happens if those get exposed or compromised somehow? I don’t want to have a static, long-lived credential that gets leaked. Someone might store it on their laptop, or they put it in a confluence doc which might be exposed to Internet, or they paste it into Slack conversations or might even get saved into a log file somewhere. Ideally why should I even give those credentials directly to an end user?. So, the question is how do we move away from static credentials and instead create a credential on-demand that’s short-lived and we revoke it when we don’t need it anymore? So there are a number of challenges in this workflow and if we have to move towards zero trust security philosophy wherein I really don’t have to trust my private network, I definitely need to do something differently.

How Boundary-Vault Integration comes to the rescue?

So, let’s see how boundary vault integration solves all the challenges we discussed with the traditional workflow.

• Authentication Using IDP: Firstly, with Boundary, every user is going to login through Identity provider. If your organization already uses Azure AD or Okta or any other IDP, the Boundary administrator can configure the same IDP for authentication. Once authenticated using this IDP, you prove that you are a valid user. Now, when we talk about on-boarding and off-boarding of users, it is just about adding or removing them from the IdP. There’s no additional step of distributing VPN or SSH credentials or rotation of SSH keys. Thus you solve the authentication very easily with Boundary.
• Authorization based on roles: Now, once the user has actually authenticated, there’s going to be an authorization based on certain set of roles and permissions. So we’re going to have a role and that role is really going to drive role-based access control. For example in our case, our database administrators, they get access to the databases and Web developers get access to web tier. But we aren’t going to specify this role for a single host or a single IP, it’s actually going to be at the logical service because that’s what we really care about now. This is really important because the logical service lets us elevate from the details that are dynamic. You can basically differentiate between these logical services in a cloud service provider using tags. So if, the database administrators have to connect to the database, the set of databases can be dynamic, the IPs can come and go, they can change, they can be scaled up and down but the overall control stays the same.
• Dynamic Services: Now, once the user sees his host, and clicks on Connect, Boundary will directly establish a connection to the target system. One of the key advantages of this approach is that you will not be specifically bridging the client onto the private network. So, we stay in line with sort of a zero trust network philosophy where we’re not bridging users onto this private network.
• Dynamic Credentials from vault: Finally, let’s talk about those scary static credentials. This is where Hashicorp Vault comes into picture. Vault is a secrets management tool from Hashicorp which is capable of generating dynamic and short-lived credentials through its different secret engines. These secret engines can generate dynamic credentials for any kind of a database, it can generate/rotate LDAP passwords for Active directory or it can even generate OTPs for linux machines. These credentials are tied to a TTL or in some cases can be used only once. Boundary integrates with Vault seamlessly to broker the credential which will then be used when the user connects to the target system using Boundary.

Demo to understand use-cases for accessing Windows & Linux using Boundary & Vault

I presented a talk on Zero Trust Security using Hashicorp Vault and Boundary in HashiTalks, 2023

Use Case 1 : Accessing Domain Joined Windows Servers using OpenLDAP secret engine

Use Case 2 : Accessing Linux servers with local accounts using SSH-OTP secret engine

Conclusion

So, if we have to summarize this blog, you will notice that

• Boundary users will authenticate to Boundary using an Identity provider.
• Once authenticated, they will see only certain set of hosts or targets which they are authorized to see courtesy roles and permissions.
• Finally they make a proxy connection to the target system using Boundary and gets the dynamic credentials for the target system from Hashicorp Vault. These set of credentials will either expire after certain TTL or can be configured to be used only once.

For more insights and to get notifications on upcoming content, follow me on LinkedIn and Medium

Originally published at https://japneetsahni.com on February 16, 2023.