Advanced Data Protection with HashiCorp Vault

Introduction

Organizations store sensitive, personal and valuable data, which must be protected. Leakage of such data can lead to financial loss, reputation risk, legal ramifications and more.

In this blog you will learn why data protection is important, which personas are typically involved to solve data protection challenges, what challenges organizations face to protect their data and last but not least how organizations can leverage HashiCorp Vault to solve these challenges.

Why is Data protection Important

According to IBM’s Cost of Data Breach Report the average cost of a data breach in 2021 is $4.35M.

Data breach in practice example: ABC Company was stuck with internal developers performing their own encryption operations on sensitive PCI-DSS data. The polyglot applications at ABC Company required developers to learn different encryption operations for different programming languages. As a result of this burden, their average number of deployments per week was down 24% and key application features were significantly delayed resulting in a customer retention rate decrease of 4%. The lack of a consistent output value resulted in a 13% productivity reduction with the DBA team as they scrambled to change database schemas to fit these output values of varying lengths. Worst of all, ABC Company failed an audit in 2018 failing to meet FF3–1 costing them $1.2M.

Personas

We can identify different personas and decision makers responsible and involved for rolling out data protection solutions and implementations in every organization and they all have different goals and expectations.

• CISO and the security teams
  A CISO and the security teams must ensure that the organization and the applications are compliant and audited. They are responsible for security after all and hence, they want to reduce risk by having more control and transparency.
• CTO and IT Managers
  The CTO and IT Managers are more focused on the cost and productivity of the implementation. They must ensure that they use and build standards and those are consistent across the organization. They are also responsible for time to market. Hence, it is also in their interest to enable their developers by offering them the right tools and processes.
• Developers
  Developers love and expect APIs, ease of use and simplicity.

Challenges

Data in most applications must be encrypted, but deploying cryptography and key management infrastructure is expensive, hard to develop against, and not cloud or multi-datacenter friendly.

The following challenges organizations typically face are:

• Increasing Risk
  With multiple attack surfaces to intercept and steal sensitive data.
  HashiCorp Vault enables you to encrypt sensitive data using centrally managed, audited and secured encryption keys. But more importantly all of this can be achieved through a single workflow and APIs.
• Reduced Productivity
  With multiple workflows/APIs to learn cryptographic standards across an organization and different projects and restricted access to HSMs. With HashiCorp Vault, you can create consistent workflows and cryptographic standards across your organization.
• Increasing Costs
  Procuring and deploying new key management infrastructure, Hardware Security Modules (HSMs) and support can be expensive. HashiCorp Vault can help reduce hardware costs related to multiple key management infrastructure solutions, HSMs, licensing and support.

How to Protect Your Data

HashiCorp Vault’s Transform secrets engine, part of the HashiCorp Vault Enterprise Advanced Data Protection (ADP) module, can encode and decode sensitive values residing in external systems such as databases or file systems. This capability allows Vault to ensure that encoded values remain safe even if they are exfiltrated from a compromised system. It does so while retaining this data’s high availability and adherence to compliance requirements such as PCI, DSS, and HIPAA.

Vault’s Transform secrets engine does not actually store the protected secret. Instead it protects only the key material necessary to decrypt the secret’s ciphertext. This maximizes encode/decode performance for applications, while also minimizing the possibility of exposure of that secret.

Vault’s ADP module seeks to secure customer data regardless of where it lives in your infrastructure. Whether it be through disk/volume encryption or two-way transformations such as tokenization, Vault ADP minimizes disruption while providing state-of-the-art data security.

Protect data such as social security numbers, credit card numbers, and other types of compliance-regulated data with one-way (masking) and two-way transformations via data type protection and use cases typically addressed by tokenization with Vault Transform.

HashiCorp Vault Enterprise (version >= 1.4) with Advanced Data Protection module provides the Transform secrets engine which handles secure data transformation and tokenization against the provided secrets. Transformation methods encompass NIST vetted cryptographic standards such as format-preserving encryption (FPE) via FF3–1 to encode your secrets while maintaining the data format and length. In addition, it can also perform pseudonymous transformations of the data through other means, such as masking.

Prerequisites

To perform the tasks described in this tutorial, you need to have HashiCorp Vault Enterprise 1.4 or later with the ADP module.

NOTE: To explore HashiCorp Vault Enterprise features, you can sign up for a free 30-day trial. We are using HashiCorp Vault Enterprise 1.11.2 with the ADP module licensed.

Let’s Get Started — Workflow

Most secrets engines in Vault must be configured in advance before they can perform their functions. These steps are usually completed by an operator using Terraform with the Vault provider and/or configuration management tool.

NOTE: For the purpose of this getting started, you can use root token to work with HashiCorp Vault Enterprise. However, it is recommended that root tokens are only used for just enough initial setup or in emergencies. As a best practice, use tokens with an appropriate set of policies based on your role in the organization.

1. Enable the Transform secrets engine
   The Transform secrets engine contains several types of resources that encapsulate different aspects of the information required in order to perform data transformation.
2. Create a named role
   Roles are the basic high-level construct that holds the set of transformation that it is allowed to performed. The role name is provided when performing encode and decode operations.
3. Create a transformation
   Transformations hold information about a particular transformation. It contains information about the type of transformation that we want to perform, the template that it should use for value detection, and other transformation-specific values such as the tweak source or the masking character to use.
   The following transformations are available for use in the secret engine::

3.1 Format Preserving Encryption (FPE)
FPE performs cryptographically secure transformation via FF3–1 to encode input values while maintaining its data format and length.

Transform secret engine FPE encode operation

Transform secret engine FPE decode operation

3.2 Masking
Masking replaces matched characters on the input value with a desired character. This form of transformation is non-reversible and thus does not support retrieving the original value back using the decode operation.

Transform secret engine data masking encode operation

Decoding a masked value is not possible

3.3 Tokenization
Tokenization exchanges a sensitive value for an unrelated value called a token. The original sensitive value cannot be recovered from a token alone, they are irreversible.

4. Optionally, create a template
Templates allow us to determine what and how to capture the value that we want to transform.
The following builtin templates are available for use in the secret engine

4.1 builtin/creditcardnumber
4.2 builtin/socialsecuritynumber

Templates currently only accept regular expressions as the matching pattern type.
It uses Go’s standard library for the regexp engine, which supports the RE2 syntax.

5. Optionally, create an alphabet
Alphabets provide the set of valid UTF-8 characters contained within both the input and transformed value on FPE transformations.

The following builtin alphabets are available for use in the secret engine
5.1 builtin/numeric
5.2 builtin/alphalower
5.3 builtin/alphaupper
5.4 builtin/alphanumericlower
5.5 builtin/alphanumericupper
5.6 builtin/alphanumeric

Custom alphabets must contain between 2 and 65536 unique characters.

Let’s Get Started — Example

1. Enable Transform engine

$ vault secrets enable transform

2. Create a named role

$ vault write transform/role/payments transformations=ccn-fpe

3. Create a transformation

$ vault write transform/transformation/ccn-fpe \
type=fpe \
template=ccn \
tweak_source=internal \
allowed_roles=payments

4. Optionally create a template

$ vault write transform/template/ccn \
typ=regex \
pattern=’(\d{4})[- ](\d{4})[- ](\d{4})[- ](\d{4})’ \
encode_format=’$1-$2-$3-$4' \
decode_formats=last-four=’$4' \
alphabet=numerics

5. Optionally create a alphabet

$ vault write transform/alphabet/numerics alphabet=”0123456789"

For you convenience we created a bash script with the above commands.

$ git clone https://github.com/cvanballegooijen/adp
$ cd adp/lab1/cli
$ export VAULT_TOKEN=<vault token>
$ export VAULT_ADDR=<vault server url>
$ ./lab1.sh

Getting Started Using the Terraform Vault Provider

The Terraform Vault Provider allows Terraform to read from, write to, and configure HashiCorp Vault.

The above five manual steps mentioned in the Getting Started above can be fully automated with a Terraform script using the Vault provider.

For your convenience we created a Terraform configuration file to configure Vault using Terraform.

$ git clone https://github.com/cvanballegooijen/adp
$ cd adp/lab1/terraform
$ export VAULT_TOKEN=<vault token>
$ export VAULT_ADDR=<vault server url>
$ terraform init
$ terraform plan
$ terraform apply

Summary

HashiCorp Vault’s Transform secrets engine provided a consistent workflow that balanced the demands of protecting sensitive data and enabling developer velocity and efficiency. You can find out more about the HashiCorp Transform secret engine here. This is the first article in a series on the ADP module.

Resources

• GitHub repository companion
  https://github.com/cvanballegooijen/adp