Is Your Organization Handling Secrets Securely?

The Secrets of Avoiding Hardcoded Secrets

Shiv Sahni
May 17 · 7 min read
Image for post
Image for post

I remember the early days of my application security journey where we used to identify hardcoded secrets in the backend code, in almost every source code review engagement and at that time I used to struggle a lot to come up with the best remediation considering the cost and overall architecture.

Based on a little experience of learning and unlearning things around this very common issue of hardcoded secrets I thought of writing something on this. In this story, I would be discussing the issue related to hardcoded secrets and the ways in which we can effectively resolve the issue.

At last, we would also quickly compare the solution provided by cloud market leader AWS and HashiCorp’s Vault to effectively and securely manage secrets.

The Problems of Hardcoded Secrets

If you have done code reviews in the past, I’m sure you would have definitely encountered the hardcoded secret issues. If not, look at the following code snippet which shows that the database credentials are hardcoded in the application source code:

Sensitive data such as secret keys, private keys, SSH keys, access/secret keys, third-party secret and API keys, etc. should never be hardcoded in the application source code or your Infrastructure As Code(IaC) i.e. Chef, Ansible, Puppet, Terraform scripts. The following are the problems associated with hardcoded secrets:

Insider Threats

Image for post
Image for post

Hardcoding the secrets in the source code violates the fundamental security principle of least privileges since anyone having access to the code/VCS(Github, Gitlab, etc.) would also be having access to the underlying services/operations through secrets. Imagine all the application developers/QA engineers having access to this sensitive information which is supposed to be a secret and thus making it vulnerable to disclosure through insider attacks.

External Threats

Image for post
Image for post

In the scenarios wherein the application’s server is compromised and the attacker has access to the code, all these hardcoded secrets are the juicy information 🤑 for an attacker which would aid in lateral movement.

Imagine once the webserver residing in DMZ(Accessible over the Internet) is compromised, the hardcoded secrets in the code would aid an attacker to reach DB server located in the internal network and thus penetrate further.

Security Vs Availability

Image for post
Image for post

Once the app/service is in production and the hardcoded secrets needs to be changed because of a breach/compliance requirement, the change becomes more cumbersome since this might now affect the availability of the system, which could lead to direct business impact.

Distributed Access Control & Management

Another problem related to hardcoded secrets is pertaining to access control and auditing. Since the secrets are distributed at various places i.e. application code, configuration files, infrastructure scripts(IaC), etc. throughout our version control systems, we neither have fine-grained access control nor the way to effectively perform audit trails to identify when, where and who accessed those secrets.

Note: There’s a wonderful video by Armon Dadgar where he very superbly talks about the Problem of Secret Sprawl-Situations wherein the secrets are sprawled all over our infrastructure in different places.

Secure & Effective Secret Management Requirements

We have various solutions available to manage secrets such as HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, Google Secret Manager, Cloud HSMs, in-house HSMs and other in-house solutions. In this section, we would talk about a few fundamental requirements of an effective and secure secret manager solution obviously apart from being highly available and scalable.

Secure Storage & Transit

The secrets should be stored and transited(TLS) securely i.e. the secrets should not only be encrypted but should be securely encrypted during rest and transit. The strong cryptographic algorithms should be used and the key used to encrypt the secrets should be securely managed!

Sounds like an Egg-Chicken Problem?

Image for post
Image for post

This is beautifully handled by secret management solutions. For instance, HashiCorp Vault encrypts all the secrets using an encryption key which itself is encrypted using a master key and stored along with encrypted data. The master key is not stored anywhere. Vault uses Shamir’s secret sharing algorithm to split the master key into 5 shares, of which any 3 are required to reconstruct the master key. Once the master key is generated, we can decrypt the encryption key and thus the secret data. You can read more about it here.

Fine-Grained Access Control & Auditing

The secret manager solution should have fine-grained access control to allow a set of user/service to have access to a particular secret instead of everyone having access to everything. Moreover, apart from access control, there should be strong visibility on who, when and how the secrets are accessed, which help us to track access requests and also makes it easier to ensure compliance with internal policies and regulatory standards.

Periodic/On-Demand Rotation of Secrets

Rotation of secrets is another essential requirement of secure secret management. Either we need to have a periodic rotation of secrets in order to comply to the organization’s ISMS policies/regulatory requirements(Ex: PCI-DSS) or we need to rotate secrets on-demand in situations such as data breach. The rotation of secrets could have a significant impact on the availability and stability of systems if the rotation is not smoothly implemented.

Some secret management solutions have already addressed this problem effectively. If we talk about AWS Secrets Manager it allows a seamless periodic/on-demand rotation with the help of AWS Lambda functions.

It natively knows how to rotate secrets for supported Amazon RDS databases and also enable us to rotate secrets for unsupported databases/third-party services. In other words, if we need to rotate password for supported Amazon RDS databases, then Secrets Manager provides the Lambda function for us else for any unsupported database or service, we must provide the code for the Lambda function. You can read more about it here.

If just like me you are also wondering how AWS solves the problem of downtime that could happen because of the propagation delay of secrets. You can have a look at this.

Now that we have discussed the fundamental properties of the secret manager, we know-how secrets can be managed securely and the ways in which it can be securely shared with user/service needing it.

Hold on! Although secrets are safe inside secret managers, did you realize we are still sitting on a risk wherein the secrets could be leaked from outside of a secure secret manager?

Image for post
Image for post

Dynamic Secrets

Even though we could have securely managed the secrets inside and while accessing secret manager but what if the entities accessing the secrets are not handling it securely. Consider a scenarios wherein the application is logging the DB password in the log files. This is the remaining piece of risk that should also be managed.

Also, with static secrets, all the entities requiring access to service would share the same set of credentials and in the case of a breach, we would be supposed to change the secret for each and every entity which could be quite cumbersome. Alternatively, in the case of dynamic secrets, we can identify a needle in a haystack easily and change the secret only for the compromised entity.

Let’s have a look at how HashiCorp Vault handles this gracefully:

Image for post
Image for post

On a high level, a dynamic secret is generated on demand and is unique to a client, instead of a static secret, which is defined ahead of time and shared. Vault associates each dynamic secret with a lease and automatically destroys the credentials when the lease expires.

Quick Comparison of HashiCorp’s Vault & AWS Secret Manager

Cost

  • HashiCorp’s Vault has both open-source and enterprise versions available. The open-source version provides significant features for secret management. The actual cost of usage depends on the server which is used to deploy Vault
  • AWS Secret Manager(ASM) is a managed service provided by AWS. The actual cost of usage depends on the number of secrets stored and the API calls used to talk to ASM.

Ecosystem

  • Vault offers a vendor-agnostic solution to secrete management. It can work in any cloud or data-centre with one common API
  • ASM is coupled to AWS ecosystem

Secret Rotation

  • Vault allows us to create dynamic secrets which can be unique per instance and support to automate rotation usings secrets.
  • ASM provides a seamless secrets rotation for a few supported databases and Lambda functions for unsupported entities.

Dynamic Secrets

  • Vault supports dynamic secrets
  • ASM does not support dynamic secrets

Although implementing a solution could very much on multiple constraints within an organization such as the current cloud platform. However, on a general note, based on the functionalities and cloud-agnostic characteristic of Vault, I believe it is a better solution to manage secrets.

References

InfoSec Write-ups

A collection of write-ups from the best hackers in the…

Sign up for Infosec Writeups

By InfoSec Write-ups

Newsletter from Infosec Writeups Take a look

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Shiv Sahni

Written by

Security Engineer |Security Consultant |Infosec Trainer | Author | Lecturer | Open Source Contributor | Learner https://www.linkedin.com/in/shivsahni/

InfoSec Write-ups

A collection of write-ups from the best hackers in the world on topics ranging from bug bounties and CTFs to vulnhub machines, hardware challenges and real life encounters. In a nutshell, we are the largest InfoSec publication on Medium. Maintained by Hackrew

Shiv Sahni

Written by

Security Engineer |Security Consultant |Infosec Trainer | Author | Lecturer | Open Source Contributor | Learner https://www.linkedin.com/in/shivsahni/

InfoSec Write-ups

A collection of write-ups from the best hackers in the world on topics ranging from bug bounties and CTFs to vulnhub machines, hardware challenges and real life encounters. In a nutshell, we are the largest InfoSec publication on Medium. Maintained by Hackrew

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store