How To Commit Your Cloud Credentials To Version Control Systems

Commit your cloud credentials to version control systems securely

___
___
Feb 24 · 5 min read

Introduction

In this article, I will share a way to securely commit sensitive information to version control systems (VCS). The focus will be on committing AWS credentials, but the method applies to Azure and Google Cloud Platform (GCP) as well.

The code to reproduce the results can be found here.

This article is intended for data scientists; you do not come from a software development background and hence aren’t aware of software engineering best practices. However, judging by this search result, it seems developers, in general, can benefit too.

Motivation

Imagine you are a junior data scientist tasked with building a pipeline to analyze text data. Naturally, this task involves moving files around S3 buckets and calling various AWS services to do things like sentiment analysis, named entity recognition, topic modeling, etc.

You are given a set of credentials that allows you to access these services. You decide to authenticate via environment variables, so you write a script called env.sh which does the following:

Figure 1: Script to export environment variables to authenticate to AWS services

Should you commit this script to your VCS?

Committing this script makes it convenient for your colleagues to carry on where you left off. They do not need to search for search the right credentials. They just need to clone your repo, and everything will work out-of-the-box.

However, committing this script is a security risk. Just imagine someone hacking/leaking the content of your company’s VCS server. Or maybe your company one day decides to open source this repo, thus accidentally exposing their private information for the whole world to see because they forgot to scrub it from the repo’s history.

Is there a solution that balances convenience and security?

Problem Statement

To make things more concrete, we will be working with the following script called main.py :

Figure 2: The contents of main.py

Calling main.py with python main.py "this movie is really awesome!" will call Amazon Comprehend to produce a sentiment classification and confidence score for the text "this movie is really awesome!" , like so:

Figure 3: Sample output of main.py

The relevant part of main.py for this article is line 8 which creates a client for the required service and relies on the existence of the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables for authentication. Therefore, you need to execute source env.sh before running main.py .

The question now is: How should we commit env.sh?

Solution

Overview

The solution involves encrypting env.sh using a tool called sops and using AWS KMS to manage the keys for the encryption and decryption process. Sops works on Windows, Linux, and Mac and integrates well with AWS KMS and similar services from Azure and GCP.

There are a few steps we need to go through before we can use sops to encrypt env.sh.

Prerequisites

In terms of AWS services, you will need access to a customer-managed key (CMK) in AWS KMS. One way to do it is to have your account administrator create and manage this key for you and create another IAM user whose sole purpose is the ability to use this key for file encryption/decryption. Let’s call this user sops-user .

As a regular account user, all you have to do is follow these steps:

  1. Install sops. Click here to download the relevant package for your machine.
  2. Update your AWS credentials file to include sop-user’s credentials:
Figure 4: Extra content to insert to your AWS credentials file

3. Define the SOPS_KMS_ARN environment variable so that sops know where to find the CMK:

Figure 5: How to tell SOPS which CMK to use

With that out of the way, we are now ready to encrypt env.sh.

Encrypting Sensitive Information

Encrypting env.sh is just a matter of executing:

Figure 6: How to encrypt env.sh

This is what env.sh looks like after the encryption process:

Figure 7: A snippet of env.sh after encryption

Decrypting Sensitive Information

Running source env.sh will now throw an error as env.sh is no longer a valid shell script.

You will need to decrypt env.sh before calling source. Here’s how you do it:

Figure 8: How to source an encrypted env.sh

Analysis

Let’s take a moment to consider what the solution in the preceding section has achieved.

The encrypted form of env.sh (Figure 7) is still human-readable but is practically gibberish. As such, it can be safely committed to your company’s VCS.

What happens if this repo was hacked or leaked to the public? It is nearly impossible to recover the credentials stored in without the right key to decrypt it, which only has access to it. Note that the credentials of sops-user is never committed to the repo.

In terms of collaboration, your colleagues need only know the credentials of sops-user to decrypt env.sh and continue developing main.py.

Therefore, this solution is a good compromise between convenience and security.

Conclusion

This article has presented a simple method to commit sensitive information by using a tool named sops and leveraging the services offered by cloud service providers.

Let me know in the comments if you have any questions or would like to share alternative methods.

Towards AI

Towards AI, is the world’s fastest-growing AI community for learning, programming, building and implementing AI.

___

Written by

___

About me: https://bit.ly/2GQX7i9

Towards AI

Towards AI, is the world’s fastest-growing AI community for learning, programming, building and implementing AI.

More From Medium

More from Towards AI

More from Towards AI

More from Towards AI

Image Filtering

More from Towards AI

___
Mar 29 · 8 min read

103

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade