Published in


Use AWS S3, KMS and Python for Secrets Management

Banner created with

So your application need to store secrets and you are looking for a home for them. You have AWS SSM, but you got tired of Rate Limits (i did), this guide will show you how easy it is to use S3, KMS and Python to achieve the goal.

High Level Goal:

From a High-Level we want to store secrets encrypted on S3 with KMS, namespaced with team/application/environment/value in json format so that our application receives the json dictionary of configured key/value pairs.

We can leverage IAM to delegate permissions on the namespacing that we decide on, for my example the namespace will look like this on S3:


We will apply IAM permissions for our user to only Put and Get on `secrets/engineering*`. So with this idea we can apply IAM permissions on groups for different departments, or even let users manage their own secrets such as:


After the object has been downloaded from S3 and decrypted using KMS, the value of the object will look like this:

{u'surname': u'bekker', u'name': u'ruan', u'job_title': u'systems-development-engineer'}


We will create the following resources on AWS:

  • KMS Key
  • S3 Bucket
  • IAM User
  • IAM Policy
  • Python Dependencies: Boto3

Provision AWS Resources

image from

First we will create our S3 bucket, head over to Amazon S3, create a new s3 bucket, make sure that the bucket is not public, by using the default configuration, you should be good.

Once your S3 Bucket is provisioned, head over to Amazon IAM and create a IAM User, enable programmatic access, and keep your access key and secret key safe. For now we will not apply any permissions as we will come back to this step.

Head over to Amazon KMS and create a KMS Key, we will define the key administrator, which will be my user (ruan.bekker in this case) with more privileged permissions:

define the key administrative permissions

and then we will define the key usage permissions (app.user in this case), which will be the user that we provisioned from the previous step, this will be the user that will encrypt and decrypt the data:

Define Key Usage Permissions

Next, review the policy generated from the previous selected sections:

KMS Review of Key Section

Once you select finish, you will be returned to the section where your KMS Key information will be displayed, keep note of your KMS Key Alias, as we will need it later:

KMS Key Description

Create a IAM Policy for our App User

Next we will create the IAM Policy for the user that will encrypt/decrypt and store data in S3

"Version": "2012-10-17",
"Statement": [
"Sid": "S3PutAndGetAccess",
"Effect": "Allow",
"Action": [
"Resource": "arn:aws:s3:::arn:aws:s3:::s3-bucket-name/secrets/engineering*"
"Sid": "KMSDecryptAndEncryptAccess",
"Effect": "Allow",
"Action": [
"Resource": "arn:aws:kms:eu-west-1:123456789012:key/xxxx-xxxx-xxxx-xxxx-xxxx"

After the policy has been saved, associate the policy to the IAM User

Encrypt and Put to S3

Now we will use Python to define the data that we want to store in S3, we will then encrypt the data with KMS, use base64 to encode the ciphertext and push the encrypted value to S3, with Server Side Encryption enabled, which we will also use our KMS key.

Install boto3 in Python:

$ pip install boto3

Enter the Python REPL and import the required packages, we will also save the access key and secret key as variables so that we can use it with boto3. You can also save it to the credential provider and utilise the profile name:

>>> import boto3
>>> import json
>>> import base64
>>> aws_access_key_id='redacted'
>>> aws_secret_access_key='redacted'

Next define the data that we want to encrypt and store in S3:

>>> mydata = {
"name": "ruan",
"surname": "bekker",
"job_title": "systems-development-engineer"

Next we will use KMS to encrypt the data and use base64 to encode the ciphertext:

>>> kms = boto3.Session(
>>> ciphertext = kms.encrypt(
>>> encoded_ciphertext = base64.b64encode(ciphertext["CiphertextBlob"])# preview the data
>>> encoded_ciphertext

Next we will use S3 to push the encrypted data onto S3 in our name spaced key: secrets/engineering/app1/production/appconfig.json

>>> s3 = boto3.Session(
>>> response = s3.put_object(

Now our object is stored in S3, encrypted with KMS and ServerSideEncryption Enabled.

You can try to download the object and decode the base64 encoded file and you will find that its complete garbage as its encrypted.

Next we will use S3 to Get the object and use KMS to decrypt and use base64 to decode after the object has been decrypted:

>>> response = s3.get_object(
>>> encoded_ciphertext = response['Body'].read()
>>> encoded_ciphertext

Now let’s decode the result with base64:

>>> decoded_ciphertext = base64.b64decode(encoded_ciphertext)
>>> plaintext = kms.decrypt(CiphertextBlob=bytes(decoded_ciphertext))

Now we need to deserialize the JSON as it’s in string format:

>>> json.loads(plaintext["Plaintext"])
{u'surname': u'bekker', u'name': u'ruan', u'job_title': u'systems-development-engineer'}

Using it in a Application

Let’s say you are using Docker and you want to bootstrap your application configs to your environment that you are retrieving from S3.

We will use a python script that will read the data into memory, decrypt and write the values in plaintext to disk, then we will use the script to read the values into the environment and remove the temp file that was written to disk, then start the application since we have the values stored in our environment.

Our “application” in this example will just be a line of echo to return the values for demonstration.

The file

import boto3
import json
import base64
kms = boto3.Session(aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key).client('kms')
s3 = boto3.Session(aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key, region_name='eu-west-1').client('s3')
response = s3.get_object(Bucket='ruan-secret-store', Key='secrets/engineering/app1/production/appconfig.json')
encoded_ciphertext = response['Body'].read()
decoded_ciphertext = base64.b64decode(encoded_ciphertext)
plaintext = kms.decrypt(CiphertextBlob=bytes(decoded_ciphertext))
values = json.loads(plaintext["Plaintext"])
with open('envs.tmp', 'w') as f:
for key in values.keys():
f.write("{}={}".format(key.upper(), values[key]) + "\n")

And our script:

$ python
source ./envs.tmp
rm -rf ./envs.tmp
echo "Hello, my name is ${NAME} ${SURNAME}, and I am a ${JOB_TITLE}"

Running that will produce:

$ bash
Hello, my name is ruan bekker, and I am a systems-development-engineer

Thank You

And there we have a simple and effective way of encrypting/decrypting data using S3, KMS and Python at a ridiculously cheap cost, its almost free.

If you liked my content, feel free to checkout my content on or follow me on twitter at @ruanbekker




We like to build things, break them and start all over again

Recommended from Medium

AWS Security Group Tools

OBIEE Interview questions by Skil App

🌀Development pace🌀

How to enable/disable a product programmatically in Magento 2?

Oracle Free Tier — Is not free.

How we built a Remote Logging system in Android

MongoDB Realm JWT Meta Data and Custom User Data

Python Gists

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ruan Bekker

Ruan Bekker

DevOps Engineer and Open Source Enthusiast

More from Medium

Setup Custom AWS lambda (λ) function dependencies using Docker containers

AWS Lambda Hands-On| Part-2

Use Python logging module in AWS Lambda to send messages to CloudWatch and SNS

Deploying Python Lambda function with Serverless Framework