Private Helm Repository (AWS S3) (Terraform)
A serverless solution.
I am compelled to share my experience hosting a private Helm repository (via S3 + API Gateway + Lambda) for use with Terraform which utilizes HTTP BasicAuth for password based auth. It wasn’t straightforward but at the same time isn’t all that complicated once you put the pieces together.
Requirements:
1. Secure
2. CI/CD managed
3. Works with Terraform
The first challenge was learning that while it is easy to create a private S3 bucket and store your Helm charts there - Terraform can’t use it. Terraform ( helm_repository plugin), when run from your command line will try to grab your helm charts via HTTPS. Your super specific terraform IAM policy be damned!
Here’s a overview of the solution:
Step 1: Setup your Repository (Git)
The first thing you will need is to setup the private repository. You’ll want to create a new git repository to manage your charts. The structure should look like this:
├── charts/
├── chart1/…
├── chart2/…
└── chart3/…
├── dist/
index.yamlYou place your charts in a sub-directory called charts, create an empty directory dist and the index.yaml will be generated automatically for you in future step.
From the parent directory, package a chart by running:
helm package -d dist/ charts/chart1
This will create a gzipped file in the dist/ folder with the helm name and version information found in Chart.yaml.
Next you need to index the dist/ folder and create the repository metadata. Run:
helm repo index .
Inside the newly created index.yaml you should find metadata for each packaged version from the dist/ folder.
That’s your repo. You can commit and setup your CI/CD to monitor and package new versions.
Step 2: Deploy to S3
The next step is to get the Helm packages to S3. Rather easy but there is one big Gotcha! Helm is very picky about the Content-Type it expects when downloading the gzipped packages (from the dist/ folder) and AWS S3 cli is not so picky when uploading! In actuality the cli will guess (wrongly) the MIME type and would normally upload the packages to S3 as Content-Type: application/x-tar. This requires us to go through the extra hurdle of forcing it toContent-Type: application/x-gzip.
Here’s a full example of the script which I use with Jenkins CI/CD to push new chart packages/versions to S3:
#!/bin/bashecho "Initializing Helm..."
helm init --client-onlyecho "Packaging chart updates..."
for d in charts/* ; do
echo "Packaging $d..."
helm package -u -d dist/ $d
done# Update index
helm repo index .echo "Syncing to S3..."
cd dist
aws s3 sync . s3://my-private-helm-bucket/dist --content-type "application/x-gzip"
cd ..
aws s3 cp index.yaml s3://my-private-helm-bucket/index.yamlecho "Finished!"
If done correctly, the dist/*.tgz files should look like this in S3 properties:
Step 3: Hosting the repository, privately!
If you want to host the repository publicly just set you bucket permissions to Public and you’re good to go. However, if you or you company wants to work a little more securely we need to add HTTP BasicAuth support so Terraform can see and utilize the repository. Unfortunately with all of Amazon’s hundreds of services, BasicAuth is not baked into S3 so we have to emulate it ourselves.
There are many steps to this but it’s not as bad as it looks. We’ll need:
- API Gateway to proxy the request
- Lambda to process BasicAuth
- IAM role for Lambda’s permissions
Let’s start by creating a IAM Policy/Role. Create a new policy (eg. Helm-Private-ReadOnly)
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::my-private-helm-bucket",
"arn:aws:s3:::my-private-helm-bucket/*"
]
}]
}Create a role (eg. API-Gateway-Helm) based on the API Gateway template. Save. Edit the role and add the new Helm-S3 policy you created.
Next API Gateway.
Click create API. On the next screen, from the dropdown Actions → Create Resource.
Click on the newly created /{proxy+} resource to configure it. Create a fake endpoint for now.
Save the setup page and then click on the resource again. You should see the following:
We’ll come back to the Method Request in a bit but let’s reconfigure the Integration Request.
Set parameters above and don’t forget to add the Role ARN. Save.
Let’s configure Method Response.
You’ll first add a new HTTP Status: 200. Then add two entries under Response Headers for Content-Length and Content-Type. Save.
Integration Response. Now you can add header mappings for the headers we just configured. What this will do is map the response headers received from S3 and carry them over to the API Gateway response headers towards the end-user.
IMPORTANT DETOUR: Because we are proxying, API Gateway performs shenanigans on gzipped files. We’ll want to disable this so that the downloaded file by the client is an accurate gzip file. This is under the API → Settings.
At this point you can click on Resources → Actions → Deploy API. You’ll create a stage (eg. charts) and get an API Gateway endpoint. At this point, you should be able to view your index.yaml file by going to:
https://<hash>.execute-api.us-east-1.amazonaws.com/charts/index.yaml
Step 4: Secure it with HTTP BasicAuth.
Create a new (Node.JS) Lambda with standard/default role. In the index.js file paste this code exactly — no need to modify it.
exports.handler = function (event, context, callback) {
var authorizationHeader = event.headers.Authorizationif (!authorizationHeader) return callback('Unauthorized')var encodedCreds = authorizationHeader.split(' ')[1]
var plainCreds = (new Buffer(encodedCreds, 'base64')).toString().split(':')
var username = plainCreds[0]
var password = plainCreds[1]if (!(username === process.env.USER && password === process.env.PASSWORD)) return callback('Unauthorized')var authResponse = buildAllowAllPolicy(event, username)callback(null, authResponse)
}function buildAllowAllPolicy (event, principalId) {
var apiOptions = {}
var tmp = event.methodArn.split(':')
var apiGatewayArnTmp = tmp[5].split('/')
var awsAccountId = tmp[4]
var awsRegion = tmp[3]
var restApiId = apiGatewayArnTmp[0]
var stage = apiGatewayArnTmp[1]
var apiArn = 'arn:aws:execute-api:' + awsRegion + ':' + awsAccountId + ':' +
restApiId + '/' + stage + '/*/*'
const policy = {
principalId: principalId,
policyDocument: {
Version: '2012-10-17',
Statement: [
{
Action: 'execute-api:Invoke',
Effect: 'Allow',
Resource: [apiArn]
}
]
}
}
return policy
}
Add environment variables for USER and PASSWORD. Save the lambda. You don’t need any fancy configurations like roles or VPC.
Go back to API Gateway and create an Authorizer. Point it to your newly created Lambda function, select Lambda Event Payload: Request. Add the Authorization header. Save.
Configure the Gateway Responses — Unauthorized (401).
‘Basic’ (don't forget the single quotes) to Response Headers.FWIW, I did the same configuration for 403 (not shown above), not sure if it has any impact but it doesn’t hurt.
Now if you go back to Resources → {proxy+} and edit the Method Request you can add the Authorizer:
Re-deploy your API to the charts stage and you should now be prompted for your USER and PASSWORD configured via the lambda.
You could stop here, but, isn’t that URL really long and ugly? If you have your domain setup as a Hosted Zone in Route53, and ACM setup for a wildcard certificate on your domain you can configure a Custom Domain in API Gateway:
While that is being setup, go into Route53 and create an A/Alias rule for your subdomain:
Custom Domain will take a bit of time to deploy but afterwards you should be able to access your chart from:
https://helm.yourcompany.com/index.yaml
https://helm.yourcompany.com/dist/chart1-0.1.0.tgz
....Bonus: Configure Terraform:
data "helm_repository" "mycompany" {
name = "mycompany-helm-repo"
url = "https://helm.mycompany.com"
username = "<username>"
password = "<password>"
}resource "helm_release" "core-api" {
name = "my-api"
chart = "mycompany-helm-repo/chart1"
...
}
Voila! I hope this helps.
I wrote this post-haste, please comment if I am missing any crucial steps or you get hung up somewhere.