Keeping Your Secrets Between Cloud Object Storage and Your Browser (Part 1)

Upload edition

Cloud Object Storage is the perfect solution for storing unlimited amounts of unstructured data in the cloud. It’s cheap, scalable and super reliable.

Imagine we have a web app that needs to upload images to the cloud — think roll-your-own-Instagram. Object Storage is the ideal destination for this data, but we definitely don’t want our web or mobile app to know the secret keys to our Object Storage service. If we hard-coded them into our client-side app, then anyone could find them and use them to upload, fetch or delete any data they chose!

One answer would be to write a serverless action that receives the uploaded file and writes it directly to Cloud Object Storage:

Masking object storage credentials with a serverless function works well! There’s an upload size limit though.

The serverless function is pre-configured (at deployment time) with the credentials for the Cloud Object Storage service so that the web app never needs to handle the keys. The disadvantage of this approach is that calling to a serverless action has a maximum request size of 1MB, so large image files would never make it to Object Storage.

Pre-Signed URLs to the rescue

A better solution, if a little more complicated, is this:

Our roll-your-own-Instagram app’s pre-signed URLs let the app safely talk directly to Object Storage.

1. The web app makes an API call to an IBM Cloud Functions action to indicate that it wishes to upload an image. (IBM Cloud Functions is based on the Apache OpenWhisk serverless platform)
2. The serverless action calculates a time-limited URL that allows the upload of a single object.
3. The website makes a second API call to write the image directly to the Object Storage service itself, using the URL it got from the Cloud Functions action.

Again the serverless action keeps our Object Storage credentials safe, but this time it doesn’t have to deal with the actual uploaded file itself, so it won’t hit any message size limits.

The URL that it supplies to the client is pre-signed, meaning that it grants access to the client-side script to upload to Object Storage for a limited time. It doesn’t give cart blanche access to the whole Object Storage bucket — only write access to one object for a short time.

Before we start writing code, we’re going to need an Object Storage service.

Sign up for IBM Cloud Object Storage

We’re going to need an IBM Cloud Object Storage service or an Amazon S3 account. IBM’s service supports a subset of the S3 API for easy migration. With a couple simple steps, we can use the same code on IBM Cloud Object Storage or Amazon S3.

Sign up for an IBM Cloud account. Once registered, you can add an IBM Cloud Object Storage service. The Lite plan gives you a generous free allocation to play with.

In the IBM Cloud Object Storage UI:

• Create a bucket — this is the “folder” where your uploaded files will be kept.
• Create a set of credentials that we are going to use in our serverless script. When creating these credentials, you’ll need to pass in a flag to enable S3-style authentication. Enter {"HMAC":true} to enable Amazon compatibility.

Here’s what that looks like:

Setting the HMAC param to true enables S3-style authentication for our IBM Cloud Object Storage instance.

Make a note of the bucket name, the API endpoint and your keys for the next step.

I found it helpful to access my storage from the command line; you can use the aws command-line tool as described here. Your credentials live in an ~/.aws/credentials file like this:

Then I can see the contents of my bucket with:

Set up CORS

Normally, a web page is only allowed to make API calls back to the server where it came from. This rule can be overridden as long as the server your page is connecting to has CORS enabled. CORS stands for Cross-Origin Resource Sharing. Our Object Storage service is going to need CORS enabling if it is to be accessed directly by a web page.

First we create a JSON file called cors.json containing the rules:

The above JSON defines which domains (all) are allowed to access to our Object Storage service with which HTTP methods (GET & PUT). We run the aws command-line tool to write this CORS configuration to our service:

Let’s see the code

The Node.js code to generate pre-signed URLs is pretty simple and doesn’t require any libraries other than the built-in Node.js crypto module. Serverless actions are just a main function where the first parameter passed in is a combination of:

• configuration parameters that were supplied at deployment time
• parameters passed in when the action is run, i.e., run-time parameters

This serverless action expects the following items to be supplied during deployment:

• access_key: object storage public key
• secret_key: object storage private key
• bucket: object storage bucket name
• endpoint: object storage URL endpoint

All of which come from the IBM Cloud Object Storage setup.

The final parameter content-type is the Mime type of the file you are uploading (for example: image/png for a .png file). Here's the code:

You can test the code yourself by calling main with an object containing the details of your Object Storage instance and the Mime type of the image you are uploading:

The URL returned should look something like this:

We can use the command-line tool curl to upload a file to this URL:

But be quick! The link will expire in sixty seconds!

Deploying to IBM Cloud Functions

We can save our JavaScript into a file upload.js and, assuming we have already installed and configured the bx wsk command-line tool, deploying this to IBM Cloud Functions is easy:

The action is now pre-configured with all of our Cloud Object Storage details. You can find the URL of your deployed action by running:

It should be something like:

The API call will generate a pre-signed URL if we supply the content type of the file we wish to upload:

We can use the command-line tool curl to upload a file to this URL:

Making a web front end

With our new serverless URL, we can create an HTML file that allows us to drag and drop files to be uploaded to Object Storage. Here’s the code:

You’ll need to replace YOUR_SERVERLESS_URL with the URL of your serverless action.

Open the HTML file in a browser and you should see a drop zone. Drag and drop a file into the drop zone and it should be uploaded to Object Storage for you. JavaScript running in the web page first fetches a pre-signed URL from your serverless action and then uses that URL to upload the file you dropped. The page is using nothing more than jQuery to create the HTTP requests.

Get in the zone! (Screenshot of the drop zone example app.)

Summary

We’ve achieved a lot:

• Signed up for IBM Cloud Object Storage to provide a cheap, limitless store of unstructured data.
• Created some Amazon-compatible credentials for our service.
• Added a new bucket.
• Enabled CORS on the bucket so it can be accessed from any browser.
• Wrote some Node.js code that generates pre-signed URLs that can be used to write to the bucket within a short time window.
• Deployed the code to IBM Cloud Functions so that the code runs in a serverless environment.
• Written a sample HTML page that uses the serverless action to generate the URL and uses the URL to write data to object storage.

Head over to part two of this 2-part series, where I cover the inverse of uploading — downloading:

Keeping Your Secrets Between Cloud Object Storage and Your Browser (Part 2)

References:

• Generating pre-signed URLs
• Source code