Multipart uploads on Sia using Filebase and the AWS CLI

Filebase
Filebase
Jun 28, 2019 · 6 min read
Image for post
Image for post

One of the most commonly used tools for transferring data to object storage services is the AWS CLI (Command Line Interface). This is a tool developed by Amazon and is largely based on the Python botocore library. The CLI features several commands that allow you to manage objects and buckets. Fortunately, it can be configured to point to any S3-compatible object storage service, including Filebase.

This tool is especially popular amongst developers and IT administrators. It is useful because it’s invoked as a command line program. This allows for it to be easily integrated with backup and other types of custom scripts. One common example would include a nightly backup script that runs via a cron job, backing up a servers hard drive to a storage bucket.

The AWS CLI is certified for use with Filebase.

To use it with Filebase, you can follow the configuration steps below. Before running through any of these steps, you will need to ensure you have the CLI properly installed. You can visit the following page for install instructions: https://aws.amazon.com/cli/

Configuration

  1. S3 API Access Key ID
  2. S3 API Secret Access Key
  3. S3 API Endpoint

The Access Key ID and Secret Access Key can be stored in a configuration file. However, the API endpoint will need to be passed in with every command. There are ways to avoid this by using 3rd party plugins, such as awscli-plugin-endpoint. We will discuss how to do this in a future blog post.

To setup our access keys, we will open a new terminal window. From there, we can run:

aws configure

This will trigger a prompt. Enter your Access Key ID and Secret Access Key. You can leave the region and output format blank by simply hitting enter.

Image for post
Image for post

Once the above steps are complete, we are ready to move onto interacting with the Filebase S3 API.

Create a new bucket

aws --endpoint https://s3.filebase.com s3 mb s3://my-test-bucket
Image for post
Image for post

Listing Buckets

aws --endpoint https://s3.filebase.com s3 ls
Image for post
Image for post

Once the above command is run, we should see a list of buckets returned. This is a list of buckets that you own. As you can see below, our new bucket now appears on this list.

Image for post
Image for post

List bucket contents

To list the contents of a specific bucket, run the following:

aws --endpoint https://s3.filebase.com s3 ls s3://my-test-bucket
Image for post
Image for post

An empty response is returned, confirming that our new bucket is indeed empty.

Uploading files to a bucket

aws --endpoint https://s3.filebase.com s3 sync my-test-folder/ s3://my-test-bucket
Image for post
Image for post

Once the upload has completed, we can list the contents of the bucket using the S3 API to confirm:

Image for post
Image for post

And of course, all files are always available from our browser-based UI console as well:

Image for post
Image for post

We can use the s3 cp command to upload a single file:

aws --endpoint https://s3.filebase.com s3 cp s3-api.pdf s3://my-test-bucket
Image for post
Image for post

Multipart uploads

To upload a file using multipart, simply try uploading a file larger than 8MB in size — the AWS CLI will automatically take care of the rest. In the example below, we will upload a 1GB file.

aws --endpoint https://s3.filebase.com s3 cp 1GB.zip s3://my-test-bucket
Image for post
Image for post

Verifying uploaded files

In the example below, we will fetch the object’s metadata from the Filebase S3 API. After this, we run a command which calculates the MD5 of the same file on our local machine. If the MD5’s match, we can be sure that our upload was successful and the service received our data properly.

aws --endpoint https://s3.filebase.com s3api head-object --bucket my-test-bucket --key 1GB.zip

To calculate the MD5 checksum of our file locally on macOS:

md5sum 1GB.zip
Image for post
Image for post

As you can see, the MD5 value of cd573cfaace07e7949bc0c46028904ff matches — therefore we can be assured the service received and processed our file correctly.

Image for post
Image for post
= Success

Conclusion

Please let us know if you have any questions, comments or concerns at hello@filebase.com

Filebase

S3-compatible object storage at a fraction of the price.

Filebase

Written by

Filebase

Blockchain-based Object Storage featuring an S3-Compatible API at $5/TB. Your first 5GB are free at https://filebase.com

Filebase

Filebase

S3-compatible object storage at a fraction of the price. No upload fees, no egress fees.

Filebase

Written by

Filebase

Blockchain-based Object Storage featuring an S3-Compatible API at $5/TB. Your first 5GB are free at https://filebase.com

Filebase

Filebase

S3-compatible object storage at a fraction of the price. No upload fees, no egress fees.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store