Introducing Stow: Cloud storage abstraction package for Go

Mat Ryer
Mat Ryer
Nov 8, 2016 · 4 min read
Stow: Cloud storage abstraction package for Go

Stow is an open-source (Apache licensed) abstraction on top of Amazon S3, Microsoft Azure Blob Store, Google Cloud Storage, Openstack Swift and more. We hope Stow will become the de facto library for interacting with such storage providers in Go.

Stow allows you to interact with various cloud storage providers via a simple single API.

define:stow

GrayMeta’s MetaFarm product harvests metadata from all kinds of files, making that data queryable and searchable, regardless of where that content lives.

When it came time to consume content from the cloud, we noticed that most providers had a similar (although not exactly identical) approach to storing content; files (or items) were stored in buckets (or containers).

Stow abstracts these concepts and provides a single API through which you can seamlessly interact with various providers.

The main three objects in Stow are:

  • Location An authenticated cloud storage location (made up of a kind, like s3 and appropriate configuration)
  • Container — A group of items
  • Item A single file

The Stow implementations are built using other open-source packages, so it certainly stands on the shoulders of giants.

How Stow works

Implementations are referred to by a kind string (like “s3”, “azure”, etc.) and an object providing implementation specific configuration values (such as credentials, the S3 region, Google Cloud project ID, etc.)

Import Stow and implementations

First, you import Stow and any implementations you wish to support:

The underscore indicates that you do not intend to use the package in your code. Importing it is enough, as the implementation packages register themselves with Stow during initialization.

Dial a location

Next, we can create (or load from a database) the kind and config data, and Dial the location:

You can see that during runtime it is trivial to pass in different kindand configuration values.

Walking all containers

Once you have dialled the Location, you can walk the Containers:

The stow.WalkContainers function is like filepath.Walk, except that it walks the Containers at that Location. In Amazon S3 each container is a Bucket.

Walking all items

Once you have found the container you’re interested in, you can walk the items in a similar way using the stow.Walk function:

The 100 argument represents the number of items to get per-page, and where we pass in stow.NoPrefix, you can optionally specify a prefix to filter the items being walked.

Downloading a file

Once you have found a stow.Item that you are interested in, you can stream its contents by first calling the Open method and reading from the returned io.ReadCloser (remembering to close the reader):

Uploading a new file

If you want to write a new item into a Container, you can do so using the container.Put method passing in an io.Reader for the contents along with the size.

Reading data a page at a time

Stow implements paging via cursors; you make a request, in the response (along with the items) is a cursor string, which can be passed into future calls to get subsequent pages of data.

You call methods that accept cursors by first passing in stow.CursorStart, which indicates the first item/page. The method will, as one of its return arguments, provide a new cursor which you can pass into subsequent calls to the same method.

When stow.IsCursorEnd(cursor) returns true, you have reached the end of the set.

The walker functions use cursors under the hood, so that’s a good place to look if you want to see a real example of how to use them.

Project status

We have been succesfully running Stow in production for a few months.

Stow is ready to use. Please vendor the dependency just in case we make any API changes before the official release.

An official version 1 release will happen early next year after the community has had time to digest it and contribute feedback, ideas, bug reports, etc.

Mat Ryer

Written by

Mat Ryer

Founder at MachineBox.io — Gopher, developer, speaker, author — BitBar app https://getbitbar.com — Author of Go Programming Blueprints

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade