Stow is an open-source (Apache licensed) abstraction on top of Amazon S3, Microsoft Azure Blob Store, Google Cloud Storage, Openstack Swift and more. We hope Stow will become the de facto library for interacting with such storage providers in Go.
Stow allows you to interact with various cloud storage providers via a simple single API.
GrayMeta’s MetaFarm product harvests metadata from all kinds of files, making that data queryable and searchable, regardless of where that content lives.
When it came time to consume content from the cloud, we noticed that most providers had a similar (although not exactly identical) approach to storing content; files (or items) were stored in buckets (or containers).
Stow abstracts these concepts and provides a single API through which you can seamlessly interact with various providers.
The main three objects in Stow are:
- Location — An authenticated cloud storage location (made up of a kind, like
s3and appropriate configuration) - Container — A group of items
- Item — A single file
The Stow implementations are built using other open-source packages, so it certainly stands on the shoulders of giants.
How Stow works
Implementations are referred to by a kind string (like “s3”, “azure”, etc.) and an object providing implementation specific configuration values (such as credentials, the S3 region, Google Cloud project ID, etc.)
Import Stow and implementations
First, you import Stow and any implementations you wish to support:
import (
"github.com/graymeta/stow"
_ "github.com/graymeta/stow/google"
_ "github.com/graymeta/stow/s3"
)The underscore indicates that you do not intend to use the package in your code. Importing it is enough, as the implementation packages register themselves with Stow during initialization.
Dial a location
Next, we can create (or load from a database) the kind and config data, and Dial the location:
kind := "s3"
config := stow.ConfigMap{
s3.ConfigAccessKeyID: "246810"
s3.ConfigSecretKey: "abc123",
s3.ConfigRegion: "eu-west-1"
}location, err := stow.Dial(kind, config)
if err != nil {
return err
}
defer location.Close()// TODO: use location
You can see that during runtime it is trivial to pass in different
kindandconfigurationvalues.
Walking all containers
Once you have dialled the Location, you can walk the Containers:
err := stow.WalkContainers(location, stow.NoPrefix, 100,
func(c stow.Container, err error) error {
if err != nil {
return err
}
log.Println("Container: ", c.Name())
return nil
})
if err != nil {
return err
}The stow.WalkContainers function is like filepath.Walk, except that it walks the Containers at that Location. In Amazon S3 each container is a Bucket.
Walking all items
Once you have found the container you’re interested in, you can walk the items in a similar way using the stow.Walk function:
err := stow.Walk(container, stow.NoPrefix, 100,
func(item stow.Item, err error) error {
if err != nil {
return err
}
log.Println(item.Name())
return nil
})
if err != nil {
return err
}The 100 argument represents the number of items to get per-page, and where we pass in stow.NoPrefix, you can optionally specify a prefix to filter the items being walked.
Downloading a file
Once you have found a stow.Item that you are interested in, you can stream its contents by first calling the Open method and reading from the returned io.ReadCloser (remembering to close the reader):
r, err := item.Open()
if err != nil {
return err
}
defer r.Close()// TODO: stream the contents by reading from r
Uploading a new file
If you want to write a new item into a Container, you can do so using the container.Put method passing in an io.Reader for the contents along with the size.
contents := "This is a new file stored in the cloud"
r := strings.NewReader(contents)
size := int64(len(contents))item, err := container.Put(name, r, size, nil)
if err != nil {
return nil
}// item represents the newly created/updated item
Reading data a page at a time
Stow implements paging via cursors; you make a request, in the response (along with the items) is a cursor string, which can be passed into future calls to get subsequent pages of data.
You call methods that accept cursors by first passing in stow.CursorStart, which indicates the first item/page. The method will, as one of its return arguments, provide a new cursor which you can pass into subsequent calls to the same method.
When stow.IsCursorEnd(cursor) returns true, you have reached the end of the set.
The walker functions use cursors under the hood, so that’s a good place to look if you want to see a real example of how to use them.
Project status
We have been succesfully running Stow in production for a few months.
Stow is ready to use. Please vendor the dependency just in case we make any API changes before the official release.
An official version 1 release will happen early next year after the community has had time to digest it and contribute feedback, ideas, bug reports, etc.
