Cloud package

CLI & API to sync data with the cloud like CloudFiles

For one of my projects I have to use CloudFiles. I decided I want to have some simple abstraction layer and CLI to sync data with console. Abstraction layer will help to stay not bound to the specific storage. That idea I took from Django as it tries to be independent of particular database specifics (and it is not so good sometimes as PostgreSQL provides a lot of useful data types and only 4-6 are used by Django whereas others are skipped). But it is good to have abstraction for the storage.

So the Go app is uploaded with fabric (yep, python fabric) and static like images, css and js should be synced to CloudFiles container.

I wrote simple package to achieve both these goals.

Sync files

• First of all install cloud package with:

$ go get github.com/OShalakhin/cloud

• now initialize configs

$ cloud init
Initializing file: ~/.cloudcore
Initializing file: .cloud
Initializing file: .cloudignore

• .cloudcore holds auth data. i.e. Cloud name (CloudFiles, S3), API key, username etc. It is generated with sample so it will be easy to read and update it.
• .cloud and .cloudignore are files created in the working directory.
• .cloud stores list of containers you’d like to sync your data (you can define more than one container). Generated with sample data.
• .cloudignore is like .gitignore. It is generated with .cloud and .cloudignore added to it as I think you won’t sync these 2 files to the cloud storage.
• Now you are ready to use it.

cloud sync

You can also pass container name to specify container and if no name is applied the first found container will be used instead.

Storage API

Thanks to the Go language I don’t have to subclass or override classes. I defined Storage as an interface with the following methods:

type Storage interface {
 Create(filename string, data []byte) error
 Read(filename string) ([]byte, error)
 Update(filename string, data []byte) error
 Delete(filename string) error
 Authenticate() error
 GetContainer() (*Container, error)
 GetURL() *url.URL
}

It is CRUD + Authenticate + Container + GetURL to use it in templates etc.

An interesting thing is

github.com/OShalakhin/cloud/storage/cloudfiles

has Storage struct with its specifics like rs.RsConnection from ncw/swift/rs package. It is used inside cloudfiles storage for CRUD, URL, Auth and is visible if you check it with

// mystorage is cloud/storage with CloudFiles provider
fmt.Println(mystorage)

you’ll be able to see specific CloudFiles fields but as Storage is an interface you’ll be able to use just these simple methods.

Currently CloudFiles are supported (implementation is based on github.com/ncw/swift package. Rackspace works on gophercloud package but it is still work in progress so I haven’t used it.).

The next plan is to implement S3 storage support and Local storage support. and add tests (hope someone likes writing tests). Feel free to ask questions.