Introducing S3X “Endless IPFS Dynamic Possibilities”: Stream Videos, Host Dynamic Websites…
S3X is a fork of MinIO that implements a TemporalX gateway, giving users the ability to store data on IPFS via the MinIO API (which is an S3 API).
The development of S3X is about 90% complete (currently usable locally, or via our dev environment, instructions below) and is built on top of the rock-solid foundation provided by MinIO. MinIO is a world-class, open-source object storage solution that is deployed by all types of companies, from blockchain startups to Fortune 500 companies. Because MinIO has been running in production for years , it provides one of the best possible foundations to build upon. By using TemporalX as the means to interface with IPFS, we have been able to circumvent the slower aspects of `go-ipfs`. That means no pinning system performance issues, no memory leaks, etc. The result is a high-performance, enterprise-grade IPFS.
Once we have completed S3X v1 iteration we will be porting the changes upstream to MinIO, in addition to all changes and improvements made moving forward. Companies interested in using S3X will simply have to update their MinIO versions once the changes are upstream in order to connect to TemporalX. With the click of a button, IPFS will be available in your application.
Why Does This Matter?
Until now there has been no effective way of using IPFS without redesigning the existing application. Alternatively, many developers have chosen to design their applications from the ground up with IPFS in mind, however, S3X has broken through this limitation. By changing the S3 endpoint your application uses, S3X will allow you to use IPFS immediately. Only a single line change is required in order to achieve full IPFS capabilities seamlessly. It’s so easy you may not even notice. In fact, unless you query our “side API” you will have no idea that IPFS is being used at all. The side API allows you to analyze buckets and objects to retrieve their corresponding content identifiers (CIDs).
Go-IPFS is great for getting started with IPFS, TemporalX focus is to make enterprise level IPFS usable in production today with amazing performance, reliability and efficiency for our clients. By using TemporalX, our custom enterprise-grade IPFS node, we suffer from none of the issues that currently effect`go-ipfs` in a enterprise environment including:
● Memory leaks: an equally loaded go-ipfs node consumes about 7x-10x the memory of TemporalX, and that’s before the memory leaks ramp up.
● Slow data ingestion at scale: the pinning system suffers a 50% pinning performance reduction when 1TB of data is pinned with a 256kb chunk size, the default size
● Unstable API with intermittent crashes
The Endless Possibilities of S3X
Due to the way S3 works, updates to objects in buckets are essentially “dynamic”. What we mean by this is that if you load the object, and after loading the object new data is added to it, the next time you refresh the object you’ll have the newly updated data. Anytime you load the object, you are guaranteed to be loading the latest version.
This functionality allows you to do things like stream video on IPFS, host a dynamic website, or even build an FTP server on top of IPFS. For a more complete list of “endless possibilities” check out our slides on s3x from Decentralized Networks Summit during ETHDenver (Use arrows to go back and forth).
Getting Started
It’s extremely easy to get started with S3X. You can either download S3X from GitHub and run it locally as shown here (the local S3X version defaults to using the TemporalX development/testing environment node, which is remote and non-local so performance might be an issue) or you can use our publicly accessible dev environment and follow the example here which uses `mc` (the Minio command line client). Ultimately any S3 client can be used Please note: The dev environment is entirely public so it should not be used for sensitive data.
Product Comparisons
Storj IPFS
Storj IPFS is an go-ipfs node that uses Storj as the S3 API endpoint that go-ds-s3 uses. As such it suffers from the same issues all go-ipfs nodes point to. Additionally it is a number of versions behind go-ipfs as development has stagnated.
● You have completely redesign your S3 application to use an IPFS API
● Need to be already using an IPFS API and point it to a storj IPFS node (aka, pointing it to an IPFS node using the storj datastore).
● Currently this is just prototype
● Dependant on go-ipfs, which is already established that production workloads can’t be supported
Textile Buckets
From the publicly available information, it is safe to conclude that Textile Buckets enables object storage characteristics with the Textile stack.
● Bleeding edge, faster and much more efficient then Storj IPFS
● Requires specific application design for, or redesign application to be compatible with Textile
● Experimental product with a non S3 API
Web 2.0: AWS S3
AWS is the gold standard for hosted S3 APIs. The platform is extremely robust, well-tested, and well-supported. The biggest downside, however, is that it requires using AWS specifically. Additionally, the charges, primarily due to bandwidth, can grow quite large and become costly. If you need to upload or download to S3 frequently, with large files, you may find yourself paying for the luxury.
Google Cloud Storage
By and large Google Cloud Storage (GCS) is equivalent to AWS S3 except “Google flavored”. As with AWS S3, costs can grow unexpectedly if your data consumption scales and requires being dependent on a cloud provider.
Wasabi
Wasabi is probably the best “web 2.0 object storage” provider. Wasabi is insanely cheap and does not come with egress charges. Unfortunately, it suffers from some of the same issues that other cloud-based web 2.0 providers face.
If you want a cloud-hosted s3 provider that isn’t powered by IPFS, Wasabi is definitely your best bet.
What is Temporal ?
The most trusted and scalable IPFS Infrastructure and open source developer tools, Temporal allows users to plug into existing applications or build production-ready applications on top of IPFS quickly. Temporal offers a free starter package featuring turnkey APIs, toolkits & 3/GB storage.
Join Temporal’s online community on Twitter or Telegram. We also have some great IPFS tutorials on our medium.
Written by Kevin Vanstone. Follow him online @KevinVanstone for the latest from RTrade Technologies.
