SharedForestStore: A New Open-Source Way to “Pin” Content on IPFS
We at Temporal are constantly working on improving and pushing the boundaries of IPFS to provide high-performance data-intensive services. Sometimes, when we encounter performance bottlenecks, such as Pinning we take a chainsaw to it.
What is SharedForestStore?
SharedForestStore https://github.com/RTradeLtd/go-ipfs-sharedforeststore is our newly-released alternative to pinning. The name describes it’s core features:
· Shared: Intended to be used in a concurrent, multi-user, environment.
· Forest: Handles a collection of hash trees.
· Store: Stores data, manage both metadata and binary data internally.
How is it different from Pinning?
In IPFS, Pinning https://github.com/ipfs/go-ipfs-pinner is an added concept on top of blockstore https://github.com/ipfs/go-ipfs-blockstore. This separation of concerns causes several problems:
· Synchronization between pinner and blockstore is done in-process, through a global lock.
· Pinner stores and processes much more metadata than otherwise necessary.
· Pin modification is slow and only reuse existing work in limited situations.
· Locking garbage collection to remove unpinned data.
SharedForestStore solves these problems by:
· Synchronization is done by using a transactional database. Currently, datastore.TxnDatastore https://godoc.org/github.com/ipfs/go-datastore#TxnDatastore is implemented for ease of adoption in IPFS infrastructure.
· Integration of CID and content data minimized additional metadata storage. Internal hashes are retrieved from content data on use.
· Store modification always reuses existing work. The operational complexity is proportional to the amount of data added or removed.
· Data deletion immediately frees up space. No garbage collection needed. If you desire opportunistic caching, you can delete from a list when storage is low.
What makes it performant?
Here is an example to illustrate how sharing and reuse works. If two users, A and B, both add a tag on the same CID. The first time a tag (Ta) is added, the whole data referenced by the CID is saved and tagged with {Ta}. When user B adds the tag (Tb), the tagged set on the data becomes {Ta, Tb}. User B’s operation happened with a single operation. Since only A can remove Ta, and B Tb, the data referenced is only removed when it is removed by both users. If A and B tags different CIDs, but they share a sub hash tree, then the same merging of operation happens internally. This example can be extended to different users, processes, or versions. It all works as long as tags are unique and namespaced to avoid malicious activity.
For developers, SharedForestStore exposes several interfaces and implementations targeting different use cases. The interfaces are intentionally designed to not overlap existing IPFS interfaces to differentiate functionality and smooth transition.
For technical details, please visit our open-source repo at https://github.com/RTradeLtd/go-ipfs-sharedforeststore. We welcome all questions and contributing interactions. SharedForestStore is still in its early dates, so there are no compatibly guarantees yet. The future is in you to shape.
What is Temporal ?
The most trusted and scalable IPFS Infrastructure and open source developer ols, Temporal allows users to plug into existing applications or build production-ready applications on top of IPFS quickly. Temporal offers a free starter package featuring turnkey APIs, toolkits & 3/GB storage.
Start uploading right away https://play2.temporal.cloud/.
Join Temporal’s online community on Twitter or Telegram.
