Development Update July 13th, 2018
Today’s development update will focus on work completed between July 8th - July 13th.
Overview of Content
- Began planning process for API that will be a key aspect for strategic commercial use cases moving forward
- Created proof of concept of fragile loop.
- Made significant backend improvements to support larger file sizes.
- Continued with frontend improvements to increase file size.
- Oyster roadmap updated.
- Korean and Chinese translations of PRL whitepaper added to site.
Oyster Storage API
This week, the development team worked on pulling shared functionality that could be useful in both the back-end and the front-end into a shared Golang library, which eventually can be made into a cross-compiled binary that could be distributed via Node Package Manager (NPM). Compiling this shared functionality is a step forward towards Oyster’s ultimate goal of building an application programming interface (API) that will significantly increase the appeal of Oyster Storage as a platform for commercial use. For example, consider Amazon’s S3 service. S3 is a standard cloud storage service that supplies a flexible API, allowing users and developers alike to interact with the Amazon S3 service without having to use the user interface provided by Amazon. If you used Dropbox during the company’s first eight years of production, you were using Amazon’s S3 service to store your files in the cloud. Although the interface users interacted with was completely Dropbox’s creation, the actual storage location of the users’ data was on cloud infrastructure provided by Amazon (Dropbox has since moved to their own cloud infrastructure platform).
The Amazon S3 API has become a sort of standard for cloud storage, and many other cloud providers build their own cloud storage services to be “S3-compliant.” However, despite the efficiency provided by Amazon S3, there is still a glaring issue associated with the service: S3 does not come pre-configured to be completely secure. In fact, some of the prominent data breaches that have occurred in the last few years have been the result of misconfigured Amazon S3 Buckets (An “S3 Bucket” can be thought of as the overarching file directory where your uploaded files are stored). To use Amazon’s S3 service securely, a user has to have some understanding of how to use Amazon’s cloud services and put in the necessary effort to secure their S3 buckets (to Amazon’s defense, they have offered several options to help in this respect via their “Config Rule” service, but that requires even more knowledge of the AWS platform!). Ultimately, Oyster will develop an API that is “S3-Compliant”, such that it provides a user-friendly interface to interact with our storage capabilities; however, unlike Amazon S3, Oyster will be able to offer secure and encrypted file storage without the need of user configuration. Users/Developers that use the Oyster Storage API interact with the very same backend used on Oyster Storage’s web interface. Thus, any file that is uploaded via the API is just as secure as the files uploaded via Oyster’s web interface.
There are three distinct benefits to building an API for Oyster Storage. First, an API will increase the appeal for Oyster to be used for commercial storage. Instead of using our web interface, companies can build wrappers around Oyster’s API to automate uploading large amounts of files, as well as managing Oyster Handles. Secondly, creating an S3-compliant API will allow companies that are currently using Amazon S3 (or some other storage service that uses an S3-compliant API) to very easily switch to Oyster. Creating the API in an S3-compliant way will give Oyster a better chance to get more commercial customers because these customers will not have to do nearly as much work to be able to use Oyster’s service. Thirdly, the Oyster API provides a whole new playground for the developer community as a whole. The Oyster community can use Oyster’s API to build whatever application they want that utilizes Oyster’s encrypted, anonymous storage capabilities. Ultimately, the Oyster Storage API will be a significant step forward in cementing Oyster’s wide-spread adoption as an alternative to current, centralized cloud storage solutions.
Oyster’s Fragile Loop Experiment
Besides the update regarding Oyster’s future storage API, the team has also been working on an experiment that showcases the ‘fragile loop’ that will be used to limit overburdening device CPUs when the Oyster web script is running. Along with the consent message, the concept of the fragile loop is the main design difference between aggressive crypto mining scripts and Oyster’s non-intrusive treasure hunting. The goal is to perform meaningful work without hogging the CPU or draining the battery of the user’s device.
Generally, this is achieved by only utilizing a fraction of the devices CPU, as opposed to going full throttle during the user’s entire stay on a website.
A CPU effectively only has the states of active and inactive. Energy saving modes aside, there is no running at, say, 30%. Instead, the CPU usage you see in the task manager means that the processor was active for a specific percentage of time over the last second or so. Thus, the Oyster team can achieve the goal of a non-intrusive web node by emulating this behavior in the browser. By splitting up the work the script has to perform into small chunks and then pausing the script between these chunks of work, the team gets an easily configurable piece of software that does not drain any batteries.
The team has prepared a demonstration of this concept for you to play around with here. Any input or criticism is welcome!
Miscellaneous Improvements
The development team has been doing some pre-deployment QA before Oyster rolls out the badger DB changes which will reduce bottlenecks on the backend and help increase the maximum file size we can support (note that the upcoming deployment of changes will not include the front-end changes which are also needed to support larger file sizes). The development team was able to upload a 200mb file using Oyster’s backend alone. Further QA is needed on the client-side of the Oyster Storage web interface before the current 5mb file size limit will be increased.
The Oyster development team has also implemented polling Oyster’s private tangle from the oyster-streamable library. Additionally, internal testing has begun with uploading files using streams on the client. Furthermore, the team has finished the rev2 client-side reference implementation, which is published here, along with writing the necessary unit tests for this implementation.
Finally, the development team has completed work on broker node/smart contract interactions with OysterPearl, as well as unit testing and QAing the associated transactions. Furthermore, the team has converted many of the React components and Redux actions in Oyster’s codebase from regular JavaScript to TypeScript.
Updated Roadmap and PRL Whitepaper Translations
The Oyster roadmap has been updated on the site, and the Korean and Chinese translations of the Oyster PRL whitepaper have been added as well.
As always, we encourage our community to post any questions or comments regarding these development updates to our Reddit or Telegram channel.
