Oyster Storage Now Supports 25MB Uploads!
The Oyster development team is pleased to announce that the initial file size limit of 5MB associated with the mainnet launch of the Oyster Storage platform has been increased to 25MB.
The development team optimized several parts of the platform to increase the original file size limit from 5MB to 25MB. For example, previously the client would process the entire file before sending all the chunks. For extensive files, this would cause memory crashes in the browser because the client had to be holding the entire processed file. Now, Oyster Storage handles the file in batches, and sends the chunks in batches, avoiding the client-side out-of-memory crashes. Additionally, all the data for a chunk was stored in a MySQL row, including the “message” field which is 2187 characters long and was burdensome on Oyster’s queries. Now, the development team has moved this large field to Badger, which is a key-value database. The key for the message field is still stored in the MySQL row, but it is considerably smaller than the message field. The team plans to move further and further away from MySQL for data map chunks to further improve optimization. Another change made was that chunk data is now stored in the Badger database and other data about a chunk (status, address, etc.) is stored in MySQL. Previously, these two databases were somewhat coupled; the badger data could not be added to the broker node until the MySQL data was added. Now, these two processes are independent of one another, which means the client can go ahead and start sending chunk data as soon as the session has been created, instead of waiting for the broker node to build the data map (which can take a long time for large files).
We ask our community to note that the team is aware of concurrency limitations. If many users upload large files simultaneously, uploads will slow down considerably. We are working on addressing this issue.
In addition to this file size increase, the development team already has some ideas to push the file size limit to an even larger limit. Currently, Oyster Storage is capable of handling file uploads of ~100MB; however, there is a concurrency issue in the infrastructure, such that if many individuals try to upload files of this magnitude at the same time, it is likely that the upload will timeout before it is finished. Once the development team has implemented their ideas to deal with these concurrency issues, they will begin further testing of larger file uploads (~100Mb) to ensure that many users can upload files of this magnitude concurrently without a timeout occurring.
Some of the changes that the development team are exploring to reach larger upload sizes and higher overall throughput are:
- implement round-robinning of uploads so that every user’s upload receives a little attention
- continue to depend less on MySQL
- switch to Hercules when it is released
- use AWS Lambdas (a form of serverless functionality) to offload the Proof-of-Work burden elsewhere
- add more broker nodes to Oyster’s network
The development team will be working tirelessly to improve the file size limit even further, and we will continually update our community regarding the progress made towards this goal.
Try out the new file size limit here: https://oysterstorage.com/
