Storj: Not a Dropbox Killer
Storj is a “storage coin” with over $30M of ICO funding. The thousands of investors would be outraged if Storj was anything less than a Dropbox killer. Or would they?
Intro
Today I’ll be trying one of the several “storage coins”. Storj aims to replace Dropbox, Amazon S3, etc. with a decentralized protocol where users are paid to store files on their computers. Storj also has two coins and one ICO.
TL;DR: Storj is too complex and I was unable to use it for file storage.
Goals for the test
- Compile Storj from source code and run it on a Linux server
- Store a photo on Storj and download it from my laptop
- Store a 5 GB file on the Storj and download it from another machine
- Store 5 GB files until I run out of money or disk space
- BONUS: Use Storj without the centralized API server (api.storj.io)
The Storj landing page
Let’s poke around in the Github
A bit further down on the Storj landing page there’s a link to their Github organization.
The Github organization has a number of projects. I don’t know what “farming data” means, so I’ll just go with “core”.
Storj Core has 1,340 commits by 24 contributors. Not much seems to have changed lately. We have to go deeper.
There are 458 closed issues, but few of them recently. The commit pulse will tell me the frequency of code changes.
A single user named bookchin has done nearly all of the work. Who is this mysterious person? Perhaps their CTO?
I won’t spend more time trying to find the person behind the username. But why has the activity stopped? My best guess is that funds have been raised. Why code when you can buy lambos?
Did someone say ICO?
There was a $30MM ICO which ended May 25th. The price of ETH was $177 at the time. Not bad! Dropbox’ first seed round was only $15,000.
Let’s make a server
I’ll create a cloud server on Amazon AWS and use it for the tests. Ubuntu is a very common Linux operating system and likely compatible with Storj.
A server with 36 CPU cores, 60 GB of RAM, and 10 a Gigabit Internet connection should be more than enough.
I’m not sure how much space will be needed. I’m starting with 250 GB. More can be added later.
The server is running in the cloud. I’ll run a couple of commands to upgrade operating system and reboot.
Finding Storj
The Storj Core README states:
However, the Storj CLI is deprecated:
Okay, so I need something called libstorj:
Surprise! There’s a completely different project than the one linked from the Storj landing page. libstorj is written entirely in C. I haven’t written C for many years and hope I don’t need to read any today. Or ever.
C is awful to code and leads to more security issues than Solidity.
Compiling Storj
The README tells me exactly what I need to install first:
Then I’ll download the source code.
To run libstorj I need to compile the source code to binary files using the instructions from the README:
Let’s see how that went:
OBJECTIVE 1 COMPLETED!
Great! But there’s something worrying me. There’s a setting named STORJ_BRIDGE which defaults to api.storj.io. I’m getting flashbacks to Ripple. Am I talking to a centralized service?
I’ll continue with the centralized server for now. I’ve added a bonus objective where I’ll run everything myself like Satoshi intended.
Registering with the centralized Storj server
To register I need a bridge password, an encryption key, and a passphrase to unlock that encryption key.
Clicking the Confirm Email Address button takes me here:
There’s a lot of information here. activated: true sounds good.
Let’s try to list my buckets. I don’t know what a bucket is yet.
$ storj list-buckets
Unlock passphrase: ********************
No buckets.I immediately regret having chosen a random passphrase if I have to enter it for every command, so I’ll just set the handy environment variable they provide:
$ export STORJ_KEYPASS=bdc1bLet’s see if that helped:
$ storj list-buckets
No buckets.Much better. I’ll try to make a bucket:
ubuntu@ip-172-31-31-81:~/libstorj$ storj add-bucket bucket-head
ID: e5ad3b426e39b581b367adee Decrypted: true Name: bucket-headI now have a bucket-head bucket. Can’t ask for more. Time to upload a file.
$ storj upload-file bucket-head bucket-head.png
Preparing File...
Upload failure: Unable to decode hex stringOkay, maybe I have to use the long-form bucket identifier:
Better, but it crashed. I don’t know what the error means. Having used computers before I’ll just keep trying:
Storj “prepared” and uploaded the file in 17 seconds. The next step is to install storj on my laptop where I’ll download the file.
While waiting for libstorj to install on my laptop, I’ll look a little at the pulse of libstorj.
The activity picked up in early 2017 and then slowed down.
Retrieving the file on my laptop
Installing libstorj on macOS Sierra works perfectly using the instructions in the README. I noticed a command, storj import-keys which could be useful for logging in from my laptop:
➜ libstorj git:(master) storj import-keys
Bridge username (email): andreas+tries+storj@brekken.com
Bridge password: ********************If you've previously uploaded files, please enter your existing encryption key (12 to 24 words).
Otherwise leave the field blank to generate a new key.Encryption key: bubble cattle leisure glue marine split cube nut couch when patrol resource bronze tonight tomorrow door skirt chicken cool problem return pudding box parent
We now need to save these settings. Please enter a passphrase to lock your settings.Unlock passphrase: ********************
Again to verify: ********************
Successfully stored bridge username, password, and encryption key to /Users/andreas/.storj/api.storj.io.json➜ libstorj git:(master) storj list-buckets
ID: e5ad3b426e39b581b367adee Decrypted: true Created: 2017-08-13T10:18:19.090Z Name: bucket-head
And now I’ll try to download the image file:
➜ libstorj git:(master) time storj download-file e5ad3b426e39b581b367adee a89e1f3d4fcdf9b69e8cfd2f bucket.png
[======================================================================] 100.00%
Download Success!
storj download-file e5ad3b426e39b581b367adee a89e1f3d4fcdf9b69e8cfd2f 0.42s user 0.24s system 3% cpu 20.475 totalLet’s see the result:
OBJECTIVE 2 COMPLETED!
Storing a single photo is great, but Storj seems like it would be more useful for larger files. Like backups or a high resolution photo collection. I’ll generate a 5 GB file with random data in it.
$ dd if=/dev/urandom of=random-five-gigs.bin bs=1M count=5120
5120+0 records in
5120+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 355.857 s, 15.1 MB/s
ubuntu@ip-172-31-31-81:~/random-files$ ls -lha
total 5.1G
drwxrwxr-x 2 ubuntu ubuntu 4.0K Aug 13 10:43 .
drwxr-xr-x 7 ubuntu ubuntu 4.0K Aug 13 10:41 ..
-rw-rw-r-- 1 ubuntu ubuntu 5.0G Aug 13 10:55 random-five-gigs.bin
ubuntu@ip-172-31-31-81:~/random-files$I just searched online for “generate random files in linux” to find the command above.
It’s time for the real test. I’ll upload the 5 GB file to Storj:
Nope, that didn’t work. The progress bar went fast to around 60%. After that it got slower and slower until the storj program crashed.
I’m not sure what the error message “Unable to receive storage offer” means. Does no one want to store my file? Was there a network error? I’ll just try again.
Nope, now it won’t let me use the service anymore. Rate limit usually means that I’m doing something too fast. I tried again after a few minutes, but the same error appeared.
OBJECTIVE 3 FAILED
Or did it fail? My storj program has been speaking to a centralized server the whole time. The api.storj.io server I noticed previously. For Storj to be decentralized I need to run everything myself.
Warning to readers
Most who read the draft of this post found the next sections too technical. They’re right. It’s about to get confusing and aggravating.
I suggest that you just scroll down to where it says “Conclusion” if you get bored.
Installing Storj Bridge
I noticed Storj Bridge earlier when I was looking at Storj’s Github organization.
The README file for Storj Bridge says that it can’t run alone:
I’ll install Storj Complex on my server first.
I don’t know what a renter node or an identity means, but let’s try anyway.
Installing Storj Complex
Storj Complex does not have any release tags/numbered versions. I’ll install from the latest code.
$ git clone https://github.com/Storj/complex.git
$ cd complexStorj Complex runs on Node.js. I’ll install Node.js first.
I also need to install RabbitMQ, a message queue program. Message queues are often used to communicate between programs.
And MongoDB, a database reasonably good at storing JSON.
I also need to install some packages which Storj Complex relies on. I’ll install these packages using the Node Package Manager, npm.
Okay, that was a lot of installing. Finally I can run Storj Complex.
That’s fair. I should have looked at the README file where it told me to specify a config file. I already noticed some example config files while poking around the code.
The example config files are the ones ending in .json. There’s a few different ones. I don’t know what a renter or landlord is. I’ll try the one named multi-config.json. It has setups for a landlord and a few renters.
I’ll try running with a config file:
The MongoDB URL is invalid. Well, it’s not even in the config file. Without a MongoDB URL, Storj Complex won’t know where the MongoDB database is running.
I’ll just go ahead and open an issue for that on Github.
Maybe running only a landlord is easier:
Something is running!
It can’t find the example private key. I’ll move it from the example directory to the project directory and try again.
Now I’m having problems with the code and not just the way I’m running it. I’ll submit an issue for this too. I’m guessing hdIndex and hdKey refer to hierarchical deterministic (HD) keys. You probably know HD keys from your Bitcoin wallet. But where is hdIndex supposed to be specified and what should the value be?
I’ll start reading the Storj Complex code.
The offending line is number 88:
The two solutions I can think of immediately are:
- Find a way to supply the
hdIndex - Not use HD at all. This way the code starting at line 94 runs instead.
I’ll search the Storj Complex code on Github code for hdIndex.
These code snippets do not help me at all. Maybe the answer is in some file.
I’ll just edit the Storj Complex code to make the default hdIndex equal to 0. What could go wrong?
It’s still crashing. Now I’m being told I picked a badhdIndexvalue. I don’t understand why. Hardened HD key indexes are really big numbers.
I haven’t looked at this since the first time I was working on it for Bitcoin in January 2014! I’ll just try 1 instead.
And run again:
Storj Complex is running! I don’t know what it does, but something is running.
And I’m connecting to peers!
Installing Storj Bridge (again)
Storj Bridge is also written in Node.js. I’ll install the dependencies like before. I already installed the MongoDB dependency for Storj Complex and hope they can share the installation.
That installed a lot of dependencies! 450 packages.
Next, I’ll start up Storj Bridge.
The Storj Bridge is running!
But it can’t connect to Redis. That’s weird. The documentation doesn’t mention Redis. I’ll just submit another issue for that and install Redis.
In case you’re wondering, Redis is a really fast key-value database used by Twitter, Weibo, Snapchat, and many others.
Redis is installed and working. I’ll try to run Storj Bridge again.
Much better. Connected to redis!
Running Storj with my own Sotrj Bridge
I mentioned previously that storj has a STORJ_BRIDGE setting. I’m not sure which port my Storj Bridge is running on. Port 8080 is already in use by Storj Complex. I’ll search the Storj Bridge code for the word port.
Maybe it’s 6383? I’ll check if something’s responding on that port:
Something is listening on port 6382! I’ll runstorj with the bridge set to that port on my own server.
I’m registered with my own Storj Bridge! But how will I receive the activation link? I never set the e-mail server settings for Storj Bridge. How could the bridge send me my activation link?
The Storj Bridge log shows:
{"level":"error","message":"failed to send activation email, reason: connect ECONNREFUSED 127.0.0.1:465","timestamp":"2017-08-13T12:27:51.642Z"}I haven’t set up an outbound email server (SMTP) in a long time. If I just run one on my server there’s a high risk that Gmail will block its emails as spam.
Maybe I can just find my user record in the Storj Bridge MongoDB database and see if it has my activation code.
I’ll start the interactive MongoDB shell. Using the shell I can query and even modify database records.
Next I’ll see what databases there are using the show dbs command.
> show dbs
__storj-bridge-develop 0.000GB
__storj-complex-example 0.000GB
admin 0.000GB
local 0.000GBThe database containing the word “bridge” is the one I want. Each database has several collections like an Excel document has multiple sheets. I’ll look at which collections there are in the Storj Bridge database.
> use __storj-bridge-develop
switched to db __storj-bridge-develop
> show collections
bucketentries
buckets
contacts
fullaudits
marketings
mirrors
partners
pointers
referrals
shards
storageevents
usernonces
usersusers is probably the one I want. I’ll use the find command to display every record in it.
The output is a hard to read without formatting. After a little research I remember there’s a “pretty” formatting option.
> db.users.find({}).pretty()I can read the output easily now:
This looks pretty promising. Maybe the long string toward the end is the activation code? I’ll look at the email I received from Storj to see what the activation link looks like:
http://api.storj.io/activations/0b629ec80837b58b38d2d0c2ea2dedc0db5fd00edf13f76aa19807d4edba534dMaybe? Let’s try:
The Storj Bridge responds with a bunch of things, including activated: true . I hope it worked.
I’ve activated my Storj Bridge user!
The moment I’ve been waiting for is here. I’ll try to upload a file.
Storj Bridge is getting stuck on Preparing File.
I start looking through the logs of everything. There’s not a lot. Storj Complex log mentions jobs timing out:
At this point I increase the amount of logging to the maximum setting in hope of finding a clue.
I find absolutely nothing.
I look through the RabbitMQ logs.
Nope.
The “Preparing File…” message shows forever and there are no clues in the log.
I have spent most of my Sunday evening trying to upload a single 5 GB file.
I have compiled 4 Storj programs. For some I had to edit their source code.
I have installed two different database systems and a message queue.
Why are there so many programs? What do any of these words mean; landlord, renter, core, farm, libstorj, bridge, complex?
I give up for now.
OBJECTIVE 3 FAILED
OBJECTIVE 4 FAILED
OBJECTIVE 5 FAILED
I’m open to trying again later. For now I’m deleting the server. Dropbox with client-side encryption works fine for now.
If you’re an investor in Storj I suggest you spend some time with the product.
Good night, blockchainers.
About The Author
Andreas Brekken is the CTO/co-founder of Helix Capital and has been obsessed with cryptocurrency since 2011. He was the CTO/co-founder of Justcoin.com and later worked as an engineer with Kraken.com.
Follow on Medium and Twitter for more reviews. Which coin should I review next? Write me in the comments.
