Storj: Not a Dropbox Killer

Andreas Brekken
Aug 24, 2017 · 14 min read

Storj is a “storage coin” with over $30M of ICO funding. The thousands of investors would be outraged if Storj was anything less than a Dropbox killer. Or would they?

Intro

Today I’ll be trying one of the several “storage coins”. Storj aims to replace Dropbox, Amazon S3, etc. with a decentralized protocol where users are paid to store files on their computers. Storj also has two coins and one ICO.

TL;DR: Storj is too complex and I was unable to use it for file storage.

Goals for the test

  1. Compile Storj from source code and run it on a Linux server
  2. Store a photo on Storj and download it from my laptop
  3. Store a 5 GB file on the Storj and download it from another machine
  4. Store 5 GB files until I run out of money or disk space
  5. BONUS: Use Storj without the centralized API server (api.storj.io)

The Storj landing page

Image for post
Image for post
The landing page markets Storj as encrypted, open-source, and sustainable. I’m not sure what sustainable means in this context.

Let’s poke around in the Github

A bit further down on the Storj landing page there’s a link to their Github organization.

Image for post
Image for post
Image for post
Image for post

The Github organization has a number of projects. I don’t know what “farming data” means, so I’ll just go with “core”.

Image for post
Image for post

Storj Core has 1,340 commits by 24 contributors. Not much seems to have changed lately. We have to go deeper.

Image for post
Image for post
Closed issues are more interesting than open issues. Open issues are often end users asking support questions.

There are 458 closed issues, but few of them recently. The commit pulse will tell me the frequency of code changes.

Image for post
Image for post

A single user named bookchin has done nearly all of the work. Who is this mysterious person? Perhaps their CTO?

Image for post
Image for post
You can also see his website at http://bookch.in. I recommend viewing the source where he uses an old-school but effective CSS vertical and horizontal align trick.

I won’t spend more time trying to find the person behind the username. But why has the activity stopped? My best guess is that funds have been raised. Why code when you can buy lambos?

Did someone say ICO?

Image for post
Image for post

There was a $30MM ICO which ended May 25th. The price of ETH was $177 at the time. Not bad! Dropbox’ first seed round was only $15,000.

Let’s make a server

I’ll create a cloud server on Amazon AWS and use it for the tests. Ubuntu is a very common Linux operating system and likely compatible with Storj.

Image for post
Image for post
Image for post
Image for post

A server with 36 CPU cores, 60 GB of RAM, and 10 a Gigabit Internet connection should be more than enough.

Image for post
Image for post

I’m not sure how much space will be needed. I’m starting with 250 GB. More can be added later.

Image for post
Image for post
There’s no place like 54.183.41.1

The server is running in the cloud. I’ll run a couple of commands to upgrade operating system and reboot.

Finding Storj

The Storj Core README states:

Image for post
Image for post

However, the Storj CLI is deprecated:

Image for post
Image for post

Okay, so I need something called libstorj:

Surprise! There’s a completely different project than the one linked from the Storj landing page. libstorj is written entirely in C. I haven’t written C for many years and hope I don’t need to read any today. Or ever.

C is awful to code and leads to more security issues than Solidity.

Compiling Storj

The README tells me exactly what I need to install first:

Adding ( braces ) makes it easy to input multiple commands, even with new lines, in the terminal

Then I’ll download the source code.

Image for post
Image for post
The latest release is version v1.0.1 which came out on June 29th
Image for post
Image for post
Downloading the source code from Github is done using the version control system “git”

To run libstorj I need to compile the source code to binary files using the instructions from the README:

Image for post
Image for post
This is a typical compile process for C programs. Autogen generates compilation instructions specific to my operating system and the program make runs the compiling process.

Let’s see how that went:

Image for post
Image for post

OBJECTIVE 1 COMPLETED!

Image for post
Image for post

Great! But there’s something worrying me. There’s a setting named STORJ_BRIDGE which defaults to api.storj.io. I’m getting flashbacks to Ripple. Am I talking to a centralized service?

I’ll continue with the centralized server for now. I’ve added a bonus objective where I’ll run everything myself like Satoshi intended.

Registering with the centralized Storj server

Image for post
Image for post
So. Many. Passwords. I make mine the command: cat /dev/urandom | head -c 10 | xxd -p

To register I need a bridge password, an encryption key, and a passphrase to unlock that encryption key.

Image for post
Image for post
I’ve never actually seen this movie.
Image for post
Image for post

Clicking the Confirm Email Address button takes me here:

Image for post
Image for post
I’m surprised they respond with this cryptic message. It would be nicer to be redirected to their website.

There’s a lot of information here. activated: true sounds good.

Let’s try to list my buckets. I don’t know what a bucket is yet.

$ storj list-buckets
Unlock passphrase: ********************
No buckets.

I immediately regret having chosen a random passphrase if I have to enter it for every command, so I’ll just set the handy environment variable they provide:

$ export STORJ_KEYPASS=bdc1b

Let’s see if that helped:

$ storj list-buckets
No buckets.

Much better. I’ll try to make a bucket:

ubuntu@ip-172-31-31-81:~/libstorj$ storj add-bucket bucket-head
ID: e5ad3b426e39b581b367adee Decrypted: true Name: bucket-head

I now have a bucket-head bucket. Can’t ask for more. Time to upload a file.

$ storj upload-file bucket-head bucket-head.png
Preparing File...
Upload failure: Unable to decode hex string

Okay, maybe I have to use the long-form bucket identifier:

Image for post
Image for post

Better, but it crashed. I don’t know what the error means. Having used computers before I’ll just keep trying:

Image for post
Image for post

Storj “prepared” and uploaded the file in 17 seconds. The next step is to install storj on my laptop where I’ll download the file.

While waiting for libstorj to install on my laptop, I’ll look a little at the pulse of libstorj.

Image for post
Image for post

The activity picked up in early 2017 and then slowed down.

Retrieving the file on my laptop

Installing libstorj on macOS Sierra works perfectly using the instructions in the README. I noticed a command, storj import-keys which could be useful for logging in from my laptop:

➜  libstorj git:(master) storj import-keys
Bridge username (email): andreas+tries+storj@brekken.com
Bridge password: ********************
If you've previously uploaded files, please enter your existing encryption key (12 to 24 words).
Otherwise leave the field blank to generate a new key.
Encryption key: bubble cattle leisure glue marine split cube nut couch when patrol resource bronze tonight tomorrow door skirt chicken cool problem return pudding box parent
We now need to save these settings. Please enter a passphrase to lock your settings.
Unlock passphrase: ********************
Again to verify: ********************
Successfully stored bridge username, password, and encryption key to /Users/andreas/.storj/api.storj.io.json
➜ libstorj git:(master) storj list-buckets
ID: e5ad3b426e39b581b367adee Decrypted: true Created: 2017-08-13T10:18:19.090Z Name: bucket-head

And now I’ll try to download the image file:

➜  libstorj git:(master) time storj download-file e5ad3b426e39b581b367adee a89e1f3d4fcdf9b69e8cfd2f bucket.png
[======================================================================] 100.00%
Download Success!
storj download-file e5ad3b426e39b581b367adee a89e1f3d4fcdf9b69e8cfd2f 0.42s user 0.24s system 3% cpu 20.475 total

Let’s see the result:

Image for post
Image for post
Brian Patrick Carroll (born May 13, 1969), known professionally as Buckethead, is an American guitarist and multi-instrumentalist who has worked within many genres of music.

OBJECTIVE 2 COMPLETED!

Image for post
Image for post

Storing a single photo is great, but Storj seems like it would be more useful for larger files. Like backups or a high resolution photo collection. I’ll generate a 5 GB file with random data in it.

$ dd if=/dev/urandom of=random-five-gigs.bin bs=1M count=5120
5120+0 records in
5120+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 355.857 s, 15.1 MB/s
ubuntu@ip-172-31-31-81:~/random-files$ ls -lha
total 5.1G
drwxrwxr-x 2 ubuntu ubuntu 4.0K Aug 13 10:43 .
drwxr-xr-x 7 ubuntu ubuntu 4.0K Aug 13 10:41 ..
-rw-rw-r-- 1 ubuntu ubuntu 5.0G Aug 13 10:55 random-five-gigs.bin
ubuntu@ip-172-31-31-81:~/random-files$

I just searched online for “generate random files in linux” to find the command above.

It’s time for the real test. I’ll upload the 5 GB file to Storj:

Image for post
Image for post

Nope, that didn’t work. The progress bar went fast to around 60%. After that it got slower and slower until the storj program crashed.

I’m not sure what the error message “Unable to receive storage offer” means. Does no one want to store my file? Was there a network error? I’ll just try again.

Image for post
Image for post

Nope, now it won’t let me use the service anymore. Rate limit usually means that I’m doing something too fast. I tried again after a few minutes, but the same error appeared.

OBJECTIVE 3 FAILED

Image for post
Image for post

Or did it fail? My storj program has been speaking to a centralized server the whole time. The api.storj.io server I noticed previously. For Storj to be decentralized I need to run everything myself.

Warning to readers

Image for post
Image for post
It’s about to get very ugly very quickly.

Most who read the draft of this post found the next sections too technical. They’re right. It’s about to get confusing and aggravating.

I suggest that you just scroll down to where it says “Conclusion” if you get bored.

Image for post
Image for post

Installing Storj Bridge

I noticed Storj Bridge earlier when I was looking at Storj’s Github organization.

Image for post
Image for post

The README file for Storj Bridge says that it can’t run alone:

Image for post
Image for post

I’ll install Storj Complex on my server first.

Image for post
Image for post

I don’t know what a renter node or an identity means, but let’s try anyway.

Installing Storj Complex

Storj Complex does not have any release tags/numbered versions. I’ll install from the latest code.

$ git clone https://github.com/Storj/complex.git
$ cd complex

Storj Complex runs on Node.js. I’ll install Node.js first.

I also need to install RabbitMQ, a message queue program. Message queues are often used to communicate between programs.

And MongoDB, a database reasonably good at storing JSON.

I also need to install some packages which Storj Complex relies on. I’ll install these packages using the Node Package Manager, npm.

Okay, that was a lot of installing. Finally I can run Storj Complex.

That’s fair. I should have looked at the README file where it told me to specify a config file. I already noticed some example config files while poking around the code.

The example config files are the ones ending in .json. There’s a few different ones. I don’t know what a renter or landlord is. I’ll try the one named multi-config.json. It has setups for a landlord and a few renters.

Image for post
Image for post

I’ll try running with a config file:

The MongoDB URL is invalid. Well, it’s not even in the config file. Without a MongoDB URL, Storj Complex won’t know where the MongoDB database is running.

I’ll just go ahead and open an issue for that on Github.

Maybe running only a landlord is easier:

Something is running!

It can’t find the example private key. I’ll move it from the example directory to the project directory and try again.

Image for post
Image for post

Now I’m having problems with the code and not just the way I’m running it. I’ll submit an issue for this too. I’m guessing hdIndex and hdKey refer to hierarchical deterministic (HD) keys. You probably know HD keys from your Bitcoin wallet. But where is hdIndex supposed to be specified and what should the value be?

I’ll start reading the Storj Complex code.

The offending line is number 88:

The two solutions I can think of immediately are:

  • Find a way to supply the hdIndex
  • Not use HD at all. This way the code starting at line 94 runs instead.

I’ll search the Storj Complex code on Github code for hdIndex.

Image for post
Image for post

These code snippets do not help me at all. Maybe the answer is in some file.

I’ll just edit the Storj Complex code to make the default hdIndex equal to 0. What could go wrong?

Image for post
Image for post
I’m excited to show off my Node.js skills.
Image for post
Image for post
To edit files on a server I have to use whatever editor is available. In this case that’s vim. I just turned on line numbers and jumped to the correct line. I’m finally being rewarded for having played so much VIM Adventures. http://vim-adventures.com

It’s still crashing. Now I’m being told I picked a badhdIndexvalue. I don’t understand why. Hardened HD key indexes are really big numbers.

I haven’t looked at this since the first time I was working on it for Bitcoin in January 2014! I’ll just try 1 instead.

Image for post
Image for post

And run again:

Storj Complex is running! I don’t know what it does, but something is running.

Image for post
Image for post

And I’m connecting to peers!

Image for post
Image for post

Installing Storj Bridge (again)

Storj Bridge is also written in Node.js. I’ll install the dependencies like before. I already installed the MongoDB dependency for Storj Complex and hope they can share the installation.

That installed a lot of dependencies! 450 packages.

Next, I’ll start up Storj Bridge.

Image for post
Image for post

The Storj Bridge is running!

But it can’t connect to Redis. That’s weird. The documentation doesn’t mention Redis. I’ll just submit another issue for that and install Redis.

In case you’re wondering, Redis is a really fast key-value database used by Twitter, Weibo, Snapchat, and many others.

Redis is installed and working. I’ll try to run Storj Bridge again.

Image for post
Image for post

Much better. Connected to redis!

Image for post
Image for post
If you haven’t seen the movie The Bridge on the River Kwai (1957) I want you to close this tab immediately. Go watch the movie! It’s spectacular. http://www.imdb.com/title/tt0050212/

Running Storj with my own Sotrj Bridge

I mentioned previously that storj has a STORJ_BRIDGE setting. I’m not sure which port my Storj Bridge is running on. Port 8080 is already in use by Storj Complex. I’ll search the Storj Bridge code for the word port.

Image for post
Image for post

Maybe it’s 6383? I’ll check if something’s responding on that port:

Something is listening on port 6382! I’ll runstorj with the bridge set to that port on my own server.

Image for post
Image for post

I’m registered with my own Storj Bridge! But how will I receive the activation link? I never set the e-mail server settings for Storj Bridge. How could the bridge send me my activation link?

The Storj Bridge log shows:

{"level":"error","message":"failed to send activation email, reason: connect ECONNREFUSED 127.0.0.1:465","timestamp":"2017-08-13T12:27:51.642Z"}

I haven’t set up an outbound email server (SMTP) in a long time. If I just run one on my server there’s a high risk that Gmail will block its emails as spam.

Maybe I can just find my user record in the Storj Bridge MongoDB database and see if it has my activation code.

I’ll start the interactive MongoDB shell. Using the shell I can query and even modify database records.

Next I’ll see what databases there are using the show dbs command.

> show dbs
__storj-bridge-develop 0.000GB
__storj-complex-example 0.000GB
admin 0.000GB
local 0.000GB

The database containing the word “bridge” is the one I want. Each database has several collections like an Excel document has multiple sheets. I’ll look at which collections there are in the Storj Bridge database.

> use __storj-bridge-develop
switched to db __storj-bridge-develop
> show collections
bucketentries
buckets
contacts
fullaudits
marketings
mirrors
partners
pointers
referrals
shards
storageevents
usernonces
users

users is probably the one I want. I’ll use the find command to display every record in it.

The output is a hard to read without formatting. After a little research I remember there’s a “pretty” formatting option.

> db.users.find({}).pretty()

I can read the output easily now:

Image for post
Image for post

This looks pretty promising. Maybe the long string toward the end is the activation code? I’ll look at the email I received from Storj to see what the activation link looks like:

http://api.storj.io/activations/0b629ec80837b58b38d2d0c2ea2dedc0db5fd00edf13f76aa19807d4edba534d

Maybe? Let’s try:

Image for post
Image for post

The Storj Bridge responds with a bunch of things, including activated: true . I hope it worked.

I’ve activated my Storj Bridge user!

Image for post
Image for post

The moment I’ve been waiting for is here. I’ll try to upload a file.

Image for post
Image for post

Storj Bridge is getting stuck on Preparing File.

Image for post
Image for post

I start looking through the logs of everything. There’s not a lot. Storj Complex log mentions jobs timing out:

Image for post
Image for post

At this point I increase the amount of logging to the maximum setting in hope of finding a clue.

I find absolutely nothing.

I look through the RabbitMQ logs.

Image for post
Image for post

Nope.

Image for post
Image for post
You’ll end up here if you searched the page for the “conclusion”
Image for post
Image for post

The “Preparing File…” message shows forever and there are no clues in the log.

I have spent most of my Sunday evening trying to upload a single 5 GB file.

I have compiled 4 Storj programs. For some I had to edit their source code.

I have installed two different database systems and a message queue.

Why are there so many programs? What do any of these words mean; landlord, renter, core, farm, libstorj, bridge, complex?

I give up for now.

OBJECTIVE 3 FAILED

OBJECTIVE 4 FAILED

OBJECTIVE 5 FAILED

Image for post
Image for post

I’m open to trying again later. For now I’m deleting the server. Dropbox with client-side encryption works fine for now.

Image for post
Image for post

If you’re an investor in Storj I suggest you spend some time with the product.

Good night, blockchainers.

About The Author

Andreas Brekken is the CTO/co-founder of Helix Capital and has been obsessed with cryptocurrency since 2011. He was the CTO/co-founder of Justcoin.com and later worked as an engineer with Kraken.com.

Follow on Medium and Twitter for more reviews. Which coin should I review next? Write me in the comments.

Shitcoin.com

There’s a lot of “blockchain” projects out there, but do…

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight.

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox.

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store