Go to the profile of Kelvin
Kelvin

google-drive-ocamlfuse + encfs = painfully slow

Long story short, I’ve been using NetDrive and Boxcryptor Classic in an attempt to upload ludicrous amounts of data to Google Drive.

Just over two months worth of uploading at 10Mb/s

So far, so well… but uploading terabytes of data at 10Mb/s is a bit unpleasant. The idea has been lingering in the back of my head for quite some time, so I thought I’d give it a try during the holiday weekend:

Since most of what I’m “backing up” is “totally legal software” that’s “freely available on the Internet via HTTP download”, why not do all of this on a gigabit VPS?

Things appeared to be fairly straightforward. First up, get a modest VPS from DigitalOcean. I chose the $20/mo option:

  • 2 GB RAM / 2 CPUs
  • 40 GB SSD Disk
  • 3 TB transfer

Then I installed google-drive-ocamlfuse and encfs:

Everything worked.

But oh man, it’s unbearably slow. Allow me to illustrate:

aaaaaaaaaaaaaaaaaaaaaaaaaawful

I began downloading directly to the encfs mount half-way through the graphs above.

The download went at 300KB/s, but pegged out the disk at 15MB/s. WTF, right?

I tried a second, simultaneous download to the encfs mount to see if I could speed things up a bit.

This resulted in a 60% spike in CPU activity and a sudden drop in disk writes.

2 requests/sec to the Google Drive API

… and caused a bunch of API requests to Google Drive, with a combined upload speed of roughly 25Mb/s.

Ugh.

Could it be a bandwidth/peering issue?

— 2015–12–27 07:35:11 — http://speedtest.london.linode.com/100MB-london.bin
Resolving speedtest.london.linode.com (speedtest.london.linode.com)… 176.58.107.39, 2a01:7e00::4b
Connecting to speedtest.london.linode.com (speedtest.london.linode.com)|176.58.107.39|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 104857600 (100M) [application/octet-stream]
Saving to: /dev/null
100%[=====================================================================================================================>] 104,857,600 107MB/s in 0.9s
2015–12–27 07:35:12 (107 MB/s) — /dev/null saved [104857600/104857600]

Probably not: gigabit speeds from DigitalOcean Amsterdam to Linode London.

Could it be a problem with the way I’m downloading/uploading?

Maybe. To test this:

  • I downloaded a 2GB test file (at 100MB/s) and saved it to my home directory
  • I then attempted to copy it into the mounted encfs directory
  • and then I waited
  • and waited
[waiting intensifies]

The same symptoms appeared:

  • sustained CPU activity
  • roughly 15MB/s of writes for an hour

… but these were followed by a burst of >100Mb/s upload activity.

I don’t know about you, but something seems a bit off here:

yeah idk

Remember: the file I’m copying/uploading is only 2GB.

smh tbh fam

The sustained ~50% CPU activity probably means encfs is maxing out one of the two cores, but the write activity is puzzling.

Even so, I manage to get at around 10MB/s on a single i5–3470S core with NetDrive and Boxcryptor.

I can’t really be bothered to investigate further, so I’ll be sticking to those two over google-drive-ocamlfuse and encfs.