Behind the Scenes: OctoSync
Disclaimer: This article is a bit long. If you just want to know how it works, skip over to fourth paragraph.
this article was originally shared on our BLOG by the CTO of Octonius, Anoop Nayak.
First of all let me rewind things to an year back. Cosmin(CEO of Octonius) called me saying, “Hey its a big mess here man! I can’t find this file for that guy. Do you know which cloud I stored it in?”. I was like “Ugh how am I supposed to know. All I see is cosmin storing files in some folder in finder, now whether that is in dropbox or gdrive I don’t know”. This happened because cosmin had crashed his mac before and had to reinstall the whole OS and now the files were just syncing to his mac.
Then immediately he asked me “Is it possible to copy a file from Gdrive to dropbox?”. And the obvious reply was a Yes. And then the stunner “Can you keep those files in sync?”. Now I was really stuck! To keep the files in sync its a different challenge altogether. But I knew it was technically possible. The mind was full of cloud api calls, polling etc. And so I said it’s possible.
It’s been almost a year after that when I was tuning the code to attain maximum performance which means transferring the files in the fastest way, lowering API calls to the minimum and ensuring the user’s privacy(yea you can say I’m paranoid about the user’s privacy) and of course the result was OctoSync.
Now coming to the point. When you share a file from your cloud(say google drive) via octonius, your friend who has octonius gets the file in his preferred cloud(as shown below). This is simple i.e the first part. Just transferring the file from one cloud to another.
It can be done in basically 2 ways.The most obvious way is to download the file from the cloud to octonius servers and then transfer it to the recipients cloud. But wait! Hold on there… Will you be comfortable with this way? Just ask yourself. Your file will stay in octonius servers for say around 5 minutes until it is transferred to the other cloud. I personally wouldn’t be happy about it.
So I had to think a lot deeper to find out the second way. The idea is to stream a file from one cloud to another cloud. So imagine your file is like water flowing through a pipe from your Google Drive to your friend’s Dropbox(if he has set his preferred cloud to Dropbox). And here as you octonius just creates the pipe and makes sure the file goes through correctly without any leaks(leaking would be analogous to a security breach here). Since its common sense that octonius doesn’t know anything about the file except for its name, size and the last modified date. Those details are just required to keep track of who changed what and when.
Now the file has been transferred safely. Now the hardest part. How on earth do you keep the files in sync? To answer that, there are 2 options again. The easiest way would be to ask the clouds whether the file is updated in a specified interval(say 5 minutes) and then if the file has been changed we can replace the older file. Suppose your friend changed the file in his cloud, then it means that you have the old file which will be replaced when octonius gets to know about it. WOW! Problem solved isn’t it. But the developers might tell you that it is not the best way. And the cloud providers(Google Drive, Dropbox etc.) would be totally dissappointed if you do this and sometimes they even block you if you cross your limits.
So here comes something known as “webhooks” or some people know it by the name “Push Notifications”. Here the idea is as soon as you or your friend change the file, the cloud provider calls the octonius servers telling that “Hey! This file owned by this guy is changed…” And this then kicks up OctoSync if it is necessary i.e if the files are out of sync the older file is replaced with the newer file with OctoSync. So at the end this is how Octonius works.
I have abstracted the deeper tech stuff just not to bore you guys more but we at Octonius want just to tell you that “We don’t even read 1 byte of your file. Period” and so your privacy is guaranteed using octonius. In the meanwhile if you haven’t tried octonius I would like you to use Octonius and let me know how it is :). I’m pretty sure you should like it if you use more than 1 cloud in your daily life. If you want to get deeper details feel free to contact me at anoop [at] octonius [dot] com. Or message me at @anoopknayak in twitter.
PS: Since you had the patience to bear me uptil this point, I will like to offer you a promo code if you can contact me via my mail (anoop [at] octonius [dot] com) with the subject PROMOCODE and a small text so as how you use octonius.