The Glorious History of How I got to the Cloud
On May 2016 I and a really great group of people started VCore, a startup focused on automated investments on BitCoin. Our greatest problem always was computational power to run our algorithm.
From the beginning, there were two ideas of how we could work this out, buy a server and run everything ourselves or put everything on the cloud and see what would happen.
The first problem was data mining, how could we keep a machine turned on 24/7? The obvious reason was to go to the cloud, and there we went. Nobody on the team had any previous experience with cloud computing e thus we made our first mistake, placing everything on an Amazon EC2.
On our head the cloud was just somebody else’s machine, so we took the first free machine AWS offered and threw all our code inside. It worked for a while, the mining was running OK, but when we tried to run the algorithm on the data the EC2 simply gave up.
We had the luck of meeting a highly specialized mentor who understands a lot of infrastructure. He put us on the path to own our server. After a lot of research on Linux dystros, hardware, CUDA, etc. On our estimates the machine we would need to run everything was going to cost us BRL15.000,00 having two for increased security would lead us to have our costs starting on BRL30.000,00 not knowing if it was going to actually work, a really bad idea!
Amazon to the rescue!
Soon after that, Porto Alegre received an AWS Experience, a full-day event about AWS’s services and solutions. During the event, we learned a lot about cloud computing, enough to change everything we had done until then. If you have the opportunity to go to such events, go.
We refactored the code, breaking everything in Lambdas and becoming serverless. Using the right pieces that Amazon offers we managed to reduce costs and increase computing power. What we would do with BRL30.000,00 could now be done with just BRL150,00/month growing as we grew.
Because not everything is flowers, when the time to run the algorithm on the EC2 data we had mined, was found out that when the computer failed at running the algorithm, all mining had stopped! A month of lost data and a return to point zero, because if don’t have data you don’t have machine learning.
The cloud is really cheaper and more efficient than a local server, but you better know what you are doing. Before you go out and throw code anywhere, anyway, take the time needed to learn what Google, Microsoft, and Amazon have to offer. They have quite a lot of solutions and each solution specializes in different problems.
Today* our algorithm lives happily on AWS divided on several Lambdas and S3s =)
The original in Portuguese, here, is from 2016, I’m no longer a part of VCore since November 2016
*Kept this way to conform to the original, I have no knowledge of what happened since I left.