Scaling Minotar
From 0 to 5000 or how Go changed the way I think about scaling.
Minotar is Gravatar for Minecraft. Minotar serves your Minecraft skin via an easy to use API, meaning websites like the Minecraft Forum or Enjin can easily embed it into their existing infrastructure.
Minotar always has and always will be a free service, we've never done ads or required end users to pay a penny to use our service. So as you’d guess, scaling with a tight budget is hard. With a normal profitable web application you’d generally scale via isolation, meaning separating your services (more MySQL servers, additional cache servers, offloading to a CDN, the like). Unfortunately that’s not possible with Minotar.
Minotar ran on a two tier scaling structure, meaning we had the “front end” and the “back end”. The front end’s job was to serve requests from cache or query the backend server if not found. And the back end’s job was to serve up new images for every request the front end sent it. This allowed us to scale to nearly infinite front end and back end servers, except for the budget.
Minotar ran like this for a couple of months, just teetering at 2k requests per second at peak, barely running with its PHP backend. Thankfully, one of our users had an idea, why not try Go? We had always heard the stories of “How Go made my application 300x faster!”, but it sounded too good to believe. Fortunately lukegb stepped in and did it for us, he rewrote Minotar in Go. The second I switched production over was like a breath of fresh air. Suddenly, without even trying our application was now capable of handling over 4k requests per second. And on a DigitalOcean $20 plan, that’s pretty fantastic. Finally instead of struggling to handle requests across 3 servers, we could now handle all of them, on one server.
Now that I look at it, I think the lesson isn't “port your app to Go and serve all the requests”, it’s actually “Use the right tool for the job.”.
Now that we use a language meant for high speed, low level applications, we get high speed, low latency applications. Minotar is now used on over 5,000 websites and serves almost 1,592,664,180 image requests per month. At peak times that’s almost 5k requests per second.
Thanks for reading my little ramble. Stay tuned for more words to melt your brain :D!
Minotar is open source at https://github.com/Axxim/Minotar/
