Built Fjord Tough
As we approach the 6th anniversary of STRATZ.com going online [during the 2016 Boston Major], it’s time to celebrate how much we’ve grown, and to set the stage for what’s to come. As the leading provider of Dota 2 analytics, there’s no better way to push forward than to get the hardware and software that powers your services up to the highest standards in esports. Onwards!
The full process of building, testing, rebuilding, and retesting the new pieces of the STRATZ backend have been going on behind the scenes these past few months. While there’s still more work to do, all of the biggest updates are now in place and online.
While today’s supdate is all about backend, the frontend crew pushed out lots of little updates and fixes across the site today as well!
We plan to provide a technical deep-dive into all the details in an upcoming post (teaser at the end), but today we’re going to share some of the more user-friendly details to whet your palette. Let’s start with some photos from building the servers…in the wild!
If you’ve never seen a server before, you might be surprised to discover that it’s literally just a computer, though usually without most of the fancy peripherals that you use with your Dota rig.
Pictured above are 5 of our 12 new servers, each one packing an AMD 7950x CPU, 128GB of Kingston FURY Beast DDR5 RAM, and two 7.6TB Kingston DC1500M NVMe drives. This makes for a grand total of 192 CPU cores (384 threads), 1.5TB of RAM, and about 168TB of [usable] storage!
The new servers have 20GB/s uplinks to each CR Node
These 12 new servers are replacing our previous single MSSQL server that ran on 14TBs worth of Seagate enterprise SSDs in a RAID 10 with 384GB of DDR3 RAM, along with our six Scylla nodes which were allotted 18TB each on old drives. MinIO and Elasticsearch shared resources with these Scylla nodes.
Fun fact: there are several other big pieces of the STRATZ backend that we won’t be discussing today, some of which also went through updates during this process.
Now onto the new software! After much diligence and exploration, KB settled on using CockroachDB to run on our new machines. While the Cockroach product itself is hugely impressive, a big motivation to rely on this technology for our continued growth came as a result of the extraordinary service provided every step of the way by the outstanding team at Cockroach. We’d like to extend a huge thank you to them for all their help and support! 💙
In addition to CockroachDB, we’ve always chosen to leverage Clickhouse to help us to completely reimagine the scope of aggregate datasets that we can provide to you. A quick example might be the ability to generate a leaderboard of heroes by wards placed and destroyed per minute, or find out the difference in how much gold is earned on average from jungle creeps across patches. The possibilities are well beyond anything we’ve been able to provide in the past!
Whether you realize it or not, if you’ve ever checked out data on SteamCharts, you’ve already experienced a bit of the power of Clickhouse.
So all of this new power and technology is great, and will obviously enable us to push deeper to uncover more undiscovered analytical goodies, but there are several additional important practical applications for these upgrades.
First, STRATZ has spent the last 6 years under the threat of a single server failure resulting in a massive outage. Thankfully, while we’ve had our growing pains, KB took care to ensure that no such catastrophic event took place! However, with our new systems, we now have proper redundancy, meaning that an entire server could explode (hopefully not) and nobody outside of our team would ever know anything happened! This is huge, and one of the most critical selling points for this investment.
Second, up until now, it wasn’t possible for us to add hardware to our servers. We either had to upgrade almost everything (which we did!) or make use of what we had. Beginning today, as the use of our resources continues to grow at an accelerating rate, we have the ability to add new servers to support the demand! Additionally, we can continue to add nodes to our existing hardware to manage various needs as they arise.
Third, our new system features triple replication, which allows for us to use a quorum to discover and resolve data corruption. In the past, if we had a data corruption issue, we didn’t have a good way to either detect it automatically or locate it after it occurred, though some situations could still be resolved.
The final important practical application to reference in this post is another result of our triple replication. Our old single MSSQL server was a powerhouse in its own right, but when it experienced a slowdown from high demand, it didn’t have a way to outsource any of the load, resulting in a more comprehensive slowdown. Beginning this week, the new CockroachDB servers can distribution load between up to three servers at a time, greatly reducing the risk that processing speeds will be affected by spikes in demand!
Psst…We also updated MSSQL from 2016 to 2022
For a final bit of relatively basic stats, let’s look at just a couple examples of how well the new system can scale. We’ll get into this more with our more detailed post later on.
Our previous system would generally run ~1,400 queries a second. The new system is currently clocking in at ~4,000 queries a second, though much of that difference is because we’re still relatively early on in the migration of storage from the old system to the new one (matches from August of this year up to new incoming matches are already on the new system). However, we now have the capacity to handle over 77,000 queries per second — this gives us growth potential for our request rate of between 20,000% and 55,000%! And with our ability to add servers on demand, our true scalability is much higher.
And lastly, I’ll leave you off with a teaser of some of the fancy data we’ll be sharing a little later on from our new Grafana suite. Behold…
See you next week for more new content!
Enjoy keeping up with what’s happening at STRATZ? Follow us on Medium, Twitter, or Facebook to get notifications of our latest Supdates! If you want to get more involved, join us on Discord!
Did you miss our last Supdate? Take a look at your new Ability Builds page!
Thanks for reading!
