Notes from GawkBox’s CTO — Part 1
Building for scale
Welcome to our very first dive into the technological side of the GawkBox story. In this continuing series, we will highlight Gawkbox’s engineering mantras and how we apply them in our day-to-day activities. We will cover our use-cases, describe how we came to our conclusions, and provide sample code snippets so you can follow along. With these entries, we want to showcase the types of technologies we are interested in, but also to provide a running log of why we do the things we do.
Before we get started, a disclaimer:
All content regarding GAWKBOX Engineering is for the purpose of sharing. In our engineering world, there are many solutions and strategies that can fit the proverbial bill. In no way are these posts a recommendation to any one type of solution or vendor.
With that stated, let’s get started.
At GawkBox, we start with (and continuously ask ourselves) 3 questions:
- What are we trying to solve?
- Are we using the right technologies for the job?
- Does it make sense?
In order to answer these 3 questions, let’s start out with the meaning of the questions themselves.
What are we trying to solve?
Here at Gawkbox, we have plenty of great ideas. When I say “plenty”, I mean upwards of hundreds of ideas sitting in a backlog. It’s not something that any one person can fully comprehend. As we focus on our vision of Bringing Live Streams to Life, we assess all great ideas or problems with this mantra in mind. As such, vetting out hundreds of ideas starts to get a bit easier when there are a few guidelines to go by. If the impact of a certain solution or feature takes more effort than the predicted return, it will get placed into a non-prioritized bucket, while ideas that do meet certain, high-value criteria end up in the “higher” priority bucket. Even rudimentary priority systems can make filtering tasks monumentally easier. What’s a must-have, and what’s a nice-to-have?
Are we using the right technologies for the job?
We don’t choose our technologies simply based on the “cool factor”. What I mean is, it’s okay to ask why MongoDB is used, as opposed to an RDBMS like MySQL or PostgreSQL (Editors Note: Here’s where you’ll know whether you’re properly versed in computer science and engineering or not). I’ve worked at too many places to see a noSQL datastore immediately picked as a technology because it was part of a known stack (i.e. MEAN). The only way to understand a technology is to read the docs, ask questions, and test it out. Get a feel for your potential technologies.
Does it make sense?
This might be the most important question of the 3. Regardless of technology choices, at the end of the day, a system’s architecture must make sense. We continuously try to find a balance between scalability and complexity. If something is simple but doesn’t scale, prepare for updates — hopefully not hacks — of never ending major refactors, or even a rewrite.
On the flip side, if you’ve crafted a highly scalable, yet equally complex design, but don’t have the “traffic” to maximize potentials, you’ve just entered the world of over-engineering.
Post-Release Hardening
After a release, we ask ourselves 2 more questions:
What are the application’s performance patterns?
At a very high level, this is really about understanding what our apps do and what types of resources they needs to perform at scale. Everything is about scale these days, isn’t it?
We start by ensuring we have proper visibility into all the core metrics (CPU, RAM, disk, logs). A lot of tools (free and paid) help us answer performance related questions so find the right solution and get it going right away.
At GawkBox, we use DataDog, LogDNA, PagerDuty and Amazon Cloudwatch.
We try to prevent sleepless nights by setting up these basic monitors and alerts early on in our development stage. Be sure that you know your app because when 💩 hits the fan, you’ll be frantically searching through logs and performance profiles until you finally run across something suspicious. You might ask yourself, “was this always a problem?” and end up chasing a solution to a problem that isn’t the root cause.
Is the underlying hardware profile sufficient?
I went to a GoLang class about 6 months ago. The instructor used a phrase I had never heard before. When he was was discussing some of the challenges he was facing at Google during his tenure there, he introduced the concept of “mechanical sympathy”. That term stuck with me ever since. I was saying the same thing earlier that week, just not with the fancy catch phrase.
What is mechanical sympathy? I believe the phrase came from F1 racing, but in our context, it means ensuring your hardware and software are aligned. Built for the cloud.
We’ll stop there for now. Hopefully, you have a brief idea of how we approach solving problems.
stay tuned for our next post about our underlying infrastructure.
— Tony Chong, CTO
