How to Break the Fourth Wall with Twitch Extensions
What is your background? Tell us a little bit about your career path.
I graduated with a BFA in graphic design at Savannah College of Art and Design, but my interests were all over the place in college. I took up a wide range of creative disciplines. I dabbled in photography, web design and development, illustration, animation, game development, product and packaging design, the list goes on. I was a professional illustrator and game dev for a number of years, but I got hooked on generative art (AKA making art with code), and that is what ultimately led to my career as a creative developer in the advertising sphere.
Generative art is all about developing custom tools to make dynamic experiences. I liked making these tools for myself and realized that it was my dream job: making tools for artists. I wanted to change how things were made.
This desire manifested itself as Lightpaint Live, a real-time light painting app. The goal was to make a sort of light painting heads-up display so artists could preview their work as they were making it (normally you click the trigger, wave lights around, and go back to the camera to see the result). The project began as a simple experiment with fragment shaders, back when Flash was popular. Since then, it has lived many lives. It became my to-do MVC for learning new platforms and languages. I rewrote it in Flash/AS3/Flex, Java for desktop and Android, Processing, a Chrome app (but Google killed the Chrome app store), an electron app for Mac and Windows, and a progressive web app (unreleased, still in beta).
If I’m being honest, it’s not very lucrative in itself, but it has been a great way for me to keep up with new tech (like the ImageCapture API), and it’s gotten me quite a few side gigs.
What is your current specialization?
I wear many hats, but lately I’ve been developing Twitch extensions. It’s a relatively new platform, just launched in late 2017. It’s a way of adding a reactive/interactive layer to a live Twitch stream. Some extensions are integrated with game APIs so they can show player inventories, team compositions, stats, or whatever else the API exposes. Others are just for laughs. One of the most popular extensions (Sound Alerts) is dead simple. Users pay bits in order to play a sound effect on the stream, which can be pretty hilarious with the right timing.
The first Twitch extension I made was for a promotional stream of The Grand Tour — Battle Cars Live. The premise was a live game of battleship with actual explosives, using cars instead of ships. The extension was mostly invisible. It was overlaid on top of broadcast graphics, allowing viewers to submit votes by clicking on the cell they wanted to detonate. The cell with the most votes at the end of a round would be detonated. We ended up with a ridiculous number of views, and a max of about 55k concurrent users. After all, who doesn’t like blowing things up?
The following extension was for Thursday Night Football. The 2018 season was the first time the NFL was streamed on Twitch, so it was a great opportunity to push the boundaries beyond the traditional viewing experience. We integrated with a real-time NFL API (SportRadar) in order to react to events in the game and made something fun and engaging. The main feature of the extension was game predictions. There were overall game predictions like the final score, total passing yards, total rushing yards, etc. Quarterly predictions were made at the beginning of each quarter, but there was a twist! Users could vote at the beginning of the quarter for the highest possible score or they could decide to wait it out and make predictions up until the very end of the quarter. The consequence of waiting was that the score multiplier decreased as time progressed, so they could make more informed decisions late in the quarter, but not end up with much of a reward. There were also bonus questions that popped up at the start of a drive. This let users predict how the drive would end: touchdown, turnover, field goal, or punt.
Beyond predictions, there was a stats browser so viewers could back up their decisions with facts. There were also weekly and cumulative leaderboards, so users could compete with friends to see who could score higher.
Developing an extension takes some getting used to, but in the end, it’s just web tech. Twitch loads an iframe with your content in various places. The content options are video overlay, video overlay components (which can be drag/drop positioned by the streamer), config (used for initial configuration), live config (displayed on the streamers dashboard, used as live controls), panel (shows in a panel below the stream), and mobile (shows as a tab adjacent to chat).
What are the most important problems your customers are met with?
Our world is increasingly driven by data, and increasingly interactive. Remember when poking a computer monitor would just leave fingerprints on it? Twitch extensions give us a way to give people what they really want out of a live video stream, even though they might not realize it themselves. This includes our clients.
The first time I watched a streamer play Hearthstone, I had an “oh sh**!” moment when I happened to mouse over one of the cards. It popped up an enlarged view and I could see all of the stats and capabilities (via Innkeeper). This was immensely engaging because I could browse the streamer’s deck and guess what their strategy might be. If I was actually good at the game, I could post my recommended strategy in chat. I could change the course of the game, even as a spectator!
Most people I talk to about my work aren’t even aware of Twitch’s existence, much less what it can do beyond traditional broadcasting. Of course, it’s not a good fit for every project, but it’s a nice tool to have around for project pitches. There is a lot of untapped potential for Twitch extensions, and it’s still a new concept to the majority of people; and in advertising, novelty sells.
What is the most remarkable experience you’ve had during your professional career in this field?
Twitch extensions, specifically for heavily promoted live streams, made my life remarkably difficult for a while. Generally, when you have a product, it will grow organically and maybe get some spikes in traffic every once in a while. You can spend time nursing it and tweaking server settings to optimize it over time. In my case, the first two extensions were estimated to get tens of thousands of concurrent users (our max target was 100k for testing purposes), so I had to be sure our services were going to be able to meet the demand. On the surface, this seems simple enough, but it quickly became much more involved than we had anticipated. We were lucky enough to be in contact with a dev support team at Twitch, and they warned us of some of the common pitfalls. I usually put a lot of faith in auto-scaling cloud services like Cloud Functions, App Engine, etc. However, they can’t actually scale up fast enough to meet the demand of 5–10k new users per minute. They choke and fail just like a regular machine would, for the simple fact that it takes time to provision new instances with your code.
The solution that worked for us was to anticipate the spikes in traffic so we could request machines before it was too late. The Battle Cars extension was simple enough, since we could manually spin up a bunch of machines and run them for the few hours we needed them to during the broadcast. For the NFL extension, we automated the command to scale up our number of instances based on kickoff times. We’d have it scale up ten minutes before the show, and scale down ten minutes after it ended.
Load testing was a big part of prepping for a production release. It was a key tool that helped us recognize bottlenecks and implement optimizations. However, we didn’t manage to accurately recreate the conditions of live production, so there were some hiccups on launch day. Looking back, we should have spent more time fine-tuning our load tests. We didn’t emphasize it enough because the original timeline was already very tight.
Integrating with a real-time API is hard. Our vendor had some test games that ran four times daily, but for some reason, they sped them up so they only ran for thirty minutes at a time. That meant we only had a total of two hours per day to test. It was constantly interrupting development and slowed us down because we had to wait until the next day to resume testing. Another thing that tripped us up was that the test data was formatted differently than the live data, so we immediately ran into issues on our first live test run. If I were to do this again (and I probably will), I would create a system that could record the data from a real game and play it back at the same speed.
What are the sources of knowledge you use to improve your skills?
I haven’t taken any formal courses in years. I learn best by actively tinkering with something, so my process goes something like this:
- Get inspired by any means necessary.
- Talk about what I’m making with my peers. They can offer great insights.
- Go to conferences. There’s a ton of really weird niche stuff that goes on at conferences.
- Write about my experiences. I only started doing this again recently, but it really helps me internalize what I learned after each project. I have a habit of only looking forward, but there’s plenty to be learned by analyzing previous projects.
- Make new stuff, even if it’s not directly connected to work. I spend a lot of my time making lamps, concocting new hot sauce recipes, and still occasionally make art.
I post my fun-time projects on Instagram and Github. Here are a few of them:
Questions asked by Alex Savinkin
Former number cruncher in investment funds & strategy consulting. One of Geekforge Founding Fathers. Blockchain and technical singularity true believer.
