An Ode to Guardian Wisp
No, not the one from DOTA.
Bad things happen. A round-the-clock service that your customers rely on, for instance, can give out like a bridge that’s carried one too many people. Sometimes, you can’t stop shit from hitting the fan (or the river below). All you can do is respond — as quickly and effectively as possible.
A while ago, the company was running a critical service that needed to be up 24/7. If the service faced issues, it was imperative that the engineering team be notified immediately — a very common scenario in software engineering. Now, there exist a host of solutions out there purpose-built for just this — like PagerDuty, OpsGenie and Cabot. However, like a lot of startups, our resources were limited and we could not afford most of these solutions.
So we did the one thing we were confident we could do. We built Guardian Wisp — our own, in-house incident reporting system.
Guardian Wisp was a simple service that did paging through SMS and email. It could page different teams or even individual people.
We set it up to read the page target data from Google Calendar — as it was easy to set up for most teams — and it integrated right into your schedule. Working with Guardian Wisp was simple: You would add the name of the person or team as a calendar event, key in their on-call duration, and then Guardian Wisp would automatically look up their email and phone numbers from our directory service and page them whenever an incident occurred.
To compensate for the sheer lack of will that plagued us in the midnights, everyone got (intrusive and annoying) apps like FireAlert 2 on their phones — so that it would really notify us when the dreaded SMS came in. We decided to go the SMS way because it was cheaper {inclusive of engineering costs} than native notifications, especially since we had people with Android and iOS devices. We used Logentries for logging our critical service, so it was just a matter of defining error patterns and setting up a webhook to Guardian Wisp.
Guardian Wisp was built using Spring Boot. We already had our own libraries for emailing via Amazon SES and sending SMS via 2Factor, which made integration simple enough. That said, it took a while to setup integration with our directory service and Google Calendar (especially setting up the project and figuring out Google’s documentation).
It took us all of a week to build and deploy the final system — It worked, it was cheap and reliable, and it provided us with all the necessary functionality that we would otherwise have had to pay quite a bit for. We loved what we had created, and it eventually evolved to be more configurable: our dashboard had options to page the engineering team if there were any technical issues, page the operations team when certain events occured, and more workflows that always helped us keep incidents in check.
Guardian Wisp eventually saw sundown, when our requirements for alerts became higher, and we just needed a more managed solution. But while it enjoyed the sunshine, it stood tall and strong, and protected us on a number of occasions.
This blog was written by Vedavyas: Senior developer, full-stack engineer, and crippling-social-anxiety specialist. He’ll deny being any of those things at any given point of time, but you can always see it in his beady little eyes.
Follow our blog for more stories from the Mitter.io team, and their quest to find the right tools and the best solutions for all things dev.
