A few weeks ago I started seeing alerts that an abnormal number of instances of our API service were failing health checks. It’s not unusual for the occasional instance to fail a health check, at which point it will be taken out of commission and replaced with a fresh…

This article describes the process of tracking down a production bug in our primary web service here at Room Key. It assumes some familiarity with Clojure, the JVM, Amazon Web Services, and Linux, but links will be provided for more…