Dyno Monitor — cloudwatch for Heroku
Heroku is a nifty cloud ‘platform-as-a-service’ provider that is ideally suited for creating your minimum viable product but that can also scale seamlessly as your project gains traction. At Carnot, we have been using heroku as the default platform, for launching new apps and services. As our products gain audience we scale our apps on heroku, and for really high volume apps where the costs are high (yes, heroku is pricey!) we migrate them to AWS or some other ‘infrastructure-as-a-service’ provider.
Cloudwatch, on the other hand, is an powerful AWS service that can monitor and record the logs and health metrics of almost every other AWS service. It can monitor not only your application logs, but also your underlying infrastructure (on AWS). Additionally one can also configure fine-grained alerts, and specify basic actions to be taken when the alert is breached.
Drawing inspiration from cloudwatch, here’s out attempt to make a simplistic tool for the heroku platform
Over time, we have identified a set of key features that we additionally build around almost all our heroku apps. These features focus on making the post-deployment lifecycle easy and automated for developers. They include:
1. Dyno-level error monitoring, alerting and automated actions
Catch memory-related errors like R13, R14 or web errors like H10, H12, H13 to alert the developers and also take auto-actions like dyno restart
2. Status check of heroku components
Watchdog mode. Check if your web dynos are live and your heroku components (heroku-postgres, heroku-redis etc) are up & reachable
3. Dyno-level metrics collection
Collect RAM-usage and %CPU-usage stats per dyno for graphing, alerting and later analysis
We found ourselves repeating these modules in all our apps, and it was turning out be fairly routine. So we thought, why not club all these services together in a single heroku app!
This led to the dyno-monitor project. This internal app was forged to do the very three things listed above. The app has been a quiet watchdog amongst our plethora of heroku apps, churning log-after-log, recording and alerting developers whenever hell breaks loose, or when, as someone aptly put it, a dyno starts becoming a dynosaur.
Today we are in the process of making this Django-based python project opensource, for public use (and scrutiny!) but mainly with the objective of conveying how simple systems can also greatly ease developer life. And also sometimes catch developers napping 😉
Do check out the project on GitHub which, at the time of this writing, contains the first of the three features. You can deploy this app to your heroku. It will auto-detect your other heroku apps and you can start using it rightaway!
WHY?
The big question is why would you want to monitor a dyno in the first place. There were four use-cases we came across, for which we thought building this out was necessary.
1. Elusive Dyno Level Errors
Heroku dynos emit errors when things go wrong or limits are exceeded. For example, when the dyno memory quota exceeds it emits R14, R15 errors. Or when web requests timeout, it gives an H12 error. These errors show up only in the application logs, and on your app’s metrics dashboard (see image above), making it difficult to capture them programmatically and act on them.
2. Memory Growing or Leaking Applications
Some apps tend to grow in memory as they handle more requests, or some may have memory leaks. Finding a leak can be tricky; it may also lie in an underlying library. A quick and effective fix can be simply to restart the dyno when its RAM quota exceeds
3. Watchdog
For any application that serves customers, it is important for the developers to know as soon as something is wrong. A watchdog service is the one that constantly looks at the status of your app or database and alerts you if the service is unreachable or goes down.
4. Dyno Metrics Collection
Two important dyno metrics: %CPU and memory are displayed on the metrics dashboard. You can only see them, but not query them, or set thresholds and alarms. We capture these and store in postgres, which can then be plugged into an analytics tool like metabase.
WHEN SHOULD I USE DYNO-MONITOR?
Well, it’s not necessary to use it, if you have other watchdog systems in place. But if you find yourself grappling with any of the above problems, you can give it a shot.
Some advantages of dyno-monitor:
- It is very quick to setup and start using on heroku
- Basic deployment on heroku will run within the free-tier dynos & free databases
- Very easy to customize (Django-based python setup)
Please note that this will not help you record your application logs. There is a host of logging addons on heroku which you can use.
Heroku has recently added a lot of monitoring addons as well. Some of these addons can serve as replacement for parts of dyno-monitor.
IN CONCLUSION
Dyno-monitor is an example of how simple monitoring can keep your dynos under control. The GitHub repo is still work-in-progress, as we make more features available in public domain. Do give it a try and stop your dynos from becoming dynosaurs!
We are trying to fix some broken benches in the Indian agriculture ecosystem through technology, to improve farmers’ income. If you share the same passion join us in the pursuit, or simply drop us a line on report@carnot.co.in
