Hookin’ it up

If you want to catch more bugs, use more hooks.
— George Allen, Sr. + me

The concept of automation is a big part of DevOps approach, to the point where some people (incorrectly) define DevOps exclusively as automation. But while there are a lot of tools and talks around automating deployment pipeline process for build->test->deploy, a few talk about utility automation for intermediate steps of that process.

As part of Open Spaces discussion “DevOps for Devs” at DevOpsDays Charlotte, we’ve talked about different aspects of end-to-end process that can help improve adoption of DevOps principals by developers. Among other things, we discussed oncall responsibilities, development-centric monitoring and process automation for developers. And while discussing the latter, I was surprised to find out that most people are either not familiar with pre/post commit hooks or not using them to their advantage. As a result, I’ve decided to outline a few of the most common use cases for commit/push hooks I’ve implemented on different projects that helped to streamline development process and reduce a number of points of failure along the way.

Testing

While it’s a common practice (and generally a good idea) to run your test suite as part of you CI workflow, there are certain tests that can (and should) be ran on every commit and/or push, depending on your deployment procedures and development processes. Running unit tests and linters on every changeset committed helps with incremental code improvement. It’s worth noting that commit/push rejection is not the only response to failed condition. Depending on the development process, the result don’t have to be represented as a fatal error, it can be a warning or a notice. Displaying (and logging) a warning message to the developer, sending an email to the dev list or posting a message into chat (more on this later) is a completely viable response. The goal here is not to have a complete test suite that dictates production readiness, but rather provide developers with an opportunity to validate (and fix) their code in small increments as it’s being committed.

ChatOps

For those unfamiliar with the term, ChatOps is a term coined by Github to describe their growing culture of “putting tools in the middle of the conversation”. In layman’s terms, it is a movement to keep engineers from app switching to get to incoming information and to consolidate all day-to-day notifications into a single channel, presumable chat tool of choice (Slack, Hipchat, IRC). This is not a new concept, but with emergence of tools with built-in integration, it has become significantly easier to integrate notifications from various channels into a chatroom. You can feed alerts, emails, even notifications from 3rd party apps like Salesforce. So why not commit notifications? Pushing triggered commit/push notifications into a project-specific channel letting the whole team know of the changes, giving them a chance to review/comment/discuss the changes in single window.

If ChatOps is a new concept and is of interest, definitely read ChatOps for Dummies by Jason Hand.

Ticket management

Chances are, if you’re using any of the popular ticket tracking tools (like Jira or Trello) you probably have them tied into your version control system already. If you don’t — stop reading and go pair them now. Even if you are not using the latest and greatest set of tools (SVN and in-house ticket tracking system, for example) you still can (and should) have these hooks implemented because having post-commit hooks tied to your ticketing system provides a lot of utility benefits. It allows you to attach particular commits to individual tickets, manipulate ticket status and even close the ticket with a commit message. This allows developers and managers to have a complete history of changes for any given ticket, making deployment and troubleshooting that much easier. It also helps to eliminate an extra step in development process — separately going into ticketing system to update the ticket. Now, unlike testing hooks, this is an example of a rule that is recommended to be very strictly enforced. If you’re diligent in using ticketing tools, rejecting any commit that doesn’t reference a ticket is a good way to prevent individual commits to fall through the cracks.

Monitoring

Identifying and fixing the issues in production is crucial, but it requires insight not only into usage patterns and trends, but also awareness of special events or triggers (business and technical) that may be responsible for the observed behavior change. One type of such events, of course, are the productions pushes. Having an ability to correlate individual deployment to the monitored data points and being able to plot deployments on the same graph as the critical metrics is invaluable for troubleshooting production issues. Not only it provides a possible root cause for the problem, it also streamlines rollback and patching procedures. Side note, having an ability (and will) to pipe business events (like email blasts, marketing promotions, special events, etc.) into the same monitoring system along with deployment events provides even deeper visibility into the state of business operation for people investigation and troubleshooting production issues.