Devopsdays Chicago 2019
On a beautiful August morning, I packed my bag and began the commute to Devopsdays Chicago. The sky was clear after a rainy evening and a breeze coming off the lake kept the humidity at bay. I was excited to ride on one of the glamorous CTA trains suspended above the downtown streets, but after walking to the station I discovered that I would be riding underground. I made a mental note to figure out a route using the above-ground trains for my commute back to the hotel. This was the seventh edition of Devopsdays Chicago and in attendance were over 500 people from the local DevOps community. As usual, the schedule for Devopsdays Chicago was broken out into a morning of keynotes followed by Ignite talks and open spaces in the afternoon.
Keynotes
Ethics in Computing
The opening keynote from Jeff Smith sought to draw attention to ethics in computing. This included topics such as the implications of informed consent, for example, whether or not agreeing to a 20-page EULA filled with legalese could be considered “informed,” and the balance between well-curated advertisements and privacy violations in the ad-tech space. While Jeff did not share many of his own opinions, I appreciated his effort to remind the audience to consider the consequences of the decisions we make on our teams, in our companies, and as an industry. As a developer, it can be easy to push responsibility for consent and ethics to others, such as product management, compliance teams, upper management, etc., but this does not remove our responsibility or address the problem of answering these tough questions.
Testing in Production
Heidi Waterhouse’s presentation made the case for testing directly in production environments using feature flags. Creating a perfect replica of a production environment comes with countless challenges and even in seemingly perfect replicas developers often find that code working in staging inexplicably fails in production. Given these challenges, why not look for ways to test directly in production?
“We are all testing in production. Some of us do it on purpose” — Heidi Waterhouse
One such way is through the use of feature flags. Feature flags fundamentally consist of adding if/else statements to your code in order to toggle between feature versions. Using the feature flag design pattern allows you to toggle on or off features for a specific subset of users in your user base. Using it you can route and shape traffic to perform canary releases and A/B tests with live production traffic and simply increase the cohort size to your entire user base when a feature is ready for release.
Introducing DevOps to a Business
Kevin Harriss, Lakshmi Baskaran, and Holly Allen all shared experiences introducing DevOps practices to their places of business. At Enova, Kevin started out by creating a team full of people who bought into the approach, providing them with a clear and well-documented vision, and sharing that vision with the broader company. He also introduced the concept of CREAM: Context Rules Everything Around Me. This concept stresses the importance of looking for the right new technology to solve the problems you are facing rather than jumping on the bandwagon of every new trend.
Lakshmi shared that she was asked by her boss to forget her plans for the year and spearhead a DevOps transformation. Her approach emphasized the importance of choosing where to place resources across three pillars: culture, process, and tools. She suggested finding and mentoring influencers in the development and operations team, identifying DevOps initiatives for the team, and incentivizing those influencers to successfully execute the initiative created for them. She also made the point that while it is not easy to accomplish, cultural transformation costs nothing. Similarly, while DevOps benefits from adopting new technologies, there are a number of open-source tools available, so a monetary investment is not required.
Holly began by tracing the history of DevOps back to Shigeo Shingo’s Toyota production system. This LEAN system was designed to eliminate waste. This included the creation of concepts like the kanban board, originally physical cards at the factory that traveled along with parts, and kaizen, a term for continuous improvement, which empowers individual workers and teams to make themselves more efficient. Holly also talked about the history of implementing DevOps at Slack.
Slack received mixed results with a centralized operations team. At first, the divide and conquer approach meant that each set of engineers could focus on their area of specialty: developers added features and moved the product forward while operations worked on efficiently deploying and monitoring infrastructure. This broke down as the size and agility of the development organization outpaced operations. Slack found success by shifting to a service ownership model. This approach involved embedding DevOps generalists with high emotional intelligence into development teams, with a result of developers becoming increasingly responsible for the reliability and performance of the services they create. Slack’s high trust environment and commitment to learning at the executive level facilitated the switch to a DevOps model.
Documentation
One talk that especially resonated with me was Karissa Peth’s “Document Like a Scientist.” The talk compared elements of cancer research to software development. While there are areas of overlap, including the use of planning boards, whiteboarding, and managing complex environments, from her perspective, software documentation pales in comparison to scientific documentation.
Karissa broke down scientific documentation in five key parts: materials list, diagrams, dates, positive control, and rationales/uses. Many of these elements have clear analogs in software development: a materials list is similar to hardware and software prerequisites and diagrams are similar to architecture documents. Dates are something often left off software documentation but can be useful to determine how recent or not recent the documentation was created and whether or not something may be out of date. A positive control, or the expected result of an action, is something else often left out of documentation. Its omission results in users not being able to verify if the step taken was the right one. Finally, the rationale/use for the document is simply the reason someone would be interested in following the document. Karissa made the point that while it is possible to find examples of documentation using individual elements on this list, nothing she could find included all of them in one place. I am planning to keep this in mind the next time I create documentation.
Ignite Talks
The rapid-fire nature of Ignite talks, 5-minute presentations with auto-advancing slides, provides a nice counterpoint to the in-depth morning keynotes. One talk that jumped out at me was Jay Destro’s talk about working at Buzzfeed during “the dress” incident. I can remember myself on the other side of the incident, as one of many hammering away on the Buzzfeed Tumblr page to see if the dress was blue and black or white and gold. Jay provided some color around this on-call nightmare, at one point describing how they were, “just throwing servers into the internet” to prevent the system from going down.
Another fun talk was J Paul Reed’s “Attempting to Bring Blamelessness to Traffic Court.” This talk recapped his experiences getting a traffic ticket at the Daly City, CA In-n-Out, a place I often found myself in high school, and his attempts to fight the ticket using principals of incident management and blameless post-mortems. Evidently, this is a valid technique, as it helped J Paul Reed beat the ticket.
One other talk I found especially relevant was Jason Yee’s “How to Shave a Yak.” Yak shaving is a term coined at MIT that refers to a task that leads you to perform another related task and so on and so forth. All of which distracts from your original goal. As someone who spends a healthy portion of my days exploring new technology, all of which requires exploring interrelated technologies and dependencies and so on and so forth, this one hit close to home. It seems like I am not the only one in this space with this issue — I found out there is, in fact, a DevOps Yak, who I had the chance to meet in person later that day. As quickly as I learned this new idiom, Jason informed me it was erroneous: yak herders do not actually shave yaks. The outer hair is coarse and would make a terrible fabric while the soft internal layers of hair can be gathered through brushing. In addition to the helpful yak knowledge, Jason suggested ways to overcome yak shaving events in the software industry. He suggested tracking your yaks: making a note of what it is, where it came from, and always keeping in mind the higher-level goal you are trying to accomplish.
Interactive Keynotes
Similar to Devopsdays Minneapolis, this conference offered an alternative to the afternoon open spaces. This time the alternative came in the form of interactive keynotes. One, “Let’s play a real-life Dungeon and Dragons” by Garima Sharma of Pivotal, involved going through the process of debugging issues as a team in practice for on-call. Another, “Site Reliability Engineering (SRE) and the Art of SLOs,” by Jennifer Petoff and Nathen Harvey of Google, focused on the process of creating practical SLOs for an organization. These interactive keynotes did a good job of splitting the difference between presentations and hands-on labs and I hope to come across them again.
Wrap Up
Devopsdays Chicago included a number of great talks about how to transition a business to a DevOps model. As usual, this event provided a great opportunity to connect with others in the industry. I look forward to the next Devopsdays I have the chance to attend.
