Business Transformation Enabled by Modern Engineering
Transformations Explained: A collection of experience, academic learning, and discussions with fellow Slalom employees.
Age of Software
As coined by the DevOps community, we’re in the Age of Software. Companies must re-envision themselves as software companies. If not, they are at risk to both the tech startups and tech giants. You must focus on value not cost. The following article describes the principles and practices needed to transform into a highly productive and profitable enterprise.
Why Transformation?
DevOps research has found a tremendous correlation to increased profitability.
Per Puppet’s State of DevOps report:
· Companies are 2x more likely to exceed in profitability and market share
· 50% high market cap growth over 3 years
Now how to do you get to this profitability? It’s simple: Customer-focused, business-aligned, full-stack two-pizza product teams focused on optimizing their value stream with the right organizational culture and modern engineering practices.
Let’s unpack that. Before we do, let’s anchor on the “Three Ways” of DevOps.
“Three Ways” of DevOps
1. Flow
· How do we accelerate the delivery of work? Firstly, make work visible, eliminate WIP or work in progress and execute in small batch sizes. Context switching is not productive. Secondly, minimize constraints and dependencies outside of the team’s domain. Believe it or not, waiting for someone to complete a ticket is not productive.
2. Feedback
· You need to think about feedback loops in everything you do. How do I get customer feedback as quickly and frequently as possible? Monitoring is a feedback loop for your operational health. Quality automation is a feedback look for your development code. You’ll hear “pull the Andon cord” in DevOps. Swarm on the problem and fix it early.
3. Continual Learning
· If you felt like some of this content may have stemmed from lean manufacturing, here’s your “Kaizen”. Focus on continual improvement and learning. Share knowledge across your organization.
Value Stream Mapping
Speaking of that principle of flow, a value stream is simply the set of steps performed to deliver value to a customer. Value stream mapping is an excellent tool to visualize wait time versus process time and percent complete and accurate or how often or things kicked back to a previous step. This is not your management consulting process flows. In layman’s terms, if I need to wait 2 weeks to get a server to run my code (wait time) and it really takes 10 minutes (process time) to provision it, it may make sense to write your own infrastructure as code.
Flow Metrics
How you measure flow in your organization is a popular topic in the DevOps community.
Categorize your work:
· Feature
· Defect
· Risk: Something performed for security, compliance, etc.
· Technical Debt Reduction: Work to increase maintainability
Categorize your states:
· New
· Active
· Waiting
· Done
Now here are your metrics:
· Flow Distribution: Ratio of each category delivered overtime period.
· Flow Velocity: Number of units of work in a time period.
· Flow Time (lead time): Time through the process from New to Done.
· Flow Load: Measure of all active items or work in progress (WIP).
· Flow Efficiency: Proportion of time items are actively being worked on.
Project to Product
Let’s talk operating model. The biggest shift occurring in a transformation is the shift from project to product. The days of large project plans with partial resource allocations are at last no more. It’s a lot easier to embrace those “Three Ways” of DevOps when you aren’t assigned to multiple value streams or projects. Product teams are about customer focus, business alignment, and bringing the work to the people.
What’s a Product Team?
· Customer Focus: I’m placing this first as usually this is the biggest challenge. If the product is customer-facing measure revenue, widgets sold, or customer satisfaction. If the product is an enablement product such as a platform or shared service, focus on adoption or net promoter score. It’s simple, product teams should either make a profit or make it more efficient for the ones that do.
· Business Aligned: Product teams should be designed to achieve the business strategy. You need the business to be engaged.
· Long lived: The team exists for the lifetime of the product. The size of the team may fluctuate over the lifetime of the product.
· “Dev” and “Ops”: The team is not only responsible for new feature development but also the operations. What a great way to get the team that builds the system to think about the operational concerns (everything from service level objectives/agreements to monitoring to recovery).
· “Full Stack”: The team should have the necessary capabilities to perform the steps in the value stream. You may need people that can provision infrastructure as code. Notice I said capabilities, not people, you don’t need a tester, an infrastructure reason, etc. Individuals may have many capabilities. Look for “T-Shaped” resources, depth in at least one area and breath in others.
· “Two Pizza”: Teams shouldn’t be larger than 8–12 people. Adding engineers to large low performing teams will only lower productivity.
· Buy doesn’t rent: Do not outsource. Product teams are long-lived teams. Leverage consultants to empower your teams.
· Agile: Product team should be focused on delivering value to the customer frequently and regular feedback. Agile is really about customer focus, reducing WIP, small batch sizes, and quick iterative feedback loops. Use agile to test a hypothesis and reduce risk. Lastly, a heavy change approval board or CAB only decreases productivity.
Product Team Learnings
There are numerous ways to start the journey to product-based teams. This may start with defining a product taxonomy, a focus in an area of the business, focus on automating operational work to allow a team to drive innovation, or an approach I strongly believe in — focus on a business-critical product and work out from that product’s dependencies. There are two biggest pitfalls to watch out for. The biggest is forgetting to focus on the customer and seeking frequent feedback from the customer. The second is to remember that product teams are cross-functional, people will need to come together from various parts of the organization.
Culture
Culture has a massive impact on organizational performance. It may seem like a hard word to define, but it’s again a set of principles. Seek to create a culture of trust, collaboration, and continuous learning. The right culture paired with a vision/purpose-focused the customer will increase organizational performance.
· Continuous Learning: Although many of principles maintain constant, technology and tooling is always evolving. All for budget both time and monetarily for training. I personally believe that learning platforms shouldn’t be standardized. Embrace the creativity of individuals and let them choose. Conferences, hackathons, and DevOps days are all ways to embrace learning. Celebrate learning goals. We do a lot at Slalom to celebrate certification. Not only is does this directly encourage learning, but more importantly, sets good hygiene around studying.
· Collaboration: Encourage means for sharing information, ideas, and support in the organization. Collaboration is about fostering communication to increase flow.
· Blameless Postmortems: You’ll hear this a lot in DevOps books. This is about having the team focus on being a team and what collective practices can make things better. Retrospectives as a form of team reflection are important.
· Collisions: Ever since I heard this term mentioned in The Culture Code, I’ve love it. Fine ways for the team or larger community to informally connect. This builds community and collaboration. You often learn much more about someone in an informal setting.
· Servant Leadership: Leaders should get their hands dirty. Their focus should be on developing and serving the people with a focus on continual improvement.
· Community / Horizontal Support: There are plenty of terms to describe driving communities of practices across a domain, e.g. Guilds or Tribes. It’s important to not just provide a forum for best practices, but also a way for people to reach out for support horizontally in that domain.
· Experimentation: Hypothesize and Test. This is a core tenant of agile and correlates to the DevOps principle of feedback. Apply this to your daily life. Don’t sit on your ideas, take action and if it fails, learn.
· Psychological Safety / Trust: A work environment in which everyone feels safe to both speak their mind / raise their hand is instrumental to a high performing team. Additionally, it allows for team members to take a risk to increase efficiency. You can help build trust by being vulnerable yourself.
· ENPS: There is a tremendously high correlation of happy and engaged employees to organizational performance. Focus on employee net promoter score.
Modern Engineering
This is the Age of Software. Along with the operating model and culture, modern engineering is the key to unlocking your business transformation.
Conway’s Law
Is there a better way to transition from the operating model and culture to modern engineering? Conway’s law states that “organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations”. Stated another way, if you create one large team of 40 people, you are more likely than not to build a monolith. Your teams should ideally reflect the bounded contexts in your application. If you have an online store with search, cart, and products, create search, cart and product teams. Further, consider the cognitive overload. It’s nearly impossible to understand all areas in an application, allow teams to focus.
Strangler Pattern
Stop, I have a large team and a monolithic application. Do I need to apply this to greenfield applications? No. The strangler pattern deserves a specific mention. While there are times that net new makes sense, the strangler pattern applies a low risk, iterative approach. Coined by Martin Fowler, a brilliant mind in the engineering space, it described a vine strangling branch by branch of a tree. Firstly, deploy a façade between your end-users and application. This often takes the form of a proxy / API gateway. You may refactor an area of the application and then reroute traffic to the refactored area transparently to your users. As you strangle your application, don’t forget to restructure your teams!
Modern Engineering Practices
There are a few key tenants to follow to ensure proper modern engineering practices.
· Loosely Coupled Architectures: A loosely coupled architecture is how you adapt to evolving customer demand more quickly. Your application should be structured by individual responsibilities or bounded contexts. Think “cart” or “search” from the previous example. This allows each of those components to be modified, deployed, scaled, recover, etc. more easily. Each component should have a clear interface to allow for communication. This loosely coupled architecture is often reflected in microservices today.
· Automated Testing: Testing serves as the feedback loop for your development. Seek to catch errors as early as possible. Write both unit tests to the individual methods in your code in addition to acceptance/integration tests to ensure functional areas of the application are working as designed. Unit and acceptance tests should both ideally be run on every commit. Bake them into your CI/CD pipeline. You should strive to perform automated testing without an integration environment — leverage fakes, mocks and stubs. Think about testing more than just your development code. Test your infrastructure, compliance, and configuration.
· Trunk-based Development: Adopting a trunk-based branching structure greatly enhances developer productivity. This allows for quicker detection of issues. It allows the team to pull the Andon cord and quickly address it. Lastly, it gets you out of the pain of complex merges by reducing batch size.
· Artifact Repository: If you rebuild your code before every environment, how can you guarantee it’s exactly the same and will behave the same way? Build your code once and externalize your configuration. Promote the artifact through your promotion cycle.
· Observability / Telemetry: There’s a high correlation between meant time to repair (MTTR) and observability. Observability is composed of four main pillars: metrics, traces, logs and alerts. As much as automated testing provides a feedback loop for your code, observability provides a feedback look for how your application is operating.
· Continuous Delivery: Continuous delivery builds upon continuous integration or simply building and testing your code upon check-in. Continuous delivery refers to building, executing tests, and deploying to a non-production environment. Continuous delivery should ideally build, test, create an artifact and deploy to a non-production environment.
· Low Risk Releases: Focus on fully automated deployment. Pipelines should always deploy your code. Leverage automation for validation. There are multiple ways to reduce the risk of a deployment including blue/green deployment (new environment deployed and then cutover occurs) and canary deployments (roll out new release node by node or slowly move traffic over to the new release). Most importantly look to decouple releases from deployments.
· Feature Flagging: Feature flags firstly allow for you to deploy your code to production without necessarily releasing it to your end-users. The power of feature flagging does not stop there, it may be leveraged for A/B testing. Furthermore, you could even monitor your production health and turn off not critical services during times of great load.
· On-demand Environments: You should be able to easily spin up and tear down your environments. Write your infrastructure, configuration, and application all as code. Consider testing your whole environment creation process along with your code.
· GitOps: Tickets decrease flow in an organization. Leverage source control as a way to maintain a consistent desired state, audit history, and reduce human error.
· Public Repositories: Make code samples and best practices available for the rest of your organization to view and leverage.
Modern Engineering Tenants
There are a few key terms to note when performing modern engineering.
· Immutable: Immutable directly refers to an object whose state cannot be changed after it’s created. Strive to immutable infrastructure. Define your infrastructure as code and ensure it’s only changed via code. Your infrastructure could be consistent between environments, get out of the habit of click ops.
· Ephemeral: Embrace the concept of ephemeral, or living for a short time, environments. Treat your servers like cattle, not pets. Design your compute to be stateless. This allows for scaling and various deployment strategies.
· Idempotent: Idempotent refers to something that can be applied multiple times without changing the results. Consider designing idempotent services for greater resiliency and the ability to replay transactions.
Modern Engineering Principles
As much as technology evolves there are a few principles to anchor around that do not change. Firstly, there is DRY or do not repeat yourself. SOLID has provided a design framework to anchor to object-oriented programming and very applicable today. AWS’s Well-Architected framework paired with SOLID sets the foundation for cloud-native design.
SOLID
· Single responsibility
· Open–closed
· Liskov substitution
· Interface segregation
· Dependency inversion
Well-Architected
· Operational Excellence
· Security
· Reliability
· Performance Efficiency
· Cost Optimization
Modern Engineering Metrics
You should measure your product team based on customer-related metrics. However, there are key metrics to analyze your modern engineering practices:
· Lead Time: The time between a work item being created and deployed to production.
· Release Frequency: How often is code released to production.
· Mean Time to Repair (MTTR): The average time it takes to recover from failure.
· Change Failure Rate: The percentage of deployments that result in failure.
Cloud
There’s been mixed messaging on cloud in the DevOps community. I strongly believe that the public cloud (AWS, Azure, GCP) is critical to a transformation. The power of cloud is that it offers building blocks to develop modern applications more quickly. The cloud reduces traditional hand offs in the software value stream and greatly increases flow. As a rule of thumb target higher level services in the cloud — serverless or containers where they make sense over traditional virtual machines to reduce operational overhead, increase scalability, and reduce cost. There is often a strong use case for a cloud platform product team to own and operate the cloud platform to accelerate customer facing teams.
Security
It’s critical to consider security across the lifespan of your development. Bake in security early an often. With automation security and speed are no longer at odds. Leverage automation to scan everything from your cloud environments, to infrastructure, to code. Create common libraries and services to help application development teams become more security. Leverage your culture of trust to ensure developers are comfortable raising their hand for potential risks or vulnerabilities. Consider a “red team” product team to seek out vulnerabilities and help the organization improve its posture.
Summary
In the Age of Software, organizations must look to improve productivity and therefore profit, by shifting to product-based teams, applying the right cultural norms and modern engineering practices.
Sources
I firstly wanted to thank Stephen Nichols, Matthew Cain and Carol Henry for their thoughts in the product and operating model space. I wanted to thank RJ Jafarkhani and Badar Qureshi for their thoughts in culture. I additionally wanted to thank Matt Mousseau, Seyed Ketabachi and Derek Poulin for their thoughts in the modern engineering space.
Kim, G. The DevOps Handbook (2016)
Kirsten, M. Project to Product (2018)
Forsgren, N. Accelerate (2018)
Coyle, D. The Culture Code (2018)
DORA. (2019). 2019 State of DevOps Report
Puppet. (2019). 2019 State of DevOps Report
