Observations from the London DevOps Conference 2024
Sharing knowledge and understanding is a key component of building a safe and healthy culture. Within the Agile Coaching Team at ASOS, we have agreed to share our experiences, not only with one another, but also the wider tech community to further support continuous improvement and learning. Here are the learnings Esha and I gained from a recent conference we attended, DevOps 2024…
Melissa Perri — Becoming Truly Product Led
The day kicked off with the anticipated keynote by Melissa Perri; who delved into essential topics such as product strategy and operations, sharing anecdotes and real-world examples from her extensive industry experience and expertise. Melissa highlighted that companies are just doing too much and not focusing on the right thing for their customers. This has led to the tech industry experiencing 28k layoffs since 2023! From this, it is clear how important it is for organisations to have a product portfolio strategy and recognising that the product IS the business.
The mention of product operations (The discipline of helping your product management function scale well. Surrounding the team with all of the essential inputs to set strategy, prioritize, and streamline ways of working) sparked our interest, it is centred around data and enabling product teams to see everything they need to (something we’re REALLY passionate about as coaches!). Making it possible for self-serve data to exist will allow people to do research, make informed decisions and measure the impact of product initiatives.
- Business and data insights — get the information out of systems and into the hands of people making decisions.
- Customer and market insights — aggregate external research
- Process and governance — leverage the skills and information for seamless collaboration
Product operations helps to translate a company strategy and goals into an actionable roadmap. Clearly mapping out an organisation’s planning processes is key for creating a balance between governance and agility.
Planning enablers
- Design the cadence of inputs, decisions and communications
- Generate alignment in initiatives
- Bring together the right product people and stakeholders for decision making
- Define key meetings: Set a fast-paced agenda with clearly defined meeting outcomes
We were so inspired by the talk we rushed to get a signed copy of Melissa’s new book ‘Product Operations — How successful companies build better products at scale’. A huge highlight for Dan 😊
Matthew Skelton — DevOps Topologies 10 Years on. What have we learned about silos, collaboration and flow
After a short break we attended Matthew Skelton’s talk on Team Topologies and his observations from the last 10 years. An interesting point and theme which appeared throughout the talk was the linking between Fast Flow (Quickly developing and releasing) and Cognitive Load (The amount of memory resources that are used or required for mental processes), Matthew clarified that success lies in addressing dynamic interactions, not building a perfect static structure.
Cognitive load appeared throughout the day in a variety of formats, however it’s clear the focus on Developer Experience (DevEx) is becoming more prudent within the industry. To support this, Matthew mentioned “The cost of tangled software” and the staggering impact (£100,000 per dev/year) of blocking engineers one hour a day on the operational cost of a company. To improve flow, we need to untangle our organisations, which in turn will decouple the software developed.
Matthew mentioned key metrics that team topologies has led to organisations uncovering as good measures to use around agility:
Key metrics (and what they monitor)
- Lead time — Flow (Elapsed time between the identification of a requirement and its fulfilment)
- Deployment Frequency — Flow (How frequently new code is deployed to production)
- Mean time to restore — Operability (Average time taken to recover a product or system failure)
- Change fail percentage — Operability (% of changes which lead to production failures)
- % of time building vs waiting — Flow Efficiency (% of time spent actively working on items vs waiting/blocked)
Understanding the problem — Matthew mentioned both Wardley mapping and Conway’s Law to highlight how complex org structures lead to highly complex products, which in turn cause increased cognitive load of developers. The greater the cognitive load on developers, the slower they are going to build and have a greater chance of unknowingly impacting another product or service.
Matthew provided an example where J P Morgan applied team topologies by mapping dependencies between teams and then actively worked to decouple teams wherever possible. This saw a 60% reduction in dependencies and a direct cost saving as teams experienced less delays. Similarly, the Home Office were able to reduce technology costs by 40% by making Product Owners account for the cost spent on their services, which instantly encouraged a cleaner split of teams and technology structure. This reinforced the message that cognitive load on developers significantly impacts flow and build and maintenance costs.
Emma Dahl Jeppesen — From assumptions to measured impact: Improving developer experience at TV2
Having split to cover more talks, Dan attended a session by broadcasting company TV2, which is used at least one hour a day by 85% of Danes. This session covered the topic of DevEx and how new measures have been introduced to highlight bottlenecks and pain points in the development process.
Emma referenced how TV2 previously introduced DORA metrics (Deployment Frequency, Mean Lead Time for Changes, Mean Time to Recover, Change Failure Rate) across the company as a way of monitoring performance. However, since the metrics were used as targets instead of simply measuring the trends (Goodhart’s Law), they noticed how teams would act in certain ways to game the metrics. An example of this was teams deploying multiple smaller chunks of code which should have been deployed together, this became apparent once tech leadership started to challenge why some teams were deploying less than others.
TV2 introduced qualitative surveys to gain a deeper understanding of how Developers are feeling and the flows work passes through before considered done. These surveys were highly welcomed and the insights allowed the organisation to focus on improving specific issues which blocked developers day to day such as streamlining the CI pipeline.
How can we utilise this approach ourselves?
- Value stream mapping — Research our developer’s journeys using both qualitative and quantitative data
- Ask Devs to walk through their process and how they’re feeling at each point, highlighting painful steps
- Review and update both As Is and To Be, continually measure if moving in the right direction
- Aim to implement changes as quickly as possible, validate and then continually improve. Don’t spend too much time planning the change (Better to monitor and adjust as you go)
Kristian Lindwall — Innovating at scale — Spotify’s experimentation journey
Later in the day we heard from Kristian Lindwall as he walked through the various ways in which Spotify focus on experimentation across the entire development journey. Throughout the talk, Kristian referred to “cultivating curiosity” by creating an environment which encourages autonomy, transparency and connectivity. An example of this was Spotify’s experimentation platform (ABBA), which enabled running tests on a far greater scale while also monitoring wider KPIs. This approach simplified how teams create and run tests, encouraging better development practices as testing was easy to do and provided insight to the wider impacts across Spotify.
The platform ABBA was formed from an experimental approach in Spotify to identify pain points in the dev process. By taking an experimental approach, Spotify was able to identify and focus on the largest problems. After several iterations, the platform was able to separate experiments from flags, measure metrics and coordinate experimentation within a single tool. Spotify’s confidence in the tool has enabled them to package and sell ABBA as a separate product entirely, this is something that we’ve talked about previously in this blog, such as @Scott Frampton (in Medium you could tag him) and the journey with fundamentals or @Callum Trounce post on how we embrace experimentation.
Whilst this talk was heavily focussed on the experimentation platform created by Spotify, it did reinforce their constant focus on experimentation and continuous improvement. By taking time to understand the developer’s experiences, they have reduced complexity and simultaneously create a product which has introduced additional revenue streams.
Sven Peters — Developer Joy — How great teams get s&#t done
The final talk we went to was all about improving developer experience, Sven Peters from Atlassian calls this “Developer Joy”. Over the past 10 years the cognitive load of developers has significantly increased as technology advanced and they became responsible for much more than just writing code. The question of how to measure developer productivity arose and almost has become an obsession with leaders.
We travelled back in time with Sven to remind ourselves that software development is not the same as a production line so productivity cannot be measured in the same way. At Atlassian, the team thought about what improves the developer experience and what brings them joy, measured it through satisfaction surveys and then handed over the information to the actual teams to solve (not a productivity committee!).
What they found was that if developers can work on great quality code and it can flow through with reduced waiting time as well as getting customer feedback, then developer joy emerges.
An example of changes made at Atlassian includes addressing the Quality Assurance (QA) ‘wait time’. The team found that they had a QA to developer ratio of 1:30 which created a bottleneck in their flow, so they pivoted the role to be Quality Assistance. This change in remit meant the QA role would do a kick off work with developers and coach them to carry out better testing and write test plans. This enabled the developers to carry out testing themselves, leaving the QA role to run test environments and reduce the developer cognitive load.
Steps taken to improve developer joy
- Ask your developers how satisfied they are — use surveys
- Give the teams time (10–20%) to solve developer issues
- Spread the learnings across teams so others can benefit from improvements
Key Learnings
We felt there was a lot of connection between the talks we went to, one of which being that unlocking flow in an organisation and a developer’s journey creates both cultural and financial benefits.
We were reminded how important it is for successful businesses to recognise not only customer but also developer experience when creating valuable products. Further supported by the role of leaders when creating organisational structure, our thinking must span beyond the products and instead encompass the supporting structures and systems to ensure we are able to pivot quickly as needed.
About Me
I’m Dan Dickinson, an Agile Coach at ASOS. I help guide platforms and domains through our continuous improvement journey, utilising data and insights to drive better behaviours, processes and mindset. Outside of work I’m a Dog Dad, Golfer and Scuba Diver.
