Moving Fast — With Confidence
At Marshmallow we’re building a world where insurance benefits everyone. This means using technology to build products that are affordable, instant, and inclusive.
To achieve this mission, one of our cultural values is Move Fast.
What Is Moving Fast?
Put simply, for our engineers, moving fast means shipping features which benefit our customer into production as quickly and frequently as possible.
Shipping features quickly requires advanced expertise and the confidence to endure micro-failures knowing they will ultimately function as gateways to real breakthroughs and genuine product improvements.
Why Does Moving Fast Matter?
Moving fast enables us to constantly deliver value early and often whilst learning and iterating quickly.
So central is moving fast to our broader mission, that we are comfortable incurring debts and associated interest by prioritising speed, knowing that the subsequent innovation will advance our product and services further.
Moving fast can afford us more opportunities to allocate time to Horizon 2 & 3 work in McKinsey’s Three Horizons of Growth permitting experimentation with more ideas in a given timeframe, taking forward those that deliver value for our customers.
Moving fast facilitates competitive advantage. The faster we iterate and experiment, the faster our products exceed the market standards and it’s our customers that experience the highest value first. Confident in the expertise of the engineers we hire, we permit the latitude to experiment in production. With less time spent speculating over unverified assumptions, the faster they can learn what truly works and, crucially, what doesn’t.
Moving fast contributes to a positive culture, a real priority for us as a company, rapid progression and frequent successes delivers a powerful cohesion among a team.
The annual state of DevOps report is one of the largest industry studies on this topic and includes thousands of companies, including Fortune 500 participants. Over a number of years the report has found four factors to be predictive of higher Software Delivery Performance:
- Deployment frequency
- Lead time (code committed to running in production)
- Time to restore service
- Change failure rate
According to these factors, participants are classified into 4 groups: Elite, High, Medium and Low performers.
High performers have been found to:
- Be 2.5x more likely to exceed profitability, market share, and productivity goals than low performers.
- Have far higher levels of eNPS (Employee Net Promoter Score), which may lead to higher retention.
- Have 50% higher stock market capitalisation growth over low IT performers over a three-year period.
- Spend as much as 20% more of their time on new work (value creation) instead of on rework, unplanned work and remediation.
These are significant statistics, demonstrating the importance of placing Moving Fast as a value at the forefront of our ideals.
Our Approach
In order to give ourselves the best chance of success, we implement the following principles to guide our approach and thinking:
- Organise our teams as cross functional delivery teams. This encourages autonomy and ownership of team outcomes, reduces communication overheads, creates better alignment, and encourages locality in decision making and execution.
- Foster an environment that encourages taking risks, psychological safety and leans much more towards Mean Time To Recovery (MTTR) over Mean Time Between Failure (MTBF). We’ll write a future post on this topic.
- Empower our engineers to have ownership across the full software lifecycle — At Marshmallow your job as an engineer does not end when a feature is in production, you are expected to monitor and support these systems. There is no Ops team to throw work “over the wall” to. This avoids costly handoffs between teams and ensures that those with the most contextual knowledge are the people supporting our systems.
- Seek to “Make Releases Boring™” — in our industry releases are all too often accompanied by ceremony and stress with late night release windows, hours of downtime and plenty of manual processes. Releases should be so routine and automated that they are repeatable, predictable and ultimately boring. To achieve this we want to keeping batch sizes small, encourage frequent code deployments, limit Work In Progress and invest in automation.
- Invest in architecture, automation & tooling — this is the area we have been weakest in so far and we’ll explain below how we are working to address this and why it matters.
Our Current Performance
It’s worth revisiting the classifications of the 2019 State Of Devops Report:
- We deploy to production anywhere between 10–90 times/week (with the 6 month median being ~58 deploys/week) equating to roughly 11/deploys/day.
- Our 6 month lead time mean is ~10 hours. This is much bigger than we want it to be. Anecdotally, it’s very possible for a commit to move from engineering workstation to production through our continuous delivery workflows in under 20 minutes on some of our services. Indeed the majority of this workflow is automated too, however higher outliers impact this result and we do not consider this good enough.
- Our 50+ microservices currently service over 380,000 external public API requests a day, each request generally fanning out to several more internal requests. The deployments we perform every day are not without incident — we have some way to go to before we achieve truly 0 downtime deployment and we haven’t yet enabled fast rollback of releases. This means we do experience brief outages more often than we’d like, although our time to restore service is generally minutes not hours.
- We don’t yet track change failure rate, but promote a culture of MTTR over MTBF.
Brooks’ Law & The Ringelmann Effect
Brooks’ Law is pertinent for us as we continue to grow. We want to ensure we grow sustainably such that each engineer who joins Marshmallow is able to deliver effectively and have maximum impact. We don’t want to over-hire and we are conscious of the possibility of The Ringelmann effect (the tendency for individual members of a group to become increasingly less productive as the size of their group increases).
The effect of Brooks’ Law observed below, suggests an organisation might expect a watershed moment at around the number of engineers Marshmallow has now — roughly a crossing point at which teams who have not invested in the practices mentioned above will start to suffer.
High performers see lower productivity per developer earlier in their lifecycle, but ultimately scale better than linearly as team size increases, especially past 100 engineers. This may be indicative of early investment in infrastructure, tooling and processes.
We are still early in our journey, having launched our first product in late 2018. During that time we have grown from 4 engineers to 10 and in 2019 grew our customer base by 3,000%, which we think makes us the fastest growing Insurtech in Europe. Anything less than performing as the high performer in the graphic above would hinder our growth ambitions and slow our pursuit of delivering on our mission. With this in mind it’s central to our thinking for us to invest, track and improve the 4 key metrics mentioned above.
Looking To The Future
Moving Fast is not without its risks and challenges, and as we continue to make mistakes and learn from them, our collective confidence will grow.
To name a few of the improvements we are committed to addressing in 2020:
- Zero downtime releases.
- Investment in more monitoring, observability and distributed tracing.
- Invest in better configuration management.
- Build more mature, stable and faster delivery pipelines and release management processes.
- Try different ways of working, such as trunk based development instead of GitFlow.
- Investing in tests higher up the test pyramid.
- Implementing synthetic transactions.
Earlier in this article, I referenced lead time as being one of the biggest predictors of performance, defined as the time between commit to production. Ultimately, we want our engineers to be able to move from idea to production as quickly as possible. Therefore we’ll also start to measure our cycle time. But we must keep ourselves grounded, we have a long way to go first in reducing the lead time per the definition above first and as Goldratt states in “The Goal”, when describing the Theory Of Constraints (TOC):
Any improvements made anywhere besides the bottleneck are an illusion.
We’re committed to continuous improvement and I look forward to sharing more about our progress in the future as we seek to build a world where insurance benefits everyone.
Be Part Of The Future
We have ambitious goals and we can’t get there without the best people. If this thinking resonates with you, you are the type of person we are looking for and would fit in well here. We are hiring and you can find open roles at www.marshmallow.com/jobs
Reading which informed & inspired this thinking:
- ACCELERATE: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations
- The Devops Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations, 2016, Gene Kim, Patrick Debois,, John Willis.
- Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation, 2010, Jez Humble, David Farley.
- The Phoenix Project: A Novel about It, Devops, and Helping Your Business Win, 2018, Gene Kim
- The Unicorn Project: A Novel about Digital Disruption, Redshirts, and Overthrowing the Ancient Powerful Order, 2019, Gene Kim
References
- McKinsey’s Three Horizons of Growth: https://www.executestrategy.net/blog/mckinseys-three-horizons-of-growth
- State Of Devops Report 2019: https://services.google.com/fh/files/misc/state-of-devops-2019.pdf Dr. Nicole Forsgren, Dr. Dustin Smith, Jez Humble, Jessie Frazelle
- State Of Devops Report 2018: https://cloudplatformonline.com/rs/248-TPC-286/images/DORA-State%20of%20DevOps.pdf Dr. Nicole Forsgren, Jez Humble, Gene Kim
- [Forsgren et al., 2018] ACCELERATE: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations, Nicole Forsgren, Gene Kim and Jez Humble, published March 27, 2018 https://itrevolution.com/book/accelerate/
- https://en.m.wikipedia.org/wiki/Brooks's_law
- https://en.wikipedia.org/wiki/Ringelmann_effect
- https://martinfowler.com/bliki/TestPyramid.html
- https://martinfowler.com/bliki/SyntheticMonitoring.html
- https://www.leanproduction.com/theory-of-constraints.html
