Macquarie embraces DevOps
Lessons from Macquarie’s journey to DevOps.
By Matthew Ellis, Head of Engineering (Banking & Financial Services) at Macquarie Group.
Over the last few years Macquarie has moved many of its engineering teams to a DevOps operating model. This has been a huge exercise that involved significant thought and planning at all levels, but we are now seeing the positive impacts of making these changes. In this post, we will share some of the things we have learned along the way.
Firstly, how do we define DevOps?
There is a misconception in parts of the industry that DevOps simply means combining Development and Operations teams or having developers doing Production support.
While this may be something that does happen, DevOps is much more than this.
DevOps is a set of principles & practices about how we can be better at software engineering. It is born out of Agile thinking and it places a focus on:
- Culture
- Automation
- Lean processes
- Measurement &
- Sharing
When implemented well, DevOps speeds up software delivery & improves quality. It’s a major shift from traditional software engineering approaches and it can be difficult for organisations to adjust. We need to have a #LearningMindset to become more mature at DevOps.
Our DevOps Principles
Each business needs to define their own approach to DevOps — there is no one size fits all approach. For us, this meant agreeing core principles that we want our engineering teams to have at the front of their minds. Through a series of workshops we agreed on the following principles in our Banking and Financial Services group:
#BreakDownSilos
We want to bring people from different specialisations closer together and minimise handoffs between teams to increase autonomy and velocity. Silos are everywhere if you look hard enough. For example, do operational reports only go to the management team not the entire engineering team? Do engineers sit with the call centre regularly to learn about what customers are saying on the phone? Find these silos and break them down.
#OwnItEndToEnd
We also want our engineers to take strong ownership of their full stack, top to bottom, even if they don’t directly own or manage the services they depend on. This also means our engineers are empowered to make their own architecture decisions and are accountable for these.
#DetectRecoverAdaptFast
Excellent monitoring, incident response and quickly learning from failure are all crucial elements of a strong DevOps culture. We have invested heavily in tooling in this space and have built some of our own tools which Facebook have asked us to demo to them. It has been incredible to witness how quickly and how well our engineers can respond to issues with this improved tooling!
#SmallChangesFrequently
The days of big weekly or monthly releases are over. We now do over 100 releases a day and use A/B & canary deployments strategies to detect issues early and rollback if required. We are always shipping value to our customers, incrementally.
#AutomateRoutineTasks
Creating space and time for teams to automate tasks is vital. We have created some amazing CI/CD tooling which supports over 100 daily releases, but we also want to automate processes that are not part of the CI/CD pipeline. For example we have developed our own chatbot on our group-wide chat platform that automates our incident management playbooks.
#MeasureEverything
You cannot improve what you do not measure. Above and beyond our improved monitoring & telemetry, every team also has defined key metrics that matter to them. These are visible on TVs everywhere and discussed daily at each scrum huddle. We have also arranged training sessions for our engineers on Site Reliability Engineering and created Service Level Objectives for all of our key services.
#LearningMindset
We believe one of the the most important elements in attracting and retaining top talent is giving engineers constant opportunities to experiment, learn and grow. To help with this we run an internal cross-skilling and professional development program for staff called “Expand Your T” which is a reference to being T-shaped. As one example, we have an internal Cloud Guild with over 1,100 staff and also have a cloud certification program for our engineers where they can choose to be certified in either AWS or GCP for free.
#ShareKnowledge
We want our engineers to always be on the look out for new and interesting technologies or ideas and discussing these with their colleagues. We also decided that email is a legacy technology and so we now collaborate entirely on social media based collaboration platforms. This has completely changed how we share information and interact with each other, and has help us promote a very flat and open culture.
Where are we now?
We have made amazing progress with DevOps in Macquarie over recent years & we now have many great examples of where our focus on DevOps has yielded major improvements. Here are a few recent examples:
- One team gained an additional week of productivity per month. This was achieved by automating routine tasks that had just become BAU.
- One area had a 75% reduction in alerts raised to on-call support staff. We did this by measuring the root cause of every single alerts over a long period, ranked the most important root cause fixes and then implemented those fixes on a prioritised basis.
- One Digital application saw a 100% improvement in website performance. We did this by listening to our engineers who wanted to move a legacy application to a cloud based environment. The results were amazing.
- A 50% reduction in Production incidents in one area. We achieved this by introducing Blameless Post Mortems for every incident and tracking remedial actions so those incident never happen again.
What’s next?
DevOps encourages continuous, incremental improvement and so our job will never be done. We like to borrow a saying from Facebook that we are always 1% complete on our journey, but we take pride in how much we have achieved so far.
