Vodafone 2020 — What a year 🎉
In only a few days, 2020 will end and what a year it has been — one with major challenges and opportunities. In a year where we’ve had a global pandemic and several lockdowns, in Digital Engineering it’s become all the more important to present our customers with a high quality, always available, coherent digital customer experience.
At the start of the year we started with a vision — writing the next chapter in Vodafone’s future. Building a world-class software engineering capability to fuel Vodafone’s transformation from Telco to TechComms. We’ve had to adapt this vision relatively quickly, accelerating much of our work due to the pandemic, to better serve our customers who had moved from interacting with us in our stores, to digital channels.
This post serves as a snapshot of some of the cool, interesting and challenging aspects of Vodafone’s journey in 2020. Before we get started on some of those initiatives, I’d like to thank all of our teams for their hard work and a congratulations across all industries to make it through an exceptionally tough year. 👏
Projects and Deliverables
Every year Vodafone has many platinum device launches, our biggest commercial activities of the year. This was the first year that our iPhone release would not only be delivered on our newly developed shop application and digital application layer (DAL) but would also be on our own instance of AWS. It was crucial for this release that we ensured we could handle the scale of traffic wanting to interact on our platform and purchase online, particularly considering the impact of COVID-19 on our retail estate.
In order to ensure a smooth and operationally silent release activity, we bought several disciplines together in order to —
- Deliver the proposition changes as well as scaling activity — fixing many of our autoscaling policies that were either inaccurately set or not autoscaling whatsoever
- Installation of APM into each service and application — by release we had 90% of our services with Datadog APM enabled
- Run several spike, soak, load and breakpoint tests — for our applications and services — we even fixed some containers that could only handle ~3TPS (Transactions per second) to handle 30+
- We used Slack for reporting and breakout channels for specific team issues
“Our traffic volumes on the day matched and partially exceeded volumes which we had exactly tested using Data driven PAT tests this year.”
David Jambor, Head of Systems Engineering
One of the biggest successes of this major release was that Digital became the biggest sales channel in Vodafone for the first time EVER. This was such a significant milestone for Vodafone’s digital journey.
Needing to Scale
The pandemic had a significant impact on demand — more and more business stakeholders were turning to Digital to get their propositions in front of customers. It meant that we as an Engineering practice had to look at how we approached our work and how we could scale, effectively, to meet that demand. We had to hold true our values whilst doing this — A quality-first approach to software engineering, a relentless focus on security and allowing our teams to drive increases in throughput, release cadence & flow efficiency.
One of the first changes that was made was a focus to towards decoupling our releases and process. Historically it was very hard for us to tease out smaller changes and release them in isolation, resulting in larger “flip” releases, where several changes were bundled together and then flipped from blue to green (or vice versa). These were time consuming, problematic and of high risk — it was likely that if 1 small part of the release was broken, the rest would need to backed out. It was a huge mental shift for our teams, focusing on small, incremental value to the customer — it meant working extremely close with our product and operation teams and also meant creating new team roles like Release Captains ⛵️. In January we managed 33 releases, which was (at the time) such a celebration for us, we were used to releasing every week or so, so this was a big change— in December so far we have done 129 releases — and with more to come.
In 2020 we’ve released over 1200 times to production, with a 98% success rate. Our aim is to continue to challenge our teams to break down releases into smaller achievable chunks — and in 2021 make even more possible with Feature Flagging (more on that later).
Another major change for us in regards to scaling up our ability to deliver value was a focus on our capability. We intended to focus on 4 key areas to scale our ability to deliver more to the business, but the key one was delivering a full CI/CD pipeline. Investing heavily in creating our own platform-as-a-service that could be used by Vodafone all around the globe — an ability to provide on demand capability, a mechanism to automate the test and deployment of code and allow us to have greater monitoring, alerting & observability.
Through this year we have made massive strides in order to make this idea a reality and although not quite there yet, we have made some really great step changes —
- Addition of Sonarqube into the pipeline — We now have the ability to run a local dockerised version of Sonar so we can see an early indication of what effect are code changes have on areas such as complexity or duplication. We also have added a build breaker so that any unit coverage below our benchmark (currently 90%) will fail the build.
- We switched to Cypress — at the turn of the year a lot of our testing was manual and we couldn’t functionally test our journeys end-to-end. We made the choice to move to Cypress and just recently had our first pre-merge check using our brand new suite in our Basket web application.
- Security scanning — With a heavy focus on security, particularly vulnerabilities we invested time and effort on selecting an appropriate tool for ensuring automated scanning of our services and applications.We tried both Synk and Whitesource and we are now adding these to our pipelines.
- Synthetics — With the ramp up of Datadog throughout 2020 we now have a fantastic suite of synthetic tests that can run on a deployment — it gives us a very quick, easy and thorough way to test the validity of a deployment and a quick way to rollback if we see issues.
Growing a Remote Workforce
It has been great to see the growth of engineering encompassing our back end developers, front end developers, agile team, business analysts, release team and our new automation engineering capability over the last year. Touching briefly on automation engineering, this new ‘chapter’ has a mandate to improve the quality of everything we deliver, and wewant them to be putting us in a position where we are finding absolutely no defects in SIT, PAT and of course Production. A lot of work this year has gone into ensuring that we are automating in a consistent manner across all teams and it’s great to be hearing such positive feedback on the difference that they are making from teams within which they are embedded.
This year we have also invested significant time in our progression framework — a way in which we can measure the progress and career development of our people. We have promoted over 10 people across the team — each promotion being thoroughly well deserved. It is so important to us that we have a way of progressing the development each and every member of our team, and these promotions are just one way of showing this.
Along with the promotions, we have made a number of excellent hires across the various different chapters. Some roles have taken a while to fill, but that’s ok — this is one of the areas where we absolutely cannot afford to take any shortcuts — we must surround ourselves with like minded, world class engineering talent. We’ve had the largest number of people involved in recruitment this year and I’d like to thank all of them for the hard work and time spent screening and interviewing candidates. In Web Engineering specifically we have hired 21 Developers, doubling our team size over the course of 9 months. Having seen over 200 candidates, with 80 individuals having done our new JavaScript coding test that allows us to validate candidates much earlier in the process and with greater success.
Beyond the internal recruitment that has been conducted we also managed to find a way to onboard a brand new supplier with well over 100+ new starters, most of which came during the first lockdown period. Reflecting on this now, it was probably a crazy thing to try, but something that we worked hard with our supplier to achieve effective onboarding for every new joiner. For example, we now have a ‘bootcamp’ library of dozens of videos we have recorded for new joiners to peruse through during their first week with us.
A Year of Tech
We’ve made significant progress in modernising our tech processes such introducing a more streamlined CI/CD, creating our very own way to server side render our applications (pre-render) and introducing Cypress tests — we’ve also onboarded several other technologies into our stack including LaunchDarkly. One of the key initiatives that we drove this year was to move to our very own instance of AWS — which is owned and managed within Digital. Migrating our environments was critical to ensure we completed the first major step on our Everything As Code journey, and enable us to have complete consistency across all the environments. Through this change we gained application and infrastructure ownership - which has lead us to full ownership of our release journey without any external dependencies. We can now replicate an entire environment in under 30 minutes.
The majority of our platform is now running on Fargate in AWS, completely removing the need for us to provision and manage servers, whilst also ensuring that we are as cost effective with our use of infrastructure resources as we can be.
Our integration with LaunchDarkly is so close to completion, providing us with an enterprise feature flagging capability. We anticipate this to be the catalyst for a behavioural and cultural shift that will see deployments become decoupled from releases, allowing us to release more frequently into production and have even greater confidence.
We have changed the way that we manage our log routing, now utilising the AWS Firelens service. Firelens not only allows us to standardise our log formats but more importantly gives us the capability to easily federate our logs to various different log services (for example, Cloudwatch and Datadog).
One more thing — we’ve dedicated a lot of time to our component library. Web Components V2 (or MAKE as it will be known by) is the second generation of our core component library. This library, written in Typescript and StyledComponents will allow our teams to share individual components in their applications, using Lerna, without the additional bloat. The creation of this library will ensure greater UX consistency, an ability to migrate into our new strategic CMS platform and an ability to theme, meaning any new channels that Vodafones can utilise the same building blocks under the hood.
Ownership
There have been some giant strides forwards in this area where we have seen development teams taking on far more ownership throughout the Software Delivery Life Cycle than we have ever seen before — and it’s something so crucial to the way we want to deliver software.
The changes to our release process has been mentioned previously, but what was so impressive about achieving this was the way our teams went about tackling the problem. Each team took it upon themselves to figure out what was preventing them from releasing independently to production, and went about sorting out those issues. The ownership and engagement we had from the teams to make a long lasting cultural change in the way we deliver software was fantastic to see.
Along with releases there has been a huge drive on achieving better sprint predictability. This was definitely an issue for us early on in the year — low sprint predicability impacts our ability to deliver, and ultimately build trust with our stakeholders. Once again, this was a challenge that every team engaged with and rose to the occasion. Teams figured out for themselves the key root causes for them that was preventing them from being predictable, came up with a series of initiatives, acted on those and as a result we made a rapid change of increasing our sprint predictability from 56% to 95% in just two PIs (6 months).
There has also been change with the way we now want to take care of our services as they run in production. With Datadog APM now in place for a large number of services, we’ve seen more development teams than ever engaged with the monitoring and observability that this tool provides us. It’s the way forwards, and as we focus on increasing the quality and resilience of our services in 2021 we very much hope that the pride we have for our services will translate to even great engagement in the way our services are actually behaving for our customers.
A Look Ahead
2020 has certainly been a challenging year, and whilst challenging years can be tough they can also bring a unity to teams that only serves to strengthen us. Looking ahead to 2021, one of the things we will be most looking forward too will be returning to the office in some form and seeing/working with our teams in person once again.
We’re going to be raising the bar on quality, with huge improvements to our CI/CD pipelines and automation & performance testing capability, as well as putting a big focus on building for failure and non functional requirements.
We know that with the amount of demand for Digital, we will have more to deliver than ever before, but we will absolutely meet that demand. What makes the year ahead more exciting than anything else is the fantastic team that we work with day in day out — after all, it’s not WHAT we deliver, but HOW we deliver that is so important for us.
