Inner sourcing at Vodafone
Inner sourcing is the practice of adopting Open Source methodologies and ideals in how teams work together on code. This GitHub article covers it really nicely and at Vodafone we decided to look at this a way to solve many of our challenges in creating a scalable modern engineering practice.
But before researching and ultimately deciding on InnerSourcing as our way to go, we first started looking at OKRs and how they could help us.
What are OKRs
OKR (Objectives and Key Results) is a goal system used by Google and others. It is a simple tool to create alignment and engagement around measurable goals.
Felipe Castro — https://felipecastro.com/en/okr/what-is-okr/
At Vodafone, like many other businesses we naturally gravitated towards KPI based objectives, that were commonly set every year. These objectives would be written SMART and in all honesty, seldom revisited. Over time these goals became more quarterly and we started to narrow our focus at a shorter time frame to make them more meaningful and achievable.
However although these objectives were delivering against some of our key targets, they were on occasion not focusing on some of the needed behavioural changes we wanted to see. The targets occasionally felt gamed or controlled and team engagement was severely lacking because of it. We needed a better way to bring teams and people on the journey and make our objectives more meaningful, more impactful and longer lasting. This is where OKRs stepped in.
One of the first challenges we tried to meet with OKRs was releasing. Our original strategy was built on blue/green deployments — we had two identical versions of our entire platform in production running, with only one of those receiving traffic from customers at any point in time. A release would involve pushing changes to the dormant production environment, running sanity tests to check for stability before then performing a ‘flip’ of website traffic to the newly updated production environment. However, the flip release process that worked well for 10 teams started to not work for 20 teams. This solution began with something small. A seed, laid out in the form of an OKR, for our development teams to be challenged by, and most importantly to own for themselves.
Each team has to release just once, in the next 3 months, independently from any other team.
We really immersed ourselves as a department in this OKR, and celebrated each and every independent release.
So, with the success and transformation this OKR bought, we decided to go again. Next we trialled teams leading releases, then increasing our sprint predictability (which is over 100% now btw) and we tried uplifting quality. Each time, with greater success and greater team engagement. So for our next OKR, we wanted to select something that could help us scale, deliver value faster to our customers and ultimately create engaged engineering teams.
Every team must accept, approve and successfully deploy one PR into a service they own, from a different team.
Why Inner Sourcing?
When we first started our digital journey, we only had a handful of teams — each with little dependency on each other. Each one could own their own application and release as and when they needed to. As we grew, more teams started to form and our ability to meet our demand grew with it. It became increasingly tough between quarters to move team members around and swarm around the business critical areas without a sharp learning curve or additional training. Each application or service could have nuanced approaches to state management, unit testing or their pipeline — we needed a way for teams to begin questioning their approach and by allowing fresh eyes on the code, challenge the way we think.
When we began too look at Inner Sourcing we asked teams to begin asking questions about their application or service.
What if someone needed to make a change to your code and had never seen it before?
What if wanted to re-use parts of your code in other areas of our estate?
With these starters for 10, teams began to consider lessons from Open Source projects, that we could learn and adapt. We asked teams to question the following areas:
- Documentation 📚
- Test automation 👷
- Tech debt 💰
- Standardisation 📝
- Logging 📜
- Agile approach (For example, rules of engagement, DoD, DoR etc.) 🏃
- Service/application resiliency, monitoring and decoupling 👐
Adopting Inner Sourcing
So what changes did we make in order to make inner sourcing successful?
We created rules of Engagement
When teams need to interact with an unknown repository the first checkpoint would be the rules of engagement — understanding how best to engage with a team and/or service. This included an overhaul of our readme’s (including a contribution.md and a changelog)from a technical perspective and our DoD (Definition of Done) and DoR (Definition of Ready) from an Agile perspective. We also overhauled our PR process including how to do it, tips and hints for review and to remind individuals to share in their success.
We focused very much on the conduct when interacting with teams outside of your day to day, especially items such as naming conventions, pull request guidelines, discoverability etc. It became important that even the most minute of challenges were tackled with the upmost care, something as small as an “_” in a name can cause some engineers to scoff — and it was important for us all to align wherever we could.
We created monitors
For all of our repositories we now have webhooks into Slack that keep everyone up-to-date with the latest changes. It keeps all change visible and allows teams to be kept abreast of any recent changes in and outside of our team(s).
We welcomed feedback
One of the key aspects of any trial is to iterate and change based on feedback. We welcomed as much feedback as possible through many different means including automated feedback surveys and feedback in retrospectives.
Challenges…
Of course, like any piece of work, inner sourcing had it’s issues that we encountered throughout.
- Shared ownership takes time
Any work (at least in the short term) was taking longer than building it from scratch. Getting new people engaged, up to speed and delivering was a big challenge and in some cases proved difficult to accept that initial learning curve. - Quality wasn’t consistent
One teams 90% unit test coverage and another’s could be wildly different — not only that but some teams approach to testing could also vary. It did make those initial changes of personnel somewhat abrasive. Many would challenge “Why was it written like this?”, “What is the value of this test?” etc. — these questions although very valuable to encourage open conversation did mean a slow start. - Consistency is challenging
UX, testing, coding standards, linting, naming — each and every thing you can imagine that could be different, typically was. It meant flushing out some relatively tough conversations that needed to happen to get alignment across our chapters. Consistency is key to allowing inner sourcing to happen. - Contributing sometimes was not seen as the priority
On occasion inner sourcing was seen as a distraction from the day job, like with anything push back would always happen and in pockets it did feel tough.
Successes
It became apparent very quickly that this was something teams wanted to engage with and participate in. As engineers we are a very inquisitive bunch and dropping into any other stream of work at any point in time was really exciting and challenging in equal measure. We saw, almost instantly teams begin to ask questions, challenge those who they work with and want to get involved. Each and every day we would see a new post in Slack talking about what work they did, how they did it and what value it gave.
In terms of measurable improvements — We significantly reduced blockages due to reduced dependencies. With teams interacting more regularly than they had before, it made managing workflow between them easier and more straight forward, resulting in improved flow. One of the other major benefits to this change was scale — and although still in its infancy the ability for engineers to now drop into a team and help deliver is welcomed. Historically we would bake into our estimates hefty onboarding, shadowing and ramp up time that would mean an engineer would sometimes be 2–3 sprints from being fully productive. That value is now much closer to 1 sprint. Our engineering engagement is increasing week on week (we use OfficeVibe to measure this)and developers are generally happier because of it (resulting in improved mastery).
What next?
So where do we take this idea from here? Teams are engaged, practices are improved and flow is on the up, it makes total sense for us to continue on this road and begin opening this up wider — this includes not only improving it in teams that have been generally slower on the practice but broadening the uptake. We want to ensure we have evangelists who can go out into Vodafone and spread the word — individuals who can help services or teams onboard and understand why. We’d like to improve our communications — open up a transparent way for engineers to call out bottlenecks, areas of challenge and even services or applications that could benefit from this change.
Ultimately we hope this will lead us to more Open Source projects in the future, where Vodafone can work not only within its own Digital Engineering capacity, but wider than that, into a global community.
For now, best to continue reading InnerSource Patterns Book and see how best we can improve 🚀
