Our journey of Infrastructure Management Transformation
Traditionally IT and Infrastructure organizations manage their technology roadmap by functions or components (for example: compute, network etc.), which automatically routes to set goals based on the technology desire of the team.
This is also how the Cloud Platform group was managed for a long time, and it was great (or at least we thought at the time that it is great). We took to the extreme the statement “build it and they will come”, thinking we know best what our users need.
But what happens when you invest, and invest some more, build a great infra which no one really needs?
A couple of years back, we decided to change the paradigm in which we operate, develop our infrastructure, and eventually set our priorities. This post is about our story of transformation and shifts to manage our infrastructure as product lines.
Before we go on about our journey, here is some background information, which is important for understanding the context of our story. We are a group of about 40 Engineers, responsible for all of Outbrain production infrastructure — including hybrid implementation of AWS presence, GCP presence, and a fleet of over 7000 physical servers and 400 network devices spread in several physical data centers globally.
On top of that we are responsible for the Build and Deployment pipeline, support hundreds of deployments daily, based on Kubernetes cluster we implemented within our physical data centers; Observability infrastructure based on prometheus and ELK stacks; Data Engineering which is responsible for all the data stores used as the backend to our software, data pipelines to assure data is available in real-time, data lakes and data analysis technologies such as BigQuery, SPARK, etc. In a nutshell — our lives are interesting!
Our journey of transformation begins with one aim in mind — we wanted to make sure the work we do generates real impact and brings value.
With that state of mind we set four ground base concepts, which guide us still today:
Customer-Centric approach -
We did not invent this concept. The novelty was to take it to the infrastructure level, and define what it means specifically for us — building the best infrastructure and tools to be used by our customers, other Outbrain Engineers.
This means we need to have a good sense of what their needs are (which is not necessarily what we think they need), and we need to start gathering this information from our customers.
Work with KPIs -
There is only one way to know if you achieved the impact you aimed for, and that is to set your performance indicators and measure them.
It can be different types of KPIs, for example adoption rate, cost efficiencies, etc. The main thing is that you set your KPIs in advance and that you can measure them.
Economies of scale -
We are a fast-growing company, so as such, we must make sure our growth and advancements are not limited to human resources (as much as possible…). Therefore we must focus on automation (no mundane tasks over and over again), and self serves — empowering our customers to be independent as much as possible.
Technological Excellence -
It was clear from the beginning that in order to progress and achieve every one of the above-ground base concepts, we MUST invest in technology, invest in the toolbox of our own engineers and continue driving innovation of our technological stack.
Once we looked at those four ground base concepts, we realized we are not developing components (like traditionally Infrastructure was regarded), but we are developing products, and those products happen to reside in the Infrastructure space.
Hence came the concept of “product lines” within our Cloud Platform group.
The transformation was gradual. Starting with our Cloud Platform Engineers taking on part-time “Product manager” roles for the product they develop.
We provided them with guidance and courses from the Outbrain Product group to support them and educate them on how product management works.
This also opened career development opportunities for our engineers who wanted to expand beyond coding and allowed them to experiment with something new.
Once we saw this direction is generating value, received good feedback from our customers, and saw that the products we develop based on this new paradigm and concept have a wider and stronger impact, we decided to double down our efforts in this direction, eventually having one of our backend engineers move to be a product manager full time (you can read more about this experience in Dafna’s great blog — “I was a developer. Now I’m a product manager”).
Over the past 2 years since we started this transformation, we have built different products generating great impact.
Products for automation of our servers install and hardware management within our physical data centers, which came in very handy when we needed to deploy in record time 100s of new servers during the COVID-19 first wave to support traffic surge; products for real time data pipelines empowering our algo engineer; products for visualization of resources usage, empowering every engineer to make sure his/her code is efficient as possible (and flagging where improvements are needed) — for all of those, we are able to see the value our products bring and measure our work impact.
Working with this Product lines approach is helping us to make better decisions about priorities and how we invest our own resources and time (probably one of the most important resources we all have exactly the same amount of… ).
It has also helped us to crystallize the impact we have within Outbrain and with that drive engineers’ satisfaction and pride in what we do.
No doubt Infrastructure management perspectives have changed for us and we will no longer look back — a true transformation!
