A Real-Life Experience Of Excellent Teamwork in Incident Handling
Project A is an Operational VC. We firmly believe that our team of about 100 experts from different departments like Software Engineering, BI, Data Science, Marketing, HR, Design, and Venture Development has a significant impact on the success of our ventures.
Currently, one of our ventures, kfzteile24, a Berlin-based multichannel retailer in the field of car parts and accessories which provides car parts for hobbyists and professionals alike, gets some support from our Software Engineering team. Side on side with kfzteile24's IT team, our experts work on the companies’ product catalog. In our experience, a collaborative culture is not only crucial in day-to-day-dealings, but especially in situations in which issues have to be handled efficiently and effectively.
Accessibility and the allocation of specific roles are essential for a smoothly functioning cross-departmental team
For months, our developers, Ops, and product managers from Project A and the kfzteile24 IT team worked on a new e-commerce platform. Finally, we brought the last puzzle piece of the new platform live: The catalog. Thanks to Microservices the rest of the platform was already running in production, but with the product catalog, the last piece of the e-commerce platform, we hadn’t had any experience yet. Nevertheless, for a period of two weeks, the platform operated without any interferences. On one weekend in August, on a sunny Saturday, my phone rang. A colleague, the Tech Lead from kfzteile24, called me.
“We have issues with the website!”
“We have issues with the site,” he said. The worst-case scenario had occurred; the site went down when someone had tried to access the product catalog. But even on the weekend, we could rely on our team — on both sides: Within 10 minutes a group of up to 7 people, colleagues from the Tech and the DevOps teams from kfzteile24 and Project A, were online. They quickly organized themselves and started to think about ways to solve the issue:
After a few minutes, we discovered that the site went down because of a Redis that had run out of the disc. We discussed possible options in the teams and agreed on one of them: Increasing the disc size and then re-deploying the service seemed to be a trivial solution; we just needed to adjust some Helm Chart. We had the assumption that the Redis would come back but wouldn’t work like expected. Some circle dependency resulted in the application trying to access the Redis when the disc size hadn’t been increased yet.
We had a running system back within 40 minutes.
Finally, we ended up with quickly setting up a managed Redis on AWS, followed by a manual configuration hack in the kubernetes cluster.
Communicate, participate, take over responsibility, distribute roles, get everyone on the same page, and find solutions as a team
Of course, there was some clean-up work to do after solving the issue. For example, we had to persist the manual changes as code in the infrastructure and to do a post-mortem incident meeting to share the knowledge about the reasons causing the incident and to discuss how we will prevent similar situations in the future. But we took some great learnings out of it:
When the conditions shift, an aligned and flexible team that smoothly adapts to the situation is vital for the success of a project.
It was a great experience to see how team members from different departments and companies communicated openly. Everyone participated in finding a solution, people took over responsibility, and, finally, solved the issue as a team. Besides a collaborative workplace culture, knowledge sharing is key to create an environment of success: Thus, once we had addressed the issues, we shared our knowledge.
