Chapter 1 — Does It Scale? Why We’re Rethinking our Engineering and Product organization

I changed my role in Ataccama recently, becoming the Head of the Product and Engineering organization — a home to about 190 people. My goal was to create an organization that will deliver a stellar product for our customers, create a motivated team that drives us toward our Unicorn vision, allowing us to scale up instead of just growing. I will be sharing a series of articles with you in which I will explain the transformation journey, the motivations behind the change, how we created a new and unique organizational model, and share real experiences and lessons learned. I believe this will be an interesting read for most product and engineering leaders, but also for engineers looking to improve the way they work. Make no mistake — this journey isn’t finished, and the only thing we know is that change is the only constant, and an important part of being successful. So, take this as an inspiration and lessons learned.

Does It Scale?

Scale vs. Growth — simply pumping resources into your organization isn’t the answer, you must find ways to increase exponentially by adding resources linearly. Take people’s mindsets into account — some people are open to doing anything, and some always find a reason to claim it’s impossible.

Ataccama used to be a growth company. And as we grew, our expenses and the size of our team were growing right along with us. Our Product & Engineering staff grew to 190 people, and our headcount grew by 80 year over year. The company doubled in size to more than 360 people.

We have 5 main products in various data management markets, covering everything from Data Governance & Data Catalog, Data Quality, Master Data Management, Reference Data Management, to Data Integration & Transformation, consisting of 100+ software components we need to maintain.

Over the last couple of years, we’d been laser-focused on developing the second generation of our Ataccama ONE platform, which launched in April 2021. It was a big success, but it also brought plenty of new challenges. It introduced the concept of Data Quality Fabric to companies and their employees, helping automate user tasks as well as providing a connected, consistent user experience across different data management use cases. It also introduced the ability to start smaller, with limited use-cases, and then migrate to additional use-cases within the same platform.

Alongside this, we’d been running a very aggressive growth strategy, aiming to reach Unicorn status within 2 to 3 years. This means a lot of fast-paced growth, on all fronts — we plan to grow the engineering team to 280 within a year and a half.

In the past, we had gone through several different organization set-ups, with different degrees of success. We learnt that as the company grows, the needs become different. Things that worked in the past aren’t necessarily the right solution right now, and also, things that hadn’t worked in the past aren’t necessarily wrong solutions for the new challenges.

In the summer of 2021, the engineering team worked in smaller, empowered scrum teams of 6 to 9 people. Our CEO was acting as the Head of Product, leading the overall product strategy with 4 separate roles reporting to him — Product Lead (covering product managers and designers), Head of Engineering, a separate Cloud Ops team, and a Product Support team. To give you a sense of the size, in August 2021 we had around 10 product managers, 10 designers, 150 people in engineering including documentarists and testers, 10 people in the cloud ops team and 10 in the support team. And during all this time, the organization was split between Product and Engineering. It made the teams feel misaligned, disjointed, like they weren’t the one team they were supposed to be.

Our setup at the time couldn’t handle the strain. If we wanted to reach our goals, we had to look at things from a completely new perspective. Instead of walking a linear path, we had to go exponential. Instead of growing, we had to start scaling.

Ataccama's CEO Michal Klaus handing over the Head of Product position to Martin Záhumenský

Reaching the limits, facing problems

During the summer of 2021, all of the key stakeholders had a feeling things needed to change to keep up, that we had reached the limits of our organization setup. Our CEO Michal Klaus decided to hand over his Head of Product role to me. My former role of Chief Growth Officer was focused on enabling the company to scale. We had many discussions about the issues we’re facing, and how to enable scaling within both Product and Engineering at the same time. We then had a joint meeting of key leaders in both Product and Engineering to discuss future setup. The main issue was the split between Product and Engineering organizations, and the disconnects it was creating, it created a lot of misalignment, didn't support the feeling of being one team, pulling on the same rope. We could proportionally grow them both, but it wouldn’t result in a scalable product and a better product. We agreed that it wouldn’t work this way. We stepped back, had an honest re-think, and re-iterated all the approaches that had (or hadn’t) been working for us in the past. We didn’t want to just blindly follow any particular methodology. Based on a SWOT analysis, we knew we were good at some things and worse at others. We later used these strengths and weaknesses to build the new organization — here’s a quick overview:

1. The concept of an ambitious, time-boxed mission worked for us. We could deal with stretches and tough goals. They challenge us. We’re like sprinters. We have to see the track and feel the finish line, even if it’s tough to get there.
   - It supports ownership instead of “I’m done once the mission succeeds.”
   - It’s easy to spot leadership when people step out of their formal roles.
2. We have the market fit, a strong product, a good vision, lots of space to grow, but we’re a bit lacking when it comes to executing fast enough
3. Our people’s attitude and culture are fantastic, they’re always willing to go the extra mile. Lots of smart and motivated people all around.
4. We had a siloed setup instead of a “flow’’ setup. Each specialist’s job was done once “my task” was done, and then it was handed over to some other specialist to do their thing, over and over again. This “manufacturing” style of development led to many issues, from miscommunication between the specialists, low delivery speed and quality, poor user experiences, to a lack of engagement from our own people. Our people weren’t seeing their achievements, or how the output of their work was being used.
5. Long, difficult, unplanned work (production incidents, issues blocking customer success), unfixed bugs and unpaid technical debt. We weren’t able to schedule these things upfront and hammer them down. Features took precedence almost every time. We simply didn’t have the visibility, alignment, or commitment to solve them, did poor capacity management and put out fires as they were springing up. Our most skilled people were dropping parachutes into the eye of the hurricane to save things.
6. As we grew quickly and focused on the development of Ataccama One Gen2, we intentionally didn’t give enough attention to some of the mature components or services. While we were building on top of these components, nobody did their maintenance and adoption into the cloud and Kubernetes age, nor had any related deep technical insights. We ended up with a bus factor of 1.
7. We were misaligned internally, even though we experimented with an objectives/goals framework. We had defined goals and empowered teams, but we struggled to pick and focus on the right things at the right time, leading to a waste of people and capital...and a drop in morale. Projects were being canceled midway because, from a goal perspective, they didn’t contribute enough.
8. Turning the steering wheel was hard. Sometimes, you need to react and adapt quickly to things happening around you. We always found this difficult in this type of distributed organization.
9. We needed experience with projects in the long run, e.g. having the first customer using the feature after 6 months of development was a recipe for disaster. It either didn’t work as the customer expected, or we underestimated the complexity, leading to a half-baked product.
10. We were saddled with a monolithic architecture. Starting from the code, the build, all the way through to testing and release itself. Teams were getting bottlenecked left and right. Even teams developing their own services had to keep the pace of the slowest one. It couldn’t scale at all. In the end, we were less and less effective even though we were growing almost 100% year over year by an Engineering headcount.

Martin Záhumenský introducing what we want to achieve with the new setup

What did we want to achieve?

We didn’t want to transform just for the sake of transforming, so we set some goals which we wanted to achieve by the time we were done, and then used this as additional input for drawing up the new set-up. It was a long wish-list

First, we had to change in order to be Unicorn compliant. We needed to scale instead of grow. Manually adding more power wasn’t the answer, we needed to set things up so that they’d scale- communicating asynchronously, decentralizing knowledge, and getting rid of all the silos, which were causing loss of ownership and separating the product, engineering, documentation, cloud and ops teams.

Ultimately, we wanted to establish total top-down and bottom-up alignment. We wanted full transparency into what our people are working on right now, who’s working on it, and precisely how it contributes to fulfilling our goals.

• We wanted to improve planning, increase visibility in the work being done across the organization, and improve request prioritization and assessment for features coming from different sides.
• We needed to ensure we were able to cascade objectives down the road, from company objectives down to business unit objectives and ultimately team objectives.
• We strived to improve delivery speed and quality, tighten up the documentation, streamline onboarding and be more timely and reliable with our delivery.
• We imagined an organization which enables disruption and innovation out of the box, and motivates people to keep innovating.
• We had to bring back the feeling of ownership, that people take responsibility for things they do, not just on their end but also how it works and feels for customers, partners, and users.
• Anything centrally managed doesn’t scale. Having both a top-down and a bottom-up alignment, with clear objectives, sets reasonable boundaries and empowers our people. We wanted to just say what needed to be done — the how would be completely up to them.
• We wanted to be more flexible and adopt changes faster, to turn the wheel at a moment’s notice. We also needed to fail faster — to discover something is going wrong much sooner, and be able to act on it, crash it, reshape or better structure it.
• We needed to improve our knowledge transfer and avoid single points of failure, i.e. individuals on which a lot of things depend.
• We had to give proper focus to all our products so they keep being stellar and leading-edge, which meant having dedicated capacity. We knew that dedicated capacity for unplanned work gives us an uninterrupted focus on what drives product innovation. We don’t have to shift from left to right.
• Finally, Engineering should celebrate what they achieved — we had to find a set-up where engineers will have a first-hand account of how their outputs are being used.

We tried learning from the best, but …

We gathered a group of smart minds around this problem, and learned from the best by doing tons of research on how other companies work. We went through set-ups used at Spotify, Apple, Pipedrive, Google, Amazon, Basecamp, looked at different methodologies such as scrum, agile at scale, SAFe, the two-pizza rule, Shape Up, we listened to a lot of books such as Inspired, Exponential Organizations, Turn the Ship Around, the Product Book, Scaling Done Right….

It’s important to understand that other models usually don’t fit other organizations one-to-one, and what works elsewhere isn’t necessarily good for you. You must consider how big you are, how your people are used to working, what kind of architecture you have, and what you’re good at. That’s why we decided to cherry-pick different approaches and create our own methodology, which we call SpaceUP.

READ NEXT: In chapter 2, we’ll explain its basics, where we took inspiration from, and why we decided to go with this particular approach. Stay tuned!

Even more SpaceUP:

• Chapter 3
• Chapter 4
• Chapter 5