When Clouds Are Not Enough: Our Journey To Scale Our Product & Engineering Organization

We’ve played with various organizational setups over the years, but for me personally, this one looks by far the most promising.

In my time in Ataccama I’ve witnessed all kinds of changes across the whole company, but recently we quite drastically shuffled the way we organize our Product and Engineering. I figured it’s a perfect opportunity to share with you a little bit of our recent journey, thinking maybe you might find it interesting. Especially if you are, like us, figuring out how to succeed with scaling your product development organization.

In this article I’ll do my best to describe where we are right now as a company, what our ambitions are, what didn’t work in our product development organization, and where we want to go from here. It’s not a short read, but hopefully, it’s a valuable one.

Where is Ataccama now

After 15 years in this business, you learn a thing or two. How to pull Fortune 500 clients, how to fix data on a near-unlimited scale, how to grow the company… I don’t even remember the last time we were called a startup, if ever? With close to four hundred people in the company, no funding, and making our own money, I’d say we are becoming something else entirely.

But you learn from failures as well. Losing key talent hurts. Cloud is an enormous endeavor that shouldn’t be taken lightly. Adjusting the organization to scale is hard. The list goes on.

As one of the top data processing vendors in the world right now, it’d be fairly easy to just support our current customer base and rest on our laurels. Thinking, that this would be enough to sustain our long-term growth.

It’s either naivety, ambition, or pure lack of self-awareness that makes us believe there’s a world where Ataccama is at the core of every data-driven business decision an enterprise makes. That’s why our focus is right in the middle of Governing, Managing, Transforming, and Providing Data in our customers’ ecosystem — and on a scale. Simply, because we’re at the point when we finally can.

You can’t fake leadership

Our product engineering team is currently over 170 engineers strong, and it keeps growing. This fiscal year, we’ve budgeted for almost 100 more engineers. A mass of such intelligence, firepower, and innovativeness needs leadership. I believe that this attracts powerful talent to our organization — people willing to take on this challenge despite the mountains left to climb towards a more ambitious future.

That’s why we recently hired the experienced executive Roman Pichlík as our VP of Engineering, got Tomas Porazil as the VP of Product Management, and shortly afterward gained a new yet familiar leader of the entire Product Organization, our Chief Growth Officer Martin Zahumensky. With so many more wonderful engineers joining us every day we finally have enough power to start making the choices that will take us further, and on a scale, while also helping us see where we should go much deeper in the data market.

The change

Honest throwback, pain, and the decision to change

Aspiring to go beyond isn’t enough. If we were going to change, we needed to know what to change, and where. We took a couple of days out of our regular schedule, packed our bags, and left our Prague offices just to look back and see what worked and what didn’t. After working ourselves into a depression with a thorough throwback analysis, we came up with a list of our personal failures and got the necessary — seemingly logical — push to change our organization and move forward. Part of it was looking at our own roles and making the decision to change them.

To give you an example from our Strengths and Weaknesses analysis, we lack readiness to natively develop and run all of our products in Cloud environments due to our long history serving mainly on-premises customers (look at me, already making excuses). And on the other hand, even in engineering, we are insanely quick to adopt, reform, and deliver value in a very short timeframe — as long as we understand why something has to happen.

This evaluation, combined with our ambition for scale and for delivering meaningful work, gave birth to an idea: change how we organize the product, as well as the engineering team.

And surprisingly enough, at the end of the day, it didn’t result in our engineers wanting to flip their tables.

SpaceUP is shooting for the stars

The organization recently adopted a ton of features. It’s an ongoing process, but there are a few concepts worth sharing. Our biggest inspiration has been Pipedrive and their Launchpad-Mission-based organization, which ensures that mission teams are dedicated only to a single feature — with no interruptions. Then, we got inspired by Basecamp’s ShapeUP and their dual-track concept, shaping an idea into its final form via pitch. We ripped it off, renamed it, violated it in every way imaginable and created something that we hope will serve our purposes.

Addressing our challenges

As a data management product, the solution obviously works with data — or at least helps the users manage their often sensitive data and metadata. Everything has to be perfectly secured, not to mention easy to deploy, observe, and operate. Also, working with data used to be mainly an engineer’s problem, but our mission is to provide an easy-to-use, browser-based frontend so that anyone in the organization can use, configure and consume everything our platform has to offer. We take the most complex DB operations, ETL systems, and integration drivers, and translate it all into web-based user flows that, again, almost anyone in the organization can manage. We’re like an engine behind those hundreds of powerful, global corporations that help them make the right data-driven business choices behind the curtain. Therefore every change we are making has to support us in fulfilling this mission.

Agility versus Ego

Agility isn’t just the speed at which you adapt your product to market needs, but also how quickly you adapt yourself to an ever-changing world. However, even with the agility of your product development out of the way, it still doesn’t mean that you’ve organized yourselves to scale.

We already had an established management structure: product and engineering management and communities of practice. But looking at it from the angle of changing in order to scale, we had to adjust everything, including the way we organize our managers.

This part of the story blew my mind. Most of the people in “key positions” were more than willing to change and take on a seemingly “less important” role, just to make sure we succeed as a company. They went from VP positions to leading delivery teams, and vice versa. Those managers were inspired by the autonomy and freedom they were offered, not by the prestige of a title.

I also realized how fake any reorganization would be without making choices like these. It inspires me to work on improving the company for the goals it’s trying to achieve, rather than dealing with the politics that naturally come into every enterprise as it grows larger and larger.

Spaceports, cleared for launch

The teams are now organized into Spaceports, with a clearly defined ownership structure and the resources to deliver a feature from end to end. Every Spaceport is given clear and measurable objectives that need to later align with the company’s objectives as a whole. Spaceport Leaders are responsible for their execution, as well as acquiring the necessary resources to successfully run the development. A Spaceport is made up of a Mission Crew and a Ground Crew.

This decision to form mission-dedicated teams came from the previously mentioned Ataccama strengths evaluation. Our product successes and the team’s high morale were due to the fact that a group of people was focused on one problem, with no interruptions from day-to-day operations.

The Ground Crew is responsible for supporting current productions, product maintenance, smaller features, and bug fixes. And then there’s a dedicated number of people within the given Spaceport that can be allocated to flying Missions. It was our key takeaway after looking at our strengths, and we’re thankful that our culture will always support dedicating capacities for something meaningful and valuable to expand our business, rather than balancing out five other priorities within the given team.

Spaceports and space-highways (just imagine..)

There were two areas that still needed to stay somewhat horizontal, without having an explicit customer output of their “features” development. One is called “product foundations,” and the second is “Infrastructure, Cloud, and Operations.”

We spent a lot of time defining objectives that will pretty much have a business-related, measurable outcome. Even though those competencies are “enabling,” as much as I hate that word, both of those Spaceports have defined KPIs. With the infrastructure team, it’s more straightforward, since most of the biggest missions are Kubernetes-adoption-related and end with a set number of customers running on this technology.

On the other hand, the product foundation Spaceport delivers frameworks and components that speed up the development process for the entire engineering team. We love the metaphor of creating highways for the rest of engineering and our clients, in terms of the product operations and other enabling frameworks.

In search of meaning

One of my colleagues recently asked me if we are doing any meaningful work whatsoever. Since then, I can’t shake the question out of my mind.

As a manager of a small team, I could define meaning for about five people. I could individually encourage them, negotiating a setup that’s challenging and satisfying for each one of them. It gets tougher with 30 people, but it’s still doable. At the 100 people mark you start to sweat, and reaching 200 and over, you’ll need superpowers to get a crowd of that size excited. Especially when you need them to pull off a thousand different feats to achieve that one goal, whatever it is. A financial incentive is one way to do it (or at least it helps), but as you might already know, it doesn’t go all the way. At the end of the day, it’s not about constantly telling people what to do. You want them to tell you.

Want to find meaning? Create.

In all the years spent in this market developing heavy-lifting B2B enterprise features, we learned that, firstly, development can be really, really, expensive. Not only through exact expenses — you could calculate that from the engineering resources — but by possibly making a mistake that years of refactoring won’t fix. And secondly, when your creation hits the right spot you can provide massive value to the market, and on a heretofore limitless scale. This puts much more pressure on the evaluation of the idea, in our case through Mission HQ, and empowering everyone in the company to pitch their Mission through a system that will hear them out.

Balancing priorities with autonomy — Missions

Missions can run both within the given Spaceport as well as across different Spaceports. Eventually, we’d like to let Spaceports themselves decide on the missions they have resources to run, and, across engineering, prioritize only those that need capacities from different teams. Currently, the prioritization is centralized, made transparent for all Spaceports by a team called the Mission HQ. They collect and process all of the mission’s pitches, ultimately deciding which mission flies and at what expense. Also, they ensure that everything runs smoothly.

Mission parameters:

• Lasts from one to a maximum of three months
• Brings massive value to the business (both for internal and external users)
• Delivers end-to-end business value, with no additional work needed after the delivery
• Is led by a mission commander (can be anyone) with a fully-equipped mission team (from developers, product managers, designers, to infrastructure and operations people, if needed).

Obviously, smaller features can be run across the engineering without this complicated prioritization.

Context is King

We were blown away by the number of incredibly valuable mission pitches that came to Mission HQ, and only a couple of days after introducing the new organization. It made us believe that people can find meaning — because they are the ones who came up with the idea and carried it out to the end. The discussions on the regular Mission HQ pitching gatherings are insanely valuable, especially for their context.

Often, you might think that what you need to solve within your domain is the most beneficial thing for the company, while there’s a different opportunity that, when solved, will help us succeed more than your own original idea. If it’s presented and prioritized in the context of other things, it’ll help everyone understand what’s really important. Not to mention that it’ll make crafting upcoming missions even better. I believe that the traditional saying “context is king” has never been more relevant. The context is given to everyone through centrally prioritizing big topics. As lame and corny as it sounds, so far we’ve had a ton of fun playing with this concept.

Putting pressure on expertise

We learned that keeping people from bringing their own product ideas to the table is not a good thing. It takes away ownership and motivation. Plus, we were never really able to build product management so thorough and exhaustive that it would always supplement the engineering with relevant, actionable ideas that fit perfectly to both the state of the current codebase as well as market opportunities. We really do believe everyone should be involved in product planning.

But that’s only one, albeit critical piece of the puzzle. Another one is simply growing in the area of expertise the particular individual is working in. Over-preparedness, perfectionism, and being comfortable with the status quo: these give away a lack of true expertise in a specific area.

Experience taught us that there’s nothing like self-development in one’s free time while applying huge pressure on that person with business features delivery. This was supposed to be addressed by our communities of practice, but it wasn’t enough. They weren’t aligned with the company and the Spaceport objectives. They didn’t have strong enough leaders. They didn’t have the actual power to make a difference.

Most of our people grew from having their standards raised by someone more experienced joining them, and inspiring those around them to be better and do better — not by talking or managing, but by example. When a professional Operations Engineer joined, all of a sudden, adopting new enabling technologies stopped being a problem. Experienced Python developers started raising standards for other developers working with them. The same happened in our Product Design team.

So, we threw the concepts of communities away and introduced “Circles,” with Circle Leaders for each Spaceport, ensuring that the horizontal expertise will be top-notch. We still struggle with it since the pressure to have tens of leader-experts for various disciplines (K8S, SRE, React, Spring, AWS, etc.) is simply a steep challenge. And with a continuously growing organization, this challenge doesn’t seem to end. We do hope, however, that we laid down the road to success for anyone who will assume the role of a Circle Leader.

Tough decisions

This new organizational structure won’t solve all our problems. It probably doesn’t even solve the majority — it’s simply an enabler to make different choices, act on them faster, ultimately figure out they sucked anyway, and try to make new ones. Again, and again, and again. For us, those choices apply both on the level of which customers we are willing to serve, as well as which technologies we’ll support, how we’ll change internally technologically, and be ready for what’s next.

Every decision, big or small, is now made with enormous pressure on scaling. And most of the time, this means letting things go. And that’s difficult since it often goes directly against all the working habits we’ve formed. Without all of that, pretending the new setup will solve all our problems is hypocritical. Combined with crucial decision-making and letting things go, a scalable and empowering structure makes the future — at the very least — hopeful.

There is obviously more depth to all these topics and many other aspects this article didn’t cover. If you are curious, please don’t be shy to reach out. If you want to shit on it, suit yourself. I’m just glad you made it all the way to the bottom of this article.