Software Organization Roles and Departments
Opinion: I think there is something myopic about many (not all) software companies and how they tend to organize themselves in the name of “scale”.
I have worked at a lot of software companies. Before my shift to Platform technologies and architecture, I was a consultant, traveling all across North America. I think I ended up working at roughly 80 or so companies in one respect or another during that 10 year period. This is all to say, I’ve seen a lot of organizations and how they are structured. I have seen these companies through the lens of a contractor, consultant, full time developer, enterprise architect, product manager, development manager, and executive. It seems important to state that my opinions and analysis is based solely on that experience and that I am not a professional organization development researcher. I say that because experts are important, and if you’re reading this and you are one, I’d love to hear from you.
First, Some Observations
There is an agreed upon truth that all technology leaders seem to know, rarely verbalize, but still hold on to as an “ace up their sleeves”.
A development/product team of 2–4 people without direct supervision will always run circles around a traditional engineering team or even the department.
This statement does not mean 2–4 people are enough to build any product. Just that 2–4 people move very fast. I know this from personal experience on multiple levels.
- First, I have my own 4 person team with United Effects and we are making fast gains. Before that, I have created entire software solutions from concept, to mocks, to monetized product launch in 3 months or less and I did it with just 3 people (counting me).
- Second, I have done what most enterprise leaders do in large companies and had my short list of people “who can get it done”. This is the “ace up your sleeve” maneuver. You have your swat team, who are usually a subset of a larger group, and you know that if your back is against a wall, you can put these people in a room, close the door, and when they come out again it will be done.
- I have been the “ace up the sleeve”. In times of unforeseen struggle, delays, and other factors that seemed to slow delivery beyond reason, I have been asked to simply do the work by myself. I’ve had this happen to me as an engineer, as an architect, and yes even as a director. And I delivered.
Knowing this fact, that small teams of capable people tend to deliver quickly when you stay out of their way and don’t bog them down with process, why do most organizations create complex delivery processes, bureaucracy and large teams? If you ask them, it’s because of scaling product delivery. I know this because for a long time I said this too despite my nagging thought that something was off.
I think intuitively I’ve always found this logical dissidence problematic. The notion is that you cannot scale small teams who work quickly and efficiently on their own, and so you need big teams. I learned big corporate scaling practices based on notions (notions being an important qualifier here) of Agile. I’m not going to get into all of them but I will call out the one you have all probably worked with by now, SAFe.
To be fair to SAFe, there are a lot of interesting templates, processes, and concepts to draw inspiration from within this delivery structure (I’m a huge fan of WSJF). SAFe purposely calls them competencies because these are concepts that when learned can help you. They are not a blind checklist for “how to delivery your late project faster”. The problem is that this is not how SAFe is usually implemented. Every time I hear someone tell me, “SAFE says we do this…” I know that they’ve missed the point, or worse, decided to use SAFe as a shield to push their own agenda. Still, this is understandable because in my opinion the problem is deeper than just processes or Agile or SAFe.
When we think about enterprise software delivery, we are forced to discuss the myriad of roles associated: Development Manager, Product Manager, Program Manager, Software Architect, Enterprise Architect, Product Owner, UX Designer, Backend Software Engineer, UI Software Engineer, Full Stack Software Engineer, QA Tester, QA Analyst, QA Automation Engineer, Release Engineer, Systems Engineer, Infrastructure and Operations Engineers, And, yes… even more.
Large enterprises have all of these roles and titles. I’ve held many of them over the years. If you’re looking for complexity, let’s go back to SAFe and look at their admirable job at attempting to bring all of these roles together into a diagram that explains what they do:
You know what I see? I see a conveyor belt in a factory.
In no way am I saying the skills associated to these roles are invalid or not needed. Nor am I saying that any one person could do all of these things. What I am saying, is that there is a logical disconnect between the reality of what we know delivers quickly (small team empowered without much management) and a structure that defines over a dozen roles to do the same thing at scale but much slower. And this brings me to my next observation:
We often create role distinctions to serve the process rather than the product.
I remember when I first gained the title “Enterprise Architect”. At the time, my title was Principal Engineer, which I took to mean “person who has some ideas people should consider” more than anything else. I was happy with that title, but the company I was in was restructuring and there was a notion that I should report to the CTO and help him implement a technology strategy. When discussing with him, he said, “I think you should be my Enterprise Architect.” I remember not really knowing what that means. I asked what my responsibilities would be and he described what to me seemed like my current role. I googled “Enterprise Architecture” and all I found was a tool to model enterprise architecture… which I realized I had been doing anyway. Still, it was a raise, a promotion to report to the CTO of a large company, and clout from being one spot away from the C-suite. Then I realized that this wasn’t about anything other than politics. They needed me to keep doing the same thing I was doing. They just wanted to position me better within a newly constructed hierarchy of authority.
I should pause here and say, this was a key turning point in my career and an amazing opportunity. I have nothing but respect and gratitude for that CTO and the opportunity he gave me.
Coming back to the point, I was fully capable of being an Enterprise Architect or a Developer or a Software Architect… but those titles were not the point. The job was the same. On the surface this may not seem like a big deal and we could simply chalk it up to the price of corporate growth, but I’d like to suggest that this is problematic because ultimately this kind of thinking represents an inefficiency which slows productivity and artificially creates new human resource costs that may not need to be there.
As an example, I once had a real conversation with a technology leader as an Enterprise Architect regarding a product I was designing. I was explaining the requirements and he stopped me and said, “You shouldn’t tell me requirements. You’re the architect. Find a product manager and have them tell me.” I responded that all that would mean is that I’d have to prepare a presentation for the Product Manager so he could then present my ideas under a new title. The leader was quietly thoughtful for a moment but ultimately did not relent. It was not my job, was his conclusion, and I should not be trusted with that accountability, specifically the trajectory of a product and user experience.
I don’t blame this leader. We (technologists) are indoctrinated into the idea that there is something special about these titles. That when you shift your career toward one, you somehow lose all professional capability in the other. We measure the success of others through the trials and success of ourselves and as a veteran of an industry that forced him to choose a pigeon holed path, it is not surprising that his thinking would be molded in the same direction when dealing with others. We tend to forget that we made all these processes and titles up ourselves.
The problem here is that these titles and these roles exist to quantify a process, not to build a product.
We see the same pattern emerge in other areas. Quality Assurance for example, is a discipline in which I spent years consulting. One of the biggest anti-patterns I often had to help unwind was the notion of a QA organization that measured success by number of tests and defects. In an industry where money is made by avoiding defects, and sales are based on the production code, not the number of tests, somehow these enterprises had evolved to incentivize increased time to market through unnecessary testing and the rewarding of defects found instead of the absence of defects in production. I suppose it’s understandable. It is easier to reward a positive (in this case tests and defects) than a negative (the absence of issues) because it is easier to quantify the former and define a dollar value. But it hurts the business, as any investment in the opposite of fast and bug free delivery would. Here we see not just role distinctions being created to serve a process, but work being created to serve the process. Work that bleeds investment from the product.
I’m on a bit of a roll so I should pause and say that I am not saying that there should not be QA Engineers or Architects. Those are skill-sets and they are vital. But roles (jobs) and skills (things we know how to do) are different. What I’m saying is that we seem to love defining roles even when what we really need is an investment in more skills.
Earlier I said that SAFe diagram reminded me of a conveyor belt. That shouldn’t be a surprise. The first software SDLC was Waterfall by Winston W. Royce and it was heavily influenced by manufacturing and construction industries. Every model we have evolved from this beginning is an attempt to capitalize on the rapid modernization of product development tooling. The disconnect between this legacy and the current reality is in the nature of the word “engineering”.
Traditional engineering fields such as civil, mechanical, or electrical require regimented, methodical, carefully planned blueprints, and often legally recognized professional qualifications beyond a graduate degree (PE). In these fields, the act of designing is painstaking and detailed because the act of implementation is expensive. If you slap together a blueprint for a bridge or a new microprocessor and teams of people or factories are requisitioned to implement your design, you can’t simply stop in the middle and say, “…hey I forgot something, let’s go back.”
These skills require a regimented and rigid waterfall like structure to ensure success and even safety. Conversely, Software Engineering is extremely forgiving to scope change (…not an invitation to or endorsement of scope creep). In the 1970’s, when waterfall was introduced, this was not the case. Software engineering projects were as difficult to rollback and alter as any other engineering project. Today, this is simply not the case. The myriad of tools now available to software developers allow them to prototype, develop and refactor functionality in hours. No other “engineering” discipline can make this claim.
Note: We are speaking about new development not legacy systems or tech-debt.
This reality has changed the way people practice software engineering. Developers today take a more relaxed approach to the creation processes, accepting the risk of mistakes in favor of rapid development and even accidental innovation because most mistakes can quickly be addressed. While software engineering still requires a clear understanding of systems architecture, data modeling, computer architecture, and mathematics, in my opinion, the act of writing code has become almost a creative activity as opposed to a factory oriented step by step process.
This brings us to the final observation:
Modern technology organization structures seem designed to cater to these roles rather than the products.
Having all of the roles we’ve discussed present in an organization forces us to consider ways to help those roles work together and keep them productive. That conversation I had with the technology leader about what it means to be an Architect vs. a Product Manager was in part fueled by a need to ensure there continues to be Product Managers (the role) and that my actions would not undermine that corporate construct.
We create logical groupings of people: Product, Engineering, PMO, Architecture, IT, etc. We silo them, give them independent career growth trajectories, and we protect their spheres of influence from excess overlap. In other words, we incentivize a conveyor belt mentality of people management, we ask them to focus on their section of the process only, and we ask that they not deviate into each other’s lanes too much. Once these physical structures are in place, we follow it up with a request that they then find ways to work with each other anyway… we call it a helix model. This incongruity of form and function is complicated and honestly I’ve never seen it be 100% effective.
Ok… this is getting long and I’ve dug a 2200 word hole.
Now What?
First a concession: I actually agree that the argument that 2–4 people move faster than an enterprise in and of itself does not provide a scalable solution to the problems I’ve described above. My purpose was to point this out simply as a beginning for a train of thought that could show the logical inconsistencies our normal processes tend to ignore.
Next, I’ll say that I do not have a single vetted solution. What I do have is a theory that I plan to flush out and implement at United Effects. Given the length of this article so far, I won’t be able to dig deep into that theory here. For now I’ll try to convey the basic idea and follow up with more detailed blog posts later.
I should also mention that concepts I want to try are not my inventions. They come from a mix discussions with peers, my own team, and readings on how some companies have experimented with non-traditional org structures. In fact, I’d like to make a shout out to my friends Chris Zangrilli and Alan Soucier. Independant conversations with them over the last year have allowed me to better verbalize these thoughts and ideas.
My general idea exists in three parts (for now):
- Early on with United Effects, instead of separating the organization into siloed organizations dedicated to the potential roles associated to software delivery, I will be creating organizational groups explicitly dedicated to the objectives of the business both in form and function. That is to say, instead of separate Architecture, Engineering, and Product and Sales organizations, I will create a Customer Enablement Organization, dedicated to the notion that everything they do is in service to the customer and product and designed to support revenue. I will extend this concept to other aspects of the business as well. Instead of separate sales and strategy groups, I will create a Revenue Enablement Organization. Instead of separate marketing, public relations, and brand we will have a Market Enablement Organization. The purpose of these names and these groups is to allow each to bring in whatever talent is necessary to meet the objective and mandate of the group in the fastest most obvious way possible without the politics of role divisions. This does not mean there will not be individual roles, but it does mean that the objectives and incentivization of each group will be clear at all times. It means unification.
- As we grow, we will work to create scalability by empowering groups to deliver fully realized solutions and revenue through single streams of accountability. Within Customer Enablement, Product lines will be individually accountable to provide Research, Design, Implementation, Marketing and initial Sales. We will want to replicate the success of small organizations by providing a mechanism to spawn and control them at scale. Like a business oriented fractal pattern.
- Growth tends to push people and organization toward specialization. This is normal and we will not fight the tendency; however, we will also endeavor to avoid distinctions in roles and responsibilities when there is no logical reason beyond “traditional thinking”. Being one team means giving everyone the opportunity to contribute across various skill-sets, being honest when training is required, and allowing your people to bring their full selves to the table without constraints. We need to stay as far away from hand-off culture as possible.
As I said, I will spend some time digging deeper into these thoughts in future articles. Does this spark any ideas for you? If so please comment or reach out as I’d love to hear from you. Either way, this is just a beginning and I will write about our successes and failures as we flush out and test these ideas.
I should also mention that a business/organizational approach to product development that focuses on value streams does not necessarily translate to a technology strategy that silos components by value stream. The technology strategy is separate and our CTO Chris Betz and I will discuss this in future writings.
If you are interested in how these ideas will come together toward a product, please go to unitedeffects.com and click ‘Join the Core’ and we will make sure you stay in the loop on all of our progress.
