Prioritization and Big Data? Think Human Nature
Not so long ago I’ve examined agile movement as a trend coming from software developers who felt that the challenges of their work are mishandled by the waterfall approaches of industrial production. Also, earlier on, in another layer of this thinking paradigm, I saw how Kanban as a method in agile software development stems from the human need to get rid of deadlines. Now, there’s another concept in project management that falls into the same pattern. I’m talking of what is known as “project portfolio management” in enterprise-scale organizations, and what we call “multi-project prioritization” in smaller companies. So, this time I want to look deeper into prioritization, and give one more example of how a promising new trend arises from nothing else but human nature.
On to project portfolio management. Just as waterfall was copy-pasted from industrial production and turned into Rational Unified Process for the sake of software development, in the same fashion, project portfolio management has its roots in the financial investment industry. As always, there’s a human need behind it. Someone must have been tired of looking for the ways to manage risks in software development projects and resorted to what seemed the closest available counterpart: financial investment. But if we look deeper, would there be any difference between financial investment and following through many projects to completion, at a software development organization? For sure, yes. First of all, projects are not shares. In investment management we want: a) to lower the risks and b) to optimize the financial gain as we compile our investment portfolio. Only that and nothing else. An average private investor is not strategically involved with a public company whose stock they’re buying.
It’s far more complicated when we deal with projects, and especially with software development projects. One key difference is that the projects are meant to be followed through to completion. Let’s take this statement as a premise. Of course, tech projects are being shuffled similarly to investment stock in giant corporations. I’m concerned with those tech companies, however, that have less leeway in taking up or dropping their projects, for one reason or another.
In the finance industry the only value indicator is ROI, while there are many more value indicators in software development. Usually, the question is not just whether the project should be picked up or given up. The questions are:
1. How are we doing in terms of budget? Do we need a safety margin to complete our projects?
2. Are we lax in terms of time? Do we need to skip on some parts of the project so as to ship a workable software in time?
3. How about people? Are they all balanced well throughout the projects? Have we made sure that the team’s collective energy is channeled into the right direction?
4. This one is the closest to where it comes with the similarities to stock portfolio management. Let’s say you work at a large organization and you oversee many projects or products. Or, at a smaller org, or at a division, one might have this multitude not with projects per se, but with what we refer to as “product or app features”. We have to prioritize and decide whether to skip this or that feature or to follow through.
On all of those 4 levels, it’s about prioritization. That’s what it’s all about. Prioritization is the toughest job of all in the world, be it in personal life, or at work. Sacrificing is the most daunting challenge, and it imposes a huge load on a person or a group of people who are supposed to decide and prioritize. By now, the buzz in the industry says that the concept of “project portfolio management” has something missing in it. The tools for multi-project prioritization are not universal, and they don’t do the magic instead of this tired human being. Either the tools have to be customized (at big costs), or they miss some instrument that is crucial for this particular organization. In a nutshell, the project portfolio management concept has outlived itself for effective prioritization, just as RUP had outlived itself previously, and was replaced by agile as a methodology in software development. But still, as a product owner, or a project manager you need to have a birds-eye view on all the risks. Still, you want to lay this burden down, and finally get a tool do the bulk of the prioritization for you, since this is the hardest job. What happens usually when some methodology is not working out as expected for those human beings? Right. They’re on the lookout for new, better — and what’s most important — easier ways to prioritize.
Voila: enter Big Data. There’s hardly an organization who does not do Big Data these days, as BI or in some other form. By now Big Data has cemented itself as an incumbent, because — supposedly — it provides a productivity breakthrough for prioritization and decision-making. If we draw a parallel with the previous occurrences of groundbreaking phenomena (like, the way agile appeared in software development, or how people resorted to Kanban within the agile paradigm, or how they looked to use investment portfolio methods for project management), prioritization seems to be the hardest job that tech professionals want to make easier for themselves, intrinsically.
Big Data is a trend that can be briefly described as follows: all the huge data about past performance and work is stored, and can be retrieved at a later time to see how past trends can recur in the present trends, thus helping decide and prioritize. Considering the meta-law of things developing in cycles, this might work, to some extent. There is a certain probability that past trends would help one prioritize efficiently in the present. There’s some financial software developed that calculates those trends. But, as far as I’ve been able to see, the big fish and the big gain in stock usually come at random. This all boils down to the intuitive feeling. Something outside data and calculations. This is not to diminish the importance of data analysis. Any data is a huge asset. Even more this is an asset as we live in the age of information, and we need to learn to get the best of those assets. But, just as the term “project portfolio” has the trace of hope instilled in copying this practice from financial investment to software development, in that it would set us free from prioritization problems, there’s something to watch out with the Big Data trend. Yes, we always strive to put burdens off of our shoulders, as Homo Sapiens, and that’s why we started with sticks as tools, then shovels, and on. Same with the heavy-duty prioritization and Big Data. To a certain extent, we can be sure that it would help us move forward, and give more sophisticated tooling for effective prioritization. But ultimately, there’s something even beyond Big Data. Some other info-tech miracle. All in all, not that we should lose hope in freeing ourselves from prioritization work altogether, but I don’t think that even Big Data would become the silver bullet for prioritizing choices. We will carry personal responsibility all the same. But the data-driven approach definitely counts as a step ahead in the evolution of methods and tools for effective prioritization across many projects and initiatives.
Related:
This story has been re-written from one of my earlier articles.
