Keeping the Human in the Loop
People, process, technology: how to harness — and not get burned by — the IT fire triangle
This is not another article on AI, or LLMs, or whatever other new technical development can be described with yet another TLA (two-letter acronym or three-letter acronym).
As impressive technologies become more widely available, they bring about equally important side effects and concerns. One such area of concern is the end of humanity, which is what I’ll be addressing in today’s article.
Just kidding.
This article was developed through experience and observation. Through various client engagements, discussions with colleagues, and third-party accounts, it has become ever more clear that in order for technologists to level up, we now need to be more humanist. As the market landscape for technology implementations is dynamic as ever, there are some key trend-proof patterns related to areas such as multidisciplinary collaboration, executive sponsorship, and approaching situations with empathy to leverage for ensuring success.
We Are Constantly Evolving
Around five years ago, almost (if not every) project involved a “lift and shift” migration from legacy to modern systems. Typically, this meant moving from on-premises data warehouses into cloud data lakes and warehousing services. The ask was for “like for like.”
Unfortunately, oftentimes when moving between distinct and contrasting technologies, “lift and shift” becomes “lift and sift.” Meaning, the migration objects can be picked up and moved but there won’t be a complete enablement or transition for the new system because it was not built to be the same as the old system.
Fortunately, many organizations now recognize the importance of re-platforming as they transition to new environments. This mindset shift increases the potential of the whole system and gives more room for learning and adaptation instead of being stuck in the old ways working.
A key part of this maturity is recognition of the importance of the whole system. This includes not only technology, but also people and process. People, process, technology is the fire triangle of information technology. All three are key ingredients, and if one isn’t present, the whole system stays cold and dark.
The Mosaic of Roles in Successful Modernizations
At the same time as technology is enabling a ground-up evolution toward a holistic system view for people working in this space as part of their day-to-day, the leaders of today and tomorrow are also making the shift to a more integrated mindset. The technology deployed in most enterprises today requires less intervention than ever before. It is also typically designed with lessons learned top of mind, so that even if the technology itself is still high maintenance, the way in which it is implemented brings efficiency and low overhead for the long term.
Data pipeline modernization efforts are a good example. Many of the legacy systems which organizations are looking to re-platform today have turned into shadow IT environments with development constrained by technology limitations. These features result in the explosion of the legacy data system to go from a well-oiled, enterprise-serving machine to a system built for ad hoc work mixed in with some business and operational artifacts along with plenty of waste. This situation is typically the result of years (if not decades) of poor communication between teams, frustration with slow and burdensome processes, and a lack of centralized oversight.
When the modernization effort comes around, the first step is a requirements gathering process; this often turns into a prioritization exercise fairly quickly. Various tools may be deployed to determine code complexity and usage metrics on the artifacts in question in order to accelerate the prioritization exercise. For entities looking to “lift and shift,” these tools may be sufficient to guide the endeavor. However, the greatest success stories typically involve conversation and collaborative design that brings together executive leadership, IT teams, and the business to perform a root cause analysis, capture patterns, and develop an intelligent and enhanced solution for the new system.
Collaboration is also important for separation of ownership. It is common for data analytics teams to want to own everything in the new solution; this desire is typically due to a combination of eagerness to get started with new and scalable technology as well as negative prior experience with long wait times for requests for updates and support. The problem is that, in the vast majority of cases, these teams are not equipped with the skills to support the solution. While infrastructure, networking, security, and operations (etc.) teams may need to upskill in order adjust to a new platform with fewer siloes than the legacy platform, everyone is still better off if these folks are involved in the new system, as opposed to the data users attempting to fill these roles. An agreement needs to be made in terms of battery limits and rules of engagement early on in the modernization effort to set a solid foundation and guide the planning as well as design.
Technical architecture involves functional and non-functional components and teams.
Coming back to the data modernization example, the data (functional) teams define the way the data moves through the system, while the platform teams implement and manage how the data moves through the system. It is important that data teams identify their enterprise production-level pipelines. This step will then help data teams design for storage (data at-rest) and transformations (data in-transit). These designs and requirements will inform the platform (non-functional) team’s design of the underlying infrastructure and orchestration. Another key part of this process is enterprise technology governance to contribute to both functional and non-functional teams’ design in areas such as service level agreements (SLAs) for timeliness of data, privacy requirements, and security standards.
Once the initial set of production-level business and operational pipelines are accounted for, the process for data teams to do their day-to-day as well as ad hoc analytics work needs to be established. A key piece of this process is the promotion of these workloads into the enterprise production-level pipelines discussed above. This brings together data teams to describe how this work is typically performed (languages, tools, platforms, etc.); platform teams to design and integrate continuous integration/continuous development pipelines, as well as the environment setup and linkages; and governance teams to define the standards which must be met for work to go from workbench to production.
Executive leadership cannot be absent from this work. Their presence is critical to the success of the solution in several regards. These are the people who typically have the North Star vision for this new system as an entity unto itself, but also in terms of how it aligns with and enables bigger, broader, and bolder visions for the organization as a whole. Less glamorous but undeniably as important is timely decision resolution, which this executive presence can provide and enable. As previously mentioned, ownership can be a contentious issue; executive leadership often needs to partake in these discussions to deliver a clear and consistent message around how the new way of working will look.
It is common for leadership as well as delivery teams to request sanity checks on the decisions and progress being made on these transformational projects. Having the guidance of someone with a breadth of experience across industries, organizations, and related projects can help navigate through these complicated scenarios to ensure the target destination is reached accurately and on time. Seeing around the corner can be tricky, and surprises (almost inevitably) happen; experience can aid in containing issues (proactively or reactively) as well as agile course-correction.
Now, for the Hard Part
Revisiting the IT fire triangle, it’s clear that in the vast majority of cases, technology is the easy part. When evaluating technology challenges, it’s especially important to maintain a “we can solve this” mindset. While people and process issues are also surmountable, they are typically more nuanced and thus more complex to approach. Furthermore, technology can (and should) typically be solved from the ground up; people and process issues often require, at least partially, a top-down approach.
Although technology leaders may be in their role due to their demonstrated technical expertise, as the saying goes, “What got you here won’t get you there.”
Evidence of this trend can be traced to how up-and-coming leaders are choosing to prepare for these roles. The Economist published an article earlier this year which called out the most popular courses among MBA students at Stanford University’s Graduate School of Business (GSB), an institution with a proven track record of developing some of the world’s most influential leaders.
In the article, the usual compulsory courses found in (credible) MBA programs are outlined: financial statement analysis, computer modelling, accounting, and so on. However, the three most popular courses are almost completely not-quantitative and focus on “hardheadedness, introspection, and diplomacy.” While performing a major enterprise-level transformation requires these traits to be present in leaders across all levels, many individuals may find they lack the experience (and thus the confidence) with programs of this magnitude. This is an area where experience-based guidance is also especially invaluable.
Around the time ChatGPT first captured the attention of the general public, a coworker and I were partaking in the unoriginal discussion around the new potential implications for job security. My colleague referenced something they had read online which made me chuckle at the time, but I find myself coming back to repeatedly. The gist of this was that only roles to which the word “therapist” can be reasonably appended to one’s title have hope for longevity.
Upon further inspection, the role — or at least the skill set — of technical therapist is increasingly critical for success. Before entering even the early design stages of a project, the true state of the situation needs to be understood and root cause analysis of any problems needs to be performed before solutioning can begin.
According to the Indeed UK Career Guide, the major skills possessed by therapists include: listening, empathy, problem solving, and communication. Under the detailed descriptions of each of these, replace the words “emotion” and “psychological” with “process and technology” and switch the context from personal and individual to teams and corporate entities, and the result is a key role for finding success in technical transformation efforts.
Delivering Real Value in the Real World
An increase in companies’ digital maturity is positively correlated to the importance of the relationship between IT and the business. In some organizations, these groups may even be each others’ perceived adversaries.
A common occurrence in design and requirements-gathering workshops is a flare-up in the tension between groups who previously viewed themselves as separate and distinct, but who now need to be able to collaborate on a regular basis. It quickly becomes evident that, while there is a technical challenge to be overcome (typically via some form of re-platforming), the real pain points are the other two vertices of the IT fire triangle: people and process.
For some situations, having an experienced neutral third party can also help settle disputes that internal leadership may be too close to. I was recently speaking with a coworker who told me about a time they were asked to write the “separation of duty” for one of their clients. Due to my colleague’s unique position of understanding this client’s ins-and-outs, plethora of related experience in organizations which had undergone similar changes, and general neutrality among the client’s various teams, they were able to accomplish this work that may have been controversial for someone on the inside.
While better technology should improve the intra-team experience, chances are that inter-team experience improvement needs to come from elsewhere. Guardrails and clearly defined (as well as communicated) procedures need to be put in place around the shared new platform to enable the organization as a whole to succeed. Having an experienced, neutral voice in the room can help facilitate these discussions in a way that drives toward the best solution. More often than not, this mediation is performed in an inquiry-based manner, thus reaffirming the notion that in today’s day and age, asking the right question is more important than knowing the right answer.
In a conversation focused on how to deliver real value, another one of my colleagues was emphasizing how we (the data engineering team) need not, and arguably should not, focus on just delivering data-related work. They shared a story with me about an interaction they had with a new client.
The client was frustrated with their current system for data analytics and reporting; the experience was burdensome and took too long. Instead of pigeonholing their focus on the data aspect of what the client was demonstrating, my colleague sat with the client and developed a deeper understanding of the situation. It turned out that the tool was not the crux of the problem, and overall inefficiency was the main culprit. My colleague described how they sat with the frustrated individual and helped identify some readily available efficiency gains in the process without any semblance of a technical overhaul. The result was a system improvement that addressed the root cause and a happy client; that is real value.
Many data teams express frustration with a lack of available resources, be it compute, data, or platform expertise (to name a few). Insufficient resources result in requests needing to be placed for, you guessed it, more resources. Oftentimes the request process itself is onerous; even if fulfillment delivers what was asked for, like entropy, frustration still only increases.
When speaking with one client, they were explaining how they were under a lot of pressure to deliver work as per the organization’s SLAs. Unfortunately, they often had to reach out to several other teams to accomplish this work; this didn’t even get into their experience with trying to roll out new initiatives. That process was even slower and often discouraged their team members from pursuing this type of work (and let’s face it, they were so engaged in getting the day-to-day done that there was barely any time for this anyway).
Despite their obvious and justified frustration, this team really just wanted to reach their full potential. The stories they were sharing illustrated how they were unable to do so with their current system. This was one of the initial conversations in the requirements-gathering process for technical design, and we did establish a key finding to inform the solution, but the finding itself was non-technical. “What I’m hearing in all this is that your team wants to be empowered to deliver impactful work through increased autonomy and increased speed; do I have this right?”
Some of the most important concepts in life were learned in kindergarten. Active listening is one of the, if not the singular, best ways to build trust and solid partnerships. The client was thrilled that what they had thought was rambling or venting was actually being carefully considered. In fact, they even sounded a little bit surprised by this takeaway and its accuracy in describing what they were experiencing, articulated in a way that they had not yet found. These became key objectives in the design of the new system.
Actual Intelligence
Any science fiction fan worth their salt knows that 42 is the answer. This is everything, the meaning of life.
But what is the question?
As comically demonstrated in Douglas Adams’ trilogy in five parts, The Hitchhiker’s Guide to the Galaxy, answers without knowing the right questions are essentially elevated trivia.
At a time when essentially all of the information on record in the world is readily available, memorized knowledge is helpful but often not necessary. After all, the human brain as a storage and query system is no match for a strong internet connection and an optimized search engine. Experience is a uniquely human trait. A well-trained model may be a good predictor of events, but human behavior is often unpredictable, illogical, and based in emotion.
While knowing the answer is useful, knowing the right questions to ask is a demonstration of actual intelligence. Warning — circular reference: this requires experience. The ability to empathize and also provide guidance based on real-world scenarios is (currently) beyond the capabilities of any technical tool.
This is not to say that people and process rule all; the IT fire triangle is equilateral, with all three components being critical. The point here is that although there is plenty of justifiable hype around new technology, this alone will not enable success.
It is not solely the use of exciting technology such as AI or LLMs that makes it so powerful. The real power comes from the vision of what it can enable: the ways in which this technology can empower people to become the 10x versions of themselves, and the ways in which previously time-consuming and burdensome processes can be streamlined and automated. In order to achieve the full potential of powerful new technology, it has never been more clear that real value comes when we keep the human in the loop.
