The Competing Dualities of Business Analytics
How to Find the Golden Mean
As with many sophisticated fundamental principles, they usually resolve to some underlying duality. No different than the North Pole versus the South Pole. Hot and cold. Sunshine versus rain. All of these present a dualism of nature that can be problematic when taken to an extreme. Per the old Arab proverb, “Sunshine all the time makes a desert.” However, when dualities are combined and harmonized, they yield a positive benefit.
This idea of the metaphysical duality of opposing polarities is represented across several cultures — to name a few, the Yin and Yang from Eastern philosophy, the Aristotelian golden mean, and the Hegelian trinity from the Western Canon. These ideas state that balance is found within a set of complementary yet conflicting tendencies. How does dualism apply to modern computing and analytics?
The Three Dualities of Analytics
Business computing today can be broadly categorized into two main spaces — systems that operationalize and systems that analyze. Systems that operationalize are classic ERP, tracking, management, and collection tools that run the business. Generally, they leverage traditional relational data stores and system architecture. Systems that analyze could be BI or EPM programs, which optimize reporting, illustrate trends, make future predictions, and even integrate machine learning and artificial intelligence. The operationalize versus analyze paradigm constitutes one of the primary dichotomies of modern-day information systems relative to design, implementation, intent, and audience. Data Mesh founder Zhamak Dehghani has referred to this dichotomy as the Great Divide. The focus of today is on the latter of the two paradigms, those systems that analyze.
When it comes to those systems that analyze, three dualities manifest themselves as tension springs of success when implementing business analytics into an organization. The correct tautness between them helps drive achievement in the domains of cost, time, and quality. Too little tension, and the project will fail. Too much tension and the spring will snap and whip you in the face. These qualities compete against success, and finding the golden mean will help give you a springboard for achievement. Before discussing the golden mean, we must first define the dualities and the strains they impose.
There are three universal tension springs when implementing an analytics system that need to be contended with:
- Duality #1 — Business Owned versus IT Guided
- Duality #2 — Institutional Knowledge versus Data Access
- Duality #3 — Spreadsheet Proliferation versus Spreadsheet Elimination
The following sections will compare and contrast these tension spring dualities and their competing nature, and seek to find harmony between them to ultimately resolve back to the realm of cost, time, and quality.
Duality #1 — Business Owned versus IT-Guided
Starting with the explosion of business systems in the 1990s, IT technical-led projects were essential. Teams of engineers, experts in hardware and software, were required to deliver organizational and technological architecture. The skills to implement these systems were too complex and inaccessible for the businessperson. While the businessperson was involved in the requirements and testing phases, this dependency on IT could not be avoided.
The method described here is considered one of the main polarities of analytics Duality #1, the IT-guided approach. The IT-guided approach, while a necessity, was expensive, inefficient, slow-moving, and cumbersome. Whole analytical projects were required to develop reports, create layouts, and change formats. What seemingly should have been simple requests from the business, ended up requiring technical teams, project cycles, additional labor costs, and bureaucracy.
As technology evolved, first moving away from the command line and into GUI, point and click-based screens, and now advancing beyond on-prem to cloud-based machinery, the involvement paradigm has been shifting, emphasizing business ownership of systems. This involvement has created a pendulum swing to business ownership primacy. At first, this approach seemed to be the panacea to the sluggish method of having to run to IT for every little report color, format, calculation change, or data wrangle, which is constant and never-ending. At long last, the businessperson is finally empowered to manipulate their data and presentation and control their destiny. The method described here is the other side of the duality, the business-owned approach.
However, as it turns out, in many cases, the expectation for business ownership primacy over systems has proved to be fragile, chaotic, and too limited in its scalability. In attempts to make deadlines, focused on pure business results, businesspeople tend to shortchange mechanical considerations, cut engineering corners, and bypass technical checks and balances. There is a skill mismatch between what the businessperson is good at and what is required to fully maximize a system’s capability. Rogue users have been known to take down systems on Friday night at 8 pm, desperately calling technical support for help on an orphaned tool from the classic IT workstream. The support person is left haplessly flailing, trying to find solutions on a system they are not adequately trained on or know little about. Not good.
To implement an analytics system, IT organizations are a vital part of this process, setting up the connectivity, implementing the software, and providing project management, quality assurance, and technical protocol to make the solution possible at scale. To entirely abandon IT yields a high degree of chaos. However, the inverse is also true. To have IT over-involvement where they are entangled in all stages of the analytical process, providing guidance at every turn, is paralytical, not analytical. This IT entanglement is how we end up with hyper-order, bureaucracy, and centralized monolithic architecture.
Each of the polarities in Duality #1 competes with themselves but is also necessary for a successful analytic experience. How can we marry the two extreme sides of the duality, both fragile, one too rigid and the other too chaotic?
Duality #2 — Institutional Knowledge versus Data Access
Institutional knowledge tends to lie deep in the heart of organizations, especially within the minds of long-term experienced employees. When it comes to analytics, seasoned, competent personnel have a vision of what they want, the important KPIs, the value drivers of the business, and intimate details of how the processes work. Sometimes these individuals seek to actualize their vision and expand it to the masses to help the organization grow and flourish.
However, institutional knowledge can occasionally be recruited for darker purposes — job security and organizational power. Thus, how institutional knowledge is wielded is dependent on intent. This battle between power and flourishing makes an organization’s pure reliance on institutional knowledge unstable. Even without dark intent, supposing an employee gets sick or quits. How does leadership scale a company relying on pure institutional knowledge? This approach presents fragility.
On the flip side, we have technical professionals collecting massive amounts of data within organizations to allow employees better access to institutional information. As it pertains to analytics, this drive to increase data access may present itself in the form of data warehouses, data marts, data lakes, or even evolving to a data mesh. Massive amounts of investment in time and capital are poured into on-prem and cloud infrastructure to realize these data access efforts. This investment involves aspects of data extraction, integration, scrubbing, transformation, hierarchization, security, governance, and quality checks. ETL pipelines are a constant feature of the data access world.
Sadly, despite this investment, data access, in isolation, while a fundamental function of IT, tends to miss the mark on delivering value to its consumers. Why? There are a few reasons for this. First and foremost, note the use of the word “data.” All of the examples aforementioned on data access and integration methods are focused on data. While data is a knowledge prerequisite, it alone cannot evolve into institutional knowledge. Something else is needed. What connects the domain of data access with knowledge?
Duality #3 — Spreadsheet Proliferation versus Spreadsheet Elimination
Ah, spreadsheets. The big green elephant in the room cannot be ignored when discussing analytical systems — Excel. Spreadsheets are a constant feature of the analytics culture at corporations, and large and small companies have a love-hate relationship with them. On the one hand, they are a necessity. On the other hand, they can be a scourge.
Many corporate cultures deal with the Excel Hell problem. While the tool Excel is an easy target due to its ubiquity, this is not a challenge limited to just Excel. More broadly speaking, it applies to spreadsheets in general, and it is more the case in how the sheet is used rather than the software maker. When we look at the spreadsheet proliferation versus spreadsheet elimination duality, Excel Hell is the far end of the spreadsheet proliferation pole. A myriad of problems can cause Excel Hell, including, but not limited to, data gaps, trust, and quality, too many disparate systems, and a lack of skills.
A common reaction to spreadsheet proliferation when it takes root in a company is a knee-jerk pendulum swing in the other direction in the form of a seismic clampdown on spreadsheet usage. Implementation of tools that boast false promises of spreadsheet replacement run amuck. This reaction takes us to the other side of the duality, spreadsheet elimination. Spreadsheet elimination, unfortunately, is not a realistic nor obtainable endeavor.
While in the case of Excel Hell, a meaningful spreadsheet reduction strategy is healthy, spreadsheets are essential to organizational success. It is a dystopian vision to think they can be eliminated from many key business processes and usually ends in disappointment, despair, and departmental siloing. How do we apply meaningful spreadsheet usage within an organization without getting too extreme to spreadsheet elimination dystopia or the chaos of spreadsheet proliferation and ensuing disarray?
The Golden Mean of Business Analytics
All three dualities presented thus far share a standard of theme. First, they are all critical to achieving business process gains relative to cost, quality, and time. Second, they are all at odds with each other, and when taken to the extreme, either side creates organizational dysfunction.
In the realm of the business-owned versus IT-guided, how can the two sides of the extreme be married without jeopardizing system scalability? In the domain of institutional knowledge versus data access, what connects these two polarities to provide meaning? In the world of spreadsheets, how can an effective spreadsheet usage strategy be implemented within an organization without creating dystopia or chaos?
The answer to these questions lies within the golden mean. The mean between the extremes is where salvation is located.
The three golden means an organization should strive for are summarized as follows.
- Self Service Culture
- Meaningful Information
- Spreadsheet Efficiency
The following diagram is a mixed martial art of eastern and western culture and illustrates the golden mean of business analytics between each fundamental duality:
In the following sections, I will walk through each golden mean. Some may seem obvious, some more abstract. Either way, the key to understanding lies within the extremes. It is on the edges where we can stress our ideas and put our preconceived notions to the test, thereby expanding our understanding.
Per the great military strategist John Boyd, “If you want to understand something, take it to the extremes and examine its opposites.”
Golden Mean #1 — Self Service Culture
Self-service culture is a people-oriented approach that harmonizes IT and the business. On one hand, it enables the business community to interact with data on demand without the guidance of an engineer or IT professional. On the other hand, it empowers the IT professional to provide meaningful information without shackling the business.
The robustness of a company’s self-service culture can be tested with a simple question: can business users conduct trusted on-demand what-if analysis without going to IT or falling into the trap of Excel Hell? If the answer is ‘no,’ an organization has not achieved a robust self-service culture.
In its simplest form, this could be the ability for a business user to slice, dice, and build robust reports dynamically. In its intermediate form, this could be future state modeling. In its most advanced forms, this could be predictive and AI features.
A key term must be mentioned here: write-back. Suppose the users constantly interact with information through a read-only pain of glass, such as a report, and cannot write changes back to the system. In that case, flexible self-service will never be achieved. Self-service must be a two-way interaction with data. Send and receive.
Self-service culture does not mean IT abandonment. It also does not mean IT-guided bureaucracy, governmentality, and control. IT instead advocates technologies that enable, promote, and reduce the monolithic centralized project structured approach for analytics and places decentralized business ownership on the data: master, fact, and presentation.
Golden Mean #2 — Meaningful Information
Information is what connects knowledge to data. Thus, the sole aim of data access is a misguided goal. Is data access a requirement? Yes, of course. It is a prerequisite. Exposing business-transformed data in a secure way to end-users is foundational. However, data access as an end state is what must change.
The development of meaningful information focuses on the overarching ability of the business to synthesize data into intelligence to help promote institutional knowledge development. The popular notion of data warehouses, data lakes, data marts, and data mesh provides ubiquitous access to a myriad of different types of data throughout an organization. While each integration method differs in its approach and structure, pro versus con, the focus is always the same — data. How can we change the conversation away from “data” to create meaningful information that can lead to insight? How do we not just focus on data visibility and granularity but instead its meaning?
A 2019 Luciano Gallón study on Systemic Thinking beautifully illustrates how data progresses to information and then to knowledge. The following diagram presents this illustration:
As the Luciano Gallón study shows, the vital component for developing information is creating relationships based on patterns that balance the mundane versus the complicated. This idea spans people, processes, technology, and even leadership. Why are these qualities rarely discussed in most IT or analytics transformation processes?
If our analytical objectives in these areas are too complicated, over-ambitious, or unrealistic, the data-information arrow will not progress toward knowledge. If our analytical goals are too mundane, short-sighted, and myopic, the data-information arrow will also not move toward knowledge.
In short, the solution rests in the hands of leadership. Quality leadership and how we marshal resources, people, processes, and technology, across the Mundane-Complexity borders, in critical areas such as master data management, data federation, business process, and distributed architecture are critical to driving meaningful information throughout an organization.
To yield the insight, knowledge, and ultimately the wisdom organizations need, understanding, acknowledging, and measuring the data-information-knowledge arrow and its characteristics can help drive more effective business value.
Golden Mean #3 — Spreadsheet Efficiency
Spreadsheet efficiency does not mean eliminating spreadsheets. It also does not mean succumbing to and being victimized by spreadsheet proliferation. In the case of an Excel Hell situation, spreadsheet efficiency can have an element of a spreadsheet reduction plan. However, spreadsheet efficiency is beyond just spreadsheet reduction. In some cases, spreadsheet efficiency means expanding and embracing spreadsheets. The key is to identify which use cases are toxic and which are healthy and put them to work correctly within the organization.
The article, Understanding the 7 Types of Spreadsheets, The First Step of Managing Excel Hell, discusses the toxic use cases versus healthy and gives you a road map to start diagnosing the problem.
In Closing
Each one of the areas of focus in this article presents a duality. There is a constant flux and tension that connects one side of the duality and the other. No different than day leads to night, then night leads to day. There is practical harmony between the tensions, and absolutism on either side of the duality leads to organizational toxicity, dysfunction, and fragility. It’s the integrated middle where success is found.
On the surface, each one of the dualities seems an independent and autonomous area of focus. However, we have learned in the field through hundreds of implementations that each of these critical tensions is conjunct with one another. Thus, you change one; you affect the others. This interdependent relationship between dualities presents a triple constraint model — a Golden Triangle. Thus, we refer to this triple constraint relationship The Golden Triangle of Analytics.
For now, adieu.
About the Author
Scott Gehring has over 30 years of experience in global enterprise information systems and holds several patents for his work in varying industries. As a pioneer in the field of analytics, he has been an influential industry leader in defining best practices around system design, implementation, integration, and operations. Scott has built hundreds of solutions for companies ranging from small-mid-business to large-scale enterprise organizations, helping to drive process improvement, tighten the link between business and IT, and provide the latest innovations in information technology.
www.scott-gehring.com
www.linkedin.com/in/scott-gehring/
Scott Gehring — Medium
Technology Whiteboard
