Analyzing Analysts — From Lighthouses to Rudders

[I thank Krish and Donnie Berkholz for inspirations on Analysis-as-a-Service and Subbu Allamaraju for being an inspiration to writing this post. I derive inspirations from independent analysts/ influencers such as Ben Thompson, R Ray Wang, Phil Fersht, Simon Wardley, Krish, Ben Kepes, Janakiram MSV, Sam Charrington, & alex williams (TheNewStack) and I dedicate this post to them. I have a tremendous respect towards industry analysts — individuals such as Lydia Leong and the firms they are employed. I am sure they know more than I do. This post is no way an attempt to throw rocks from a glass house. ]

TL: DR — Software is eating the world; Business of Industry Analysts is no exception.

One would have read analyses from Michael Cote to James McGovern and might wonder why yet another analysis on analysts? This post summarizes my observations on current industry trends and my expectations of their impact on the business of industry analysts in a decade or so.

Industry Analysts now: Lighthouses behind Closed Data, Closed Insights

Who are industry analysts and what do they do?

Institute of Industry Analyst Relations (IIAR) defines ICT industry analyst to be

“a person, working individually or within a firm, whose business model incorporates creating and publishing research about, and advising on how, why and where ICT-related products and services can be procured, deployed and used”

Though this definition applies to individual analysts as well, this post refers industry analysts to those that work with reputed industry analyst firms. Industry analysts are independent, neutral players between sellers and buyers. They help both buyers and sellers, albeit in different ways. They are characterized by their ability to see farther on the horizon and an ability to forecast where the industry is heading. They also collect and publish data on usage patterns, spending patterns, market forecasts etc.

Industry analysts share their insights through published research reports, 1:1 calls, webinars, presentations and many other channels. Currently, most of these reports and access to time with analysts are behind subscription paywalls. It is to be noted that some firms like RedMonk are different in this aspect that they make all their research reports free. Some other firms make part of their research reports free. However, almost all data behind their research and access to the time with analysts and their insights are available only through subscriptions.

To summarize, by helping clients (buyer or seller) understand what is next, industry analysts act like lighthouses — they guide you to the right destination. It is to be noted that they are lighthouses and can only guide you, but not make you travel to the destination. You need to sail your boat.

Industry Trends

We are currently witnessing the next big industrial revolution which is evolving at an exponential phase. While various technological breakthroughs are, both cause and effect of this revolution, some of the interesting trends in the way industry is shaping in response to this revolution are summarized below.

Software is eating the world — As-a-Service Economy, Shared Economy, API Economy

It’s almost a cliché now to say that software is eating the world, we’ve seen lot more than that.

In the past decade or so, the way we consume software or procure computing/ storage resources has changed fundamentally. Even though the term ‘Software-as-a-Service’ (SaaS) was used as early as 2001, it is Salesforce and Amazon Web Services (AWS) that have been pioneers in pushing us towards consuming software and infrastructure as services. This shift to as-as-service delivery/ consumption has spawned an As-a-Service Economy where a wide variety of tools/ products/ platforms ranging from computing to productivity to business applications to cognitive computing are delivered and consumed as services.

As-a-Service economy has accelerated innovations dramatically, in turn spawning a new Sharing Economy which is built around sharing of human, physical and intellectual resources. As -a-Service economy has also changed the way businesses innovate and grow their internal cultures, so much that disruption through sharing economy has become a question of when, not if.

If one pays attention to the rise of As-a-Service and Shared economies, it is not difficult to see the role of APIs in both — without clear implementation of APIs, AWS wouldn’t have been as successful as it is now. We now have APIs for ordering pizzas, civic data, finding camp grounds, finding cloud jobs, finding your way around in a new place without knowing local languages or to get your latest dose of xkcd comic. Soon we will have APIs that will take you from point A to point B without any human intervention. It is still early days of API economy though and we have a long way to go.

An undercurrent to all these is that open source software is eating the world and developers are the new heroes.

Rise of Modern Enterprises

Technology driven companies like Netflix, Uber, AirBnB have either disrupted their incumbents or are in the process of doing so. Even behemoths in traditionally non-technical sector are becoming largely technology driven — Capital One, GE Capital, Bloomberg and more.

Technology driven companies are nimble and agile; they are strong proponents of open source and many also have open sourced their internal tools (such as Netflix). They prefer to release their products/ services often. They are known to select their technology stack based on their applications and business needs rather than external recommendations. They are not shy talking about them. They are not scared to adopt the latest in tech (be it cloud, containers, bots or anything) if they seem to serve their needs; they are also not scared to reverse their choices if needed. In short, technology driven companies are characterized up traits such as agility, scale, preference to open source, technology driven culture, data driven decisions and lack of fear to experiment with the latest. ‘Modern Enterprises’ is the term commonly used to refer to such technology driven companies.

We are already seeing the rise of such modern enterprises, large enough to replace what are enterprises now (‘traditional enterprises’). Technology is already becoming the largest sector in terms of market capitalization, with 5 out of Top 10 global companies in 2016 already being technology companies. Just like how enterprises of internet age replaced enterprises of earlier generations, modern enterprises will replace enterprises now. In about ten years from now, modern enterprises will be the top enterprises with ‘traditional’ enterprises been either replaced or re-born.

Rapid pace of technological innovations

Current industrial revolution is characterized by an exponential pace of evolution and ever increasing expansion of its scope. To provide you with a perspective, consider Moore’s law — for the past 5 decades, much of the third industrial revolution was guided by it. But we are already approaching its limits now. We now are in an era of exponential growth in everything — generated data, data storage capabilities, computing capabilities, number of connected IoT devices, number of articles we share on social media, battery capacity, market size of chatbots and more.

The rapid pace of innovations has also changed technology adoption/ hype cycles. What used to take decades is now taking years. What was trendy two years ago, is no longer trendy and what is trendy now will not be so in two years from now. A simple look at search trends on some popular technologies in the recent years would confirm this.

Search Trends — OpenStack, Kubernetes, Serverless, Docker
Search Trends — ML, DL, AI, Cognitive Computing, Neural Networks
Research Coverage — Time to Critical Mass: OpenStack
Research Coverage — Time to Critical Mass: Docker/ Containers
Research Coverage — Time to Critical Mass: Serverless

With rapidly changing technology innovations and shrinking adoption cycles, industry analysts are increasingly playing catch-up game rather than being ahead of the curve. For example, consider the number of research publications/ reports published by major analyst firms on some of the popular technologies/ innovations in recent years. For each of these popular technologies, there is a definitive catch-up phase before analyst coverage begins to attain critical mass. Though various factors such as client interest, maturity of technology etc. factor into selecting topics for research, it is clear that industry analysts need time to gain expertise in these topics as techniques that used to work earlier, such as matching patterns, extrapolating from past industry trends etc. fall short. With the rapid pace of innovations, it is inevitable that industry analysts find themselves trying to catch up, rather than being ahead of the curve.

Rise of Bottoms-up Decision Making

Another interesting trend we are observing now is the shift in decision making.

In traditional enterprises, decisions are made at the top levels of management (fondly called the ‘C-Suite’) and pushed top-down. Decisions are almost always driven by financial/ business needs. The C-Suite decides (or delegates) everything — from vision, mission, culture, hiring, expansion to day to day operations. Different parts of the organization are looked at as cost-centers and spending on these cost-centers are decided by the C-Suite as well. C-Suites made technology choices as well and controlled technology spend. Any desired change — be it cultural, procedural, behavioral, operational or anything trickled down from top. In short, decision making is a top-down process with C-Suite being the decision makers and change catalysts and business needs influenced culture and technology choices.

Culture & Technology driven by Business

On the other hand, technology driven companies are witnessing an increased trend of bottoms-up decision making. Their engineering groups are getting more empowered to make technology related decisions. Their technology decisions are also influencing their culture, enabling them to achieve agility and efficiency not possible before. This, in turn opens business opportunities not possible before. In short, there is a shift towards empowered, bottoms-up organizations from being hierarchical, top-down organizations. Experiments such as open workspaces, holacracy, unlimited vacation policies, flexible work spaces, etc. are all indicators to a new style of organizations, albeit with their own success/ failures.

Technology driven Culture & Business

Influencers of the future — Rudders using Open Data & Self-Service Insights

To summarize, we are witnessing the rise of modern enterprises which are technology driven with decisions driven bottoms-up internally, rather than driven from outside. Such organizations are more likely to make decisions based on data than just listening to expert recommendations.

Modern enterprises will make their decisions through Self-Service Insights using services and data available to them. Their influencers will not only be pointing them to destinations but also be leading them through transformations. They will be the rudders guiding their ships.

This won’t happen overnight but will take a decade or so through following not-necessarily sequential steps.

Rise of Independent Analysts

As of now, the influence and market share of the industry analyst firms is insurmountable. Even among the top analyst firms, size disparity among them is as wide as the difference among top cloud service providers. Though the industry is predominately influenced by industry analyst firms, the rise of independent analysts is already happening. They are gaining more influence than industry analysts among niche areas, for example, RedMonk among developers, Heavy Reading among Telcos, and HFS among BPOs. Analysts from independent analyst firms such as HFS, Constellation Research are also increasingly voted on par or higher than their peers working with industry analysts firms.

Independent analysts are also the ones that respond quickly to rapidly changing technology innovations and newer business models. This trend is expected to continue, with more independent analysts gaining influence in such niche areas. Over course of time, such niche areas will expand large enough to be recognized as mainstream. Consider the ‘tongue-in-cheek’ Tragic Quadrant published by IIAR based on a small data set ranking industry analysts on their impact and relevance. I expect to see independent analysts moving upwards on their impact in upcoming surveys.

IIAR Tragic Quadrant 2015 with annotations

Rise of Cognitive Computing-as-a-Service

We are seeing a rapid growth in the availability of cognitive capabilities to end users as a service. What has been accessible to high end research organizations is becoming available to all. In the past two years, all major cloud service providers have started offering cognitive capabilities as services. Virtual Assistants such as Alexa, Cortana, Siri, x.ai have leveraged advances in cognitive computing to offer human-like-services. Based on developments fueling the growth in machine learning, deep learning, computer vision, NLP, visual recognition and the general push towards API economy, I postulate that:

Cognitive computing capabilities available as a service will double approximately every year

Open Data, Open Insights

As we saw earlier, data and insights that influence technology choices and purchasing decisions now are largely behind paywalls/ subscriptions — close data and closed insights. We are witnessing a reversal in this trend.

Influencers and decision makers are increasingly considering data and signals available out in open. As Big Data explosion continues, new and unexpected sources of data are being looked at to make decisions such as product feature prioritization, technical choices, purchasing decisions, market identifications, fashion trends, disease outbreaks, calamity preparedness, etc. Disparate sources such as Stackoverflow, Hacker News, popular technical blogs, LinkedIn are becoming new troves. Signals such as the tweets from the president impacting stock market are being considered in financial models.

We already observed the tendency of technology driven companies sharing out their tools, learning, and findings openly. More independent analysts are also making their research available without paywall. In short, more and more insights are becoming available outside subscription paywalls.

Such trends are expected to continue to reach a state where open data and open insights would be significant sources of influence of change.

Analysis-as-a-Service & Self-Service Insights

How far are we from availability of Analysis-as-a-Service capabilities through which one can derive insights on their own using data sources they prefer through APIs/ services?

With Analytics platforms such as Microsoft PowerBI, Tableau, Praescient end users can now play with data and derive some insights on their own. Some analyst firms such as ESG are also exposing some self-service capabilities to their clients through their interactive portal. However, it should be noted that such capabilities are focused more around data visualizations than insights. We are also witnessing more NLP capabilities becoming available to end users through bots such as Summarize and translation APIs. Considering these trends and the expected growth in the availability of cognitive computing services, we are not far from the availability of true Analysis-as-a-Service capabilities. Self-Service insights then become a reality.

Equipped with such self-service insights and data sources available in open, decision makers and influencers of future would be driving organizations through desired transformations rather than being guided. They will be the rudders.

Disruption?

Paraphrasing Bill Gates we always tend to overestimate changes in the next two years but underestimate the disruption that will occur in next ten. The business of industry analysts as we know is going to get disrupted. It doesn’t necessarily mean that the giants will go out of business. If a simpleton like me can think about this, they must have already started working on it.

The disruption could happen in one or more of following ways

Adapting to different research methodologies

Industry analysts must adopt different research methodologies — hands on testing, test labs, looking at unconventional data sources, querying open data sources or more. And they will — though this might face some initial resistance, but it will be a necessity. They will also start picking up new areas of coverage rapidly. For example, 451 Research was among the first to cover DevOps and soon larger firms followed with all large industry firms covering DevOps now. I also expect acquisitions of independent analyst firms in the lines of tech-acquisitions to acquire research methodologies, mindshare, and talent.

Adapting to different business models

Access to industry analyst firms are largely through bundled subscriptions to agreed number of access accounts, commonly referred to as ‘seats’. While they vary in the type of bundling, they typically include access to premium content, access to time with analysts, training, webinar, consulting services or more. Subscription rates would vary on the type of bundling and the number of seats. Some firms like RedMonk are already following an open access model where their research content is available to all, but additional services like access to analysts’ time or expertise are billed. Some industry analyst firms also offer pay-per-research models where research reports are available a la carte. Eventually, I expect the industry analyst firms to adopt Pay-as-You-Go model, going with the industry trends. Industry analyst firms will also start offering as-a-service capabilities at varying degrees.

Adapting to different methods of influence

Independent analysts use different venues to influence — online presence, blogging, social media, webinars and more. While some popular industry analysts use such mediums, most of them don’t, primarily due to their content being available only behind paywalls. As more industry analyst firms start providing open access to their content, industry analysts will also start using different methods of influence.

No more Corporate Strategy groups

Groups that drive corporate strategy — identify market opportunities, evaluate competition, set product road maps, or anything that corporate America loves to call as ‘strategy’ are going away. Period. Such decisions will be taken by product groups themselves using data and services available to them. Though it is not the industry analysts that are getting disrupted here, this will be one of the first impacts of the impending disruption to the business of industry analysts.

Summary

Software is eating the world; Business of Industry Analysts is no exception.

It is entirely possible that I will be proven wrong or disruption happens in ways that I didn’t foresee. Nevertheless, it is going to be an interesting decade. Buckle up!