Thinking of adopting an analytics platform? Ask your local analyst!
Analytics tools exist in abundance. They often look great and come in slick packaging. But the best analyses are produced by experienced analysts. Buying software without consulting your analytics expertise can be a waste of money at best, and lead to costly errors at worst.
It sounds really attractive…
“We helped them build a major machine learning algorithm with this — three hundred thousand features, over billions of records over transactional data! That’s what we’re capable of!” This was the ultimate up-sell. My organization at the time was evaluating the warehousing capabilities of a large enterprise software vendor. As one of the few data analysts on staff, I had been asked to provide my perspective on where analytics would fit into this process. This was something new — someone asking me to think about where my technical work fit into the broader strategic goals of the company. And the vendor we were evaluating had no shortage of “push-button” systems to make analytics faster, better, and easier (or so they claimed). It wasn’t until a few years after this experience that, aside from being a good learning experience for me as an analyst, the company also made a good decision by viewing analytics as a technical process requiring technical input.
Drowning in information…
There is now a near deluge of analytics tools now available to organizations, serving a wide variety of needs — web analytics, IT analytics, advanced analytics/machine learning/predictive modeling/artificial intelligence, CRM, Social Media analytics, etc. Gartner’s (in)famous Magic Quadrant evaluates major private (i.e. not free) vendors in both advanced analytics and business intelligence & reporting. I have used some of these tools in my own work. Then there’s open-source: R vs. Python; Pig/Hive/Spark; Apache, and the proprietary vendors that support these open-source platforms… it’s a long and convoluted list.
Some of these platforms are IT & engineering focused, others focused on BI & reporting, & still others on advanced analytics. Each is attempting to fill a niche, and many are trying to up-sell wherever possible. Going back to my warehousing anecdote (I’ll get to the end of that conversation later), we came looking for better data processing and left with materials for a bunch of “point & click” analytics tools.
Clearly, these companies and projects are trying to fill a need — businesses are continuing to accumulate data; yet many leaders feel that they don’t have sufficient insight into those data, or that they aren’t fully leveraging the full offerings in their data sources. But the real question is: will this tool actually drive value for your organization? A good sales pitch will give use cases and examples what other similar organizations are doing with the vendor’s product (I’ve seen sales pitches that didn’t show that — they felt very academic). So you can see that the analytics tool or buy-up might be useful, but it still doesn’t actually answer the question of, will it be useful to your organization? And this is because every organization is different: analytics teams are organized differently, you might have different tools already in use, or your business is just slightly different from where the platform is designed to provide value.
Seek & Ye Shall Find…
I feel that there are two primary concerns for whether or not to adopt a platform: technical concerns and process concerns. How (and how well) does the software work, and where does it fit in the analytics process?
From a technical perspective, Rick Sherman highlights a few features that BI tools should have. Advanced analytics tools have different requirements (e.g. what kinds of algorithms are supported, can you pull apart model objects for model diagnostics), although there is some overlap with BI tools. But another important important question — which I feel is generally less discussed — is: does this platform make sense in your analytics process?
And in both of these respects, I believe that there’s no better way for decision-makers to get an understanding of whether an analytics tool will be useful, than to just ask their analysts. When I say, go ask an analyst, I don’t mean just pitch something to them at their desk, while they’re grinding away through code or drawing randomly on paper trying to figure out some business problem. Those are definitely not good times to ask an analyst to think about anything other than data. What I’ve found is helpful, is to bring an analyst and/or analytics manager into the entire decision-making process. Have them go to whatever sales pitches the decision-makers or executives are going to; ask them to be engaged in the process of deciding on the tool; have them pilot any product before actually adopting it.
Having a technical user taking part in the evaluation is good for a number of reasons. They can help you get to specific questions or formulate challenges to the vendors that you might not otherwise have thought about. In the sales pitches and evaluations in which I have participated, a few of the primary questions that I often ask are:
What is its purpose and value? — Is the platform designed to do business intelligence, reporting, predictive modeling? This seems like a straightforward question, but many times it’s not. One example is the data warehousing vendor also selling “analytics capabilities” embedded in the warehousing process. Maybe it’s a function that allows a user to visualize relationships between customers, based on semantic tags being assigned to the data as they come in. That sounds very complex and interesting, but why do you need it? Is it going to fit in somewhere when your data scientists are creating models? Will it contribute to tracking and reporting? Or, is there someone in your data services organization who could hack up a Python script to handle that task? If someone in your organization can get the job done with existing tools (and there’s no clear need for more efficiency or capabilities), then it’s worth reconsidering.
Will it be adopted? — What I refer to here, is whether the analytics/BI/business members will actually use the platform. This is where the input of your analysts is most important. Analysts need to pilot whatever tool is going to be adopted, to determine if and where they feel they can use it. If it’s just duplicating functionality of your existing systems or lacks key functionality, then it’s likely to simply sit on the shelf and collect dust, so to speak. Additionally, one tool may provide differing levels of value to different groups in the organization. A visualization package may not be right for your data scientists, but may be what the Data Services team needs to better understand its work.
On the other hand, without providing the right tools, “shadow IT” can crop up. I have lived in this world — it’s inefficient at best and dangerous at worst. If analytics teams don’t see the value of software and continue using what they have (whether supported or unsupported), then there’s no point in buying. For any platform that is going to be adopted, training also needs to be provided for analysts. Everyone is busy; providing new analytics options into the organization may seem like it’s opening doors, but it can often end up feeling like an additional burden to the people expected to use and produce work from it. In this regard, it’s usually best to expect a general decrease in productivity as analysts spend time on training and familiarizing themselves with new software.
Is it dangerous? — I include this specifically in reference to advanced analytics/predictive modeling/machine learning software. There is no shortage of vendors that provide “automated” or “push-button” analytics functions. From a business perspective, “automated machine learning” sounds like a great way to implement predictive analytics into an organization. Give a point-&-click software to any business user and there’s no need to hire an overpaid data science “unicorn”… except when things don’t go according to plan. The quality of any model is dependent on the underlying data being fed into it. If those data are bad or not properly investigated before the model build, then model quality will suffer. Data exploration, feature engineering, variable creation — these are all different terms that generally describe the process of creating a quality dataset for modeling. That is a task that requires the domain knowledge and expertise of a trained analyst. There are other issues, as well, such as variable/feature importance and model diagnostics which may or may not be available through point-&-click software. Diving into model details is the job a data scientist or advanced analytics user. Trusting a “black box” to deliver results without deeper investigation is simply dangerous. Discovering and testing these issues is another reason that analysts need to be engaged in the software adoption process.
Coming Full Circle: People > Platforms
What I have tried to illustrate here, is that choosing analytics software — whether a stand-alone software or add-ons to an existing environment — is inherently a technical decision, designed to aid a process that should be led by qualified technicians. As a result, organizations should leverage their technical expertise (i.e. analysts) to aid in the selection process. Fancy visualizations and promises of automated insights are either only telling half a story or an outright lie. A well-designed dashboard or highly accurate machine learning pipeline relies primarily on the quality of the people making them, and less so on the actual tools they are using.
To illustrate my point above, and bring the story I started telling in the beginning full-circle: the warehousing vendor we engaged had three or four analytics extensions that they were trying to spin to us. As a way to demonstrate their analytics prowess, they provided a case study of a major model they had created for a large e-commerce company over transactional data (using 300,000+ variables over billions of records). And they built that model using… four data scientists and open-source software! So, while they were pitching us “push-button” analytics, they and the e-commerce company had made a serious investment in human resources. Now, do I believe the vendor was being purposefully misleading by making that kind of appeal? No. But it was misleading. We (or at least, I) were unconsciously conflating the results of a highly customized project with a those of a pre-packaged “black box”… Buyer beware. Ultimately, we ended up finding a different warehousing platform and I stuck with my homegrown solutions.
Involving analysts in the decision-making process isn’t just a measure to ensure that the organization is purchasing useful software. Analysts and other technical positions — either through their own personalities, the perceptions and priorities of management, or a combination of the two — tend to get boxed into their specific work. This not only limits the skill development and career opportunities of the analyst, it ends up hurting the organization by sheltering employees from understanding the organization’s broader strategic direction. In my professional work, I have focused very much on being a technician. Participating in department- or executive-level initiatives has helped me get broader experience outside of my immediate work.
Ultimately, the power of analytics lies in the skills and experience of the analysts, not the software. The prevailing mindset in analytics needs to change from one where software will give us easy access to instant insights, to one where software is simply a tool to augment the capabilities of the analyst. Focusing on people and processes rather than platforms can help not just create better decisions and insights, but also better purchasing decisions.