Evaluating and Implementing Analytics Platforms

In the two previous installments of this series, we’ve talked about how the role of analytics has changed and what steps you should be taking to bring your enterprise’s analytics platform into the modern era. In this article, we’ll walk through the last two major steps of analytics platform selection: evaluation and implementation.

You know that performance, timeliness, scalability, operational efficiency, and cost-effectiveness are key factors to consider, but evaluation can still be difficult. After all, most vendors are happy to say they can support any scenario you throw at them. How can you make them prove it before you sign the contract?

Even if you do find the right solution, the hard work isn’t done yet. Failed implementations can kill the value of even the best solutions. Keep reading to learn how to sidestep the pitfalls of these final two steps in the analytics platform buying process.

How To Conduct Technical Evaluation on an Analytics Platform

By this point, you’re well-prepared to take a look at several analytics platform products and assess them based on the criteria that matter most to you. Once you’ve done that, it’s time to build your shortlist and move on to the technical evaluation phase.

Phase I — Technical Evaluation

Depending on the solutions you’ve shortlisted, your organization’s IT restrictions, and the technologies you’re already working with, you will most likely end up going down one of these three paths:

• Open Source Products — Open source software projects (or the open source core of a commercial product) tend to be the easiest path to start investigating. In most cases, these projects offer a free download of the software on their community website. Additionally, you can often get any basic questions answered about the technology and its deployment in the project’s associated forums, slack channels, or on Stack Overflow.
• Cloud Products — Cloud products also tend to be easy to get started with, though you can hit a snag when it comes to paying to try them out. Fortunately, most have a flexible pricing model that allows you to pay for only what you use, and some may even offer a free trial period. Be aware that the duration of the free trial may not offer enough time to fully evaluate all of the solution’s features.
• Commercial Free Trials — For many enterprise software offerings, you’ll usually be able to work out a deal with the vendor’s sales team and arrange for a proof of concept (POC). The POC should come free of charge when it comes to the evaluation license and technical resources supporting the POC. It's important to stress that POCs are a much higher commitment for you and your organization than the two aforementioned options. Ensuring a successful POC means having dedicated resources from the vendor as well as your organization. In some cases, you may have to extend the POC into a pilot project. If this happens, you will likely need to pay for the vendor’s service personnel.

Phase II — Standard Validation and Proof of Concept

Once you’ve settled on the solutions you want to look into, you’re ready for the next two phases of your evaluation: Standard Validation and Proof of Concept.

• Standard Validation — In this phase, you should use a standard test benchmark such as TPC-H or TPC-DS to get a general sense of how a product works and performs.
• Proof of Concept — Once you’ve got a decent grasp of the product’s benchmark performance, it’s time to put it to the test in your own environment with your own data. Below are the recommended tasks you should complete in this phase, though you may have additional tasks based on your specific use cases:

— Pick several test cases that are the most challenging ones in your current workflow.

— Identify tables and views used by these test cases and understand their relationships.

— Obfuscate any sensitive data if needed.

— Produce the queries used in your test cases, make adjustments for different SQL dialects.

— Prepare the expected results for validation purposes.

— Execute the queries and validate results and performance numbers.

— Integrate with your front-end applications or BI tools (depending on how much you are willing to change your front-end tools, this step may not be necessary).

— Integrate with other key systems in your IT landscape such as your security framework, scheduling system, and data observability products.

Understand that POCs, even with open-source products, are not free. It is a very involved process that requires a lot of labor hours. For a POC to be successful, you will have to assign engineers and analysts to work with the vendor personnel side-by-side to get good results. Speaking of results, it is also critical that you clearly define the scope and success criteria of your POC ahead of time to make your final decision process as smooth as possible.

How To Implement an Analytics Platform

Finally, you’ve made your selection and it’s time to make it available to your stakeholders. It has taken a lot of work to get to this point and you probably can’t wait to roll things out, but this is not the time to rush into anything. A successful rollout is a mindful one.

The rule of thumb here is: Crawl, Walk, Run.

Start with Only a Few Scenarios

First, identify a couple of scenarios, ideally new projects, and plan to use your new platform for these projects. Starting with new projects will reduce your data migration risks and avoid compatibility issues. It also gives you quick wins to build trust with the new platform from your stakeholders. Work with your vendor’s support team to identify the best projects to start with and leverage their expertise to support any new development work.

Develop a Communication Plan

Second, develop a communication plan for your stakeholders. Explain why you are making the changes, the benefits for the users and for the company, and any action items users will have to take. Make sure you have prepared a feedback channel for people to report any issues during the transition.

Focus on Your Power Users

Third, start with a small group of ‘trial’ users. This group will be your power users who understand the business and the application more than most users. They will switch existing apps to the new platform and be your beta testers.

Ensure You Have Enough Support

Finally, you are ready to make the big switch! Having a support team standing by to address any issues will be critical. Make sure you work with your vendor’s support organization so they are prepared for this roll out with a dedicated support team standing by.

If your experience is similar to most businesses, the path to rolling out a new analytics platform has been challenging and time-consuming. Now is the time to make sure all of that work was worth it. You may think that the worst is behind you, and if you’ve been following this guide, much of it is. The truth is that selecting the right solutions and rolling them out is only half the battle. In fact, Gartner recently found that only 20% of data analytics solutions deliver satisfactory business outcomes for the companies that adopt them. How can you ensure that your new investment pays off?

The answer lies in cultivating a healthy data culture in your organization. You can read more about that here.

Join Us on Slack

If you’re interested in the StarRocks project, have questions, or simply seek to discover solutions or best practices, join our StarRocks community on Slack. It’s a great place to connect with project experts and peers from your industry. You can also visit the StarRocks forum for more information.

RELATED ARTICLE

Understanding Analytics Platforms
The 5 Most Important Criteria for Choosing an Analytics Platform
How To Reduce Snowflake Costs by 80%