Pivoting from Strategy to Execution: Best Practices for Business Intelligence Platform Development

Over the past decade working with early round funded executives, I’ve noticed a disconnect between investor commitments and the daily processes and KPIs used to measure success.

While fundraising creates focus around key questions founders must answer to raise money — the market opportunity, competitive landscape, unique value proposition, monetization model and revenue objectives — the skills to achieve proof-of-concept and product-market fit aren’t necessarily those founders need to build scalable businesses and track progress toward success.

Translating business objectives into go-to-market infrastructure — talent mix, operational processes, enabling technologies, and success metrics — is where problems arise.

The first post-funding hires are usually functional go-to-market leads — marketing, sales, and operations. There is rarely an obvious candidate to design a system organizing enterprise data points into diagnostics of progress toward revenue and growth objectives.

As a result, founders often fly blind and find themselves in one of the following two scenarios:

Static periodic performance: manually aggregated static enterprise performance snapshots, normalized across disparate data and visualized for executive consumption. It is the result of a massive aggregation exercise. Because of the enormous investment required to complete aggregation it’s generally completed at most, quarterly.

Partial or functional performance: turnkey tools that aggregate data from a set of related functional tasks — usually within a single team. Think engineering team performance measuring total throughput, team member efficiency, etc.

Both of these scenarios are non-optimal and don’t provide the real-time guidance today’s executives require to identify front and back office barriers to growth.

As a result, it is absolutely imperative to prioritize the investment in a robust BI tool to guide your firm’s growth path.

Before designing a BI platform, develop the metrics most representative of progress toward growth objectives. Metrics will vary by firm but will generally fall into product delivery, demand generation & customer acquisition, and business operations buckets. Typical data inputs and source repositories are listed below:

BI Stack Design

While firms often possess unique business model characteristics, internal data input configurations and key performance indicators, all BI stacks require three functional components — data preparation, data warehousing and data visualization.

Data Preparation Layer

The most important step of BI stack design is data preparation: the process of combining, cleaning, and creating data sets ready for executive analysis.

In the pre-cloud era, this was generally a batch process managed by an on-premise ETL solution (short for extract, transform, load) that takes data from one system, transforms it, and loads it into a data warehouse.

Today, due to the nose-diving price of cloud server storage and proliferation of SaaS solutions — high power, low cost and easily configurable data preparation systems are now available to startups without dedicated IT or devops functions.

When evaluating partner capabilities, it’s critical to consider present and future requirements:

• What is the potential range of product and customer data inputs?
• What is the potential data volume I’m dealing with?
• At what cadence will I need to analyze data?

There are three data preparation approaches, each providing ‘best-fit’ depending on the answers to the questions above:

Extract, Transform, Load (ETL): Traditionally, ETL tools were on-prem hardware-heavy tech solutions. No longer. They range from ‘lite’ ETL solutions integrated in end-point dashboarding tools to stand-alone cloud ETL solutions that support streaming data processing, are relatively scalable and have pre-built integrations with core business systems.

Most dashboard and visualization solutions offer integrated ‘lite’ data prep functionality. Like standard ETL tools, they allow for data editing and easy logic statement building (e.g., split and join) across variables from different databases, and metric calculation development. However, they only support certain input type (e.g., databases), and offer only batch data availability. Cloud ETLs allow data extraction from more sources beyond databases (e.g., SaaS, cloud storage, SDKs) includes advanced data transformation logic development, allow customizable data availability and provide air-tight regulatory compliance.

If you are dealing with fully mappable relationships across data inputs, require compute-intensive computations and/or are dealing with smaller datasets, ELT preparation is likely your ideal approach. If you have static database-only inputs, consider an integrated ‘lite’ solution. If you require real-time data availability and an array of database and third-party integration points (e.g., mobile apps, commercial services), or operate in a highly regulated vertical, a cloud ETL may be an ideal fit.

Extract, Load, Transform (ELT): A newer data preparation variation, ELT meets many advanced use-cases better than ETL. Fundamentally, ETL is a linear, batch process — as data size grows, transformation time increases. ELT is a parallel process and speed is never dependant on incoming data volume.

With ELT, raw data is transformed within the target database system — not a stand-alone solution — and only on an as-needed basis. When someone queries data, it’s transformed only for that purpose. ELT is also distributed across servers enabling almost unlimited scalability — better supporting high-volume business models like eCommerce and real-time bidding solutions.

One of the downsides of eliminating a dedicated data preparation provider is that you lose the neat visual interface and data preparation features that ETL tools provide to ease the data transformation process. Transformation instructions must be written and maintained by programmers, usually in Python.

When you have the technical resourcing to build and manage data transformation instructions, are working with unstructured data and need to process large data volumes, ELT may provide an ideal solution.

Integration Platform as a Service (iPaaS): iPaaS is an emerging solution category that provides SaaS for building and deploying integrations between business systems — cloud and on-prem. With iPaaS, a code-free UI for rule setting and logic statement development lets users develop integration flows connecting applications residing in the cloud or on-prem and then deploy them without installing or managing any hardware or middleware. iPaaS bundles the ETL process, which previously existed in standalone products, into a broader integration ecosystem.

If you have complex upfront data inputs, have evolving business data requirements due to business model iteration, or maintain on-premise data warehouses, an iPaaS solution is likely a strong fit.

Data Warehousing Layer

When it comes to warehousing databases, there are two main types: SQL and NoSQL — or, relational databases and non-relational databases. Selecting the optimal database partners depends on several business model dimensions of your enterprise:

Transaction Volume: The data structures used by NoSQL databases (i.e. JSON documents) are more lightweight than those used in relational databases, making most operations faster in NoSQL than their SQL cousins. NoSQL databases are also cloud-based and distributed across servers allowing for easily scaling processing volumes. Finally, they don’t have to process logic rules like joining disparate tables holding various data points. Remember, if you’ve chosen ETL data prep, you’ll need a NoSQL database system

Object Logic Complexity: If you have complex KPI schema that will require complicated querying, are working exclusively with database inputs, and routine analysis of data, you’ll likely want to stick with a relational database. In relational databases, there is an built-in and foolproof method of enforcing referential integrity and rules at the database layer. The tradeoff with the data richness noSQL databases provide is the requirement for custom data transformation instruction.

Business Model Specificity: If your data requirements aren’t clear at the outset or if you’re dealing with massive amounts of unstructured data, you may not have the luxury of developing a relational database with clearly defined schema, meaning a non-relational NoSQL database is your best choice.

Visualization Layer

While possibly the least technical and most straightforward part of a the BI stack — there are several dimensions to consider before selecting a solution partner.

Embeddability: Visualization tools are generally available as standalone or integrated solutions. Integrated visualization allows you to put the benefits of BI directly into a service that are already part of existing business processes. Integrated tools are either partially or fully integrated. While aesthetically similar, partially integrated solutions as a rule are less secure than fully integrated tools, and should be avoided as much as possible.

Analytical Assistance: Visualization tools are rule-based dashboards translating data filters into various visual formats. A newly emerging class of visualization tools apply machine learning to enable descriptive analysis like alerts for data volume spikes, significant metric swings, or other anomaly detections. In the near future, these tools will estimate the impact of specific scenarios on key KPIs like sales close rate, return on marketing investment, and customer churn.

Interactivity: When it comes to dashboard customization, more is almost always more. While most solutions offer some level of customization like changing presentation formats, look for tools that facilitate both real-time filtering and data pivoting to facilitate meaningful, real-time strategy discussion.

Final Thoughts

I’d like to offer a couple final thoughts for the road ahead:

Prioritize BI Development: For small firms with overstretched teams, building a cutting-edge BI tool may feel daunting, but I firmly place this in the ‘80/20’ bucket — a 20% resource investment that generates 80% of performance. Once built, executives often consult their BI tool on a daily — and sometimes hourly basis — and find it an indispensable tool in strategy development and iteration.

Plan Ahead: In the high intensity trench warfare of startup world, it’s easy to get caught up in immediate business requirements. Myopic BI design can incur significant long-term tech debt and associated remediation costs. Make sure you incorporate long-term product planning and associated data use cases into your design to ensure long-term value from your initial investment.