BI in Startups
How People.ai Setup BI/Analytics Function in Under 6 Weeks
One of the growth challenges that startups face is the need for every decision to be data-driven. But to be data-driven, you have to have a Data and Business intelligence (BI) function that can power your analytics and inform your decisions. With data spread across tens of systems, providing holistic insights using spreadsheets is not only difficult but not scalable. Setting up Analytics is necessary because this is a key foundation of your business and one on which future growth depends. We thought we’d share the story of how we created our BI/Analytics function in a period of six weeks. This is the technical story behind it and we hope it and the lessons learned will be helpful to other startups preparing to establish their BI/Analytics function.
4 Basic Components of Your BI/Data Infrastructure
To begin with, at a very fundamental level, there are four logical components that go into building most BI/ Data Warehouse systems:
- Ingestion and compute layer to ingest and transform data (ETL/ELT)
- A persistent data layer to store the raw/modeled data (Data Lake or Data Warehouse)
- A visualization layer to provide insights via various BI tools
- Additional components like workflow automation, Data Quality, CI/CD, etc.
How to Evaluate Infrastructure Vendors
The first challenge is to identify the technology/vendor for each infrastructure component. In today’s BI/data space, there are hundreds of options, with pros and cons of each selection. So, how should you go about selecting components that will scale as the company matures from early-stage to growth-stage? Here’s what we learned:
Step 1: What fits in the company?
A few questions can help answer this:
- Cloud or On-premise: Are the company’s engineering and business systems hosted on cloud or on-premise? What are the organization’s plans in one to two years (any big migration in the pipeline)? What cloud vendor is being used?
- Data Security: Are you planning to process and store sensitive data? What level of compliance (HIPAA/PII/PCI/SOX/GDPR) is needed?
- Budget: How much money is your company willing to spend on BI (both on people and technology) before seeing an ROI, and how do those budgets scale up and down depending on company growth? This helps in choosing systems with less stickiness and more elasticity, to scale up/down quickly.
- Delivery Speed: With multiple departments needing data, realistically, what are the pressing and critical needs that can help move the needle? This will help in choosing components that scale versus provide speed, in the interim. On a slightly related note, a big factor that helps in identifying and prioritizing the business needs and urgency is having an Analytics champion in the C-suite and every function. Partnering with the champion helps you to take a holistic view of the ‘true’ urgency and priority of needs, and that in turn helps in selection.
Step 2: Understand the business use cases and 3 V’s
- Volumes: Create an inventory of data-producing systems and what volume of data each one produces. Have a multiplier of 10 to account for future use-cases.
- Variety: This is very critical, as it helps to come up with the Ingestion layer. You likely have multiple data-producing systems (departments that use dashboards, such as Product, Sales, Finance, Marketing, HR, Engineering, Customer Success). What type of data do they produce? Is it structured or semi-structured?
- Velocity: Identify how often decisions need to be made in various functions and levels (from C-staff to Sales Rep). With the exception of a few use cases, this may not be super critical at an early stage company. However, as the company matures, this will become a huge value add. Consider having near-real-time (15 mins) use cases and use cases for weekly, monthly, and quarterly dashboards. For most use cases, a daily cadence for a data and reports refresh will suffice.
Making the Vendor Selection
Armed with the above information, these are some example requirements of vendors that you may establish:
- The solution needs to be in the cloud
- All components must scale linearly in both directions
- Minimal resource contention, especially between concurrent ETL writes and dashboard reads
- Minimal to no maintenance cost
- Has all compliances and satisfies security needs
- Can store and process any file and data formats
- Is within budget considerations
- Offers less vendor risk and components with less stickiness (easily portable)
Data Store: The above requirements will help you narrow down possible vendors, for example: AWS Redshift or AWS RDS (If you are an AWS shop), AWS S3 + Hive/Presto Layer, Snowflake, etc.
ETL/ELT: Next comes the ETL/ELT layer. With the choice of tens of on-premise and cloud options along with building in-house frameworks (using Python/Java), it is hard to narrow the field. Go back to your requirements. If your business needs something quick, however exciting it is to set up an in-house ETL/ELT framework, you may have to say no to on-premise tools and in-house frameworks. Many startups simply don’t have the resources or time to maintain and build them.
If you are a cloud-based startup with multiple systems to integrate quickly, you may consider a managed data pipeline platform. Look for solutions with little to no maintenance and that comes with pre-built connectors for hundreds of applications. The goal is to streamline this process and be able to quickly set up integrations.
BI Tools: For the BI layer, look at cloud solutions that offer maturity, richness in visualization, user’s choice, cost, and in-house experiences.
Setting it Up
The above process helped us to identify a good solution for our BI/Data infrastructure. Once identified, all the vendor evaluations, POCs, and contracts took three weeks and we set up our complete BI infrastructure in less than six weeks. We started delivery of dashboards and surfaced rich analytics from week four! These choices made our analytics easier, faster, and more reliable; but, as People.ai continues in its growth stage, the complexity increases. There will always be a need to evolve our infrastructure (systems and processes), to adapt to the growing needs of the business. As it goes, the only thing that is constant is change!
We hope sharing our story gives you a place to start and a framework for how to get your BI/Analytics function up and running in no time!
ABOUT PEOPLE.AI
People.ai accelerates enterprise growth through the power of AI. With the industry’s only Revenue Intelligence System, People.ai frees all customer-facing teams, including Sales, Marketing, and Customer Success, from manual data entry by automatically capturing all contact and customer activity data, dynamically updating CRM and other systems of record, and providing actionable intelligence across management tools to realize the full selling capacity of the enterprise.
