Making sense of data: Building an analytics platform for teams at Fasal
At Fasal, we collect and store a lot of data. Data comes from IoT devices, customer interactions on the app, and via various external integrations. Everyone at Fasal loves data and we all generate insights out of the data to understand and learn more about our company, customers, and products. But accessing all the data that we collect and store in various forms, shapes, and places was always a challenge, and making sense out of it required a lot of back and forth work between teams.
Data if utilized correctly offers a wealth of opportunity to individuals and companies looking to improve their business’ intelligence, operational efficiency, profitability, and growth over time. Failure to leverage digital data to our advantage in today’s world — it’s like walking down a busy street wearing a blindfold.
What is an Ad-hoc analytics platform and why we needed one?
We are drowning in data, it’s crucial to work with the right tools to not only segment, curate, and analyze large data sets but also uncover answers to new questions that we didn’t even know existed. And when it comes to finding actionable answers to specific questions, ad-hoc analysis and reporting are essential.
Ad-hoc analysis and reporting is a dynamic, real-time report generation on an as-needed basis designed to answer questions in response to an event using current data. In our case, the product team wanted to understand more about user behavior, feature performance, etc. The engineering team was interested in engineering KPIs like device performance, app performance, etc. Sales and management wanted to understand customer behavior, sales, subscription renewable, etc. Agri research team is always dependent on sensor data and crop data for research and modeling of disease and pest. This was the need of the hour to set up a platform to help all our internal teams with analytics and reporting. And after evaluating multiple tools, technologies and ways to implement an efficient system, we ended up building our own data pipeline and using Redash as our data collaboration and visualization engine.
The analytics platform architecture at Fasal
The Adhoc analytics platform consists of the following components (we use MongoDB as our production database)—
Analytics Node
Read-only node exclusively targeted for querying. The analytics node is run on a MongoDB’s Altas cloud platform. It is a replication set of our production database. We use a separate realtime replicated, analytics node to make sure our heavy analytics queries don’t have any effect on our production level transactional database, and user experience on the app remains as smooth as it can.
BI-Connector
BI-Connector serves as a bridge between the MongoDB cluster and BI tools. It allows users to query with SQL on MongoDB data. That is something that our engineer team and product team loves because they know querying using SQL. But not the sales and agri research team of-course, they want a drag-drop solution to their analytics need. Hold on! we solved that below.
Data Collaboration & Visualization
We use Redash as our data collaboration and visualization layer. This enables anyone in our organization to build dashboards and visualize the data by simple drag and drop, and share it with the required team members. This also allows us to plug data from multiple data sources to build a unified dashboard for a specific purpose.
Redash Deployment on AWS
You can refer to the diagram above that defines the AWS deployment setup. You may also want to check out this repository which contains settings files to host Redash on AWS.
More about Redash
Supports multiple data sources.
Redash connects to multiple SQL, NoSQL, Big Data, and API data sources and helps run queries against these data sources to help us answer complex questions. It gives us access to all the important data in a single location.
SQL query editor
It has a SQL client which makes it easier to browse schema, click to insert queries, create snippets, and perform other query tasks efficiently.
Create amazing dashboards
We can gather information from multiple sources and display visualizations into dashboards. Share those dashboards with colleagues and external partners.
Get alerts
We can also set up alerts and get notified of certain events on your data. The alert can be notified to us via mail or on our slack channel.
Open-source, self-hosted and easily manageable
Redash provides an open-source setup that we can configure and customize according to our needs and deploy it through the cloud, and provide access to our users anywhere and on any device.
Enabling teams to extract value out of data
We use reports to track progress against targets and goals to improve performance. We generate KPI reports to understand and help us answer some of the critical questions like —
- How many customers we have and how many customers have we converted from demo and trial to paid during a defined time period.
- How many total devices that we have deployed in the fields and what is the ratio of healthy Vs non-healthy device.
- How many customers are utilizing our disease and pest models and how many of them are taking actions to our notifications which will help them in reducing the chances of pests or diseases in the foreseeable future.
Decision Making
We make data-informed decisions to back up our intuition. When a question gets asked internally, our answers will most likely be returned with a Redash link of a dashboard, table, or visualization.
Also read —
What's your data story and how you analyze data? Leave in the comments below.
