Leveraging GenAI to superpower our analytics platform’s users
Unlocking insights: How we used GenAI to boost our analytics experience!
In the last year, we’ve experienced a great technology emerging and becoming publicly accessible: Generative AI. Encouraged by the excitement around this artificial intelligence, we decided to harness the power of GenAI to provide our users with a tool for understanding both their data and our analytics capabilities.
As a small startup, we try our best to keep our tasks focused and efficient, providing value to our users sustainably and with limited resources. GenAI came as a great prospect for this kind of mentality, providing a powerful solution while being easy to integrate and interact with. Plug some text in, and get some clever answers. (But it wasn’t that simple, was it?)
I had the honor to lead the development of this feature. In this post, I’ll dive into our experience and share how we got from a thought to data storytelling — generating insights from charts and tables.
The challenges we face at Sightfull
At Sightfull, a challenge we face as a business data analytics platform is how to provide useful and informative content about our client’s business data. We would love to allow our clients to speak with our business analysts, so why not clone them using GenAI?
Exploring possibilities
While implementing GenAI into our product, we started by narrowing down which part of the product it should be incorporated into. From personal use, we already knew that trying many different tasks together would result in a suboptimal experience, reducing the value it could provide. To narrow it down, we mapped where we knew our clients would like personalized assistance and the different tasks that could be done with our product.
Through this process, we ended up with 3 main ideas,
- Discovery — Find what you need quickly or what you didn’t know you needed.
- Productivity — Do more in less time and interact with the platform more efficiently.
- Explainability — Understanding your data and what you see or might have overlooked.
To select the ideas that best supported our case, we mapped the different inputs and outputs for each idea, whether it worked with structured or text-based data.
From the ideas above, we chose explainability, leveraging both controlled and structured data as input and a free-text response that we believed would be most suitable for our business users. Called “Data storytelling”, its goal was to provide meaningful analysis of the different metrics and tables clients see, summarizing and marking points of interest.
Our process
To create our data storytelling feature, we were missing a few components:
- Queryable GenAI endpoint to interact with
- User’s current view and underlying data
- Powerful prompt to properly convert the view into user-facing insights
To create the GenAI endpoint, we started by creating our own microservice, tasked with interacting with the FE and OpenAI’s ChatGPT. The service would receive the current user’s view: the metric, metadata, and any other user customization such as filters — and would return multiple summarized insights based on the metric. I won’t go into detail on this part because it was a pretty standard service.
Prompt engineering techniques
Prompt engineering was, to my surprise, one of the most interesting parts. I found a large amount of complexity embedded into this, although it seemed simple at first. Our first attempts of “throwing everything together in the pot,” or as we understood later on, “zero-shot” prompting, didn’t provide very accurate results. So, we started diving deeper into how processing prompts and interacting with GenAI looked.
Iterating on the prompt structure, we found a few techniques that could be applied to us:
Few-shot
Expecting the generation to simply work was somewhat optimistic. By providing a few examples in the system prompt, we were able to generate much more predictable results, keeping the responses in the same structure and length.
Chain-of-thought
“A long answer is more accurate than a short one.” This technique reminded me of asking for a ‘whole answer’. Instead of expecting the prompt to create meaningful insights straight away, we added a step to summarize the data beforehand, explaining all the different trends and anomalies. We used the fact that the generation could be separated into ‘analysis’ and ‘summary’ instead of trying to juggle both at the same time.
Prompt-chaining
The chain of thought was a significant improvement, but still, doing two things simultaneously was causing inconsistencies in the result, hallucinated numbers, and inventing data that wasn’t actually there. To resolve this, we completely separated the prompts, creating one prompt to elaborate and act as a translator between data and text and the second to refine the text and provide actual insights based on the input.
Transition to preprocessing
Something wasn’t clicking
Our feature was working, but response times were quite high, even reaching 20 seconds. (Hopefully, users love spinning loaders!) After conducting some research, we found that in the model we were using, the time taken to generate tokens increased linearly, with an average of 28 milliseconds per token. Therefore, we needed to generate fewer tokens to reduce the response times.
As a band-aid, we started streaming the results, giving the user a sense of progress, although the generation took the same amount of time. But it just wasn’t good enough.
That wasn’t the only problem. Responses were also too vague or inaccurate, omitting numbers or hallucinating due to bad math.
Taking a step back
We tried tweaking the prompts but eventually decided to mix it up. We were controlling the inputs, and as we tried to embed our data team’s ability to analyze the metrics, we were trying to generalize too much. Instead, we decided to perform preprocessing of the data to help the generation with preprocessed data, providing a code-generated summary instead of forcing the GenAI to get good at math.
This change generated a summary in less than a second and was 100% accurate. Because we were now code-generating, we could specify even more and create different summaries for different kinds of metrics, providing even more specific data for creating actual insights.
Eventually, our prompt looked like a large template containing a code-summarized metric, all the relevant metadata explaining the context in which the metric lived and the different parameters the users selected to view this metric. Using this template, we could generate accurate insights based on the different kinds of metrics we displayed and keep the generation fast and responsive.
All of this iteration wouldn’t have been possible without creating a quick feedback loop, and that’s one of my key takeaways from this. Our initial setup was very manual. We invested some time in creating an easy-to-run setup with customizable prompts so we could iterate quickly and test our different configurations, time that was returned very quickly.
In addition, we documented the different iterations and remarks to make sure we were keeping track of our prompts. As we’ve seen earlier, even slight modifications caused drastic changes in the generated responses.
Future steps
Creating the data storytelling feature was a great experience, but it was only an appetizer for us, getting our foot in the door and starting to experiment with GenAI.
We’ve had our fair share of questionable decisions (DataFrame modules in Typescript aren’t pandas…). We lacked one of the best aspects of Generative AI, and that was the conversational part, allowing you to pursue your own questions and ideas. That marked a large upcoming goal: instead of getting insights about your data, actually interacting with it.
Making that leap required introducing new concepts and methodologies that weren’t popular at the time, including unstructured user input instead of structured input and RAG (Retrieval augmented generation) to retrieve the correct information in real-time. Keep an eye out for future posts to learn more!
Thanks for reading, if you are still curious you can experience it yourself at Sightfull.ai.
