LLM and BI System Synergy — First results and Mid-Term Perspectives

The New BI Challenge

Business Intelligence team often acts like a superhero for companies now, guiding management through the entire journey from raw data to analysis on the top of dashboards. But here’s the catch — not many companies are using BI System as much as they could. Factors such as poor data quality, mismatches in information needs, a trust gap among users, bad delivery quality, and more contribute to an adoption rate of around 15% (active BI users to the total number of employee), as revealed by a BARC/Eckerson Group joint survey.

The big talk about ChatGPT and other fancy language models last year, has led big BI vendors to race against the clock to make BI even better. Tableau, Power BI, ThoughtSpot are in a rush to improve BI and have already shown off their plans and sneak peeks of how things will look for users in the future.

The challenge lies in the fact that the urgency to incorporate LLMs into BI has outpaced the development of a clear plan on how they should work together with numerical data, multiple data contexts, and hidden assumptions, as well as what interface to provide for the user. There are also formal blockers, such as prompt length limits and corporate data security concerns.

Our Vision: Making Good Old BI smarter step by step

In response to the uncertainties surrounding the integration of LLMs and BI, our Reporting & Analytics and AI Teams at Luxoft have taken on the challenge and begun testing the Conversational experience for our Tableau users by leveraging the capabilities of Azure OpenAI.

Here are our 2 scenarios we started with:

1. Glossary Terms Consultant: The bot helps explain tricky company terms in a way that everyone can understand. Answering questions like “What is [term]?”, “What does [metric] mean?”
2. Report Navigator: The bot guides you to the relevant reports based on what you need. Queries like “What report can I use to monitor [metric]?”. It not only recommends the reports but also helps to understand them better. Queries like: “Tell me more about this report” or “How should I read this report”

The value of these use cases is apparent — bridging the gap between the user and existing certified reports. Addressing navigation issues is crucial, as it is one of the key reasons for challenges faced by casual users in our company.

To build this functionality, the project team used a stack that included Semantic Kernel, Azure Cosmos DB, and Azure OpenAI. As a source of information, we utilized the BI Reporting documentation wiki and the Corporate glossary:

Both scenarios at the end work pretty fine. The bot is rephrasing the initial text and providing information without overwhelming users with non-relevant details.

We found that approximately 9 out of 10 recommendations were fully correct, with only 10% being partly correct. Here are the examples:

Sometimes, the Bot generates links to non-existing reports, which can be amusing. This issue can be resolved by providing clearer inputs in the source data for this specific question.

We’re working on enhancing the precision of metadata linkage between terms and reports from the data catalog. This will improve accuracy in the next stages of the project. There’s also ongoing work to further templatize wiki pages and make adjustments to glossary content to ensure better accuracy.

What’s Next? Advanced Data Assistant

In our grand plan, we envisioned an advanced Data Assistant that provides detailed metric values with reasoning for deltas. Users can make simple data queries such as “What was the revenue value of [Account] in [Region] last month, and how does it differ from the plan?”.

While it seems technically feasible, this will required additional work at least in two areas:

1. Test the quality of numerical data perception and conversion into the correct response in limited length of the promts;
2. Prepare datasets with high-quality metadata and semantic linkage with a glossary.

Another feature we are considering is adding an option to “Call an Expert” into the chat, similar to what you find in Bank mobile apps when their chatbot fails to answer your question. In our workflow, the bot can connect the user with the relevant duty expert who will assist with technical report-related issues or with data stewards/SMEs who will provide valuable insights into the figures from a business perspective. This feature can ensure a seamless and robust experience with interactive feedback.

Concerns and Mid-Term Perspectives

So, we are in the process of evaluating what’s working and what’s not, refining our approach, and dreaming about the potential impact our project could have on BI adoption and user experience.

However, despite the allure of next-gen BI, there is no certainty that it will profoundly impact the way managers work with analytics. BI, as a function that translates data into understandable dashboards, is unlikely to disappear in the next 10 years. This resilience is attributed to:

1. Poor data quality continues to impede many insights generation initiatives.
2. Hidden Contexts and Assumptions that distinguish real data from “datasets for pre-sales” taht vendors uses in their inspiring demos.
3. A trust gap between managers and machine-generated insights is significant.

Facing challenges in BI adoption, we have a 100% probability of encountering even more significant adoption issues with this kind of conversational BI solution. It’s crucial to proceed with experiments in user experience in a ‘Test and Learn’ iterative manner, gaining proof that the toolset provides real value and you have active users offering comprehensive feedback.

In the upcoming 1–2 years, I anticipate the following trends in this area

• BI vendors will release and fine-tune their own GPT-based workflows in collaboration with third-party LLM products.
• Growth in internal initiatives led by BI and proto LLMOps teams to enhance and expedite the UX for end-users. This includes semi-automation of simple data ad-hocs using LLMs, corporate knowledge bases, and well-annotated data sources.
• Massive development of small, reasonably priced, and use-case-oriented custom corporate LLMs based on open-source AI as part of the competitive struggle among IT-oriented enterprises that can afford such initiatives.

This can finally truly levels up the self-service analytics workflow of casual users in BI projects. However, the path to achieving this is not as straightforward as it may initially seem

Stay tuned for more exciting updates on our quest!