Best Practices for AI/BI Genie Spaces on Databricks
Authors:
Dr. Philip Laserstein — Solutions Architect @ Databricks
Sebastian Kollmann— Solutions Architect @ Databricks
Introduction
AI/BI Genie empowers business users to interact with data using natural language. It is powered by Unity Catalog, which enables it to use the semantics of your data: By leveraging the rich context provided by Unity Catalog — including data lineage, documentation, or query history — Genie delivers insights to a broad set of business questions. Genie provides the flexibility of natural language and GenAI combined with trustworthiness to truly democratize Data & AI powered by the unified governance of the Databricks Data Intelligence Platform.
AI/BI Genie Setup
Genie spaces are typically prepared by data analysts by selecting a focused set of tables from Unity Catalog and exposing their metadata, by adding instructions that transfer organization-specific information (such as business logic) into the Genie space, and by providing well-defined example questions and answers (e.g., queries) to guide the business users on their first steps.
Data: choosing the right dataset
We recommend starting with a small set of tables (5–10) with a manageable number of columns (<50) for one specific topic for a reason: the more coherent and semantically connected your data set is, the better Genie will work. What do mean with a coherent and semantically connected data set?
- The relationship between the different tables reflect real-world connections or logical associations. You should add primary key and foreign key constraints to your tables so Genie knows how the tables are connected and can be joined.
- All tables and columns of the dataset contribute to the overall topic of this dataset. If there are excess columns which do not fit you should create views to clean the data and make it more coherent.
- Typically the data for a Genie space are curated business-level tables from the gold layer, similar as you would provide it to a BI tool to build a dashboard on top of it. This is often done using a Kimball style star schema: a multi-dimensional data model used to organize data in a database so that it is easy to understand and analyze.
Also, make sure that any user who is interacting with the space has at least SELECT privileges on the data used: as the data is governed by Unity Catalog they will otherwise get an error message due to missing privileges.
Documentation: document and annotate your data
Make sure the data is well-annotated. Because without documentation — which adds business context to your data — everybody (incl Genie) would be just guessing. Simply said: the better your documentation and annotations are, the better the results will be. You can add annotations to your data with AI generated documentation to save time and reduce manual effort while still being in control to ensure the descriptions align with the specific use case and domain knowledge.
Process: preparing the space
During preparation the Genie space should be tested to check response quality and ensure it is providing the expected responses. You can rephrase the provided sample questions and add and refine instructions until Genie provides the expected response.
Genie is translating the questions from natural language to SQL and therefore needs a SQL warehouse for query execution. We recommend using a serverless SQL warehouse for faster startup time and intelligent workload management to process queries quickly and cost-effectively. Users interacting with the space require CAN USE access to the SQL warehouse designated to the space.
Enhancing Response Quality
This step ensures that you transfer all the business context you have about your data into information Genie can use to enhance the response quality. There are currently three ways to do that: instructions to add additional semantics with natural language; SQL Examples to guide Genie on how to query your data and retrieve information; and Trusted Assets which are predefined queries to enable a single version of the truth.
Instructions: transfer your business Language to Genie
Instructions provide additional information to help Genie learn your specific business language. They provide guidance to the large language model to understand company-specific jargon or domain-specific concepts.
- For example, we at Databricks start our fiscal year in February which is deviating from the common fiscal start in January. Thus, it is necessary to give this company-specific detail as an instruction to Genie so it can aggregate numbers correctly:
- Fiscal Year starts in February - Another possibility is to tell Genie how it should apply filter logic or search for values in your columns if you know that some columns are case-sensitive:
- always convert strings to lowercase and use a like operator when applying filters
- countries in the column “country_code” are stored with two characters (e.g. US, AT) - Furthermore, you can use other prompting techniques such as “one shot learning” to teach Genie a bit more about specifics on how it should treat columns and can extract information from them:
- the column “People_Name” is in the format “Firstname Middlename, Lastname”. If the content of the column is “Francis Ford, Coppola” then the Firstname = Francis, the Middlename = Ford, and the Lastname = Coppola.
Make sure your instructions are clear and concise. The same is true for your data: the more coherent and simple your instructions are, the better the results Genie will provide.
Trusted Assets: predefined queries for predefined questions
To help illustrate the different capabilities between Trusted Assets and SQL Examples, let’s consider an analogy: Imagine teaching a student to solve multiplication problems. First, you teach them basic multiplications like 2*3, which they will typically memorize. Later you will teach them the tools to calculate more complex multiplications like 123*64. Most students will memorize recurring multiplications, like 8*8=64, while also learning the tools required to calculate 123*64, which they leverage independently for new multiplication tasks.
Trusted assets are user-defined queries triggered for specific questions. Genie executes the pre-defined query and provides the result with a “trusted asset” tag. The association between Trusted assets and specific questions is implemented via the function comments. In our analogy, this would be similar to memorizing the result of 8*8 without doing the calculation.
- Define a query that addresses a specific question, in this example converting beer prices from EUR to USD
SELECT
o.year,
o.beer_price,
(o.beer_price * e.EUR2USD) AS beer_price_usd
FROM
main.default.oktoberfest_gold o
JOIN main.default.eur_usd_conversion e ON o.year = e.Year
WHERE
o.year = 20052. Register the generalized query (adding the filter as a parameter) as a Function
CREATE
OR REPLACE FUNCTION main.default.eur2usd_oktoberfest_function (yeartime ARRAY < INT > DEFAULT NULL) RETURNS TABLE
COMMENT 'This function facilitates the conversion of beer prices in eur to usd for single years and multiple years' RETURN
SELECT
o.year,
o.beer_price,
(o.beer_price * e.EUR2USD) AS beer_price_usd
FROM
main.default.oktoberfest_gold o
JOIN main.default.eur_usd_conversion e ON o.year = e.Year
WHERE
array_contains(eur2usd_function.yeartime, o.year)3. Add the registered function as a trusted asset
4. Retrieve trusted results to questions
SQL Examples: guiding Genie’s queries
On the other hand, by leveraging SQL Examples, users can provide examples of how to retrieve information from your dataset. In our analogy, this would be like showing examples of how to do complex multiplications. In contrast to trusted assets, the SQL Examples provided by the users are not necessarily executed; rather, they help Genie understand how to query your data and provide additional quality assurance.
Improving Genie
Improving Genies response quality has two aspects: rating answers will help Databricks to improve the overall quality of Genie whilst it also helps the creator of the Genie space to understand what the users are asking and enables them to incorporate the feedback and reiterate on the instructions and predefined SQL queries.
Feedback: rating Genie responses
You can rate every Genie response with a thumbs up 👍 or thumbs down 👎 button. This feedback is used by the Databricks product team for future improvements. If you downvote a Genie response, you can provide a description of your feedback and capture a screenshot. The Databricks product team will only review the information you’ve provided in the description field and the screenshot, not the original question or generated SQL query. Providing feedback helps to improve Genie over time and will ultimately give you better results for your business questions in the future.
Monitoring: understand what your business users are asking
The monitoring page provides a comprehensive record — the Messages feed — of all questions and answers exchanged in a Genie space, which can be filtered by various criteria such as time, rating, user, and status.
By regularly reviewing the monitoring tab, you can identify areas where your Genie space might need improvements. Are there certain types of questions that Genie struggles to answer? You can leverage these insights to improve your instructions or to provide Trusted Assets. By clicking on the links in the “Question” column, you can inspect the complete thread related to that question.
Conclusion
One of the most common use cases we see is offloading super-specific queries from dashboards to Genie. Traditional dashboards have limitations in terms of flexibility and interactivity, making it hard for business users to easily access and analyze data. Instead of creating numerous dashboards to answer every question business users may have, focus on building dashboards for your key KPIs and standard questions. Then, provide business users a genie space based on the same semantic data, equipped with trusted assets and proper instructions, to explore more detailed questions. This approach will free up time from your data team while empowering business users to ask in-depth questions and uncover further insights. In conclusion, Dashboards are ideal for the top 10% of recurring everyday questions, while Genie is for digging deeper into more complex questions that dashboards can’t answer.
To sum it up, focus on these major components to ensure success with an AI/BI Genie space:
- Data: choose a clean dataset — semantically connected and topic-specific — and make sure it is well-documented, modeled, and annotated
- Instructions: transfer your business-specific information to Genie via natural language
- Predefined Queries: leverage Trusted Assets to enable a single version of the truth as well as SQL Examples to guide Genie’s responses as they are powerful tools to improve the accuracy of Genie’s responses to common and recurring questions
- Feedback: monitor questions and their rating to further enhance the provided samples, business context, and standardized questions and ultimately improve the Genie space
If you are interested in more examples, have a look at how onboarding a Genie space is akin to hiring a new analyst for your data team or how you can improve your genie space based on an example of baseball data.
