Under the Hood: Leveraging AI for Smart Insights in Typeform
Co-authored by Daniel Caldas and Naveen Reddy
At Typeform, we’re always exploring ways to enhance our platform and provide our users with more powerful tools for data analysis. In this blog post, we're thrilled to dive into the technical details of our latest feature, Smarts Insights, which leverages artificial intelligence to generate comprehensive, visually rich reports from Typeform responses.
Scoping and Solution Modeling for Smarts Insights
To deliver a transformative data analysis experience tailored to our users' needs, our team embarked on a thorough scoping and solution modeling process for Smarts Insights. We faced several key challenges that guided our approach to developing this innovative feature, all with the aim of enhancing your data analysis capabilities.
Challenges with Typeform Response Data
Typeform response data can be broadly categorized into two types: structured data from question types like multiple-choice, dropdowns, and ratings, and unstructured data from open-ended text responses. Handling unstructured data, with its inherent complexity and nuances, poses the following challenges:
- Privacy Concerns: Unstructured data may contain sensitive personal identifiable information (PII), which cannot be sent to external LLMs such as gpt-3.5-turbo without proper anonymisation; hence, we scope our analysis to the following types of questions: yes/no, NPS, opinion scale, rating, dropdown, multiple-choice, and picture-choice questions.
- Cost and Performance: Running all text responses through large language models could escalate costs unpredictably and impact performance.
Guiding Principles
To address these challenges, we established the following guiding principles:
- No PII Exposure: Ensure no sensitive respondent data or PII is sent to external language models.
- Constrain Variability: Reduce the variability of language model outputs by constraining the scope to specific formats, question types, chart types, and metrics.
- Questions: Instruct the AI to scope reports to questions in existing typeforms.
- Question Types: Pre-filter certain question types like payment, calendar, short text, or long text questions to keep inputs chartable.
- Chart Types: Constrain the AI to work with specific chart types.
- Metrics: Instruct the AI to work with absolute counts and percentages.
Safeguarding Customer Data
To uphold our commitment to data privacy and security, we implemented several measures to safeguard our customers’ sensitive information. Following the principle of “privacy by design and by default,” we deployed a PII question filter that identifies and excludes any question types that could potentially contain personally identifiable information, such as email addresses, phone numbers, or open-ended text responses. By pre-filtering these question types, we prevent raw respondent data from being exposed to external language models.
Another decision we made when developing this feature was to leverage only aggregated metrics and statistics for report generation rather than share individual-level data. For example, instead of revealing the age groups of specific respondents, we provide summary insights, such as “10 respondents are in the 20–30 age group, and 30 respondents are in the 40–50 age group.” This approach allows us to derive meaningful analytics while strictly preserving the privacy and confidentiality of our customers’ data.
Through rigorous data security measures, we aim to deliver the powerful Smarts Insights without compromising our customers' trust.
Report creation workflow (How)
Here’s a brief overview of our new system architecture. We’ll refer back to this diagram throughout the blog post as we explore the technical details of each component.
Choosing the LLM model
In September 2023, during the Smart Insights project, we assessed two potential models from OpenAI: GPT-3.5-Turbo and GPT-4. Given our existing working relationship with OpenAI and the rapid advancements in AI technology, these models were the natural choices to consider, given they were the only reliable production-ready LLMs available then. While GPT-4 demonstrated superior performance during our evaluation process, the performance gap between the two models was insufficient to justify the significantly higher cost associated with GPT-4, which was approximately ten times more expensive than GPT-3.5-Turbo.
It’s important to note that what is considered optimal can change rapidly in the fast-paced world of AI progress. However, based on the landscape in September 2023, after carefully considering the cost-benefit analysis, we opted to proceed with GPT-3.5-Turbo. This decision allowed us to strike a balance between achieving satisfactory performance and maintaining reasonable operational costs within the context of that specific time period.
Finding order in chaos
Our end goal is to generate a report structure that outlines exactly what to chart and how to chart it. The structure must be machine-readable and predictable, domain-constrained, and easy to use to program business logic and user interactions.
The cornerstone for generating these reports is the typeform structure containing the questions. In essence, a typeform is a list of questions.
Here is an example of a multiple-choice question and a rating question:
[
{
"id": "nhU1AZSE3XO2",
"title": "Pick a flavour",
"properties": {
"choices": [
{
"id": "IeJY1ZP6lhN0",
"ref": "50988421-0bdb-4654-9f12-c76a1ec58090",
"label": "Classic"
},
{
"id": "jR4JUGbdMKiO",
"ref": "880b1295-d414-4de7-91f3-96d51ac55c42",
"label": "Orange"
},
{
"id": "PkpJIFnwL0vT",
"ref": "579c914a-e629-436e-a665-6b4f2f2e7c44",
"label": "Lime"
}
]
},
"type": "multiple_choice"
},
{
"id": "eABwC8lYJx5Y",
"title": "What is your overall rating for this flavour?",
"ref": "51830bd0-bd0f-4c13-b4af-33f8dd3f967f",
"properties": {
"shape": "star",
"steps": 5
},
"type": "rating"
}
]Here’s a JSON snippet of a real-life production example of a report:
{
"id": "01HQPJKX5GVG1TY8M249M6XYCH",
"visualizations": [
{
"id": "01HQPJKX5GRMP458MA9FF1DACM",
"title": "Overall Rating Segmentation",
"type": "stacked_bar",
"xAxis": {
"id": "nhU1AZSE3XO2",
"label": "Pick a flavour"
},
"yAxis": {
"id": "count",
"label": "Number of responses"
},
"zAxis": {
"id": "eABwC8lYJx5Y",
"label": "What is your overall rating for this flavour?"
},
"description": "Segmentation of overall rating by flavour"
}
}
]
}How can we instruct AI to chart the “Number of responses” on the y-axis? How can we decide which questions are relevant to chart results?
We use AI to interpret the list of typeform questions, making correlations between those questions and finding a suitable way to present them to creators.
Here’s how we achieve this:
- We prompt the model with the typeform questions, supported question types, and visualization types.
- Through an OpenAI function call, we force a structured output from the model.
- We use the OpenAI evals framework to evaluate the output for its quality and structure.
- We achieve the final artifact, a JSON structure that resembles the previous example 2).
Here’s what our Open AI function definition looks like:
[
{
"name": "get_charts",
"description": "Create charts for data analysis based on the chart properties given as input such as type of chart, what to chart on various axis",
"parameters": {
"type": "object",
"properties": {
"00_step_by_step_reasoning": {},
"01_univariate_distribution_charts": {
"type": "array",
"items": {
"type": "object",
"properties": {
"id": {"type": "string"},
"title": {"type": "string"},
"type": {"type": "string", "enum": ["bar", "pie"]},
"x_axis": {"type": "string"},
"x_axis_id": {"type": "string", "pattern": "^[a-zA-Z0-9]{12}$", "description": "Only Field Id of the question is allowed in this field"},
"y_axis": {"type": "string"},
"y_axis_id": {"type": "string", "enum": ["count", "percentage"]},
"z_axis": {"type": "string"},
"z_axis_id": {"type": "string"},
"chart_order": {"type": "integer", "description": "ranking of how crucial is the chart for the analysis"},
"chart_description": {"type": "string", "description": "description of what is being displayed for non data stakeholders"}
}
}
},
"02_grouping_and_segmentation_charts": {}
},
"required": ["00_step_by_step_reasoning", "01_univariate_distribution_charts", "02_grouping_and_segmentation_charts"]
}
}
]Aggregate data for a report
Now that we have the report structure, we need to perform some aggregations on the collected responses for our customer’s typeforms to enable the report to chart the information that answers questions like: “How many people prefer the Orange flavor and rate it on a 3–4 range?”.
To achieve this we have a step in our platform data pipeline that pulls in the latest report structure for a given typeform and aggregates data according to the instructions in that report — the Report data aggregator component in the initial diagram.
AI-powered insights
Now that you’ve seen how we generate the report structure, we can share that we use the same strategy to create short AI textual insights for each chart. Here’s an example.
Keeping reports alive
As customers collect new responses, typeforms change frequently, requiring adaptation in two main areas:
- Users may modify questions in their typeforms by rephrasing, tweaking, or removing them.
- Live typeforms continually collect data until closed, or a certain limit is reached.
To accommodate these changes, our platform utilizes Typeform’s event-driven architecture. When a creator updates a typeform, we listen for changes and regenerate the report structure with AI, replacing the old structure with the new one. The report data worker refreshes the data, considering the latest user responses. This process repeats every 48 hours for each typeform.
UI & UX
Here’s the user journey for Typeform creators to access reports:
- Our platform assesses typeforms based on specific criteria, such as response count, language, and other more advanced criteria, to determine eligibility for a report.
- Users are alerted with an empty state when a report is generated. The report is presented within a new sub-section called “Smart Insights” in the existing results section. We display an empty state until the report is ready.
- Once ready, interactive reports become accessible on the same page and we also notify users via email.
- As mentioned, to keep data up to date we refresh the reports’ data every 48h.
Visualizations are interactive, with tooltips explaining chart sections and correlations with typeform questions. The UI remains lightweight, offloading data-heavy tasks to the server, including bucketing and limiting groups in complex visualizations. AI insights are summarized as bullet points at the bottom of each chart, and interactive visualizations are built using the visx library, leveraging existing expertise.
Learnings
- Artificial Intelligence is a powerful tool, but it’s not flawless. By carefully defining constraints, such as filtering out certain question types and focusing on specific chart formats and metrics, we can harness AI’s capabilities to generate meaningful and accurate reports tailored to our needs.
- When time-to-market is crucial, leveraging cloud-based AI solutions like OpenAI’s APIs can provide a significant advantage over self-hosted large language models (LLMs). This approach allows us to quickly leverage cutting-edge AI capabilities without the overhead of setting up and maintaining complex infrastructure in-house.
- Integrating AI providers into our workflows requires striking a balance between functionality and data privacy. To mitigate the risk of exposing sensitive personal identifiable information (PII), we implemented a PII question filter that excludes question types that could potentially contain such data. By working with aggregated metrics and statistics, rather than individual-level data, we can derive valuable insights while strictly preserving the privacy and confidentiality of our customers’ information.
- When deploying an AI-powered product at scale for a large customer base like Typeform’s, the inherent latency of AI operations can potentially impact the ability to deliver real-time interactions to users. To address this challenge, it becomes crucial to clearly communicate the user journey and provide timely notifications throughout the process.
What’s next?
While we have addressed the immediate challenges with structured data, we recognize the need for more robust solutions to handle unstructured data responsibly. As advancements in language model technology continue, we will explore feasible solutions such as deploying in-house language models for processing unstructured information while preserving privacy.
One of the natural enhancements we’re pursuing for our Smart Insights report is to introduce greater flexibility in adding visualizations and conducting further analysis. To facilitate this, we’re excited to announce the upcoming rollout of our ‘Ask Me Anything’ (AMA) feature, powered by AI technology. This innovative feature will empower users to pose questions in natural language; in response, they’ll receive answers, relevant data visualizations, and insightful observations. By leveraging the power of AI, the AMA feature will enable users to explore their data intuitively and conversationally, without the need for complex query languages or extensive technical knowledge. By articulating their questions using everyday language, users can unlock valuable insights, uncover hidden patterns, and gain a deeper understanding of their data. This seamless integration of natural language processing and data visualization capabilities promises to revolutionize how users interact with and derive value from their data. Whether seeking clarification, exploring hypothetical scenarios, or delving into intricate details, the AMA feature will be a powerful tool, empowering users to extract meaningful insights efficiently and effectively.
