How DecisionFacts leverages Generative AI to converse with unstructured content and unlock institutional knowledge with Google Cloud

Background

Over 80% of the information and data captured in the organization resides as unstructured data. While technology exists to extract data from text, video, images, charts, and embedded tables, and build a search layer to navigate these diverse data sources, the rate of adoption in the enterprise has been slow. There are 3 major reasons for the lack of prioritization of these projects:

• Time, effort, and the skillset to build an end-to-end system
• Ongoing operational challenges of a traditional data model, intent-driven search
• Lack of ROI measurement (such as total cost to productivity gains benefits)

The emergence of Generative AI has sparked interest by companies to retrieve customer, market, and product insights from existing knowledge management systems. Once trained, Large Language Models (LLM) have unlocked cost-effective methods to improve business efficiency. This transformation is possible because the technology is blended with an intuitive user experience. Users are excited to converse in natural language and interact with LLM-generated responses, which previously required significant technical expertise. At the same time, organizations are wary of the risks of enterprise-wide adoption without proper guidelines and safeguards.

DecisionFacts, a B2B startup, has a mission to ensure Responsible AI for decision-making. It understands the challenges of adopting LLM-based AI applications in business. Accordingly, DecisionFacts (DF) empowers enterprises to build and use AI Apps with integrated process management and governance. It abstracts the complexities of cloud infrastructure and models to deploy apps in a secured environment. In addition, DecisionFacts provides ready-to-use cutting-edge DF LLMs for business operations use cases around interpretation from charts, and images with corroboration for data-driven decisions.

This blog illustrates DecisionFacts’ approach to an enterprise use case using LLM modules on the Google Compute Platform (GCP).

Business Challenge

A global manufacturer has over 40,000 documents in a corporate knowledge repository — including standard operating procedures, performance reports, process control manuals, and more. These documents reside in multiple, enterprise knowledge management systems.

The company wanted to use Generative AI to revolutionize how they manage, retrieve, and share the wealth of knowledge from their internal systems. At the same time, the business unit was concerned with:

• Risks of sharing sensitive company information on a publicly hosted LLM.
• Since most of the documents had information in charts, graphs, and tables, the current LLMs were not able to extract business-related information such as insights or trends from charts.
• Factual response of the information and privacy/ access of certain data.

Consequently, the company was looking for an enterprise-grade application without large in-house development and operational overhead of managing different technologies.

DecisionFacts Platform and Approach

DecisionFacts (DF) provides a rich set of features for indexing data corpus without replication of data, customizing models with fine-tuning based on prompts and feedback loop on improving the accuracy of responses. The following core offerings of the DecisionFacts product was instrumental to meet customer needs:

Technical Overview

DecisionFacts stack runs on GCP cloud and leverages GCP Vertex AI’s Generative AI models for content embeddings and stores those embeddings as vectors into Vertex AI’s Match Engine ANN Service.

• Data Extraction and Indexing: The indexing module of DecisionFacts was connected to the customer’s different knowledge management systems, including Microsoft Sharepoint, to access the content. As part of this step, the module used a pipeline to first classify content between text and images/charts/graphs. The DF image module was pre-trained on a proprietary dataset on different chart types to generate responses from interpreting charts. All the data was processed and indexed on the customer’s own cloud instance. This satisfied the customer’s main concern of privacy and security of information.

DecisionFacts Index Module with image, text content extraction

• DF LLM with Integration to Data Systems: DecisionFacts LLM module accessed the indexed data from the vector database via a search inference service. The DF LLM included a chain of sub-modules for prompt engineering, retrieving search results from the database and embedding using PaLM2 to create a response. The search results not only provide the most relevant information based on the questions asked but also a reference to the document with page numbers, and images. This allowed transparency and audibility of where the response was generated from.

In many cases, the search result was generated from multiple sources and the system provides a reference of all documents. It also kept the context of conversation to furnish only the most relevant information when asking questions. Certain documents were labeled as sensitive and required permission to view the content. DF model mapped the permission labeling of the documents during indexing and provided appropriate responses on whether to show content or not.

In addition, the framework allows appending data from a dynamic query to connect to internal data systems such as BigQuery. The system stores the content of the response in an audit table, which is also used for checking the accuracy of the response through a feedback mechanism.

User Experience

• Conversation based on image content:The DF LLM module also addressed the customer’s requirement to get insights from charts, and images. The custom chart model of DecisionFacts interprets images and computes the values from figures, which may or may not have explicit data labels. This allowed users to ask questions not only from the text, but also more complex questions related to computation from charts, and trend analysis. The DF LLM combined the search results from text and images into a unified response to the UI.

Interpretation of charts, graphs, and combined embedded response

• Responses with Reference: The users interacted with an application using “DF Knowledge AI” App. This app was deployed on a Google Cloud instance that is dynamically scalable based on the number of users and search requests. The interface allowed users to ask questions, which sent the request to the LLM module and subsequently received the response with reference to the document. The DF Knowledge AI not only enabled interactions with a corpus of documents indexed but also gave the ability to users to search on a specific document (within the corpus) to get the relevant information. This ensured that the responses are factual and also allowed users to provide feedback on the response. The data collected from the feedback is used for retraining the model, fine-turning the prompts, and more.

UI for Document Search with computation from charts and references

DF Knowledge AI Conversation Feedback

DF accepts feedback from users based on the search response to improves the search. DF model retrieves raw response from vector database based on the relevancy, if the response given by the vector database based on similarities is not appropriate or information is not accurate, authorized users can provide feedback by like/dislike the response content or recommend for the change to modify the response content.

The reinforcement learning model based on the feedback loop mechanism is very useful for the users to validate the response and improvise the search response for better accuracy. Response content like & dislike feedback stores into Google big query along with content metadata. When a search retrieves a response from the vector database, the feedback details are also considered to improvise the scores.

Technical Architecture with Google Cloud Platform

To unlock the value for businesses to run LLM models and data extractions, DecisionFacts partnered with Google Cloud Platform (GCP) to build a system that is highly reliable and scalable. Leveraging the GCP services such as Google Kubernetes Engine, Virtual Machines, Google Buckets, Big Query, and more, DecisionFacts built the application and integrated it with various API services, not limited to Google Document AI, PaLM-2 and Google Cloud Datastore. Further, the fully packaged application is deployed on the customer cloud environment that runs as a Kubernetes cluster using a Google Cloud account.

DecisionFacts (DF) is a platform that runs on GCP with the following components:

• Projects for running LLM/ Indexing/ Data processing and tracking runs for prompt governance
• Integrated Notebook to build LLMs using transformer models, datasets on Tensorflow or Pytorch framework
• Storage UI for uploading files and storing system-generated files, logs
• Application for search, conversational interactions, or other Generative AI apps. The different apps are deployed on a server that can be accessed using the DecisionFacts interface or an API end-point that can be integrated into customer applications (such as existing Chatbot to Search UI).

The DecisionFacts control plane is responsible for authentication, and orchestration. It runs on Google Kubernetes Engine with runtime engines for Python, Spark depending upon the modules. The data is stored in Google private bucket, which is encrypted. The application uses GCP virtual machines for deploying the service. The data processing and indexing use a combination of high compute virtual machines and GPU instances in GCP. Additionally, incremental indexing of documents is done using a persistent Kubernetes cluster that runs multiple jobs in different pods.

The extraction pipeline reads the document based on the document type and creates a text document to write content. If the input source document detects images, the module runs the classification algorithm to categorize the image type charts, graphs, Tables, and other types of images. DF’s proprietary image processing model runs the OCR engine using Google Document AI to extract content based on the image classified type and generates a summary based on the input image and updates it into the document to index.

Before indexes into the Vector database, the DF pipeline module uses Google’s PaLM API embeddings — ‘embedding-gecko-001’ to convert texts into vectors and indexes into the vector database along with document information such as file name, category (Image, Text, Tables) as metadata along with the vector dense matrices.

DecisionFacts Knowledge AI Architecture using GCP cloud & PaLM API

The inference APIs run into GCP VMs for search service. DF’s Knowledge Search AI gets the input query as POST method as an inference API request from users. The following activities happen before sending a search response to the requested query.

• Search query converts as embedding using the same PaLM embedding model
• The search module hits the vector database with query embedding matrix data
• Using Cosine similarity, the vector database returns top K relevancies of content to the DF search module
• The retrieved raw response converts as a human conversational response using PaLM LLM and sends the response to the request.

Outcome for Business

The following are the benefits that users of DecisionFacts customers are highlighting:

• Self-service Research — The solution enabled quick turnaround for desk research instead of manually aggregating scattered information. The summarization and conversational aspects of the DecisionFacts Knowledge AI helped the organization to enhance the cross-pollination of the information instead of relying on dedicated training sessions.
• New Opportunities — Having an indexed corpus of internal data, the company plans to generate new content. For example, creating a new functional specification for a product based on current operating procedures. The platform provides the ability to build custom LLM modules and publish them for specific use cases.
• Data Security and Compliance — The DecisionFacts on-premise setup helped to mitigate any corporate policies around the use of Generative AI tools. Further, the ability to integrate with internal systems for access control solved the concerns of permissions or privacy.

Conclusion

Organizations considering Generative AI use cases with enterprise-grade security, have an out-of-box stack with inbuilt collaboration and governance from DecisionFacts using the Google Cloud Platform. DecisionFacts’ goal is to enable Responsible AI with Google as a trusted partner.