Democratize Data and AI in Financial Services with Databricks AI Functions & Unity Catalog
Author: Pavithra Rao, Delivery Solutions Architect @ Databricks
Introduction
In the rapidly evolving financial services sector, integrating Generative Artificial Intelligence (GenAI) and large language models (LLMs) is heralding a new era of innovation and efficiency. This shift is notably marked by the democratization of Data and Artificial Intelligence (AI), powered by innovative AI functions that promise to revolutionize how financial institutions operate.
One of the core challenges in Fintech has been making AI capabilities accessible to a broader range of business users. Traditionally, AI in Fintech applications has been constrained to chat-based interfaces designed for analysts and business users, with a limited focus on utility libraries that could handle ad-hoc summarization, translation, classification, and more.
However, the introduction of Databricks AI functions is set to change this landscape. By providing these utilities through simple SQL, AI and analytics are now woven into the fabric of every business analyst’s daily tasks, freeing data science teams to focus on complex AI projects instead of being bogged down by routine tasks that can be easily managed with a simple SQL service.
AI functions are particularly transformative in facilitating access to previously inaccessible information, empowering analysts to make more informed decisions, manage risks, and maintain a competitive edge through data-driven innovation. This is achieved by embedding AI directly into analysis workflows, thereby streamlining the process of leveraging AI for business insights. Most importantly, Unity Catalog can be leveraged as a single unified governance layer to manage data, functions (as well as AI functions), and models in single place, making the productization of AI-embedded products scalable for the Enterprise.
Unity Catalog — single catalog for Data and AI
Unity Catalog serves as the governance platform for all data products from tables and materialized views, to functions/code, models, and even open data shares. This catalog, combined with AI functions and Databricks Mosaic Model Serving, empowers enterprises to create end-to-end data products all governed in a single place with unified ACLs.
The ai_query Function: A Game-Changer
The ai_query function is a built-in Databricks SQL AI function enabling analysts to easily access ML models directly from SQL queries. This function allows the serving of Machine Learning models and Large Language Models using Mosaic AI Model Serving, and querying them without needing to leave the SQL environment. Such an approach not only simplifies the querying process but also enables seamless integration of AI capabilities into financial analyses, including the crucial area of financial fraud detection.
Financial Fraud Detection
The application of the ‘ai_query’ function in analyzing transaction data for potential fraudulent activity illuminates its value. By combining rule-based business logic with ML models, sophisticated systems can be developed that identify anomalies and patterns indicative of fraud. This method significantly enhances financial institutions’ ability to detect and mitigate fraud risks efficiently.
Consider a ‘detect_fraud’ function created using Databricks SQL APIs, which examines transactions based on specified criteria — such as transaction amounts exceeding a certain threshold or unauthorized overdrafts — and labels them according to the level of risk they represent. The culmination of this process not only offers a streamlined method for identifying potential fraud but also enriches the decision-making toolkit available to financial analysts.
The above code creates a function named `detect_fraud` using simple Databricks SQL APIs. The function returns a string indicating the level of risk associated with the transaction.
The function uses a `CASE` statement to check the following conditions:
* If the transaction amount is greater than $10,000, the transaction is labeled as `suspicious`.
* If `isUnauthorizedOverdraft` is 1 the transaction is labeled as `high_risk`.
* If none of the above conditions are met, the `ai_query` function is used to predict the fraud label by querying ML model name `fraud_label` registered in Unity Calog and served using Mosaic AI Model Serving endpoint
The function and the ML model used in the function are registered in Unity Catalog, which provides centralized governance for all Data and AI assets.
End users (Financial analysts or Business analysts) can easily use this function with a simple select statement by providing the account_id details in the select statement. This will apply the `detect_fraud` function to each row in the `transactions` table and return a new column named `fraud_label` indicating the level of risk associated with each transaction.
Furthermore, the utility of AI functions extends beyond backend analytics. These functions can be integrated into front-end applications with minimal effort, offering real-time analytics capabilities. This is particularly beneficial for financial institutions seeking to enhance their customer experiences through immediate risk analysis and fraud detection responses.
More AI Functions
In addition to ‘ai_query’, Databricks offers a suite of AI functions through its Foundation Model APIs, enabling tasks like forecasting, sentiment analysis, classification, and more. This comprehensive toolkit underlines the potential of AI functions to transform the financial services industry by providing an easy-to-use interface for a wide range of AI applications. It simplifies the querying process for models, supports multiple model types, and offers flexible request and response formats, thereby allowing analysts and data scientists to concentrate on crafting innovative solutions to business challenges.
For more code examples on how to leverage out-of-the-box Databricks AI functions check out the Databricks Demo Centre tutorial
Governing Your AI Systems
One cannot discuss the advancements in AI capabilities without considering the governance of AI systems. As organizations rapidly adopt state-of-the-art foundation models to empower their data practitioners, the importance of centralized governance frameworks becomes increasingly apparent. Registering AI functions and ML models in Unity Catalog ensures that all Data and AI assets are centralized, governed, secure, and compliant with relevant regulations. Further to enable customers to have a unified interface to easily manage, govern, evaluate, and switch models Databricks announced new capabilities in Mosaic AI Gateway
Mosaic AI Gateway for Centralized AI Governance
Mosaic AI Gateway sits on Model Serving to enable rate limiting, permissions, and credential management for model APIs (external or internal). It also provides a single interface for querying foundation model APIs so that customers can easily swap out models in their systems and do rapid experimentation to find the best model for a use case.
Key Features of Mosaic AI Gateway
- Rate limiting: Prevents abuse of model APIs and ensures that usage is within acceptable limits
- Permission management: Controls access to model APIs and ensures that only authorized users can query models.
- Usage tracking: Tracks who calls each model API and captures what data was sent in and out.
- Inference tables: Captures data sent to and from model APIs, enabling auditing and chargebacks.
- Mosaic AI Guardrails adds endpoint-level or request-level safety filtering to prevent unsafe responses and even PII detection filters to prevent sensitive data leakage.
- system.ai Catalog: Provides a curated list of state-of-the-art open source models, enabling customers to deploy and fine-tune models easily.
Unity Catalog empowers everyone to create AI functions while providing much-needed clarity and transparency. With its centralized governance, fine-grained access control, and robust auditing and lineage capabilities, it’s now easier than ever to harness the power of AI while maintaining control and visibility. This means that AI functions can be utilized to their full potential, with clear explanations and accountability every step of the way.
By leveraging Unity Catalog and Mosaic AI Gateway, organizations can ensure effective AI governance, prevent issues like data leakage and cost overruns, and enable rapid model experimentation and deployment. To learn more about Mosaic AI gateway check out blog post!
Conclusion
Databricks AI functions and AI gateway are powerful tools that unlock the potential of AI in financial services. By simplifying the process of querying traditional ML models, LLMs, or simple rule-based logic and providing a unified interface for AI governance, these innovations enable analysts and data scientists to develop creative solutions to business problems while ensuring the secure, compliant, and cost-effective use of AI systems!
