How we built our first loan application chatbot using LLMs: Part 1

At Funding Societies, our mission revolves around providing substantial support to small and medium-sized enterprises (SMEs). We consistently face these issues:

• Promptly addressing visitor inquiries regarding our lending products.
• Gaining a better understanding of the SMEs through follow-up questions when visitors reach out.
• Offering relevant product suggestions based on the information provided, along with an easy link to apply right away.

To solve these issues, we utilised Large Language Models (LLMs) to develop an AI chatbot, Shane (SME Hub, Aligning Numerous Endeavours). Available across all our text-based omni-channel support, Shane assists with financing and investing queries whenever needed. Besides answering questions, Shane evaluates financing eligibility and guides customers through initial application steps. It learns from various sources like our websites, helpdesk, and internal documents to provide accurate and prompt responses.

This is a 2-part series of how we made it happen:

• Part 1: This article, suitable for audiences from different backgrounds, provides an overview of the project’s inception, the critical decisions made along the way, and its impact.
• Part 2: Tailored for engineers, this segment will offer a technical deep dive into how we built the entire stack.

The Inception

Our journey began in late 2022:

• Exploration: The engineering leadership started exploring the potential of generative AI, which was gaining traction in the tech community. This exploration was aimed at identifying how AI could be harnessed to enhance our Financial Technology (FinTech) operations.
• Hackathon to Prototype: In mid-2023, a hackathon event was organised, serving as a catalyst for our engineers to build the initial Proof of Concepts (POCs). The innovative solutions presented during the hackathon showed promise and laid the foundation for what was to come.
• POCs to Production: The POCs didn’t just stay as prototypes. They captured interest across the board, leading to a decision to move towards productionising the solutions.

Management Decisions

• Experimental Undertaking: This project was experimental, yet it was seen as a high-leverage activity by the executives. Given the manual-intensive nature of FinTech processes, the promise of streamlining operations using AI was an appealing prospect.
• Support from Above: The executive sponsorship provided the necessary backing and resources, ensuring that the project received the attention and support required to move forward.
• Team Composition: A nimble team of skilled, independent, and entrepreneurial software engineers was assembled. The aim was to keep the core team small to reduce friction and expedite decision-making processes.

Technology Decisions

• Build vs Buy Dilemma: A pivotal decision on our journey was whether to build the AI solution in-house or opt for an off-the-shelf product. One of the challenges we faced was the business serves two very distinct segments of customers — SMEs and Investors. Each segment has unique needs and requires a different approach. An off-the-shelf product might not cater to the diverse requirements of both segments effectively. After evaluating the need for a tailored solution that works for all parts of the loan lifecycle, we found that building the solution ourselves was the most suitable approach.
• Built-By-Agency vs Built-In-House Dilemma: Recently, many new chatbot agencies have popped up, each offering their rendition of “enterprise ChatGPT” solution. The tech behind this is changing fast with new features being added regularly. We wanted to launch our chatbot quickly but also ensure it would stay relevant as technology advances. We believed the best way to do this was to understand the AI ecosystem ourselves and build the chatbot in-house instead of relying on an agency. This way, we could control the source code and add new features when needed, without waiting on a third party.
• Choosing an LLM Provider: Among the wide array of LLM providers, we selected OpenAI, owing to its superior experience and quality. Additionally, a collaboration with Microsoft was forged to mitigate risks and avail private cloud access along with other generative AI features, providing a balanced and robust technological backbone for our project.

Success Criteria

Three-month Target: As an experimental project, we aimed to achieve a measurable impact within a three-month period.

By focusing on the upper funnel of the loan lifecycle, we aimed to bolster visitor engagement and lead generation. The broader objective was to enhance user experience while also driving an increase in loan disbursement volumes through this initiative. We track a few categories of metrics to ensure we are on the right path.

Customer Experience Metrics — Evaluating the performance and effectiveness of our chatbot was crucial. The metrics included:

• Response Time and Customer Satisfaction Score (CSAT): These metrics were gathered directly from our customer messaging tool, which keeps track of response durations and collects customer feedback post-interaction.
• Quality of Response: This metric was assessed manually by our Customer Experience team, who reviewed a sample of chatbot interactions to evaluate the accuracy and helpfulness of the responses provided.

Sales Metrics — To gauge the chatbot’s impact on our sales, we focused on:

• Enquiry-to-Clicks Ratio: This metric measures how many inquiries lead to clicks on provided links, indicating a customer’s intent to proceed with an application or share contact information. It helps assess the chatbot’s effectiveness in guiding customers through initial loan application steps.
• Contribution to Loan Disbursement Volumes: While guiding customers through the application flow, we utilised shortened links embedded with UTM codes. This allowed us to track the user journey from inquiry to action, giving us insights into the effectiveness of the chatbot in driving sales actions.

We held weekly reviews of these metrics for a real-time grasp on Shane’s performance, swiftly pinpointing areas for improvement. This nimble approach allowed for data-driven adjustments, keeping us on track towards our three-month target. Monthly metric aggregation gave a broader impact view, aiding strategy refinements for upcoming weeks. Through careful metric collection and analysis, we maintained a structured monitoring approach, aligning our efforts to boost customer engagement and sales performance.

Risk Mitigation

• Early Engagement: Engaging business stakeholders, along with risk, compliance, and security teams from the outset was crucial. Their input helped in shaping the product and ensuring it adhered to the necessary standards and regulations.
• Quality Assurance: Multiple rounds of manual QA were carried out to collect feedback and fine-tune the system. This was then evolved into a structured testing framework to continuously monitor and ensure the accuracy and reliability of Shane.
• Model Hallucination Mitigation: Large Language Models like the one powering Shane are prone to hallucination, where they might generate incorrect or nonsensical information. To mitigate this, we put measures in place to monitor the generated responses and correct any inaccuracies, ensuring a trustworthy user experience.
• Safety Measures: A kill switch was integrated to allow a quick fallback to human support when needed. Moreover, clear disclaimers were put in place within our chat communication to manage expectations and mitigate any risks associated with incorrect information dissemination.

Impact Achieved

After a month of deploying Shane to a limited audience (around 15% of our Malaysia market), we observed these impacts:

Enquiry-to-Clicks Ratio: We noticed a remarkable ratio with an average of 74.86%. This metric demonstrates a strong engagement and a high level of interest among the users, encouraging them to take the next step in their financing journey.

Contribution to Loan Disbursement Volumes: Shane’s integration showcased a solid conversion rate from leads to disbursement at 8.5%, demonstrating that employing AI did not compromise on lead quality.

• Before Shane’s integration, inquiries were filtered based on certain criteria and then passed on between teams for further follow-up. However, there was no direct guidance provided to customers on the loan application, and a gap existed between inquiry and follow-up.
• With Shane now part of the process, customers receive immediate, high-quality responses to their inquiries and are guided through the initial steps of the loan application process. This continuous engagement, available 24/7, not only maintains customer interest but also facilitates a smoother transition from inquiry to application, all while upholding the quality of leads generated.

Round-the-Clock Support: With Shane, we can provide 24/7 customer support, and the response time was dramatically reduced to an average of 15 seconds. This ensured that customers could get assistance whenever they needed.

Quality and Satisfaction: The quality of responses was maintained at a high standard with over 95% accuracy. Moreover, customer satisfaction hit the ceiling with a 100% satisfaction rate, and notably, there has been no dissatisfied customer escalation so far.

Code Less, Engineer More

At Funding Societies, we embody the ethos of “Code Less, Engineer More” through projects like Shane. This initiative wasn’t merely about writing lines of code but engineering a tool that would have a real, positive impact on small and medium-sized enterprises (SMEs). It’s about utilising technology to solve practical challenges, ensuring our efforts translate into tangible benefits for our customers.

Through this article, we hope to provide a glimpse into the journey of launching an LLM project. From the critical decisions of building in-house versus outsourcing to assembling a focused team and setting clear, measurable targets, each step is crucial for the success of such a project. The learnings shared here could serve as a guide for those looking to embark on a similar venture within their organisations.

In the upcoming Part 2 of this series, we’ll dive into the technical side of things. We’ll discuss the challenges we encountered, the solutions we crafted, and how we improved Shane over time.