Get to know our LLM Gateway — and how it provides a secure and reliable space to use generative AI

We built our own LLM gateway so employees could explore and use generative AI in a responsible way.

In November 2022, OpenAI released ChatGPT and changed the way the world interacts with artificial intelligence. Suddenly, anyone with an internet connection could access large language models (“LLMs”).

While LLMs have been available for some time, there are two key factors for why ChatGPT prompted more mainstream adoption: hardware improvements allowed LLM training on much larger datasets, and modeling the LLM through a chat user interface simplified how users could interact with the technology. As a result, any employee or team could add LLM to their toolkit and unlock more efficient ways of working.

It also exposed a problem for businesses: what happens to the proprietary company data or personal information shared with an external LLM? Is this data at risk of being leaked or accessed by third parties through the LLM?

We saw an opportunity, not only to leverage LLMs to further simplify our own workflows at Wealthsimple, but also solve key issues for companies eager to add an LLM to their internal toolkit. We believe this space holds massive unlocks across industries and wanted to give our teams the freedom to experiment and explore this technology, while upholding our privacy practices and commitment to protecting our clients’ data.

Introducing the Wealthsimple LLM Gateway

A few months ago, we launched our own internal LLM gateway which addresses two pain points:

• Security: Ensure the data shared cannot be used by third parties (some current LLMs usage policies state that data shared with non-API consumer products can be used to improve their own products)
• Reliability: Help prevent errors in production due to rate limits and overloaded servers

In addition, we saw an opportunity to address accuracy issues at the application level, like ‘hallucinations’ (when an LLM generates nonsensical outputs), minimizing the risk of LLMs providing incorrect information.

Our LLM gateway is comprised of two components:

• A re-creation of the chat frontend using OpenAI’s /chatCompletion endpoint. This restricts communication to the API and disallows OpenAI to use the data to improve their products.
• An API wrapper on top of OpenAI and Cohere endpoints (with plans to extend this to more providers in the future).

We track all the data sent externally (from both the frontend and through the API) and apply a series of heuristics to redact potentially sensitive information. Internally, the redaction step is enhanced by a PII removal model we developed in-house.

Since launching internally in April 2023, our LLM gateway has made over 72,000 requests to OpenAI and Cohere. Use cases include code generation, content editing and general question asking.

Accessing LLMs responsibly

AI and applications like LLMs have enormous potential for changing the way we research, build and innovate. Which is why we built the Wealthsimple LLM Gateway as a productivity tool and playground for our employees to further explore this technology in a responsible way.

For those interested, the Wealthsimple LLM Gateway is available to anyone looking to solve similar challenges and can be found on Github.

Written by Mandy Gu, Engineering Manager at Wealthsimple, in collaboration with Diederik van Liere and Sam Talasila.

Wealthsimple offers forward-thinking financial tools used by over 3 million Canadians. “Maker Stories” is an inside look at how we get things done. Interested in joining our team? Visit our “Work With Us” page to learn more and view open roles.

The content on this site is produced by Wealthsimple Technologies Inc. and is for informational purposes only. The content is not intended to be investment advice or any other kind of professional advice. Before taking any action based on this content you should consult a professional. We do not endorse any third parties referenced on this site. When you invest, your money is at risk and it is possible that you may lose some or all of your investment. Past performance is not a guarantee of future results. Historical returns, hypothetical returns, expected returns and images included in this content are for illustrative purposes only. Copyright © 2023 Wealthsimple Technologies Inc.