Get to know our LLM Gateway — and how it provides a secure and reliable space to use generative AI
How we built our own LLM gateway so employees could explore and use generative AI in a responsible way.
In November 2022, OpenAI released ChatGPT and changed the way the world interacts with artificial intelligence. Suddenly, anyone with an internet connection could access large language models (“LLMs”).
While LLMs have been available for some time, there are two key factors for why ChatGPT prompted more mainstream adoption: hardware improvements allowed LLM training on much larger datasets, and modeling the LLM through a chat user interface simplified how users could interact with the technology. As a result, any employee or team could add LLM to their toolkit and unlock more efficient ways of working.
It also exposed a problem for businesses: what happens to the proprietary company data or personal information shared with an external LLM? Is this data at risk of being leaked or accessed by third parties through the LLM?
We saw an opportunity, not only to leverage LLMs to further simplify our own workflows at Wealthsimple, but also solve key issues for companies eager to add an LLM to their internal toolkit. We believe this space holds massive unlocks across industries and wanted to give our teams the freedom to experiment and explore this technology, while upholding our privacy practices and commitment to protecting our clients’ data.
Introducing the Wealthsimple LLM Gateway
A few months ago, we launched our own internal LLM gateway which addresses two pain points:
- Security: Ensure the data shared cannot be used by third parties (some current LLMs usage policies state that data shared with non-API consumer products can be used to improve their own products)
- Reliability: Help prevent errors in production due to rate limits and overloaded servers
In addition, we saw an opportunity to address accuracy issues at the application level, like ‘hallucinations’ (when an LLM generates nonsensical outputs), minimizing the risk of LLMs providing incorrect information.
Our LLM gateway is comprised of two components:
- A re-creation of the chat frontend using OpenAI’s /chatCompletion endpoint. This restricts communication to the API and disallows OpenAI to use the data to improve their products.
- An API wrapper on top of OpenAI and Cohere endpoints (with plans to extend this to more providers in the future).
We track all the data sent externally (from both the frontend and through the API) and apply a series of heuristics to redact potentially sensitive information. Internally, the redaction step is enhanced by a PII removal model we developed in-house.
Since launching internally in April 2023, our LLM gateway has made over 72,000 requests to OpenAI and Cohere. Use cases include code generation, content editing and general question asking.
Accessing LLMs responsibly
AI and applications like LLMs have enormous potential for changing the way we research, build and innovate. Which is why we built the Wealthsimple LLM Gateway as a productivity tool and playground for our employees to further explore this technology in a responsible way.
For those interested, the Wealthsimple LLM Gateway is available to anyone looking to solve similar challenges and can be found on Github.
...
Written by Mandy Gu, Senior Engineering Manager, in collaboration with Diederik van Liere, CTO and Sam Talasila, LLM Manager.