AI At Doctrine
Our first LLM-based features are now released to 100% of our customers! The journey has been incredible, starting six months ago when we formed a task force with the goal of releasing our first GenAI-based feature to last week, when we achieved this goal by rolling out three features to more than 13.000 customers.
I’d like to take this opportunity to reflect on the use of AI at Doctrine.
AI, particularly its NLP branch, has been central to Doctrine from the very beginning. It has been and continues to be crucial in building our most valuable assets, such as:
- Our Legal Graph: A knowledge graph that connects all legal documents and entities, bringing context and intelligence to the platform.
- Our Search Engine: Especially our query understanding workflow, which interprets users’ true intent to retrieve the most relevant documents.
- Our Recommendation Engine: Matching embeddings that represent users and all the legal content available on the platform to provide users with reading recommendations on their subjects of interest.
In developing these assets, we have been early adopters of various Machine Learning models, from Recurring Neural Networks to Transformer architectures in recent years. We currently have 30 of those models in production and counting.
More recently, with the advent of LLM models, new opportunities have emerged:
- Automating and enhancing internal tasks such as dataset or benchmark creation.
- Accelerating iteration on existing features by relying on LLMs instead of training our own models.
- Most importantly, bringing to life opportunities that were previously out of reach without LLMs.
Initially, we decided to focus on the last point and work on two key opportunities:
- The ability to summarize legal documents, including court decisions and enterprise deeds.
- The ability to answer questions on legal matters and enterprises.
For the first opportunity, we focused heavily on prompt engineering, working with legal experts both for prompting and benchmarking results, tailoring the prompts to various scenarios. For example, the best prompt for court decisions might differ from one court to another.
For the second use case, we implemented our own Retrieval Augmented Generation (RAG) architecture. This architecture relies on Doctrine’s state-of-the-art legal search engine and legal graph, incorporating multi-stage reasoning, embedding-based reranking, and post-processing to provide answers with clear references to sources.
In both cases, one challenge has been designing benchmarks to measure the impact of changes on the quality of the results. Doctrine’s legal experts have been instrumental in this process.
Following Doctrine’s “Release Early, Release Often, and Listen to Your Customers” value, we chose not to spend too much time beforehand on selecting a particular LLM. Instead, we decided to use either GPT-3.5 or GPT-4, depending on the complexity of the use case and the required prompt size, as these were the predominant players in the field. This approach allowed us to quickly develop a first version and put it in the hands of users to iterate and improve. This rapid development cycle enabled us to reach a point where we could step back, survey alternative LLMs, and decide if they offer a more impactful cost-to-result ratio. The benchmarks established during the earliest stages of the project allow us to proceed confidently without the fear of introducing regressions.
This is only the beginning and I’m very excited to see what we are currently working on released more widely in the coming months!
