Generative AI’s Power: Discovering the “Sweet Spot” of its Applicability in the Legal Industry
In the context of generative artificial intelligence (AI), I often wonder whether we, as an industry, are asking the right questions. In our rush to quickly assemble panels, products, and webinars, have we stopped to really think about where this tool is currently and how it may fit into the overall scheme of things? Or, have we rushed to judgment on one end of a spectrum or another? Amidst heated debates and viewpoints, it is crucial to take a moment to delve deeper into the practical applications of generative AI as a tool and explore its true capabilities in the legal industry. With nearly 25 years of industry experience, including a specialization in legal data processing, I attempted to do just that by embarking on an exploration with the goal of only trying to learn more so that I can better understand generative AI’s potential in the legal field.
In this post, I dive into the practical applications of leveraging the OpenAI API in the legal industry. Specifically, I will explore how the API could build an instructive Chatbot, summarize text-based responses, distill a summary from a case opinion, and create a knowledge graph that extracts valuable data. The overall message should not be doom and gloom about replacing humans, nor over-hype that ignores the current limitations of this tool, but instead, identifying certain discreet tasks where this tool can be leveraged to make certain parts of a process easier or as a substitute for rote human-performed tasks where using humans to do the task would not be practical.
General Background
In the legal industry, there has been much debate about the potential impact of technologies like ChatGPT on the roles of lawyers, paralegals, and other legal professionals. While I don’t consider myself an “expert” necessarily, I am more of a generalist, I have led teams handling text-based data extraction and processing, I have worked on machine learning initiatives, and I have pursued self-study in data science, machine learning, and natural language processing. During this AI frenzy, I didn’t want to settle for merely consuming articles and posts, and I am generally skeptical of extreme positions of any kind. Instead, I wanted to explore and experiment for myself to better understand which tasks generative AI excelled at, where it struggled, and how it could enhance existing processes in a practical way. I did not do so with the intention of discovering an earth-shattering solution, but rather to better understand how to leverage the OpenAI API to simplify certain tasks done within an overall process.
In my research, I discovered a now-(mainly) private YouTube channel by David Shapiro. Luckily, I watched his many instructional videos before they were taken down. These tutorials taught me how to implement various processes using prompts in the API. Through my exploration, I found there may be a real “sweet spot” where generative AI tools can help with laborious tasks difficult to address through other coding methods.
These tasks require an account with OpenAI and an API key. They were written in Python using Notepad++ and run in the terminal on my machine. These tasks require one or more Python files and .txt files for the API key and prompt. For the Python code, I mainly used what David Shapiro provided in his tutorials, though I experimented with some of it or applied the code in different circumstances. More than the Python code, the prompt files are where I further experimented with creating guardrails and providing more explicit instructions.
The following, therefore, goes through four of my so-called experiments. By doing so, I intend that the reader have a better understanding of how a generative AI API might be leveraged to do certain tasks in the legal industry and beyond.
1. Building a Basic Chatbot that Leverages the OpenAI API
The first experiment I undertook was building basic chatbots that leveraged the API. One was a Legalytical chatbot meant to mimic a discussion one might have with me, another was a chatbot that could discuss Minecraft with my son [controlled test conversation with myself typing], and a third was a chatbot that could answer questions a patron might have about a local museum (such as current exhibits, operating hours, and other basic information).
These chatbot-type tasks can be performed by a human, but most companies, nonprofits, and small businesses do not have the time or resources to staff multiple customer service representatives, 24 hours a day. This is also a task currently performed by coded chatbots, but those chatbots typically only answer very specific and predetermined questions and cannot replicate a more human-like conversation.
Concerns about such a use would be whether the bot would make up or provide inaccurate information if it did not have the answer. And, I found it did both so I experimented with the prompt to establish and keep it within certain guardrails. I was also concerned about whether someone chatting with the bot could steer it into dicey territory that could tarnish the reputation of the business or organization it helped, so I instructed the bot to only respond to questions on the predefined topics.
Thus, most of my time was spent experimenting and revising the prompt to prevent the ramifications of these limitations. As you can see, I kept it on topic and instructed it to answer that it didn’t know rather than fabricate information. Even though it is possible to work with its current limitations, someone creating a chatbot should spend significant time to make sure the bot performs as expected. But, I do believe the potential is there for the expanded use of chatbots to provide information on various topics in a way potentially far more conversational and flexible than ever before, while also not requiring significant resources that most businesses and smaller organizations do not have.
2. Summarizing Text-based Reviews, Survey Results, or Similar
Whether you are a product manager, survey taker, or business owner, you would be lucky to have extensive feedback to refine your product or business. But working with product reviews and other text-based responses can be overwhelming. It requires reading and bucketing the responses so you know which are the most important and which are more one-off suggestions. To do so, these responses need to be reviewed and distilled down into actionable work, focused on the most critical areas.
A human can read through reviews, and I have written before about how one can make sense of them, but if there were truly thousands, it would be better to leverage a tool that could iterate through the reviews and summarize them into a form that gets to the most critical issues and improvement points.
For this experiment, I used a publicly available dataset that David Shapiro used in his tutorial (Amazon Kindle reviews available on Kaggle). I used his code that randomly samples the reviews and played with the sample number (for example, 25 random samples done 10 times). I also experimented with the prompt to get it to summarize those reviews and create Agile-style user stories for actionable product improvements.
Although usually products and companies suffer from having too little feedback versus too much, this code and prompt could be changed slightly to handle many scenarios where you need to distill text-based responses into common themes. I could see changing the code and prompt to handle survey responses, industry guidance, or other instances requiring the categorization of large amounts of textual data into useful nuggets.
The limitations here would be confirming that the tool is using a proper sample size that identifies and summarizes the most critical points. I would spend significant time at first refining this process to make sure it is operating as expected. Still, the API could likely be leveraged reliably to weed through a lot of text and get to the heart of the matter, so humans can move on to solving these problems, rather than figuring out what problems there are to solve.
3. Summarizing Court Opinions and Other Legal Documents
Attorneys love to be kept aware of what is happening. There are multiple current awareness-type services out there to summarize opinions, legal complaints, and other legal filings. But to do so requires extensive time to find the notable documents, read them, and then summarize their contents. These tasks require either an extensive editorial team or coding the summary in a reliable and consistent way, an extensive dataset, and significant trial and error.
This process could therefore benefit from having generative AI create the summaries if it can do so reliably and the token limits continue to increase. The overall process would be to obtain the document (usually in pdf form from government websites), convert that document into a txt file, then send those files to the API to summarize them. In this experiment, I used Supreme Court opinions, converted them to .txt files, broke them into chunks to get around the token limit, and then prompted the API to summarize the opinions. The following is an example of the result it produced.
This case is well outside of any area I practiced in, and may not be the best summary, but it is demonstrative of how a current awareness process could leverage an API, especially after more experimentation to confirm accuracy. Even with its current limitations, this will be an important tool that will expand current awareness while automating the most tedious parts of the process. And, by thinking about it as providing a summary from which someone can access and read the entire legal document should they so choose, it is relatively low risk if it got it wrong. Any humans working in this space could focus instead on what these opinions and legal documents mean and how they might affect legal professionals and their clients, rather than expend energy on summarizing basic information.
4. Data Extraction and Knowledge Graph Creation
In his tutorials, David Shapiro also showed how data could be extracted from a court opinion and used to create a knowledge graph. When I saw this, I immediately thought about how useful generative AI could be within the realm of legal publishing and technology to leverage as the backbone for a citator or headnote-type product that required significant data extraction from court opinions and other legal documents. This could also be used for all kinds of legal analysis products that depend on identifying the citation and its context from various legal documents.
Extracting citations and text from legal documents manually to create such a product is exceptionally tedious work requiring significant resources. And, coding this extraction is possible to an extent, but it is challenging to code all the ways that citation appears and to also extract the surrounding text and save it in a format that would be useful for building additional products leveraging that data.
Now, this will not be easy and would require extensive experimentation to confirm accuracy and consistency, but I think that a generative AI API can extract data from text and save it into a knowledge graph that could then be leveraged for building other products, like how a human might manually extract these data. To test this, I took supreme court opinions, converted them to .txt files, broke them into chunks, and played with a prompt to create a knowledge graph that might be useful for a legal research product, but that would be exceedingly tedious for a human to do and difficult to code.
The results were inconsistent and would require extensive additional experimentation and perhaps the evolution of the technology to be reliable, but there is potential here. What was especially interesting is that it seemed able to extract and identify cites and even connect a short cite with a past mention of the full cite.
Conclusion
We are still in the early, discovery stages and need to make sure we stay focused on the right questions. While I have seen the transformative potential of AI in tasks that are both tedious for humans and challenging to code, there is much more to uncover. Significant exploration and experimentation are necessary to comprehend how generative AI can effectively and reliably help legal professionals and build legal products.
However, amidst this ongoing journey, AI remains promising in its potential to widen horizons, allowing for the development of more robust and sophisticated products in the legal field. By harnessing the power of AI, we can enhance efficiency, streamline processes, and unlock valuable insights from vast amounts of textual data, such as case opinions and legal documents.
To realize this potential, it is crucial for legal professionals, researchers, and technologists to continue collaborating, pushing the boundaries of what is possible. By combining domain knowledge with AI capabilities, we can shape the future of the legal industry, making sure the tools we develop are reliable, secure, and ethically aligned. Thus, while there is much more to discover and understand, the promise of generative AI in the legal industry is undeniable.
