Teaching the Legal System to Fail Better

“Evolution forged the entirety of sentient life on this planet using only one tool, the mistake.” — Robert Ford, Westworld

Last week marks one year after I officially started working as an Innovation Fellow with the ABA Center for Innovation. Getting the opportunity to speak about my experience at the ABA Annual Meeting was something I never imagined I would have the opportunity to do. I truly cannot believe how much the inaugural group of fellows was able to accomplish. What follows is a story that I hope will provide some perspective on how the legal industry could learn to do better.

What would the rule of law in the United States look like if we started over and built it up from scratch? Would we require as much paper as we currently do? Would courts and government offices have chatbots that could help reduce costs by allowing people to navigate more effectively through the court system? Would we be fill out forms with the same pieces of information over and over again in order to verify we are who we say we are?

In many industries, technology has completely changed the way professionals and organizations operate. Turbo Tax has given us rules based automation of the tax code. Google Translate has figured out how to replicate the cognition that allows us to understand different languages, by taking a statistical approach and identifying patterns of language between translated documents and search result outcomes. The popular dating app, Tinder, has presented a viable mechanism for connecting individuals looking for relationships through a combination of location data, reputation information, and encrypted communication channels. And Wikipedia has created an outsourced network of 34.1 million users who have contributed over 100 million hours to the maintenance of a free and shared knowledge database.

Perhaps the most interesting thing about all of these developments is that they are incredibly complex and deeply personal. This begs the question, if other industries can do it, why can’t the legal industry move on from entrenched in procedures and rules that were not designed to optimize the unique strengths of the internet and computer science? The current build of the legal system is by no means perfect.

The legal system is not affordable. According to the 2017 Justice Gap Report, 86% of the civil legal issues faced by low-income Americans received inadequate or no legal help.

The legal system is unclear. Harvey Silvergate, a civil liberties lawyer from Boston, famously suggested that the average American commits three felonies per day.

Further, the legal system is not even that accurate. A 2017 study from the Urban Institute found wrongful convictions in around 11–12% of cases where physical evidence produced a DNA profile of known origin.

Compared to other professions, this puts the legal industry farther behind other industries failing to live up to the promise these technologies have brought to other industries. But what is to be done?

In the outside world, especially at startups and tech companies, when a program doesn’t work as it is supposed to or bugs to the program are identified, the bug is identified, new code is added to the codebase and some sort of update or new version is introduced. In order to develop a model that accounts for all these known errors in the legal industry, there needs to be an update to the operating systems that are not working.

The legal system needs to learn how to fail better. That is to say, the legal system and those operating within the legal system need to learn how to own up to failures, diagnose them quickly, and leverage them so that we can understand how not to fail in the future.

Over the rest of this article, I ruminate on the concept of how failure can be beneficial, summarize the successes and failures of a recent project I undertook as a fellow with the ABA Center for Innovation, and offer up the idea of a community of those interested in legal tech, specifically designed for the internet, as a possible mechanism to help unlock more of the potential of the legal system.

I. Using Failure to Find Success

In the first week of my law school orientation, one of the Deans of my school told a courtroom full of baby 1Ls something that made absolutely no sense to me at the time. She said, “Plenty of people have failed their way to success.” As I sat there, the idea sounded incredibly ridiculous. Failing to succeed seemed impossibly paradoxical and terribly inadvisable, given we were being introduced into an environment where success was almost exclusively determined based on the result of a handful of tests.

Even in the broader context of the legal profession, the idea seemed ridiculous. The legal system is an area in which even the smallest mistake can be penalized with some of the worst consequences imaginable. This type of failure can cause a client to go to jail, it can cost a client lots of money, and it can secondarily result in damage to your reputation as a professional. It is this same type of failure that meant a party in a labor dispute case from Maine lost $5,000,000 because of the lack of an Oxford comma.

After letting it simmer in my brain for a few months and especially as I sit and type this out right now, the idea of failing to success is actually one of the most empowering things I have learned. Redefining failure in such a way that turns it into success takes agency over a situation where performance was less than expected and leverages that as a stepping stone to some different and greater success. And while this iterative process focused in the direction of improvement is good and desirable, it does have its fair share of challenges.

An unfortunate characteristic of failure is that it has the ability to deter people from putting forth more effort because failure can be really embarrassing. I know this because I have failed quite a bit. Some of my biggest and most extravagant failures occurred after I decided to go to law school.

At the end of my first year of law school, my class rank was 163 out of 167. My GPA was so law that I was not even permitted to complete the application to be on law review.

Like nearly everyone who goes to law school, following graduation, I sat for the bar exam. I studied more than I thought I could study and, ultimately, I failed. I missed passing the bar exam by approximately three multiple choice questions. It was during this time, when I found myself in one of the darkest and deepest holes that I’ve been in. It was at the bottom of that hole when I cam across another quote from Henry Ford:

“Failure is simply the opportunity to begin again, this time more intelligently.”

For me, this provided more perspective on how to fail better. Ultimately, I figured, if I am going to make the commitment to do something worthwhile, I will have to be perpetually under construction, learn from my mistakes quickly, and do everything I can to avoid them. And this is what has worked for me. In spite of all of the seemingly damning failures I went through during that period that suggested this was not the right field for me, I have been able to do some things I never would have imagined and things that honestly would not have been possible without having gone through those failures.

After my terrible first year of law school, when I started the“fail your way to success” period of my life, I found some noteworthy successes. I made it onto the Dean’s list for my last two years of law school. I got connected with Legal Hackers and founded a chapter in Kansas City with the help of Ellen Suni, Michael Robak, Tony Luppino, Dazza Greenwood, Tony Lai, and Jonathan Askin. I authored and submitted an article that was accepted by the same law review publication I was unable to apply for.

After failing the bar, when I started the “fail better” period, I found my way into even better fortunes. I got connected to a full time job at RiskGenius, an artificial intelligence startup that permits me to build my legal and technical knowledge on a daily basis. I got the opportunity to work as a teaching assist for an interdisciplinary Computational Law course at MIT. I was recognized by Legal Tech News as one of eighteen millenials changing the face of Legal Tech.

I say all of this because I think it demonstrates a couple of points that need to be understood if the legal industry is going to meaningfully advance.

On a personal level, if you want to do something, never give up. You are more than your class rank or the scores you get on some tests. Take your lumps, figure out how to do better, and do it. You will surprise yourself.

Learn to enjoy the process of doing things better than before and take solace in the fact that important and innovative opportunities will always fail some way. Nokia used to be a paper mill. YouTube started out as a dating website. Here, the important thing to remember is to fail in a positive way. Change your strategy. Fix your errors. Keep moving forward. In the next section, I walk through what some of this looks like, as I explore some of the nuance of one of my latest failures.

II. The Rise and Fall of the DFENDR Project

Over the last year and a half or so, I worked on the DFENDR Project as an Innovation Fellow with the ABA Center for Innovation. Like lots of other things I have been involved with, the project was a failure in a lot of ways.

I applied to the center outlining a project that would function something like a sharing economy app for wrongful conviction investigations. The idea was that the different pieces of a wrongful conviction investigation could be atomized into discrete components. This new model of working is something that is happening all around. Uber drivers login to the app and pick people up. Workers for Amazon Mechanical Turk (MTurk) sign in and start completing Human Intelligence Tasks. In the short term, this would take advantage of the idle capacity of those within the legal system who need work. In obscure or difficult cases, it helps leverage the institutional knowledge of specialist contributors. And as a network, it helps monitor performance as it unfolds to help aggregate information to improve policy insights.

The way it as specced out initially, law students could gain experience with e-Discovery software by logging in and tagging documents as responsive or non-responsive. Forensic science experts could provide information about the integrity of their results by having auditable trails of information. Attorneys associated with the cases could have all the information stored in one location for their convenience. Academics and others could also use this information to create models about which factors most heavily contribute to a wrongful conviction. Then, as investigations are conducted, each of these different nodes within this network could receive updates letting them know which parts of their work directly contributed to someone obtaining their freedom.

I submitted this proposal to the Center for Innovation and, fortunately, my proposal was accepted. The three goals I set for myself in the proposal were to 1) reduce the costs associated with sharing the information in a case, 2) speed up the process of conducting wrongful conviction investigations, and 3) develop a framework for understanding and processing data from wrongful conviction investigations.

Image for post
Image for post
Dan Linna explaining the KATA Method of Improvement during Innovation Bootcamp at Seyfarth Shaw LLP

One of the best experiences of my fellowship with the ABA was the ability to work with so many great people. Early on, the center hosted an Innovation Bootcamp in Chicago where our inaugural class of fellows got the chance to learn from some of the best experts in the country working at the intersections of law, technology, ethics, and design. Everyone was so helpful and generous with their time, insights, and networking, during this time period. Something that still stands out about this whole process was how genuinely fun and enjoyable everything was.

To figure out the most effective way to achieve the goals each of us had set for our projects, we developed roadmaps using Design Thinking and Lean Thinking. For those unfamiliar with the core concepts of Design Thinking and Lean Thinking, the most basic idea is to create a systematic way of 1) understanding what problem you are trying to solve and 2) developing different ways to determine what works, what doesn’t, and do so in an efficient way. Below, I have included a basic framework for Design Thinking, from the Stanford d.school, which I drew heavily from.

Image for post
Image for post

The first step in the project came about before I applied for the fellowship, when I was having coffee with Tricia Bushnell, the Executive Director of the Midwest Innocence Project and my former boss. Tricia outlined a challenge she had with working on cases with different experts across the country. Through my work with MIP as an intern, I knew firsthand how costly this process could be for the organization and how slow it could be to get the right files transferred to the right people. So I started poking around.

After several lengthy conversations and google hangouts with Summer Farrar, the MIP’s case coordinator, and Paul Bieber, the Founder and Director for the Arson Research Project, we began developing a list of the various requirements the project would have, it became clear that designing a new operating system for working on wrongful convictions cases would be somewhat challenging and complex because of all the mechanics that such a system we had. The support we received and the plan we developed, however, made everyone involved think that success was possible.

To reduce the amount of up-front development work that would have to be created, I surveyed some innovative projects across the country that functioned similar in a similar way to my vision for the project. I started having more conversations with people who had used a combination of e-Discovery software and analytics tools to review different pieces more efficiently and identify different connections within that data. Through these conversations, I was fortunate enough to obtain a grant for hosting and use of a customized workspace for a year from e-Discovery giant, Relativity.

Image for post
Image for post
You should check out Relativity.com

After learning more about all of the cool things that could be done in Relativity, I began connecting all the pieces of what we had been doing with a function that could be completed within relativity. The requirements we identified at the outset involved 1) conducting phone calls with the forensic experts using the software to make sure the information they needed would be accounted for with a coding layout in Relativity, 2) setting up permissions for the different users that would be involved with the system, 3) getting those users familiar with how to use the system, and 4) setting up a dashboard that would collect and model usage data in a user-friendly way that could help represent the work involved.

For each of these steps, I looked back to the design thinking process and started having conversations with subject matter experts for the different types of components that would be needed for this project. For a better understanding of the wrongful conviction work, I had conversations with Jon Gould of American University, Sarah Chu of the Innocence Project in New York, and Lucy Guarnera, a PHd candidate in the University of Virginia Cognitive Psychology Department. For some of the computational legal framework components, I had conversations with Dan Linna of Legal RnD, Dazza Greenwood of the MIT Media Lab, and Tony Lai of Legal.io and CodeX, the Stanford Center for Legal Informatics. For better understanding of how to implement some of the data-driven policy portions of the project, I talked with Kira Antell from the Office of Legal Policy at the Department of Justice and Chris Griffin of Harvard’s Access to Justice Lab. I also spent time weekly learning about how to navigate around Relativity with Eamonn Coyle and how to build my project management skills with Janice Hollman.

These conversations provided deeper insights about the various components involved could fit together to solve the challenges we identified and showed me what next steps would look like for moving things forward. We began prototyping for the various pieces of the project and came up with 1) a coding layout in Relativity that could be used to evaluate Arson cases, 2) a coding layout in Relativity that could be used for double blind forensic audits that could be used to better understand how cognitive bias affects wrongful convictions, and 3) a series of potential dashboard models using pivot tables in excel, tableau, and Relativity that model data from the wrongful conviction registry and user data to help monitor work being done on the project.

Image for post
Image for post
A visualization of a sample dashboard

Yet, for several reasons, the project ultimately failed to meet the initial goals we set out. We did not reduce operating costs of wrongful conviction cases. We did not create a system that was able to collect data about wrongful conviction cases in real time.

The biggest reason for this is that I was unable to get all the different pieces fit together completely. I did not have the ability, nor the capacity to accomplish this. So, as the time to renew the license for our workspace came around and, with the project in the state it was in, we all decided it would be best to archive the project until such a point in time where it would be more cost effective to work on it again.

III. Legal Tech Book Club: a community to discuss new ideas, diagnose failures, and unlock the potential of Legal Tech

In having looked at what went wrong with the project at a deep level, there are several things stand out. Innovative project ideas can benefit from the history of other innovations. Drawing from analogies, interviews, discussions, and other communal experiences to collaborate can help improve the likelihood of any project achieving true and lasting success. Some of the most helpful things for the project were simply reading books about innovation, blogs on legal tech, white papers from various companies, and scholarly articles. One article in particular stood out to me.

Image for post
Image for post
You should check out this blog on the ROSS Intelligence website

Earlier in the year, Vanessa Butnick Davis, Vice President of Legal & Managing Product Counsel at LegalZoom posted a guest blog for ROSS Intelligence where she calls for a Legaltech Startup Graveyard, an endeavor that would seek to investigate the life and death of legal startups. It is in that spirit, I wanted to publicly diagnose my project with the ABA Center for Innovation.

It is also in that spirit that I started building out Legal Tech Book Club, an open-sourced repository of information that can hopefully contribute to making the legal system more accessible through conversation and discussion of projects.

Through the failures that I have run into, I understand that the first step toward achieving large goals will be very incremental. Legal Tech Book Club seems like a good start to this because it can help serve as a model that anyone can contribute to, it can increase collective knowledge, bring a spotlight to different issues, and operate at a leaner and more sustainable level.

An early effort that I am undertaking is to generate conversation and identify interested collaborators for Legal Tech Book Club through a regularly occurring series of discussions on innovative ideas, interviews with subject matter experts, and any other way that people may see fit. All of this will take place on natively digital mediums, like over an issue on our GitHub or through Google Hangouts or Telegram. Further, by partnering with organizations like Legal Hackers, groups will have an opportunity to host local discussions, post links to book reports and make their knowledge available to people all over the world. These conversations will be archived and made available to help provide context about how the legal industry is changing.

As this endeavor gains momentum and legitimacy, Legal Tech Book Club can also turn into the go-to repository for hopefully successful projects, but definitely for failed projects. Beneath all of this, it is my hope that by adopting the open-source ethos for project ideas and knowledge sharing, this community of people working to solve deep and complex issues and teach the legal community how to learn from failures.

Image for post
Image for post
The cool logo for Legal Tech Book Club

Many groups have already started to do important work across the legal tech ecosystem. There are a ton of great projects that we have linked to on our page that more people need to know about, including CodeX’s Legal Tech Index, Legal Tech Innovation’s Law Firm Innovation Catalog, Law Firm Index, and Law School Index, and Sarah Glassmeyer and Chase Hertel’s Index of Legal Tech and Innovation Conferences. This list, however, is not exhaustive. I am sure I have failed to include things. But, because it is on GitHub, anyone can log in, make a pull request, and submit to add to a growing body of knowledge in a meaningful way.

Aside from initial investments of time, this effort will not require a large amount of work by any single node. Further, because the internet rewards the creation of content, work done in small bits and pieces will be available for a long period of time with attribution credit through the history of pull requests made on GitHub, as well as the recorded participation of those in the discussions.

**For those interested, our first digital book club meeting will take place in September.**

IV. Closing Thoughts

Image for post
Image for post
The ABA Center for Innovation crew (from R-L): myself, Irene Mo (@imokx), Amanda Brown (@BDashAmBrow), Athena Fan (@AthenaFanLT), and Toby Franklin (@tobyafranklin)

The concept of failure is something that has an understandably negative context — failure is the act of not achieving a goal. However, the ability to improve the way that failure happens is something that can be incredibly valuable and that is not limited to one brief moment of time. Failure has taught me a lot over the course of my life and is basically the fuel that got me to where I am. Through the creation of Legal Tech Book Club, I hope to provide a framework that allows people to understand and diagnose the failures and successes of Legal Tech projects in an environment that encourages experimentation and rewards those who have the courage to think outside the box. Looking at my work at the Center for Innovation, by providing context about what went into it, and shining a light on some of the complexities our project faced, it is my hope that people later on can benefit from all the work and effort that went into something like the DFENDR Project. The opportunity is one that I learned a great deal from and want to share with others. In spite of what we were not able to achieve, this experience is going into my book as a success that fills me with pride and knowledge.

If you want to join the preliminary list of people involved with Legal Tech Book Club, fill out our Google Form: https://goo.gl/forms/Psnb1ZExH2srl1sE3

If you’re interested in staying up to date with the latest happenings of Legal Tech Book Club follow us on twitter: @legalbookclub

I owe a thank you to everyone who offered their time to help me work on this project. I did my best to include everyone in the article. I am thankful to Relativity for their support of this project, without their help this would not have been what it was. A shout out to my friends, bosses, and co-workers at RiskGenius for their support — we have a great team and I am excited to see what we can build. Finally, special thanks to my family and friends for being there for me through this whole process. It is also my little sister’s birthday today. Happy Birthday, Mallory!

Click the links below to follow what I’m up to.


Written by

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store