Why prediction in legal tech needs to die quickly

TL;DR — predictions about outcome diminish in value the more complex a process is. If you’re right about an outcome but it takes 8 years to resolve and costs 10x what the dispute was about who cares if you were right.

Part I — Humans are irrational

Im tired of people ‘predicting’ the ‘outcome’ of cases. It’s a dangerous idea. It’s a scary approach. It minimizes the legal system. It diminishes what judges do. But most importantly it’s useless.

I’m going to write two posts:
1) Human factors that render prediction useless
2) Statistical reasons that prediction is useless.

Button up your friggin’ power suit cause we’re about to go deep.

What right do I have saying prediction is useless?

For starters I ran a company that sold divorce predictions as a part of our service offering. We showed people what the outcome of their divorce was most likely to be based on historical trends. As any family law practitioner would tell you — in Canada you don’t need some fancy algorithm. There is literally a formula.

We took it one step further and looked at cases in an effort to tighten the range the government formula would spit out. After almost 5,000 users went through our product maybe 4 actually accessed the prediction piece. We sold an access subscription to our service. We thought this feature would be wildly popular. No one gave a shit.

Finally I am an ok mathematician, programmer and statistician. I’ve written the software and built these types of models. I will outline why predictions in the legal sector are particularly imperfect.

Why didn’t people care? aka ‘The Human Factors’

For starters a machine stating a probability is less appealing than a human providing comfort and counsel. We underestimated this element of human behaviour. People in the midst of a legal dispute do not think rationally or reasonably.

Second — knowing what the likely outcome of something is misses a critical component — how to get to the outcome. The prediction itself ignores that the path to the result is long, winding and perilous. It ignores that if the other party is not behaving rationally no amount of forecasting will help you. If the legal system was rational people wouldn’t take two hours off work to fight a $30 parking ticket. Human’s are inherently irrational. The legal system is run by humans.

I assume under the phone is the prenuptial agreement since the person is behaving rationally when they get married too. Or maybe its just their vows. Or maybe its an empty folder since this is a stock photo. Also there is a small plant on the table.

Third — disputes are messy. Litigation is messy. In our case we were trying to provide a tool for people to negotiate with their spouse with in an effort to keep their divorce in the ‘uncontested’ stream (meaning they recognized that the outcome was a foregone conclusion so why argue about it). We solicited feedback from our customers relentlessly.

The accompanying divorce stock photo. Same people on the same day with the same photographer commemorating the event. They saw the prediction about the chance of their marriage succeeding and said ‘fuck it lets get this over with’ or something like that.

Due to regulatory challenges this feedback was collected via email and explicitly avoided any commentary or discussion around their particular circumstance. Inevitably the only thing people would talk about was their position and perspective on the separation, the reasons they were leaving and how the other side was wrong.

What we learned was that predictions are meaningless when humans are irrational. You can’t make a client behave rationally. To state the human prediction problem directly:

There’s a 10% chance you will win this case.

Finally — in law school we train zealous advocates not rational compromise machines. Those zealous advocates bring their own irrationality to the table. They may instinctively know they have a tough case. They may discuss or even encourage settlement. The challenge is that both sides need to have the willingness to compromise. Both sides can’t be Lloyd.

The legal process remains human driven. Humans do not behave rationally. Human lawyers frequently take on their clients cause as their own. Exercises in prediction ignore the human component of the equation. While I intend to tackle prediction more technically in part II I would note that as of yet there is no model I am aware of capable of accurately predicting human behaviour.

Predicting human behaviour is a noble goal. Except that’s not what companies are selling. They’re selling a giant helping of bullshit. The easiest way to cut through it all is ask them to insure your litigation against loss. If they’re that certain their predictions are good they should be willing to put up the cash. Except they wont :)

In the end if you could predict the outcome of litigation there are plenty of companies on the public markets that are the subject of ongoing litigation. Except these questions are messy from a human perspective. The outcome is unpredictable. They exist in a low information environment.

Which leads to part II: The math doesn’t make sense either.