How AI Will Change Reputation As A Proxy For Quality

What you do will matter more than who people say you are

Every day individuals at all levels of the economic spectrum ask the same question, “who would you recommend?” When it comes to services that we have trouble evaluating or when data is unavailable, we turn to a proxy. “Who would you recommend” is a way of asking about reputation. You are my doctor and I trust you, so I’ll trust your recommendation (at least until I can judge for myself).

In law, this is a time-honored tradition, especially at the upper-economic levels. As a general counsel, I would get calls from my friends in other companies. They would give me a brief rundown of the the practice speciality, geographic location, and other necessary constraints and then ask for recommendations. In fact, I still get those calls today. As long as there wasn’t a barrier (e.g., a conflict of interest) I would give the best names on my list.

Economists have studied the proxy concept, since it is widely used. With the use of online tools (e.g., Yelp, Amazon reviews), we also have a new take on an old practice. Reviews on these sites are the modern incarnation of the old “call your friend and ask for a recommendation.” Now, we can get crowdsourced recommendations. But, what if you are new and just starting to build your brand, so the number of reviews is low. Or, what if you had a disastrous event and need to re-build your brand. Call in the fake reviewers. They may never have eaten at the restaurant or purchased the product. Their reviews get mixed in with legitimate reviews, pumping the score.

Fake reviews have spawned a new sub-industry of review filtering. These tools try to spot the fake reviews so sites can eliminate them from the stack. Nothing shatters the integrity of the crowdsourced review system like finding out half of the reviews were made up. Fake reviews were harder to do and easier to spot under the old system. One or two people might claim to have used a service and blackball the provider, even though they never stepped across the threshold. AI can help us spot these frauds in the new system.

Lawyers Use Proxies

Lawyers have taken the reputation proxy and moved it from a nice way to help a friend to an art form. Independent groups survey those who do the recommending to find out who they recommend and why. Lists are prepared ranking who gets the most recommendations. Law firms hunger to be on the list and rise in the rankings.

Being on the “most recommended” list is a short-cut for the actual recommendations. A general counsel can read the list and, instead of having to poll his or her friends, see what a much larger sampling of general counsel reveals. A list is like polling hundreds of friends. If a firm is in the top 10 on the list, then just call the firm and skip the calls to your friends especially if the matter is sensitive (“happen to know anyone skilled at handling a massive money laundering case?”).

From the law firm’s perspective, the goals are to be on the list and then run up a string of years on the list. The more years on the list, the more your firm’s approach to practicing is validated. Who cares what the pundits say about how the practice of law is changing, clients love you for wasting time on things and increasing your prices each year.

One hitch in law is that the lists are for firms, not individual lawyers. When one general counsel calls another, she typically wants more than just a “go to firm X,” she wants a name. In an era when firms have thousands of lawyers and lawyers come and go, knowing more than the reputation of a firm can help. This personal quality separates the lists from the calls.

The Missing Link

Missing in the recommendation approach is any real measure of quality. We can start with the person making the recommendation. We assume they have worked closely with the firm and even the lawyer who they are recommending. Since we have no objective quality measures in the legal industry, that simply means that the recommender liked the lawyer and thought he or she did a good job.

Of course, the breakdown in data does not stop there. We may not have worked with the person making the recommendation. Meeting at a conference and having a two hour chat during dinner is not the same as working together on a matter. Sharing our interests in backcountry hiking does not give us a valid base for measuring whether the other person knows a good brief from a bad one. Yet, we still ask for their recommendation.

That prior paragraph may sound dismissive, but consider it for a moment. We haven’t measured whether the appellate brief was good or bad, the contract sound or flawed, the advice worthy or unworthy, or the lawyer themselves brilliant or addled. We have too few facts to make a meaningful judgment about the recommendation or the recommender.

This is where AI could be a game-changer. Assuming we can find and use good quality data sets, those hanging questions (“was it a good brief?”) may start yielding to objecting or less-subjective answers. Evaluating the performance of a professional is not all about the numbers. But, it also should not be all about opinion without facts.

We could start with contract quality. With AI, it is easier to ask whether a contract conforms to industry norms or has many outlier clauses. Outlier clauses may not have inherent “good” or “bad” qualities, but we could measure how often a clause leads to a dispute. Some software has moved in this direction. We still do not have quality standards, but we have the beginnings of a comparison system.

With appellate briefs, we could measure the extent to which arguments in a brief track with the decision on appeal. Winning a case, but for reasons separate from those argued to the court, does not mean a lawyer did a good job. The party may have won despite, not because of, the work of the lawyer. We can also track the performance of judges. This last concept raises a new area seldom explored. Measuring the body of a judge’s work has been challenging and filled with bias. AI will help us sort that out.

We can capture and analyze data on strategic advice (something few clients do today). Think of this as similar to capturing what happens when you follow the doctor’s instructions. When our children were young, my wife had her system. When an ailment similar to something we had seen in the past would appear, she could refer to her notes and give the doctor feedback on what did or did not work. Over time, the practice eliminated many trips to the doctor.

Some of the quality analyses will come from third parties. Using AI to look at bits and pieces from many databases, the third party can assemble a “quality profile” by firm and by lawyer. The Free Law Project has just announced that it has available every free written order and opinion from PACER. As more open source projects like this one take off, more data becomes available and that means third parties can develop new models for measuring legal work.

Again, data is not the beginning, middle, or end of the story, but it is an important part. Simply having great quality stats should not get you the recommendation. A lawyer may be good at the technical stuff (to use a well-defined term), but not so good at the empathy part of the relationship. The problem today is that we get a lot of feedback on empathy and very little sound data.

The Takeaway

AI is making waves around the world. Some of those waves are crashing down on professions, such as law. Old world practices are giving way to new world technology. Some of the customs are changing as well. As AI gives us ways to turn guesses into objective reality, how we find professional services may change.

For those who think this may represent form over substance, there may be some shocks. We would like to believe that reputations earned over decades will withstand AI scrutiny. Perhaps they will. Or perhaps those reputations were never on solid foundations. Any way you look at it, we will have some interesting times ahead.

If you enjoyed reading this article, please clap (click the the clapping hands). The more you click (or the longer you hold down the icon), the more you enjoyed the story.

About: Ken is a speaker and author on innovation, leadership, and on the future of people, process, and technology. On Medium, he is a “Top 50” author on innovation and leadership. You can follow him on Twitter, connect with him on LinkedIn, and follow him on Facebook.