The Legal Services Innovation Index isn’t about legal services or innovation
Back in high school and college, I worked as a bike mechanic in Anchorage, Alaska. I loved that series of jobs. I got to hangout in a cool environment working on bikes, talking about bikes … nerding out about bikes. It was the ideal job for me at the time.
At the last shop I worked at, right before law school, my employer and a competitor in town were having a no holds barred SEO (search engine optimization) battle royale. The newcomer had overtaken my shop’s place for top Google result for fat tire bikes (it’s Alaska, people wanted to ride in the winter), and my boss was not going to take this affront lying down. In an act of subversive retribution, my boss filled our website with invisible keywords that the two shops were using to boost their SEO. To a person on the site, you wouldn’t know this trickery was afoot. However, the Google search algorithm did. My boss was rewarded with a jump in Google results, and victory was ultimately his.
This story is less about the chicanery of two Alaskan bike shops and more about the algorithm they were trying to impress. Throughout society, similar battles of different scale and importance are taking place. Entering this fray is the Legal Services Innovation Index from the Michigan State University College of Law LegalRnD center, which is an attempt to measure law firm innovation.
The project falls short of analyzing legal services and innovation. In fact, it runs the risk of being a well intentioned algorithm that creates a horserace at the expense of client services.
Launched in August, this project’s mission is “to rank and assess [law firms] on their use of technology” instead of “revenue and profit.” This, the site says, will “accelerate innovation and technology adoption that will lead to legal-service delivery improvements and increased access to legal services for everyone.” A laudable goal.
Created by Professor Daniel Linna at MSU, the project ran a Google search analysis of law firm websites by looking for particular buzzwords associated with “innovation.” These terms include blockchain, AI, and data analytics, among seven others. The index rates law firms by collecting and aggregating the number of times these words appear.
After the project’s launch, my legal tech Twitter feed filled with accounts praising this project:
At the moment, this project may seem innocuous and even worthy of praise. However, when a set of criteria catch fire and are used as an industry benchmark, a potential for the Index, organizations start chasing the wrong goals. Just look at the impact U.S. News and World Report have had on colleges.
While treated as authoritative, the college ranking algorithm is the outcome of an opinion that doesn’t tell us about the quality of education. However, administrators around the country have made expensive and aggressive efforts to change their institutions so that they can get a bounce in the rankings.
Universities have to spend lavishly to impress the U.S. News algorithm. In a paper from 2014, researchers found that if the University of Rochester, then a mid-30s ranked school, wanted to move up just two spots it could increase the average faculty salary by about $10,000 and spend about $12,000 more for each student. Collectively, this would cost $112 million a year. To get closer to the coveted top 20, Rochester would also need to enroll more students from the top 10 percent of their high school graduating classes, increase alumni giving, see its graduation rate increase by two percent, decrease the acceptance rate, and increase the SAT and ACT scores of incoming students.
The popular delusion around meeting U.S. News’s standards has caused some universities to lie about and juke their numbers to get an edge on similarly situated schools. Claremont-McKenna lied about its SAT scores. Iona College, just outside of New York City, lied for years about graduation rates, test scores, and freshman retention. To juke their stats, Baylor went as far as paying incoming students to retake their SATs to make a jump in the rankings.
While all this skullduggery to appease an algorithm might seem crazy, the worst of it is that these administrators undertaking these acts were not working to improve the education of their students. Why? Because the U.S. News rankings don’t actually judge education quality!
This brings us back to the Legal Services Innovation Index. It doesn’t measure legal services or innovation, yet it creates the same set of circumstances for a horserace like the U.S. News rankings.
To the project’s credit, the site says that this keyword search “might be a weak or poor proxy” to show innovation. However, the site also says, “the extent to which a law firm talks about an innovation category on its website is meaningful information and worth assessing for discussion purposes.”
Is this true though? This project does not look for context of the words it tracks. Contrarily, a law firm could post the sentence “AI will never replace a good lawyer,” and, while this is presumably antithetical to what the Index is trying to measure, the algorithm will count the use of “AI” towards the firm’s innovation score.
At its core, this project tells us how frequently a firm uses ten words.
There’s no argument presented that talking about AI and blockchain correlate with progress in the legal services industry. In fact, the website states, “There may be a significant gap between what a law firm says it is doing and what it actually does.” In an attempt to fill this gap, the LegalRnD center does have a companion catalog of law firm tech projects.
Without a doubt there are great potential applications in the legal field for the maturing technologies that the Index anoints, but the Index doesn’t account for client satisfaction or client retention on account of innovation. These factors are what will determine a firm’s long term success, not whether or not they opine often enough about specific topics.
Think about it from a client’s perspective. Say a client needs help with a property acquisition or copyright issue. I have a hard time believing that client will be persuaded to contact one firm over another because a firm’s blog has some interesting thoughts regarding blockchain’s application in multiparty property disputes. Concerning itself with tech language is something that resonates with geeks like me and legal tech Twitter, but whether or not this is something that will correlate with better services and advocacy is a very different question. One that isn’t answered by this project.
Nonetheless, if the industry decides to put weight behind this formula, then we will see an increase of posts on “blockchain” and “expert systems”. What does that do for legal services? I don’t know, but I imagine it has something to do with increasing a young associate’s blog writing responsibilities.
During that great Alaska bike store SEO feud of the late-Aughts, the bikes we sold didn’t improve, our customer service stayed the same, and our clients got the same mechanic work they’d grown to rely on. The only thing that changed was the number of keywords hidden throughout the website to trick an algorithm and stick it to a competitor.
Aware of this, let us not fall prey to the cult of algorithms and buzzwords. Instead, let us work together to identify and measure meaningful improvements to legal services. Without a doubt, this is a challenge. However, if done well, it would truly be an innovation.
