Search Audit 2023 and the AI Online Legal Information Assessment Tool
It is well known by now that most people use Google and other search engine platforms to search for legal information. A question that we are therefore always interested in at the Stanford Legal Design Lab is:
What are people seeing on the Search Engine Results’ Pages (SERPs) from their search locations?
This is where search audits come in. Search audits allow us to gain insight into what users are seeing (from their locations) on SERPs when they type in a legal query. This is incredibly useful for researchers and professionals in the access-to-justice community, as it allows us to understand what kinds of resources, help, and legal information are presented to people. This information helps us create better resources and make better decisions on how and what to present to users to assist them in their legal journeys.
Last year, we ran an extensive audit. We looked at search results for almost 70 zip codes in 14 states. That accumulated in 88,429 results. This Medium article will walk you through our high-level findings, the AI legal quality assessment tool, and our future plans for this research project.
Search Audit 2023
In 2023, we examined how legal queries were performing on search engine pages in different zip codes across the country. In total, we accumulated 88,429 results for almost 70 zip codes in 14 states. Before we walk you through the results of the audit, let’s take a look at the data points we used to run the search audits.
We examined legal queries for four different issue areas: debt collection, eviction, domestic violence, and flood contractor-related legal issues. These were the legal queries we examined for each legal issue area:
We ran these legal queries for 70 zip codes in 14 states. Below you can look at a map which highlights all the areas we examined in our search audit. The numbers that you see on the map are the total number of results for each state that we received through the search audit.
The following map shows all the zip codes that we ran for the 2023 search audit. This means that for each of these zip codes that you see on the map, we ran the full list of queries and analyzed the results. You can click on the link to visit the interactive map on Tableau if you want to take a closer look at the zip codes and the count of results for each zip code.
Now let’s break down the data. In total we had 88,428 results.
If we further break it down, we can look at the total amount of results per state. As you can see in the chart below, Texas and Indiana generated the lowest number of results while Washington and Louisiana had the highest number of results.
If we go one step further, we can look at the total number of results per zipcode.
Now let’s look at the frequency of the URLs, meaning: which URLs show up the most in the full dataset without filtering any of the results (we’ll get to that in a minute)?
These individual webpage URLs show up most frequent in the total number of results. At this point in our research we are not filtering the websites by legal issue or jurisdiction yet, but by looking at these URLs we can see that mostly commercial websites are showing up, with the notable exception of hotline.org which is a national domestic violence resource.
What we can also see in this list is that the frequency of the URLs is still a small percentage of the total number of results. This might indicate high personalization by the search engine based on location, SEO and other factors. The search engine algorithm tries to provide the most relevant results based on each individual query and other factors. The lack of an outlier with a high percentage of frequency reflects this.
Another reason why the level of frequency is so low for each URL, is because there is a high variety of URLs on the Internet. Search engines index an incredible amount of webpages. This is why an individual URL is just one result in the massive ocean of the Word Wide Web.
Here we are looking at URL domains. This is not a specific webpage as we showed in the previous section, but the full website. The list does not seem to differ that much from the URL list with the exception of a few domains, such as ‘Nolo’.
Now let’s look at the top 10 URLs for each legal issue: which websites show up most often for each legal issue? The aspect that we are most interested in is whether the top URLs show jurisdiction and legal issue specific information. For these charts, I removed all the null results. If you hover over the URLs in the interactive dashboard you can see several interesting individual pages show up.
As seen in the chart, some of the URLs that are ranked in the second spot, offer no helplines, are PDFs (this makes the information less searchable) or only provide organization specific resources.
It gets interesting if we delve deeper. Do the URLs provide jurisdiction specific information if we sort the results by state? We assume that the results for each legal issue area shift if we add the filter for each state to the results as the results should be jurisdiction specific.
What we can see for the legal issue debt collection is that the URLs remain fairly the same across all states. There is a bit of shifting in the placement here and there but no new additions, except for spot 10 in Minnesota that refers to a new solosuit link instead of wallethub.com which is the tenth most frequent URL in other states.
Let’s see if the results differ for domestic violence. Most of the results remain the same compared to the nation wide URL frequency list for domestic violence, but there are a couple of jurisdiction specific pages popping up for CT, FL, MN and MI. And on the Massachusetts list we see the first appearance of a legal help website.
Now let’s look at the legal issue of eviction next. The charts below show that in contrast to the previous legal issues, the top 10 frequency of URLs in each state is significantly different and mostly jurisdiction specific.
The top 10 most frequent URLs in each state are not only jurisdiction specific, but also produced by mostly legal aid, court or other governmental websites.
The URL results for flood contractor related queries are less jurisdiction specific than eviction. However, 8 out of the 14 states have 1 or 2 jurisdiction specific results.
The above charts illustrate that the top 10 URLs for domestic violence, debt collection and flood contractor legal issues, do not change that much once they are filtered by state. This means that the information on those pages is less likely to be jurisdiction specific and this, on its turn, can also impact the quality of the information.
Online Legal Help Cohort Dashboards
There are plenty other data metrics to discuss about the search audits, consider the Average SERP ranks for each domain, the frequency for URL domains, the top ranked pages, etc.
This past year, I worked on designing and developing search audit dashboards that showcase all these charts and provide a user-friendly way to filter the data.
The dashboard with the full dataset can be found here: https://public.tableau.com/app/profile/n.ra.al.haider/viz/SearchAuditReport2023/Home
The dashboards for each individual state can be found here: https://legalhelpdashboard.org/audit/
Due to space constraints we cannot show each chart, but the dashboards provide an overview of the full analysis for each state.
The question that we are interested in looking at when analyzing the data is:
how do we know when information on a website is high quality legal information?
AI Online Legal Information Assessment
In our paper ‘Does Googling Justice Work? Auditing Search Engines’ Performance as Intermediaries of Legal Help Information Online’, the following criteria are described to assess the quality of legal information:
“Quality Criteria 1: The site has jurisdiction-correct, accurate, current information.
Quality Criteria 2: The site presents specific, detailed, actionable information about rights, process, and services.
Quality Criteria 3: The site puts minimal burden or cost on a person to access information about rights, process, and services.” (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4414305)
There are also two indicators that give us more guidance in what ranks as a low or high quality legal information website
“Indicator 1: The site is run by a public interest organization, not a commercial one.
Indicator 2: The site is run by a legal organization rather than a non-legal one.” (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4414305)
Let’s look at an example, this InStyle article is ranked as the second most frequent URL for the legal issue domestic violence: https://www.instyle.com/threatening-breakup-emotional-abuse-5364547
Let’s assess this URL based on the quality criteria:
“Quality Criteria 1: The site has jurisdiction-correct, accurate, current information.
The site does not have jurisdiction-correct information.
Quality Criteria 2: The site presents specific, detailed, actionable information about rights, process, and services.
The site does not present specific, detailed, actionable information about rights and services.
Quality Criteria 3: The site puts minimal burden or cost on a person to access information about rights, process, and services.”
The site is freely accessible and not behind a paywall.
As for the indicators, the URL is from a commercial, non-legal organization website. Although the website does not provide false or misinformation, it would still be categorized as low quality legal information on the basis of the criteria. An individual looking for legal information regarding domestic violence would not be able to find actionable next steps, resources or even information about their rights in that article.
It would of course take a substantial amount of time and resources to manually review every single website, even if we would take a sample size. However, AI models nowadays allow us to automate the process.
As an experiment, I developed the AI Online Legal Information Assessment tool. This tool runs on ChatGPT4o and assesses websites based on the quality criteria and indicators mentioned above. If you give the tool an URL, it will indicate whether the website is high, low or medium quality and provide the reasoning based on the indicators.
If I let the website assess the InStyle article, it will provide the following assessment:
Here is the evaluation of the website at the provided URL, based on the specified criteria.
### Website Assessment
#### URL
- **URL**: [https://www.instyle.com/threatening-breakup-emotional-abuse-5364547](https://www.instyle.com/threatening-breakup-emotional-abuse-5364547)### Quality Metrics
1. **Jurisdiction-correct, Accurate, Current Information**
— **Rating**: Low
— **Explanation**: The article lacks jurisdiction-specific legal information and does not provide accurate or current legal advice.2. **Specific, Detailed, Actionable Information**
— **Rating**: Low
— **Explanation**: The article provides general advice and emotional support but lacks specific, actionable legal steps that someone can take to address issues related to emotional abuse and threatening behavior.3. **Minimal Burden or Cost**
— **Rating**: Medium
— **Explanation**: The site is free to access but contains advertisements and pop-ups that can burden the user experience.4. **Public Interest vs. Commercial Organization**
— **Rating**: Low
— **Explanation**: The site is run by a commercial organization, which may prioritize generating traffic and ad revenue over providing unbiased legal information.5. **Legal vs. Non-legal Organization**
— **Rating**: Low
— **Explanation**: The site is a non-legal organization, focusing on lifestyle and fashion rather than legal matters.### Summary
- **Overall Quality Rating**: Low
- **Detailed Breakdown**:
— The website provides general advice on emotional abuse and threatening behavior in relationships but lacks jurisdiction-specific, accurate, and actionable legal information.
— The presence of advertisements and the commercial nature of the organization may introduce biases and burden the user experience.
— For legal issues related to emotional abuse and threats, more specialized legal resources or nonprofit organizations would be recommended.
The Legal Information Quality Assessment Tool needs to be refined and tested throughout the next months, but the initial experiments are promising. The tool also responds to lists and provides a table as a response, where the assessment is summarized. For example, a list of the top ranked SERP results for the legal issue debt collection are assessed as follows:
Here is the updated table with an additional column summarizing the assessment for each site regarding debt collection:
| URL | Quality | Summary |
|------------------------------------------------------------------------------------------------------|---------|-----------------------------------------------------------------------------------------|
| https://www.yourworkplacerights.com/harassment-in-the-workplace/ | Low | Not relevant to debt collection, focused on workplace harassment. |
| https://www.ylfbankruptcy.com/tucson-lawsuit-defense-and-litigation-attorney/ | Medium | Useful information on lawsuit defense but primarily promotional. |
| https://www.womendeservebetter.com/8-tricks-to-handling-debt | Medium | Provides general advice on handling debt but lacks specific legal guidance. |
| https://www.virginiadebtrelief.org/l/gs/options/ | Low | Limited actionable information, focused on promoting services. |
| https://www.turnanewleaf.org/services/housing-and-shelter/ | Low | Not relevant to debt collection, focused on housing and shelter services. |
| https://www.topconsumerreviews.com/debt-consolidation-programs/index.php | Medium | Offers reviews of debt consolidation programs, potential commercial bias. |
| https://www.thenesbittlawfirm.com/assistance_with/credit-repair-lawyer-columbus-ohio/ | Medium | Useful for credit repair but primarily promotes legal services. |
| https://www.taxreliefcenter.org/l/gs/update/ | Low | Lacks detailed and unbiased information, focused on tax relief services. |
| https://www.takechargeamerica.org/credit-card-help/ | Medium | Provides practical advice on managing credit card debt, some information may be outdated.|
| https://www.supermoney.com/quotes/tax-relief | Medium | Offers comparisons of tax relief services but includes promotional content. |
| https://www.ssa.gov/ssi/ | Low | Not relevant to debt collection, focused on SSI benefits. |
| https://www.simonfitzgerald.com/lp/shreveport-la-bankruptcy-ppc_lp/ | Low | Primarily focused on promoting bankruptcy services. |
| https://www.seattle.gov/utility-bill-help | Low | Not relevant to debt collection, focused on utility bill assistance. |
| https://www.russmanlaw.com/drunk-driving | Low | Not relevant to debt collection, focused on drunk driving defense. |
| https://www.quickenloans.com/alf/wham/loanType | Low | Commercially driven with limited debt collection information. |
| https://www.quickenloans.com/alf/wham | Low | Primarily promotional with minimal relevant information. |
| https://www.qualify4disability.com/evaluation-2/ | Low | Not relevant to debt collection, focused on disability qualification. |
| https://www.prosperikey.org/ | Low | Not relevant to debt collection, focused on financial assistance resources. |
| https://www.polarislawgroup.com/bay-area-employment-law/ | Low | Not relevant to debt collection, focused on employment law. |
| https://www.polarislawgroup.com/bakersfield-employment-law/ | Low | Not relevant to debt collection, focused on employment law. |
| https://www.pennsylvaniadebtrelief.org/l/gs/resources/state/ | Low | Limited actionable information, focused on promoting services. |
| https://www.oal-law.com/cincinnati-bankruptcy-lawyer/credit-report-dispute/ | Medium | Useful for credit report disputes, focused on promoting legal services. |
| https://www.nevadadebtrelief.org/l/gs/update/ | Low | Lacks detailed and unbiased information, commercially oriented. |
| https://www.nerdwallet.com/best/mortgages/refinance-lenders | Low | Not relevant to debt collection, focused on mortgage refinancing. |
| https://www.melmedlaw.com/practice-area/unpaid-overtime-and-minimum-wages/ | Low | Not relevant to debt collection, focused on employment law. |
| https://www.marrsterry.net/bankruptcy-overview/debt-negotiation/ | Medium | Provides useful legal insights but primarily promotes services. |
| https://www.louisianadebtrelief.org/updates/ | Low | Limited actionable information, focused on promoting services. |
| https://www.legalshield.com/consumer-finance/ | Medium | General legal guidance but commercially oriented. |
| https://www.lawinfo.com/credit-repairs/minnesota/minneapolis/ | Medium | Offers useful information on credit repair but primarily promotes legal services. |
| https://www.knevel.com/bankruptcy/credit-card-debt-defense/ | Medium | Useful for credit card debt defense, focused on promoting legal services. |
| https://www.jacksonhewitt.com/tax-resolution/cp/pma-owe-collections/ | Low | Provides practical tax resolution advice but not focused on debt collection. |
| https://www.investopedia.com/articles/personal-finance/091615/difference-between-restructuring-and-refinancing.asp | Low | Not relevant to debt collection, focused on financial restructuring. |
| https://www.indianadebtrelief.org/l/gs/update/general/ | Low | Lacks actionable information, focused on promoting services. |
| https://www.indianadebtrelief.org/l/gs/resources/state/ | Low | Limited detailed information, commercially oriented. |
| https://www.indianaconsumerlawgroup.com/debt-defense.html | Medium | Provides useful guidance on debt defense but focuses on promoting legal services. |
| https://www.incharge.org/options/debt-settlement/ | Medium | Offers practical advice on debt settlement, some information may be outdated. |
| https://www.ilrc.org/public-charge-safe-use-list | Low | Not relevant to debt collection, focused on public charge issues. |
| https://www.helpmebounce.org/ | Low | Not relevant to debt collection, focused on financial assistance. |
| https://www.garmokiste.com/legal-services/debt-collection-defense/ | Medium | Useful for debt collection defense, primarily promotes legal services. |
| https://www.fastofferpro.com/ | Low | Commercially driven with limited relevant information. |
| https://www.employmentlawhelp.org/cpc/general | Low | Not relevant to debt collection, focused on employment law. |
| https://www.employmentattorneyhome.com/free-evaluation/ | Low | Not relevant to debt collection, focused on employment law. |
| https://www.discovertoday.co/ar?q=stop%20wage%20garnishment%20immediately | Low | General advice without specific legal guidance. |
| https://www.debtreliefplanners.com/ | Low | Limited actionable information, potential commercial bias. |
| https://www.debtmd.com/ | Medium | Offers practical advice on debt relief but includes promotional content. |
| https://www.davtyanlaw.com/work-harassment-attorney-dmg-sem/ | Low | Not relevant to debt collection, focused on workplace harassment. |
| https://www.davtyanlaw.com/employer-not-paying-on-time-dmg-sem/ | Low | Not relevant to debt collection, focused on employment law. |
| https://www.csd.ca.gov/Pages/Assistance-PayingMyEnergyBills.aspx | Low | Not relevant to debt collection, focused on energy bill assistance. |
| https://www.crsoftware.com/ | Low | Primarily promotional with limited relevant information. |
| https://www.corvias.com/our-services/financing | Low | Commercially driven with limited relevant information. |
| https://www.calmba.org/help-with-medical-bills | Low | Not relevant to debt collection, focused on medical bills assistance. |
| https://www.californiadebtrelief.org/l/gs/update/ | Low | Limited actionable information, commercially oriented. |
| https://www.californiadebtrelief.org/l/gs/resources/ | Low | Lacks detailed information, focused on promoting services. |
| https://www.bankofamerica.com/info-home-loans/purchase-search1/ | Low | Not relevant to debt collection, focused on home loans. |
| https://www.azconsumerlaw.com/ | Medium | Provides useful guidance on consumer law issues but focuses on promoting services. |
| https://www.arkansasdebtrelief.org/l/gs/update/ | Low | Limited actionable information, commercially oriented. |
| https://www.advantageccs.org/blog/the-best-online-credit-counseling-system-in-the-industry/ | Medium | Offers practical financial advice but includes promotional content. |
| https://wamortgagerelief.com/contact/ | Medium | Provides useful information on mortgage relief but includes promotional content. |
| https://wamortgagerelief.com/contact-lm/ | Medium | Offers practical advice on mortgage relief but potential commercial bias. |
| https://usafundingapplications.org/v9/c/ | Low | Commercially driven with | Medium |In total, out of the total 112 URLs evaluated, 82 were rated as low quality, 30 as medium quality, and 0 as high quality.
Hopefully in the future, this tool will also allow us to pinpoint recurring low-quality information URLs, spotlight high-quality URLs, and showcase best practices of online legal information for stakeholders working in this field.
My goal for the upcoming months is to refine this assessment tool and set up an annual search audit as it is only by deepening our understanding of online legal information that we will be able to work towards a Better Legal Internet.
