Cleveland Legal Aid Learned the Outcomes of Its Brief Services Through Texting
Legal aid programs invest significant resources in delivering legal advice, brief service, and extended representation to clients. The problem is, for the considerable number of advice and brief service cases, they seldom know whether the help they provided solved the client’s problem. Having a streamlined system in place to measure outcomes would address this issue and improve programs’ effectiveness, efficiency, and services to clients.
In 2017, the Legal Aid Society of Cleveland (Cleveland Legal Aid) successfully implemented a pilot project to text clients who received advice or brief service so it could learn the outcome of the help provided. The pilot collects outcome data related to housing conditions, eviction, foreclosure, simple divorces, criminal record sealing, and debt problems. It also collects feedback on the program’s community education presentations and shared information about services via text — facilitated by Cleveland Legal Aid’s new tech-savvy approach to communication.
The Brief Service Problem
Approximately 80 percent of the cases Cleveland Legal Aid closes are brief service. This number isn’t unique to the program; it’s a significant blind spot for the legal aid community.
In an effort to understand how helpful its services were, Cleveland Legal Aid started sending out paper surveys a few years ago to its brief service clients. Roughly half of its former clients have received them.
However, upon looking closely at them, Cleveland Legal Aid identified several shortcomings with the surveys. For instance, they had questions not specific to the problems people face, low response rates, and low client-reached rates. They were also resource-intensive, costing time and money that could’ve been spent elsewhere.
So, when presented with an alternative option, Cleveland Legal Aid welcomed the chance to try another communication method: texting. (The organization now both sends paper surveys and texts clients; the surveys are mailed quarterly to a randomly selected sample, months after the client’s case concludes, at their last known address.)
The Outcome Texting Project
Cleveland Legal Aid’s Outcome Texting Project was created out of a 2015 LSC Technology Initiative Grant (TIG) to automatically text questions to clients who received brief service or attended a community legal education seminar. It was a direct attempt to address the brief service blind spot.
“With legal services, we measure the impact of our extended representation for clients regularly to understand whether the representation is effective in attaining clients’ goals,” said Anne Sweeney, Managing Attorney for Community Engagement at Cleveland Legal Aid, who oversaw the Outcome Texting Project.
“We have not been able to do that generally when it comes to the brief advice we provide clients because we usually talk to them once or see them once and don’t have any follow-up contact. So we don’t have a good way to know whether what we gave them, what we told them, and what we advised them to do actually helped them solve whatever problem they brought to us. And we thought that using text messaging to follow up might be a solution to that dilemma.”
In regards to the last consideration, Cleveland Legal Aid partnered with FrontlineSMS, an award-winning social enterprise that helps organizations define and implement two-way communication through texting. Led by its CEO, Sean Martin McDonald, the FrontlineSMS team helped the program set up its system. While geared toward collecting information about outcomes, this new approach also emphasized promoting texting as a way to communicate with Cleveland Legal Aid so its staff could engage with clients around more than just outcomes.
Cleveland Legal Aid and FrontlineSMS developed three different types of text messages that Cleveland Legal Aid would send:
1. Outcome messages
2. Information messages
3. Community legal education and outreach (CLEO) messages
Each of these messages has its own purpose and is sent to different recipients. The potential pool of recipients for the outcome messages is all the applicants who qualify for services from Cleveland Legal Aid.
As a snapshot, the program had 19,668 applicants from May 2016 through October 2017. Of those people, 17,000 — or 86.4 percent — provided mobile phone numbers; 14,354 of the 17,000 — or 84.4 percent — also gave permission to receive texts.
Cleveland Legal Aid started sending outcome messages on a weekly basis starting February 7, 2017, where the case met the criteria for one of eight “activities.” Each activity is based on a specific problem code and the specific brief service delivered by Cleveland Legal Aid to applicants who call with that problem. There are three to five text questions for each activity, and all recipients receive all questions in the sequence, regardless of whether or not they respond to any message.
For example, someone who calls about filing for divorce receives three texts associated with “Activity 32A” (the program’s designation for simple divorce). A caller who needs help with their housing conditions receives three texts related to that legal activity. (See below for these texts. For a full list of the eight activities with their corresponding messages, timing, and criteria, click here.)
Five of the eight activities include text messages that offer a person some additional help related to their original problem and the brief service provided. These “help messages” ask a person to text a specific phrase back to Cleveland Legal Aid if additional help is needed. If they text back, an automated email is sent to a designated attorney, who then calls the person.
For Cleveland Legal Aid, as Anne Sweeney revealed, building the texting system and creating the different types of messages — the outcome messages in particular — took a lot of work. That’s where the time, money, and energy came in. However, the process for identifying clients, sending messages, and collecting data went smoothly once implemented.
That process started with Cleveland Legal Aid’s data analyst, who created automated reports that now run weekly, and identify the appropriate group of clients for each activity. The list in a CSV file is manually uploaded into FrontlineSMS’ system, where a click of a button triggers the appropriate series of questions for the relevant activity.
The timing of when each question is sent was predetermined and part of the programming done by FrontlineSMS. Through FrontlineSMS’ system, Cleveland Legal Aid staff can view the messages sent and responses received. The system also compiles high-level summary data in tables for easy viewing and makes detailed data available by exporting a CSV file.
At the outset of Cleveland Legal Aid’s project, FrontlineSMS staff recommended that it begin developing a culture of texting by offering general information via text to the public and its client community.
These so-called “information messages” offer basic information about Cleveland Legal Aid’s intake hours, office locations, brief advice clinics, and more. They are automatically delivered to a person when they text “LAS” followed by a keyword to the program. For example, someone can text “LAS hours” to learn about the intake hours. “LAS clinic” informs about the location of the next brief advice clinic. (To see the full list of information messages, click here.)
More recently, Cleveland Legal Aid expanded to provide more substantive information for people who need it. For example, “LAS conditions” triggers the following message in response: “When a landlord does not keep property in safe condition, tenants may be able to pay rent to the court. Find out the steps required to rent deposit at https://lasclev.org/my-landlord-hasnt-responded-to-my-requests-for-repairs-what-can-i-do/.”
Cleveland Legal Aid’s third and final type of text message is its community legal education and outreach (CLEO) messages, designed to gather feedback on its community education presentations and trainings. These messages are specifically for service providers who attend a “Legal Aid 101” presentation and job training participants who attend an “Employment: Know Your Rights” presentation.
At the end of one of those presentations or trainings, participants are invited to text a keyword to Cleveland Legal Aid’s number. By doing so, they are automatically enrolled in a group to receive specific, scripted questions via text. After receiving a confirmation welcome message, they receive about one message per week for four weeks. (To see the full list of CLEO messages, click here.)
After a couple years and six phases, Cleveland Legal Aid’s Outcome Texting Project has largely been a success.
Looking at the statistics, between May 2016 and October 12, 2017 (the date in which the most recent data were downloaded), 14,354 applicants — 73 percent of the total applicants and 84.4 percent of applicants who provided a cell phone number — consented to receive text messages from Cleveland Legal Aid.
For the outcome messages in particular, from February 7, 2017 to May 1, 2017, the total number of people identified to receive any of the text messages was 995 people for 1,016 cases. Of this group, 616 responded to at least one message (other than opting out), for an overall response rate of 61.9 percent. By contrast, the response rate for the paper surveys Cleveland Legal Aid sent out was 6.4 percent.
Above all, these numbers show that texting is a much more effective communication method for gathering outcome data on brief services than paper surveys — Cleveland Legal Aid’s previously relied-upon method. Not as easily quantifiable is the effect that Cleveland Legal Aid’s own staff had on the success of the project. Without their vision, thoughtfulness, and readiness to adopt and engage — in concert with FrontlineSMS’ technical assistance — the results wouldn’t have been as good.
As FrontlineSMS CEO Sean Martin McDonald explained:
“It’s worth saying that the credit for the numbers goes to — as all credit should — the implementing team. One of the things that you learn about when implementing technology across a lot of different organizations is that the individual relationship between the organization and their community is by far the most important driver of high response rates, of high satisfaction, of constructive relationships.
Speaking specifically to text messaging, when done correctly, text messaging is short, convenient, and actionable. And if you can take those as guiding principles and frame the relationship and communication that you have with a client, in that way, you’re almost never wasting your time. You’re asking them things that they’re prepared and able to answer and, ideally, based on that answer, you’re giving them avenues to materially improve their situation.
I think that a lot of what we’ve seen as a platform and using SMS broadly, is that the demographics of people who need legal aid assistance overlap very heavily with the populations that are not served particularly well by other platforms. And so, starting with messaging, which is of course imperfect but still is better than almost everything else — if not everything else — that’s out there, for particular things, ends up being a very strong way to maintain or add value to a relationship.
It’s not a great way to be introduced to people — you don’t build trust necessarily with text messaging, but you can take a small amount of trust and make it a big amount of trust by being a thoughtful designer in the interaction. And I think that this particular pilot has really proven the value of that thoughtfulness and the value of working with and doing the research around the program to make it successful.”
Over the course of the project, Cleveland Legal Aid has had a few discoveries, too. For example, it found that low-income people will agree to receive text messages from a legal aid organization — such as Cleveland Legal Aid — and respond to questions about the help they received.
Although not extended and comprehensive, brief services can effectively resolve some people’s legal problems. Cleveland Legal Aid wasn’t assured of this before, having only a small window into the case outcomes of its clients.
And texting clients for outcome data can be accomplished by legal aid programs with a moderate investment in technology and time. This last point, which underscores the project’s feasibility, is even more true after the continued success of Cleveland Legal Aid’s pilot project.
“Because of the time and effort that Cleveland Legal Aid has put in, they’ve made it substantially cheaper for everybody else involved, because so much of the build is done,” said McDonald. “There’s always more to do and there’s always ways to make the experience richer, the data analysis richer, and the number of workflows supporting the system more plentiful. But I think that a lot of it is just getting started because of this initial TIG. Frontline will be publishing a version of the platform that other legal aid providers can sign up for and take advantage of in a generalized way.”
So, in piloting the project, Cleveland Legal Aid has left the door wide open for replication. Other programs can use Cleveland Legal Aid’s system as a model, but there might be more to draw from soon. The program is exploring how else it can improve its texting service and make it even more robust.