Giving Legal Teams Better Tools to Represent Asylum Seekers
How UNHCR is experimenting with artificial intelligence (AI) to develop quality legal arguments more efficiently.
By Amy Lynn Smith, Independent Writer + Strategist
When you think about lawyers working to defend someone’s rights, you may picture them sitting in a law library, poring over books the size of an unabridged dictionary searching for legislation and legal precedents to make their case. Or, at the very least, you imagine them in their office, at a desk strewn with massive case files, books, and memos, digging for the facts they need.
That scenario is no longer the norm, but the computerized equivalent still holds true: When attorneys are preparing a case, they must sift through volumes of case law, searching through databases to find relevant legal decisions, policy documents, and case law that can support their own legal arguments. It’s true for all lawyers, but the stakes are especially high when it comes to lawyers who help refugees and other displaced people seeking asylum in a new country, after being driven out of their homeland by violence, persecution, or other factors that make it impossible to continue living there. That includes Refugee Status Determination (RSD) and Protection Officers working at the UN Refugee Agency (UNHCR), but also attorneys and legal aid providers in countries of asylum, for example.
To determine whether someone has a substantiated claim for international protection, an important information tool used by these lawyers is refworld.org, a UNHCR global database of law and policy documents relating to refugee law. The database is also used by asylum decision-makers, members of the judiciary, academics, and others outside UNHCR who are working to help refugees make a home in a new country.
Asylum decision-making is complicated under any circumstances, and procedures differ by country and region. To achieve high-quality decisions within the limited time available, easy access to relevant information is crucial. Lawyers need to know about the legal framework applicable to a case, such as the asylum law in the country of asylum. They also need to understand laws that are in force in the applicant’s country of origin, since that information may help prove that an asylum-seeker has good reason to claim asylum. Other relevant information includes jurisprudence that shows how these laws are applied and interpreted, as well as UNHCR policy and guidance on the correct interpretation of international refugee law. All of this information is available on refworld.org, but isn’t always easy to navigate the database to find it.
For instance, an attorney may find a reference to a potentially useful legal decision in a document on Refworld and wants to review that decision. The attorney would have to either return to the homepage to start looking for the referenced decision or turn to a search engine such as Google to look up the case. As a result, the legal research journey becomes cumbersome and time-consuming. To improve internal navigation and make Refworld more intuitive, UNHCR is assisted by lawyers, working on a pro bono basis, who summarize the cases and tag the decisions with relevant metadata, which stores information that links back to sources of other data. Part of this tagging involves the manual extraction of all references to other decisions that are mentioned in a document and then adding the relevant Refworld hyperlinks to these references. For long documents, this can take considerable time — time that could be better spent on tasks such as summarizing the legal considerations of the case.
All of this manual work raised the question of whether it would be possible to automate the process of extracting the references and providing the relevant hyperlinks.
“We want to help all of the people using refworld.org save time and find the most relevant information to help them write the highest quality argument on behalf of asylum seekers,” explains Iris van der Heijden, Project Coordinator for Refworld’s Case Law Collection. “We also hope this will make it easier for people to look for decisions in other jurisdictions, to see what kinds of legal arguments have been used and what judges have been deciding.”
Combining Artificial Intelligence With Human Intelligence
Reconstructing a website is a complex and often expensive process, especially due to the database structure like refworld.org. That’s why the team applied for a grant from UNHCR’s Innovation Service Innovation Fund, which was awarded in 2019.
The team had plenty of lawyers on hand to provide input on the legal aspects, but needed specialists who understood all of the technical details — especially because the proposal supported by the Innovation Fund specifically designated the use of artificial intelligence (AI) to help simplify the process of identifying and tracing the location of the appropriate documents for each case.
“With AI you can do a lot of things that help navigation and usability, which is why we applied for the grant to help bring on the expertise and personnel to get this done,” van der Heijden says. “The legal world is catching up with AI so there’s a lot of great opportunities, but it also poses a lot of challenges.”
For example, she says, it can be complicated to establish a partnership between the legal experts and information technology (IT) experts working on the project.
“You need to have someone who is able to translate the legal information into the needs that would be provided by the IT department,” van der Heijden says. “That’s a general struggle for many people working on either side of the business: legal and IT are like two different languages, so someone has to translate.”
But the team has found strategies to overcome this challenge, such as learning to formulate questions in ways that help the IT experts make the website as user-friendly as possible. The project team includes staff from the RSD Section and the Protection Policy and Legal Advice Section on the legal side and staff from the Digital Engagement Section, who are responsible for the development and maintenance of refworld.org on the IT side. In addition, the team is supported by the Division of Information Systems and Telecommunications. According to van der Heijden, having a team with expertise from all sides of the business is proving effective.
How AI Works: Opportunities and Challenges
With AI already being used in the legal world for tasks such as legal research and dispute settlements, UNHCR wanted to use AI to facilitate the search of the database for every document necessary to build a strong case on behalf of asylum-seekers in need of international protection. This includes extracting citations, finding related cases and decisions, searching for UNHCR policy documents and relevant legislation, and much more. The use of AI can save vast amounts of time and effort — scanning documents of 100 pages or more to detect citations and information, rather than someone having to do that work manually.
AI is more pragmatically known as a “machine-learning approach.” Simply put, artificial intelligence is the broader concept of machines carrying out tasks in a way humans consider “smart.” Machine learning (ML) is a current application of AI, based around the idea that we should be able to give machines access to data and supervise their learning.
The team is collaborating with two external partners that specialize in AI, to figure out the best way to teach an ML algorithm how to recognize various legal decisions on Refworld to train it on what patterns of text it should be looking for. Once the output is received from the model, UNHCR then validates the results.
Depending on the “cleanliness” of the text or the inputs provided to the model, there could be a difference of a single letter in a citation that the machine would recognize as an error or a separate reference, or one paragraph in a document might mention the full case citation while the next paragraph only uses its abbreviated form. So the team is essentially training the model with data from refworld.org to recognize all of these variations. Especially when citations come from different countries, this can pose a particular challenge, because not every country has the same citation standards. In every instance, the team is running tests to make sure the machine is delivering accurate results. In addition, different machine-learning models may deliver different outcomes, which is why the team is testing two different models to better understand the advantages and disadvantages of both solutions.
Real Results With Artificial Intelligence
Through experimentation with AI, the team has made good headway on key evaluation metrics, including the ability of the model to produce results that are relevant and correctly classified by the algorithm. In some cases, the accuracy rate is between 85 and 90 percent.
Ultimately, the goal of the redesign and the use of AI is to simplify the process in a variety of ways. This will mean faster searching and much more targeted searching — and even the clustering of similar decisions about a certain topic from different jurisdictions, all in the spirit of improving the efficiency of the process and the quality of the legal argument. This kind of solution will also benefit staff uploading the information on Refworld, who no longer have to manually extract and link the references from the documents.
UNHCR’s Innovation Service’s Innovation Fund is supporting the project not only financially, but also helping the team learn innovation processes specific to AI.
“Because of the Innovation Fund team and their input, we’ve been learning a lot about how to approach this project from an innovation perspective,” van der Heijden says. She explains that the Fund team has provided valuable advice on prioritization, the actors, products and services on the AI market, various procurement and partnership opportunities, validation processes, and the continuous need for human involvement.
“We’ve realized how important it is to really include the end users,” van der Heijden adds. “You can have a great idea, but if you don’t include the end users to make sure it is useful, usable, and delightful for them, then you’re basically doing it for nothing.”
To that end, the prototype will be tested with end users to better understand the added value, the desirability, and especially the trustworthiness and potential risks of the solution. For example, if the machine returned a reference with an error, such as a typo, would the end user object and, if so, how much? And would the end user consider it an even greater issue knowing that the error is caused by a machine instead of a human being? Depending on the outcomes, an assessment might be made of how to mitigate any potential risks.
In the future, additional challenges related to the accessibility of legal information could be tackled with the assistance of AI, such as that of multiple languages. As van der Heijden explains, legal translation is a world all its own, but with the help of AI it might be possible to provide translated summaries of legal decisions so users can determine whether a document might be useful for their legal research purposes and, as a result, better assess whether it would be worth requesting an official translation.
“There are so many ways AI could improve our processes even further, beyond the scope of the project supported by the Innovation Fund,” van der Heijden says. “Ultimately, we want to provide easy access to quality information related to law and policy, so that those who are working with asylum-seekers who have claims for international protection can build a very strong argument on their behalf.”
