The impartiality of some French judges undermined by machine learning
The French administrative judge must be independent and impartial (Art. L721–1, R721–1 et seq. of the Code of Administrative Justice on the disqualification or the famous Article 6 of the European Convention on Human Rights). Is it seriously conceivable to expect from a man, however virtuous, a perfect impartiality? Probably not, and this is what “predictive” algorithms show us.
1 / A new context and new possibilities
As part of its open data policy, the French State  has made available to all over 350,000 judicial decisions corresponding to the content of the Legifrance website . Due to the quantity of texts, the existing research tools are sometimes inadequate.
At the same time, the technological community (universities, research center, small and large companies) has dramatically improved predictive algorithms dedicated to the understanding of human language. The state of the art allows us to extract reliable and accurate information from documents as a human would.
This new combination, where legal data and powerful algorithms are available, offers the possibility of massively analyzing jurisprudence in order to extract a synthetic and unpublished vision.
Supralegem.fr applies those algorithms to extract, among others, the quality of the plaintiff  and the defendant, the nature of the legal provision  or the theme of the decision .
To do this, we use algorithms to read many texts and then we ask them millions of questions, all texts combined. When the answer is false, the algorithm update itself to avoid that error in the future (or at least make it happen less often). The procedure is repeated a hundred times. A specially equipped server, whose gross power is equivalent to approximately fifty conventional computers, takes between 5 and 6 days to run everything. Finally, the results can be queried by lawyers in a few seconds.
The accuracy of our results is between 90% and 99% depending on the extracted field. It is to our knowledge the first time that an algorithm having a form of understanding of the meaning of words and sentences is applied to French jurisprudence.
The rest of the methodology is simple. Decisions dealing with a specific kind of legal case are selected and aggregated to calculate statistics by judge or court. We can then compare, for example, the rejection rate between several judges.
This approach is truly innovative as the trends are based on objective and verifiable data and not on subjective points of view and rumors. This is a real paradigm shift.
2 / The case of Obligation to Leave the French Territory (OQTF in French)
A possible application of this approach concerns requests to cancel Obligation to Leave the French Territory (OQTF below). This is the expulsion measure that may be imposed on foreigners in irregular situations resulting in deportation to the border. When the application is rejected, it means that the foreigner must leave the French territory. This dispute is the most important in volume in administrative jurisdictions .
The partiality of certain judges in the matter of OQTF is a sensitive subject several times denounced by different actors of the legal system like the judicial assistant @bismatoj on Twitter in 2015.
The judicial assistant is a young lawyer who assists a magistrate by drafting decisions. The judicial assistant is entrusted with files by the judge rapporteur, and is himself responsible for analyzing the exchanges of documents and the documents produced before the court. From then on, he forges a legal conviction like the magistrate, and proposes a sense of judgment that the administrative magistrate is free to follow. The magistrate has the last word, but on simple cases, he will most often follow the work of the assistant.
According to this justice assistant, some presidents of Chamber will, in the case of OQTF’s cancellation applications, systematically request a draft of a rejection judgment before the case has been examined independently of the legal and factual context. A judgment of rejection is proposed by the justice assistant, even when it is contrary to his conviction and that the case is strong and in favor of the foreigner. For those judges, according to the justice assistant, the discharge would, in fact be pronounced in 90% of the cases.
The algorithms of Supralegem.fr allow us to verify this assertion by selecting decisions that concern OQTF cancellation requests and calculating the rejection rates per year, per judge and per court (as explained before, they are simple statistics of the data extracted with machine learning algorithms).
We select decisions that deal with the right of aliens which contain the terms “leave the territory” (“quitter le territoire”), “alien” (“étranger”) and “asylum” (“asile”). We only keep cases where the applicant is an individual and where the defendant is the administration.
We can extract the lowest and highest rejection rates among the judges who made the highest number of decisions. “Juge” means the President of the Chamber who has ruled in the remainder of this article.
The average discharge rate for these jurisprudences, for all judges combined, ranges between 78% and 81%  over the period 2012–2015.
The difference between the extreme rates is striking: more than 40 points!
The rejection rates per judge are rather constant over time, indicating substantive trends. In each court, the cases are randomly distributed among judges. Therefore, there is no judge specialized in a specific country or any similar explanation for these rates variations.
Mme Tandonnet Turot has the highest rejection rate, and during these years, she was the president of the French administrative court of Paris, meaning the biggest and most prestigious administrative appeal court of France. Her case is interesting because Mr. Krulic is working in the very same court, at the same time, and has a much lower rejection rate. Again, cases are distributed randomly between judges of a court. Both have judged hundred of cases during this period.
Only a careful reading of the support documents of each case would enable us to decide the merits of each decision, which is impossible since those documents are not disclosed in France. However, we have very significant statistics indicating that there is an apparent bias  among some administrative appeals judges.
To check the consistency of the results presented above, we can conduct the inverse experiment by calculating the rejection rate when the requester is the administration and the respondent is an individual (the other selection criteria are identical to those used previously).
We find that the strict judges with the individuals are the most conciliatory with the administration and vice versa. Thus, there seem to be some OQTF judges biased to administration and others more defiant.
It is important to note that we are showing judges having the most extreme rates.
3 / The extension to other questions of law
The project is based on a simple observation that the judge remains a human being. Subjectivity is, therefore, possible.
Supralegem.fr takes  the administrative jurisprudence on Légifrance and proposes research and analysis tools based on its automated analysis.
The search engine allows you to create statistics for each administrative appeal judge in order to reveal new insights. As a result, the user knows the doctrinal position of his judge on his question. This knowledge can reduce the risk associated with the practice of litigation.
The website also facilitates jurisprudence searches using different filters based on the extracted data, such as the nature of the legal provision, the nature of the plaintiff and defendant, or the category of the decision.
In administrative law, the administration wins more than it loses. More than 70% of motions submitted by a private party are rejected. Removing 70% of applications often allows to keep the interesting part, such as cases where the application for annulment of an administrative act has been made, where the taxpayer has been relieved of tax payment, etc.
Specifically, a lawyer of an asylum seeker will be interested to know the decisions where a foreigner from the same country that his client won against the administration in an OQTF case. The filters will allow the lawyer to only see the decisions of cancellation of an OQTF where the plaintiff is an individual, the defendant is an administration and the decision contains in its text the expression “leave the territory” (expression Which can be found in all OQTF decisions) and the name of the client’s country. Thus, the lawyer can avoid reading thousands of unnecessary decisions for his case.
It can also, only see the decisions where the administration is the applicant and see its application rejected, as in our analysis of OQTFs.
To our knowledge, this project is a first in France, both by the number of decisions on which the algorithms were applied and by the sophistication of the algorithms used and the quality of the results found. 
If you want to go further, do not hesitate to read our article “6 Ways to Use Predictive Algorithms to Improve Your Jurisprudence Research”.
 Via the website of the Etalab mission
 Decisions available in open data represent one to two-thirds of all the decisions issued by each Court of Appeal over the period under review.
 Private person, government, public power …
 Cancellation, rejection, discharging of taxes, suspension of proceedings …
 Among a forty, like right of the foreigners, taxation, urbanism …
 More information on this dispute here.
 The Tweets in question are available via this link.
 The high average discard rate is explained by the nature of the litigation and the fact that the administration issues largely based OQTFs.
 It is only apparent because we do not have more elements than these statistics.
 From 2000 to the present.
 The site is in constant evolution and we invite the reader to make returns to the authors (on Twitter @supralegem or by email firstname.lastname@example.org). We are as interested in the uses that you envisage as a constructive criticism of the existing functionalities.
This article was first published on the Village of Justice.