Who writes WTO Panel and AB Reports? A tentative stylometric analysis
Introduction
International trade law figures highly in the news today, for better or worse. An essential part of the story, however, resides in what arguably constitutes the summit of the current international trade system: the WTO’s dispute-settlement system.
This system has been the spoiled child of recent empirical legal studies, with brilliant papers by, e.g., Pauwelyn (“Minority Rules: Precedent and Participation Before the WTO Appellate Body”) or Daku & Pelc (“Who holds Influence over WTO Jurisprudence?”) based on its output to date: more than 450 panel and Appellate Body (‘AB’) reports, an exhaustive source of data to be mined and analysed.
As in other fields of international dispute-settlement, the personality of the decision-maker matters in the decisions taken and the development of a jurisprudence. More than in other fields, however, the WTO system confers an important role to its Secretariats (one for the first-instance panels and one for the AB). As has been examined elsewhere, however, a relatively “strong” Secretariat might be counterbalanced by the appointment of relatively “weak” pannelists, and maybe even AB members. On the other hand, the legitimacy of the whole system might suffer were the Secretariat to be perceived to be too strong.
One area where the Secretariat(s)’s influence can be expected to be felt is in the drafting of WTO ‘judgments’: the Reports issued by Panels and AB members (as well as the arbitration awards) are often very long pieces of legal, and technical analysis —in this context, it is understandable that pannelists and AB members rely on the expertise of their respective Secretariats to get it right.
Yet, the criticisms levelled in other legal fields (notably in investment arbitration) in terms of genuine authorship and the value of non-delegating the drafting of judgments could apply, to an extent and mutatis mutandis, in this context as well.
In what follows, I try to retrace the role of the Secretariat in drafting WTO panel and AB reports by subjecting these documents to a stylometric analysis.
Methodology: Dataset
The texts of the decisions were downloaded from Wolfgang Alschner and Aleksander Umov’s excellent database of international economic disputes documents (see here); additional data was collected from the WTO’s website. The code and data underlying this analysis are on file with me.
The dataset preparation was straightforward; I:
- Collected metadata with respect to every document in the corpus (names of the pannelists/AB members, type of document, date, etc.); and
- Annotated the .xml documents so as to delineate their various parts: intro, facts and procedure (“INTRO”), arguments of the parties (“ARGS”), and reasoning of the pannelists (“SUBS”) (in what follows, I focus only on the INTRO and SUBS categories).
The dataset thus included the following unique documents:
Furthermore, I counted a total of 273 different pannelists, AB members and arbitrators involved in adjudicating these disputes.
As in many other networks of international adjudicators, however, the frequency of appointments at the WTO sees important balances between a small number of repeat players and a large set of individuals appointed infrequently: thus only 49 adjudicators have received five appointments or more, and 25 of them received nearly half of all appointments (nearly all of them AB members, as should be expected for the AB’s narrower pool).
Methodology: Stylometry
Stylometry is “the statistical analysis of variations in literary style between one writer or genre and another.” It is most famously known for attributing unsigned texts to presumed authors. In international law, stylometry has so far had been in the spotlight at least once: in the fight over annulment of the Yukos v. Russia award, where a stylometric analysis attributed most of the award’s text to the tribunal’s secretary.
Stylometric analyses are relatively straightforward to perform, notably thanks to packages like Stylo().** Shortly put, Stylo() analyses a given number of features (“Most-common words”) between different texts, as well as other elements to figure out the degree of stylistic proximity between two texts. Only the raw material (i.e., the words) is used in the analysis.
To make sure that style is not dependent on the topic of a document, words are culled to keep only those words that appear in a significant portion (60%) of the documents. The lists of features analysed for each part of the decisions (INTRO and SUBS) are available here.
Stylo() generates several types of output, but I will be using only two here:
- “Edges” and “Nodes” tables to be used for a network analysis using Gephi (see below); and
- The table of proximity score between each pair of documents.
From the first tables, I create network analysis maps, which allow me to visualise the stylistic distance between the decisions: closer pairs will attract each other compared to documents that are further apart stylistically speaking. (To be sure to capture only stylistic proximities, I discarded all links between decisions that occurred in the same dispute.) Thus, if every point on the map represents one given decision, it should be closer from the other decisions with which it shares stylistic traits.
Findings
I find that for the INTRO and SUBS parts of the decisions, the identity of the president of the panel (or the tribunal) has no clear impact on the stylistic similarities. By contrast, as shown by the network analysis maps, it is the type of proceedings (Panel, AB, arbitration) that is much more likely to indicate stylistic proximity.
Introduction, procedures and factual sections of the documents (INTRO):
Merits parts of the documents (SUBS):
Finally, a large version of the latter map, this time according to the type of proceedings (panel, Article 21.3(c), Article 21.5, Article 22.6). Notice how the compliance reports (which can be issued by both panels and the AB) are dispersed between the AB and panel “galaxies.” Notice also how the arbitral awards in Article 21.3(c) proceedings — in which the tribunal is always a member of the AB — are indeed closer, but not mixed with the AB galaxy (which is displayed in orange in the map above).
(The same maps, in higher resolution, and with node labels, are available here in .svg format — click and then open in new tab.)
These results are borne out by the assortativity scores. Assortativity (or “homophily”) refers to the probability that two nodes in a network are close to each other on account of a shared trait. As shown in Table 2, the personality of the people on the bench has little impact on the stylistic closeness of two decisions, whereas the stage of proceedings, the type of proceedings (merits, compliance or arbitration), or even the identity of the respondent has more predictive power in this respect (absolute numbers below are not meaningful, the difference between the categories is).
However, this does not exactly tells us that adjudicators do not write the decisions, for at least two reasons: (i) if the AB-member universe shares no individual adjudicator with the pannelist universe, we should expect decisions to gather along the AB/Panels divide as they do here; and (ii) contrary to arbitration practice, the role of co-adjudicators (as opposed to the President) might be greater in WTO proceedings, such that focusing on Presiding members is misguided.
To check for these possibilities, I further looked whether the decisions that are closest from one another in terms of style share the same adjudicators and if in this respect they differ from a random pair of decisions at the same stage of a dispute (I leave arbitrations apart for this purpose).
In other words, if every decision had a wholly different bench one from another, every pair of decision should have involved six individuals as adjudicators; a lower number for the decisions that are close (stylistically-speaking) would be an indication of the impact of the individual adjudicators, however their role on the bench.
Table 3 shows the result for this analysis. For both AB and Panel stages, I selected a random sample of 10 decisions to compare with the ten closest — i.e., stylistically — pairs of documents (for each decision).
As expected, the average number of adjudicators is lower for AB decisions, reflecting the fixed pools of AB pannelists available. When looking only at the ten closest documents for every decision, the average number of adjudicators working on these two documents fall — albeit by little.
Conclusions
What to make of this?
First, an important caveat: all analyses here are imperfect: the dataset is imperfect (I did not root out the quotations from report to report, for instance); the stylometric analysis is imperfect (be it only because it is ill-adapted to shared authorship documents); and the findings themselves rely on proxies that are, yes, limited.
Yet, as expected, it seems that adjudicators in the WTO model have a limited impact on the text of their decisions. As a result, and given the clear distinction between AB and panels decisions, it is a fair inference that a number of hands at the Secretariat(s) are responsible for the drafting of decisions. (Although the clear divide between AB and panel decisions can also be explained, to an extent, on the distinct boilerplate language used at each stage.)
The analysis also does not rule out the role of the pannellists and AB members, especially to the extent that they would discuss, revise and adopt a first draft made by someone else: indeed, in general I find that this kind of second-stage interventions on the text has less weight for the purposes of the stylometric analysis, providing the revisions are not too deep.
As explained above, however, the outsized role of the Secretariat(s) is a known fact among WTO participants and observers; yet, if further studies bear out the results of this one, its magnitude might elicit a modicum of surprise.
.
* See Pauwelyn J, “The Rule of Law without the Rule of Lawyers? Why Investment Arbitrators Are from Mars, Trade Adjudicators from Venus” (2015) 109 American Journal of International Law 761.
** Eder M, Rybicki J and Kestemont M, “Stylometry with R: a package for computational text analysis” (2016) 8 R Journal 1, 107–121, url: https://journal.r-project.org/archive/2016/RJ-2016-007/index.html