When mass deportation becomes an ordinary survey question
By Mauricio Suárez and Helen De Cruz
Which outcome do you prefer?
A: All EU nationals living in the UK can stay if they continue to work while all others must leave
or
B: All EU nationals must leave
Quite a difference from everyday survey questions such as “Who do you think should succeed Queen Elizabeth”, and distressing to read for people who have made their lives in the UK. Yet, this survey question has been presented as if it is an ordinary option for people to consider in a recent LSE study on British participants’ attitudes to Brexit.
The study by Sarah Hobolt, Thomas Leeper and James Tillier of British attitudes to the Brexit negotiation has received widespread attention in the media. It was initially reported by Buzzfeed, has percolated into Sky news, the Independent, and even the BBC.
Unfortunately, it seems to have been misreported and as a result has generated completely out of place expectations amongst supporters of a “hard” Brexit, while genuinely demoralizing supporters of the UK’s continued membership in the EU.
We won’t comment on the flawed exercise in outreach, which in this case has been obviously messed up and mismanaged, but would like to stress that there are deontological limits and norms for statistical research into public opinion that have not been carefully or appropriately applied in this case. In particular, there are two ways in which we believe the authors of this study have failed to follow the appropriate ethical guidelines. In doing so, they have ensured damaging consequences for thousands of people directly affected or referred to in the study (besides the unpredictable political implications that a change in public discourse may have in the forthcoming Brexit negotiations, which will affect generations of Europeans to come).
First of all, and this is well known, it is unethical to pre-report the results of a study still undergoing peer review for the only sake of maximal public attention or interest, i.e. with an eye on maximizing exposure in the media at any cost. Regardless of how the story was fed to the press and to Buzzfeed in the first instance (and there are obvious questions regarding what interests or purposes it may have served), it is quite clear from the tweeted reactions of the authors that they were initially delighted by the media success of the story.
More worryingly, they seemed to have been happy to go along with the headlines. For instance, the Independent piece run under a headline that read: “29% of Remain voters would accept expulsion of all EU nationals after Brexit” which is misleading as a summary of the results. Nevertheless, one of its authors, Thomas Leeper, was happy to circulate in twitter under a caption: “I am quoted extensively here and hope I do a good job of explaining how to interpret our findings”.
In that piece, Dr. Leeper goes on to assert that “Remain voters are very favourable to the rights of EU citizens, but they’re not so favourable as to oppose that outcome completely”, and he is quoted as following this for effect with the statement that this shows that there is “losers’ consent”.
Both statements are a misleading way of describing what the study, if anything at all, shows. This cannot but make you wonder whether the author was at that point failing to interpret his own results appropriately, or was just happy to run with a headline that would produce a public and notorious splash in the media. If the latter, that was an act not just against the elementary deontology of statistical research into public opinion, but more generally that of public dissemination of any scientific research.
But there is another, more worrying, way in which this incident displays what on the face of it looks like a breach of deontological norms regarding research into public opinion. In general, it is not ethical to conduct research into public opinion under impossible or improbable premises, particularly if those impossible premises are offered to the participants as if they had a genuine choice over them.
More particularly, when the issue at heart is something so sensitive and divisive as the forthcoming Brexit negotiations, it is most inappropriate to introduce options in the survey “for dramatic effect”.
There are two options that the participants in the study were offered that appear dramatic and turn out to be impossible — hence probing questions must be asked as to why they were included in the first place. The two “choices” we have in mind are of course, the putative choice for the UK to exit the EU without a payment on the remaining commitments (“no exit bill”), and the putative choice for the UK to simply expulse all EU citizens (“all must leave”).
As to the first one, most lawyers agree this is legally impossible — and certainly the European commission has made it clear that on their part they believe it to be mandatory on the grounds of the UK’s past legal commitment and undertakings under the treaties. Now the UK could choose not to pay, but it would be pursued in international courts until it does — and, on the counts of most expert opinion, it would end up having to pay. So, the idea that there is here a free choice for the British people is tendentious, to say the least.
The other putative “choice” that the participants were presented with that seems impossible under the law is the choice to force all EU citizens in the UK to leave. The UK has no such powers under international law, and it should never be presented as a free choice for UK citizens to want to demand of their politicians that they exert such a choice.
We now come to what seems to us some of the most evident methodological faults of this study. These impossible premises on their own may seem to fundamentally compromise the scientific integrity of the study, at least as regards its claim to portray faithfully public opinion on this issue of huge significance. In the type of study that has been carried out (a conjoint study, i.e. one where different packages of options are bundled together and offered in order to elicit information regarding the relative priorities of the participants) it is important that no impossible or unrealistic option is included, and that all realistic options are included*.
Failing to do so has the potential to maximally distort the results — since the participants are not asked to provide their optimal absolute measure of support for any option, but are rather asked to implicitly rank them amongst the options available. The introduction of just one unrealistic premise vitiates the remaining choices — since they are only chosen in relation or with regards to whatever else is on offer.
Thus the presence of a choice “not to pay” any of Britain’s legal commitments affects the inclination of the participants for all the other options ranked alongside this (and remember that the study forces participants to choose in all cases). It is plausible that the presence of a “zero payment” option will make all the participants more likely to reject higher sums and to accept smaller sums closer to that zero figure instead. (The other figures also seem bizarre and arbitrary, since they are not in the range of the £36–100 billion figures that have been discussed in the press).
The choice of “all must leave” when applied to EU citizens is similarly tendentious: Even though the results of the study show most people (leave or remain sympathiser) are very unwilling to accept this option, it nonetheless stands to reason that the other “hard” options against EU citizens (such as e.g. “leave unless you have work”) would receive greater acceptance in light of the inclusion of such an extreme option.
Importantly, the exclusion of an option “to stay” fully in the EU is unjustified, since by all present accounts — both in Europe and in the UK — it remains a possible (some say likely) outcome of the negotiation process. The only options offered to participants at present range from soft Brexit, to hard Brexit, to a no-deal Brexit. This erasing of the option to remain is demoralizing for the Remain movement. The current government has made the political decision to take Remain off the table, but academic researchers should make their decisions to design surveys in a spirit of free inquiry and political independence. While it is not known how large the percentage of “Hard Remainers” is (polls suggesting over 20%), it is surely large enough to include in a survey alongside the extremist options of no exit payment and mass deportations.
For all these reasons, it would seem that the methodology of the study is flawed for its purpose, and nothing much can possibly be reliably inferred from it other than i) the fact that most people, regardless of whether they are leave or remain supporters, are indifferent or ignorant about the details of the negotiation (the border in Northern Ireland being an egregious case), and ii) that leave and remain supporters differ in some of the key aspects of the negotiations.
It would be desirable in any case that the authors of the study show the humility to offer some retraction of the views they expressed on twitter, and that the LSE undertakes to formulate a press release correcting the very significant and potentially damaging misinterpretations that are circulating in the media.
—
*The bundle options also make it difficult to draw straightforward conclusions about levels of support. One could design a bundle study that would allegedly show that vegetarians would be willing to eat foie gras. See here for an explanation.
Originally published at medium.com on August 14, 2017.
