The Problems with Animal Charity Evaluators, in Brief
The animal protection group SHARK recently released a video essentially accusing Animal Charity Evaluators (ACE) of deceiving donors in order to funnel money to a few charities associated with Nick Cooney. I found that this video, which includes a short clip from an interview I did with SHARK’s Stuart Chaifetz, overstates the case against ACE a little bit. That said, I generally agree that ACE’s work is not what donors may expect, that it is not rigorous or objective, and that it is strongly biased toward a few nonprofit corporations, especially those in which Cooney is involved. ACE’s work is complicated, so I thought it might be helpful to explain, briefly, what a few of the biggest problems with what it does and attempts to do actually are, in my opinion.
ACE is not a charity watchdog
One of SHARK’s core objections is that the charities ACE evaluates are limited to a very select few. This is simply because ACE, despite its name, is not a charity watchdog. Rather, ACE is a product of the utopian, ultra-utilitarian Effective Altruist subculture. Effective Altruism is all about doing the most good per dollar, which, depending on whom you ask, may mean anything from preventing malaria deaths in developing countries to giving Eliezer Yudkowsky money to fund mostly-unpublishable research on the far-future safety of artificial intelligence. For ACE, it mostly means donating to farmed animal welfare nonprofits, since these affect the most land vertebrates, and few nonprofits deal mainly with marine animals. (Fish Feel is one of them.)
Additionally, ACE requires that charities have “room for more funding,” which in practice means requiring that they have already attained a certain size (unless ACE decides that a charity can use donations many times greater than its budget, as it did when it first recommended Animal Equality.) Unacknowledged are ideological considerations: ACE will probably never recommend United Poultry Concerns, a medium size nonprofit dedicated to advocating for farmed birds, because of the latter’s stance in support of animal rights as opposed to utilitarianism. ACE would likely defend this by saying that such charities are not “effectiveness oriented,” or something similar.
The upshot of this is that ACE may not be at all what donors expect, unless they are Effective Altruists. Whether this is a problem for you largely depends on whether you are an Effective Altruist, and whether you understand and accept that ACE’s recommendations are based on an opinionated concept of “effectiveness.”
ACE relies on unscientific research and distorts the results
Since its beginnings, the form of activism ACE has shown the greatest interest in evaluating has been vegan or meat-reduction advocacy. ACE reasons that since the average American consumes about 30 land animals per year, one may try to measure the number of vegans created by some advocacy program, multiply by 30 times the expected number of years they will remain vegan, and count the result as the number of “animals spared.”[n1] It’s a nice theory, except that it turns out to be extremely hard to measure the effects of advocacy on people’s behavior, and in fact, despite several years of effort and dozens of unpublished “studies” (most of which were unscientific) Effective Altruists have struggled to produce convincing evidence that meat-reduction advocacy efforts have any effect on people’s behavior at all, let alone to determine whether one type of message is better than another.
Currently, ACE estimates that The Humane League spares 3.4 animals per dollar by leafleting, expressing their “uncertainty” by giving a range of estimates which go as high as 6.1. If the mean estimate were correct, it would indicate that giving ten dollars to The Humane League, which they would use to distribute leaflets that encourage people to consider vegetarianism, would spare more animals than being vegan for a year. Their online ads are estimated to spare 18 animals per dollar (again, expressed as a range) making a two dollar donation better than a year of veganism.
As I previously explained in greater detail, the effectiveness estimate for leafleting is solely based on one grossly flawed uncontrolled study, despite ACE having another study which found no effect compared to control, and in actual fact the uncontrolled study found that just one of 489 respondents became vegetarian. ACE then used an arbitrary construction to estimate that 5 out of 489 people would. (This wasn’t meant to represent meat-reducers, who were estimated separately.) ACE even provided a “leafleting calculator” which allowed one to calculate how many animals could be spared per dollar based on ACE’s manipulation of this study. Though it remained on ACE’s website for years despite the complaints of several critics, it has finally been taken offline. However, ACE hasn’t retracted any of its work, instead asserting that the problem was “confusion” about its degree of confidence in the cost-effectiveness estimate for leafleting, which it had described as the “top intervention.”
Of course, the real problem here isn’t with the strength of the evidence on leafleting, but with the way ACE misuses data to arrive at transparently absurd conclusions. It also doesn’t make much sense to group advocacy efforts according to the delivery system: there are huge differences in content between the out-of-print Your Choice, and Vegan Outreach’s new leaflet What is Speciesism? Similarly, there is a huge variety of online ads, and I’m sure some are effective and others aren’t, although it’s almost certain that none of them spare 18 farmed animals per dollar. ACE’s evaluation of their effectiveness is incredibly complicated, but mainly rests on guesswork, since, as ACE admits, there is no good evidence wherewith to quantify their impact.
ACE evaluates the effects of corporate outreach in an unrealistic way
In ACE’s jargon, “corporate outreach” refers to campaigns targeting animal-using corporations, which ask them for typically nonbinding agreements to phase in welfare reforms, often after ten years or more. Whereas the evaluation of vegan or meat-reduction advocacy requires nonexistent data about its effect on consumer behavior, evaluating the effects of corporate welfare campaigns may seem more straightforward, since one may try to count the number of animals directly affected by a reform commitment. With a number of caveats, this is essentially ACE’s approach to evaluating corporate welfare campaigns. In practice it also results in exaggerated figures.
In fact, ACE estimated that The Humane League’s overall efforts spared 500 animals per dollar in 2016 (expressing this as a range from 95 to 1,700) and the main reason for the gigantic figure is an agreement with United Egg Producers to eventually introduce sexing technology which would replace the practice of grinding up live male chicks in macerators or suffocating them in bags. The new technology was developed by the industry because it is more efficient than the current method. The agreement states that the industry will introduce the technology by 2020 “or as soon as it is commercially available and economically feasible” — in other words, when it wants to. For getting the industry to agree to introduce a more efficient technology at such time as it’s profitable to do so, ACE determined that The Humane League was 90% responsible for the change. So, it estimated that the industry’s new equipment would spare 180 million male chicks annually from culling, and credited The Humane League with 90% of that figure. In reality, while the new technology will end a terrible practice, the impact of the meaningless policy agreement on animals is probably zero. It nevertheless generated good press for both The Humane League and the egg industry. In a congratulatory article, Vox’s Dylan Matthews announced: “This solidifies the Humane League as one of America’s most important animal rights groups.”
Of greater concern, to me, is the ongoing campaign to promote Global Animal Partnership standards for broiler chickens, which involves both Mercy for Animals and The Humane League, as well as other nonprofits. One goal of this campaign is to introduce gas chambers, which PETA has advocated for 15 years as a “less cruel, more profitable method of chicken and turkey slaughter.” Gas chambers are “more profitable” in terms of marginal costs, so anyone who wanted to open a new large scale chicken slaughterhouse would use such a system, but companies that already have investments in older equipment have long resisted the change. However, now that much of that equipment is aging and needs to be replaced anyway, the industry is more willing to phase in the “more profitable” system.
The real impetus for the broiler campaign, though, is some retailers’ desire to introduce slower-growing chicken breeds, which they can market as “ethical” and better tasting, and sell at premium prices. Perdue, the only company with which Mercy for Animals had negotiated a policy as of ACE’s last review, has shown interest in feel-good marketing of slower-growing chickens since 2014, so this, too, may fall in the category of a nonprofit negotiating a change that was already happening. To me, though, the most troubling thing about this campaign is the direct harm it may cause. According to an industry document, the meat yield of slower growing breeds may be about 15% lower than the conventional ones. This is not an exact figure, because different breeds and practices will produce different results, but if it’s even in the ballpark, it means that hundreds of millions of additional birds would be farmed to produce the same amount of marketable body parts. As ACE has always been intensely concerned with reducing the number of animals used and killed in agriculture, I find it shocking that it has made no effort to address this pitfall at all. The prospect of additional birds being raised and killed is not mentioned in its calculations of the effectiveness of this campaign.
To calculate the cost-effectiveness of cage-free campaigns, which advocate aviary systems for egg-laying hens, as opposed to battery cages, ACE tallies the number of hens that may be affected someday by the non-binding agreements (93 million in the case of Mercy for Animals last year) and then multiplies this by
- The proportion of the campaigns it expects the nonprofit is responsible for (33% in the case of MFA)
- An arbitrary assessment of the suffering-reduction due to switching to the new system (valued at 10% of the total suffering in a battery hen’s life)
- A duration of time, which functions as a cutoff (5.6 years).
The product of these figures becomes the “equivalent hens spared,” and is 18 million in the case of MFA. One problem with this analysis is that it isn’t clear whether these reforms offer improvements at all. The main reason why they might not is the horrifying phenomenon of cannibalism, a learned behavior which can emerge in highly stressed and crowded groups of birds. When chickens have less than 1.5 square feet each, the risk of a cannibalism outbreak significantly increases. One might hope that organizations advocating for the transition to aviaries would fight for basic floor space standards, but they have not. Many of the policies, like Walmart’s, require just 1.0 square feet per bird. ACE doesn’t differentiate between policies according to their content, supporting the nonprofits’ emphasis on fast deals rather than good deals.
Last year, The Humane League announced over 100 “victories.” Reflecting just how closely aligned ACE’s views are to those of the nonprofits it evaluates, it uses the same language in its evaluation: the number of “corporate policy victories” is an actual statistic in ACE’s cost-effectiveness analysis of The Humane League.
To seriously evaluate the results of welfare campaigns would require a very different kind of analysis to that which ACE conducts. It would require considering corporations as intelligent actors and predicting how they would behave in the absence of animal welfare advocates. The impact of The Humane League’s agreement with United Egg Producers, for example, should be valued as zero because it only requires the new technology to be introduced when it makes financial sense to do so, which is the default behavior of for-profit corporations. A serious analysis would require examining the substance of policies, rather than just tallying them, to avoid rewarding dysfunctional behavior like The Humane League’s perpetual victory machine. Most importantly, it would consider the corporations’ motivations for entering into agreements with the nonprofits. Often , corporations do this because they can exploit the credibility of activists to earn public goodwill. Such corporate cooptation is a common problem across many fields of activism, and can be exacerbated by “bottom-line” measures of effectiveness. Make no mistake: the corporations do get their pound of flesh, which is how the co-founder of a major vegan advocacy group ends up announcing a butchery competition.[n2]
Of course ACE is not at all equipped to resolve these issues, but that’s also part of the problem. The basic lie of animal-focused Effective Altruism is the idea that we can objectively determine how best to advocate for animals using quantitative analysis. The real world doesn’t work like that: while data and experiments can be useful to advocates in various ways, there is no science of politics. The false narrative of effective pragmatists versus unteachable ideologues, which is maintained by ignoring or caricaturing the latter’s arguments, does nothing to advance a necessary conversation about the direction of the animal rights movement.
~
At the Animal Rights National Conference last weekend, I stood up during a question-and-answer session to ask an ACE researcher to respond to fundamental criticism of the organization’s legitimacy. I cited psychologist Casey Taft, who has run large scale randomized controlled trials of behavior change interventions, for the assertion that some of ACE’s work is pseudoscientific. I also cited sociologist Corey Wrenn, who recently has written extensively on factionalism within the animal rights movement, for criticism of ACE’s connections to the organizations it recommends. Finally, I requested a written reply to the moderately detailed critique I wrote in December.
I felt a bit bad asking the speaker this, partly because I knew that she was not in a position to productively respond, and partly because I’m aware that a Q&A session is not inherently an invitation to make such demands. However, I could not think of any other way to get to the heart of what appear to me to be the complicated, numerous, and pervasive problems.
After the speaker politely stated that it was someone else’s call, ACE president Jon Bockman stood up behind me, smiling, and asked me for the microphone. Naïvely, I expected him to say that ACE would be issuing a written statement on the subject. Instead, he said that ACE “has many enemies,” brought up an out-of-context facebook comment I had made to the effect that if I were in charge of ACE I would close it down, and stated that this demonstrates that I don’t “have the interests of animals at heart.” In the heat of the moment, I touched the microphone, asking him to let me clarify. Bockman immediately made me regret it: “And he’s now trying to grab the microphone from me,” he said, “which I think shows what kind of person he is.” When he was done, the moderator, who works for Mercy for Animals, did not allow me to respond.
Afterwards, I briefly spoke to two other ACE employees. I found that they believed that, even though ACE has refused to publicly address its harshest critics, it has an argument which it chooses to keep private. When I narrowed the discussion to ACE’s cost-effectiveness estimate for leafleting, which I consider a particularly egregious and clear example of its abuse of data, one of them asserted, flabbergastingly, that he thought some useful information could be derived from the study on which it was based. I am willing to entertain the possibility that I have been wrong, and if ACE really does believe it can defend its work, I hope it will do so publicly.
The most fundamental problem with ACE, though, is revealed by Jon Bockman’s assertion that someone who doesn’t value ACE must not care about animals. ACE has two conflicting roles: it is both a research organization that aspires to determine the effectiveness of animal welfare advocacy, and an advocacy organization that promotes farmed animal welfare nonprofits. To argue, as I would, that the two nonprofits it has promoted most extensively are inefficient and dysfunctional, endangers the latter goal. When supporting certain organizations is equated with doing good, it becomes impossible to objectively evaluate them.
Notes
[n1] ^ Actually, ACE’s analysis also includes an accounting for elasticity, which reduces the expected effect of a year of veganism on the number of land animals used and killed to less than 9. If both this and the mean effectiveness estimate for online ads were accurate, it would imply that online ads create the equivalent of two vegans per dollar.
[n2] ^ Miyun Park, pictured here, was acting on behalf of Global Animal Partnership, the same pro-meat-industry organization that wrote the welfare standards which The Humane League and Mercy for Animals are currently promoting through their broiler chicken campaign.