Thought blog — Analytic Activism

I don’t know this to be a fact, but my hunch is that Donald Trump’s campaign did very little A/B message testing and Hillary Clinton’s campaign did tons. There are countless differences between the two candidates and their campaigns so the outcome of the election shouldn’t mean that A/B testing was ineffective, but I wonder if the short-term gains of testing (more money raised, more volunteers, etc. ) can come with a long-term loss in perceived authenticity. Obama didn’t have this authenticity problem because he came across as a very natural and genuine candidate. Hillary Clinton, on the other hand, was widely seen (by supporters and opponents) as being disingenuous and managed. It may take a candidate or a spokesperson who comes across as particularly genuine in order to mitigate the possible negative effects of testing.

With that said, it is not like a supporter feels like they are being manipulated when the donate button changes from red to blue. The problem arises when focus-grouped language is used over and over again, and everything a candidate does is tested — then it can feel unnatural. Even if one test, in isolation, doesn’t feel manipulative, I wonder if the overall feel that comes with the “let’s test it” attitude doesn’t begin to feel a little fake and doesn’t cause the candidate to second guess themselves when there isn’t time to test. In short, can the testing of tactics have a negative effect on the long-term strategy?

Karpf addresses this question and acknowledges that it is much harder to A/B test the long-term effects of a particular action or strategy. Just because option A got more people to volunteer, doesn’t mean that it contributed towards the overall goal. For example, if anger and fear recruited more volunteer canvassers than nuance and understanding, then the former would be seen as the more effective message. But, if the volunteers show up at an undecided voter’s door and are righteously dismissive of the opposing candidate, then this could have a negative effect on the campaign, even though the angry and fearful message recruited more volunteers. Even though this is entirely speculative, my point is that a successful tactic does not automatically advance the overall strategy. In addition, if the long-term effects of repeated exposure to angry and fearful messages leads to divisive and hateful politics, then this could also harm the overall strategy. There may be a danger in looking too narrowly at any specific tactical outcome, for a change in one area often influences other areas. A victory in one area does not necessarily advance the overall strategy or get you closer to the goal. The message that may resonate with volunteers or donors, may turn-off undecided voters or may energize opposing voters.

Karpf quotes Brian Christian, who says “A/B testing is not simply a best practice — it’s also a way of thinking, and for some, even a philosophy. Once initiated into the A/B ethos, it becomes a lens that starts to color just about everything — not just online — but in the offline world as well” (p. 12). On first reading, I felt like this was a little overstated, but after thinking about the statement and the broad principles behind A/B testing, I think I understand what Christian is saying. First, A/B testing or the scientific method requires us to have an open mind in order to test our assumptions. We have to be willing to question what we believe and look for evidence that goes against our beliefs. When we think something should be done in a certain way, our conclusion is often based more on an unarticulated gut feeling then on specific evidence, which is often incomplete. That instinctive pull often gives us enough certainty to act or to express what we think. The ethos of A/B testing encourages us to go beyond our initial assumptions. Even if we never run an A/B test, by questioning our assumptions and grappling with counter-evidence, we have done a little to escape the effects of selective exposure. Keeping an open mind and attempting to see things from an alternative perspective is beneficial even if an A/B test is never conducted. Because life unfolds in an uncontrolled environment where there are countless influences, it is not typically possible to isolate a single variable and conduct a controlled experiment. This doesn’t mean that the scientific method is inapplicable; in a highly complex situation, the scientific method tells us that we should have a high level of doubt about our conclusions. By recognizing the complexity and the incompleteness of evidence, it allows us to hold our conclusions more softly and be willing to change our mind as new information arises. Doubt and uncertainty gets a bad name, particularly when it comes to leadership because doubt gets in the way of action, but inaction is sometimes appropriate. Doubt, as unsatisfying as it may be, is often the most honest attitude to take.