FTFY: Wikipedia tested something that probably didn’t work out.

While I don’t fundamentally disagree with this being a “mistake” the argument is flawed and incomplete.

First there’s the title (completely lacking any nuance) that might as well be “I am right and Wikipedia is wrong”. Come on–just throw a “probably” in there. The very topic of picking apart something and judging it in a vacuum, with no knowledge of the business context, no knowledge of the why and the how and the where, is itself flawed and is always going to lead to an incomplete picture–so at least leave some room for doubt in there.

Then there’s the idea that making mistakes is inherently bad. It’s not. And this is probably my biggest issue with this sort of article…

Let’s take it one point at a time. Most of this is based on an example from “Yes!: 50 Secrets From the Science of Persuasion”. For the reasoning to be true the example should be correct and the example should be equivalent to the situation that is under consideration.

So is the example valid?

Cialdini is a pretty smart guy and he normally knows what he’s talking about. So I’m inclined to believe him… But:

1. Methodology

a. Is pretty hard to argue that this a fair test. Does the test “change one factor at a time while keeping all other conditions the same”? Not really. Both the message and the graphics are changed. The “winner” also has a strong extra symbol described in the book as “a red circle with a diagonal bar in the middle”. So not exactly a fair test, is it?

b. Distribution looks problematic. The signs were placed on different paths with no control of factors like: composition, size and number of groups walking those tracks. Even with varying the signs it doesn’t look like there was enough control over visitors or over the environment to have clean results. It’s not even clear that the number of visitors on each path was equal.

c. And then there’s statistical confidence. There’s no information about how many visitors saw the signs or how long the test was run. But there’s a chance a small sample size and high noise (point b.) skewed the results enough that even a huge margin (2.92% vs 7.92%) is nothing but random variance.

So there’s a big question mark over the statistical significance of this test.

And is the example equivalent or at least similar enough to this situation?

Even if I have serious doubts about the example being as text book as it sounds the question remains if the two are similar enough.

a. Context — being in groups makes people stupid. Seriously, just look at any crowd of people leaving a sports event. Most will behave in a way they would never do in normal circumstances. And tourists generally act like they shut down a part of their brain they usually keep on. So the strength (and even the direction) of the social proof message could be a lot different for them than for someone that is alone in front of a Wikipedia screen.

b. Target–are the groups compared similar enough? Are the people taking a walk through a petrified forest the same people that are likely to donate to Wikipedia? Probably not. And, while they are fundamentally not that different in the way they react to the kind of persuasion Cialdini preaches, it could be that there are just enough more people on Wikipedia that would like to stand apart from the social norm rather than conform to it for this to be the right message. Maybe…

I’d say with enough doubt over the validity of the example and over the equivalence of the two situations there’s a chance the demonstration is just wrong.

Admittedly proving the demonstration to be wrong does not invalidate the conclusion. And the conclusion here is most likely correct. But that’s really not the point.

The point is this was probably run as a test.

A much fairer test than the one in the example with a large enough sample size and great control over visitor distribution. And whoever ran this test now knows exactly how the message performed over their default message, and with that learning, even if the test failed, they are one step further in achieving their goal of getting more donations.

Seriously–You should test that!