Thanks for sharing, Brian. It’s great to read about how you are using lean experiments like this to make product decisions at Buffer. I hope that this will encourage others to do the same.
Running experiments is fairly easy, fast and cost effective as you described in your post above. However, every experiment needs a falsifiable hypotheses, i.e. they can be proven incorrect, so you are clear about whether the experiment has been successful or not (if it is not, the experiment still provides great insight and may get you more prospects for future experiments as you got through the “Notify-me” button). Creating falsifiable hypotheses is what I see people struggle with when designing experiments. Mostly, their experiments are vague, optimistic and untestable, e.g. “Managers of swimming pools want to save water”.
With that said, would you care to share your hypothesis for your experiment? I think this would be really helpful to anyone wanting to design their next experiment.