Our experiments typically have relatively large sample sizes so our statistics are CLT (Central…

Jonathan Parks

62

Thank you Jonathan! This is very helpful. With metrics formed from Bernoulli trials (which is what I think you’re referring to in the Numerator/Denominator sense), the first two moments are sufficient. In fact, with only the first two moments, you are bound to use the Normal distribution.

While I can see a metric like “searches per searcher” being reasonably well-behaved, it certainly doesn’t need to be normal, although I suppose the normality assumption is sufficient?