Beyond Significance Testing

A key focus of market research is the interpretation of statistical differences and whether they actually matter. When marketing teams want to know how pleased or worried to be about a change, or whether a difference between two numbers confers  leverage, they routinely ask researchers, “Is it significant?”  By which they mean: Is this difference real or merely the product of chance?  Although the question can be answered based on the simple calculation of a p-value, the relevance of the answer is less straightforward. P-values have long been a North Star for guiding decisions based on data, but they can also lead us astray. It’s important to understand what p-values offer, what their limitations may be, and what else we can do when p alone cannot provide adequate decision support.

Significance testing has infiltrated even our popular culture – so much so that an American voter who keeps an eye on polls is familiar with terms like “statistically significant” and “within the margin of error.” Ironically, researchers and statisticians have been growing more prone to challenging its use in a variety of situations. Uneasiness about the way p-values can be misinterpreted (and abused) has led prominent organizations like the American Statistical Association and the American Psychological Association to largely abandon the use of Null Hypothesis Testing (NHST) in favor of a different estimation framework that shifts the emphasis toward the magnitude of difference between numbers and away from the probability of observing that difference by chance. One respected academic journal has gone so far as to say it will not publish p-values at all. So after years of colorful p-thrashing by statisticians themselves, there is growing consensus in the sciences that we need a shift in focus from significance to meaningfulness. But despite a much-trumpeted focus on methodological innovation, the “insights industry” island has been largely insulated from the debate about p for several good reasons.