Not statistically significant does NOT mean insignificant | Rebecca A. Morgan | AME Target

“My fully-loaded 2012 Audi A6 had an intermittent frustrating problem since the day I bought it. No diagnostic codes indicated a problem. Escalation to German engineering had me ready to move back to Lexus. Their response was ‘it must not really be happening. Our codes would indicate if it were.’ That obnoxious response was based on the assumption they had thought of every cause of failure in developing the diagnostic codes. FMEA is not 100% and never will be. Do you have customer data that you’re not actively using to improve your product Four years after I first reported the issue, Audi issued an urgent safety recall for the problem that I had been experiencing. Why the delay?”

Sourced through AME Target

Michel Baudin‘s comments: I am sure many have had similar experiences to Becky’s with customer service in many companies. They tell you their product is used by millions and it’s the first time anyone reports this problem. You are probably using it wrong, or misreading its output,… This being said, it’s not really related to the concept of statistical significance.

Whenever journalists or politicians want to say that a change of any kind is meaningful, they borrow the phrase statistically significant from data science without realizing that, taken out of context and stripped of quantitative levels of significance, it means nothing. Significance is a concept that statisticians have invented to support decisions when outcomes are unclear.

Some experiments have obvious outcomes: as you turn on the ignition the car starts or it doesn’t; others experiments aren’t so clear. It the test plot with fertilizer A yields 3% more carrots than the one with fertilizer B, how firmly can you conclude that A is more effective than B? Whichever call you make, you run the risk of either ranking A as better than B when it’s not or ranking them as equal when A is better.

The statisticians’ levels of significance help you quantify these risks. For example, you may set a threshold in differences between A and B so that a yield above that threshold has a 1% probability of occurring if both fertilizers have the same effect. Then you say that the difference is significant at a 1% level of significance. It matters whether the level is 1% or .001%. Unless you specify the level, statistically significant has no meaning.

Statistical significance is a matter of probability, not volume. In the celebrated case of the Cuckoo’s Egg of the 1980s, Clifford Stoll, the system manager at the Lawrence Berkeley Lab in California, noticed a 75-cent error in the computer center’s books. Since the books were kept using the center’s computers, this small error could only be consistent with normal operations if the computers couldn’t add. Stoll imputed it to hacking instead, and eventually traced it to a German spy ring working for the KGB that had penetrated numerous US military databases.

“Not statistically significant” cannot be used to justify a lack of attention to details or rare events. Within a company, attention to detail is a matter of engagement. It withers where employees are expected to just do as they are told because details at any level are invisible to higher levels in the hierarchy. It thrives where employees are called upon to put all their abilities to use on the job, including their senses and their intelligence, in other words where management shows respect for their humanity. What is a minor detail to their bosses is major to them, and they will take the initiative to address it.

#StatisticalSignificance, #AttentionToDetail, #RespectForHumanity

Leave a Reply

Your email address will not be published. Required fields are marked *