I’m covering the International Stroke Conference in San Diego. It’s an excellent meeting, sponsored by the well-respected American Heart Association. The AHA has long been a pioneer in communicating science to the public through the science press. The meeting is chock full of important news, and there’s no lack of interesting studies for the assembled news media to cover.
Which is why it’s so puzzling that at one of its news conferences today the AHA chose to feature a study whose results did not meet statistical significance. Now I do think journalists and scientists should pay more attention to negative results, but that’s not what happened here.
The researcher presenting those results never mentioned that the results weren’t statistically significant, presenting her results as if they were significant. I noticed this and challenged her on the statistics. Other reporters pointed out that her conclusions were not as strong as she was implying even if the results had reached statistical significance!
Here’s a link to the abstract, by Lynda D. Lisabeth, Ph.D., et al. It was Dr. Lisabeth, an epidemiologist at the University of Michigan, Ann Arbor, who presented the results at the news conference. According to Dr. Lisabeth, the study shows that women who have strokes are 42% more likely to have non-traditional stroke symptoms than men. If true, this could be important, and public health professionals should warn women to look for these symptoms.
Dr. Lisabeth’s main result reached either a p value of 0.09 (according to the abstract and the slides) or 0.07 (according to Dr. Lisabeth’s answer to one of my questions during the press conference). Results are not considered to be statistically significant unless the p value is under 0.05 (meaning a 1 in 20 possibility that the results arose from random chance).
One result did, apparently, reach statistical significance, with a p value of 0.03. The investigators looked at comparisons between men and women on 13 different symptoms, and according to this analysis, women were more likely to experience mental confusion when having a stroke. But there’s a problem in statistics when you make many comparisons like that. If you make 20 comparisons, you’d expect at least one to reach a p value of 0.05 purely by chance. Statisticians have a way of correcting for multiple comparisons. I asked Dr. Lisabeth if she applied that correction, and she admitted that she had not.
What was her explanation for presenting these results as important when they were not? She said, “I, as an epidemiologist, do not go by those strict 0.05 cutpoints.”
What a remarkable statement. It’s well known in medical research that all epidemiological studies have to be taken with a grain of salt, since epidemiologists can’t conduct randomized, placebo-controlled studies. And that grain of salt is necessary when epidemiological studies do reach statistical significance. When even an epidemiological study fails to reach statistical significance, I think most scientists would feel justified in ignoring it. Why then was it featured in a news conference by a respected organization at an important scientific meeting?