Statistical Versus Clinical Signifigance
In a preceding section, we established the basic link between the magnitude of an observed difference and the calculated probability that such a difference could have arisen by chance alone. The game plan is to determine this probability, and if it is sufficiently small, to conclude that the difference was unlikely to have occurred by chance alone. We then end up saying something like, “Readers of the book are significantly smarter than the general population.”
And that is the meaning behind statistical significance; that is, the probability of the observed difference arising by chance was sufficiently small, and therefore, we can conclude that the IV had some effect. It’s really too bad that someone in the history of statistics decided to call this phenomenon “statistical significance” as opposed to, say, “a statistically nonzero effect” or “a statistically present effect” because the term is, somehow, so significant. The basic notion has been perverted to the extent that p < 0.05 has become the Holy Grail of clinical and behavioral research, and that p < 0.0001 is cause to close the lab down for the afternoon and declare a holiday.
Let’s take a closer look at what determines that magical p level. Three variables enter into the determination of a z score (and as we shall see, nearly every other statistical test): (1) the observed difference between means, (2) the SD of the distribution, and (3) the sample size. A change in any one of these three values can change the calculated statistical significance. As an example, we can examine the results of the reader IQ experiment for sample sizes of 4 to 10,000. How would this variation in sample size affect the size of difference necessary to achieve p < 0.05 (that is, statistical significance)? Fig.1139displays the results.
So a difference of 10 IQ points with a sample size of 4 is just as significant, statistically, as a difference of 0.2 IQ points for a sample size of 10,000. We don’t know about you, but we would probably pay $100.00 for a correspondence course that raised IQ by 10 points (from 140 to 150, of course), but we wouldn’t part with a dime for a course that was just as statistically significant but raised IQ by only two-tenths of a point.
Figure 1139 – Table 3-1: Relationship of Sample Size and Mean Values to Achieve Statistical Significance
Some figures may not display clearly when rendered as a PDF or printed.
Sample size has a similar impact on the beta level. If the true IQ of readers was 119, a sample of 4 would have a beta level of 0.37; that is, only a 63% chance of detecting it. The beta level for a sample of 100 is 0.05 and for a sample of 10,000 is less than 0.0001. So, if the sample is too small, you risk the possibility of not detecting the presence of real effects.
The bottom line is this: the level of statistical significance—0.05, 0.001, or whatever—indicates the likelihood that the study could have come to a false conclusion. By itself, it tells you absolutely nothing about the actual magnitude of the differences between groups.
Up to now, we haven’t talked about clinical significance. Basically, this reduces to a judgment by someone of how much of a difference might be viewed as clinically important. The topic consumes zillions of journal pages in the Health-Related Quality of Life literature, basically because no one can agree anyway. However, Jacob Cohen, a statistician and pragmatist, framed differences in terms of a simple concept called the effect size (mean difference / standard deviation of the sample) and said, based on reviewing some literature, that a small effect size was about 0.2, a medium one was about 0.5, and a large one was about 0.8. That seems to be as good as any other approach.5343
Content on this page was last changed on March 19, 2009.
© 2002 BC Decker Inc. Show Disclaimer
| 5343. | Norman GR, Streiner DL. PDQ Statistics . 3rd ed. Hamilton, Ontario: BC Decker Inc.; 2003. |