How To Determine If Sample Size Is Statistically Significant

Anyone who has designed anything -- whether that be a new medicine, a pattern method, or fifty-fifty a new recipe -- has faced the question: "Is this better than what I had before?" If you're just deciding whether or not you like a new recipe, getting an answer is straightforward. If you are in an bookish or industrial setting you must besides answer an even more of import question: "Can I prove that this is better?"

Whether this procedure involves interviews or experiments, designers ofttimes need to show with statistical significance that their new design is an comeback over the old. Statistical significance requires a big enough sample size to prove a meaningful difference between ii groups. Picking a sample size is an exercise in walking the line between picking the smallest, to the lowest degree plush sample size without risking inconclusive or statistically insignificant results that waste the experiments. The sample size y'all choose needs to be in the "goldilocks zone" of not too much, not too little, but just right.

Thankfully, picking a sample size doesn't have to be a guessing game. The statistical tool of power analysis determines how big of sample size is needed based on a few statistical parameters. In this article, we explore how to perform a ability analysis to pick a sample size, and what can be done to increase statistical power if there are limitations on the sample sizes available.

Many online calculators exist for doing power analysis calculations automatically. You can check out two of my favorites here [1] and here [2]. While you will probable use a computer, it'south important to empathise what's going into the figurer.

Power Analysis Statistics

Figure 1: Illustration of Statistical Values Used in Power Analysis — Effigy ane: Illustration of Statistical Values Used in Power Analysis

The footing of ability analysis is that if 2 sample means are close, information technology's likely the underlying population distributions have a lot of overlap. This means it's probable that the samples came from the same source, or the two sources aren't significantly different. If the two sample means are far autonomously, it'due south likely that in that location isn't much overlap between the population distributions, and that the samples likely came from ii significantly dissimilar sources.

Figure one illustrates two hypothetical distributions H0 and H1, which could correspond ii different materials, or 2 design processes, or results from a control group versus a handling group.. Hither we see that the means are spread quite far apart, but due to the standard deviations of the sample distributions, at that place is a fair amount of overlap between the two. Power assay is a statistical tool to determine whether the divergence in performances of the 2 distributions are more probable due to existent differences or randomness. Power assay can be used to analyze the differences betwixt ii distributions, and assistance us predict how big of a sample size nosotros will demand to show the ii distributions are actually different. Power analysis calculations take in three main inputs: statistical power, a significance threshold, and an effect size which is a summary variable that describes overlap.

Statistical power is the likelihood of an experiment correctly rejecting the zilch hypothesis (which is the assumption that 2 sources aren't different) and determining that the two sources are different. If yous tin prove that 2 sources are different, and ane has a higher value than the other, so you tin country that one source is improve than the other. A typical value for statistical power is 0.8 or eighty%. Some sample size calculators use Beta, which is equal to i - power. Alpha is the threshold for significance and is equal to the odds that y'all would get the same results due to gamble. Blastoff is typically set to 0.05. In Effigy ane, power is shown to be the surface area under the normal curve to the right of the disquisitional t-score or z-score. This disquisitional t or z score comes from the value for H0 where the area under the H0 distribution is equal to alpha.

Effect Size is a way to summarize the departure betwixt two distributions. While consequence size is hard to visualize, i way to recall virtually information technology is to think about the overlap between distributions. IF the overlap is small the effect size is large. If the overlap is large the effect size is pocket-sized. Information technology is equal to the differences in the means divided by the pooled standard difference. If yous have data from previous experiments, y'all can hands plug in the sample means and standard deviations to get your sample size. If you don't, at that place are several ways of estimating the effect size.

What to do When You Don't Have Enough Information

Running a preliminary experiment is the platonic option. This provides actual data well-nigh the effect size. This greatly reduces the take a chance associated with estimating a sample size. However, performing these experiments may be costly or non possible, such equally when a very large preliminary sample size would exist needed to determine effect size, or if a sample size is needed before experiments for funding or experiment approval.

Thankfully at that place are some reliable options for estimating required sample sizes. Many fields used standard effect size estimates based on whether they believe that the difference betwixt two things is small (0.2), medium (0.v) or large (0.8) [3]. Modest effect sizes crave large samples to testify significance, and large effect sizes merely demand minor samples. Effect size values of 0.2, 0.five, and 0.8 originated from the field of Behavioral Science, merely have wide application. These effect sizes correspond to required sample sizes of 394, 64 and 26 respectively. Other fields take their own upshot size standards.

When standards aren't available, we can typically make a pretty practiced judge about the differences betwixt the means of ii distributions and the variability. Some areas have high variability and small upshot sizes such every bit slightly improved medical treatments. Pattern methods typically have loftier variability due to human factors and variability. This can arrive difficult to show that a difference betwixt ii different methods isn't due to take chances. For this article I researched the issue sizes in Design Theory & Methodology Papers and constitute that many pattern methodologies accept effect sizes much larger than the typical estimates of 0.2, 0.5 and 0.8 (See Effigy ii).

Illustrating the Effects of Distance Between Means and Variability on Sample Size

Figure 2: Effect Sizes in Design Theory & Methodology Papers — Effigy 2: Effect Sizes in Blueprint Theory & Methodology Papers

The greater the difference in performance betwixt ii design methods, the smaller sample size is needed to show a departure. If you are developing a method, product, or treatment, improvements in your design not only pb to greater results, information technology is also easier to bear witness that one method is improve than another. Some other gene to consider is how you are measuring the difference between groups. If you are measuring performance using a 1-4 rating scale, you will not be likely to show a strong difference because the measurement doesn't permit for large differences. And then if yous are having a hard time showing statistical significance, a alter in how you are measuring performance might be what you demand.

The other factor in effect size is the variability which is measured by pooled standard deviations. The greater the variation inside a grouping the larger sample size is needed to show a divergence. This variability e'er exists and tin come from a diverseness of sources. Manufactured parts typically have very footling variability. Humans are all so unique and different that experiments involving humans typically take much higher variability. While some level of variability will always exist, sure measures can assist limit measurement variability. Variability can be decreased by controlling for a variety of factors and by running experiments more carefully,

Conclusion

Hopefully this article has helped accept the guesswork out of picking a sample size. Answering the two big questions "Is this better than what I had before?" and "Can I evidence that this is better?" are much easier when you take the statistical measures to support your findings.

References

Power and Sample Size Calculator, https://www.gigacalculator.com/calculators/power-sample-size-figurer.php
Inference for Means: Comparison Two Contained Samples, https://world wide web.stat.ubc.ca/~rollin/stats/ssize/n2.html
Cohen, Jacob. Statistical power analysis for the behavioral sciences. Bookish printing, 2013.