Generally, a power of . 80 (80 percent) or higher is considered good for a study. This means there is an 80 percent chance of detecting a difference as statistically significant, if in fact a true difference exists.

The power of a test is the probability of rejecting the null hypothesis when it is false; in other words, it is the probability of avoiding a type II error. The power may also be thought of as the likelihood that a particular study will detect a deviation from the null hypothesis given that one exists.

The power of a study, pβ, is the probability that the study will detect a predetermined difference in measurement between the two groups, if it truly exists, given a pre-set value of pα and a sample size, N.

Increasing sample size makes the hypothesis test more sensitive – more likely to reject the null hypothesis when it is, in fact, false. Thus, it increases the power of the test. The effect size is not affected by sample size.

Like statistical significance, statistical power depends upon effect size and sample size. If the effect size of the intervention is large, it is possible to detect such an effect in smaller sample numbers, whereas a smaller effect size would require larger sample sizes.

Consequences of a type 1 Error Consequently, a type 1 error will bring in a false positive. This means that you will wrongfully assume that your hypothesis testing has worked even though it hasn’t. In real life situations, this could potentially mean losing possible sales due to a faulty assumption caused by the test.

When the null hypothesis is true and you reject it, you make a type I error. The probability of making a type I error is α, which is the level of significance you set for your hypothesis test. An α of 0.05 indicates that you are willing to accept a 5% chance that you are wrong when you reject the null hypothesis.