• Most of our modules are about tests for statistical significance for various types of research questions and designs. However, obtaining a statistically-significant result is just the outcome of a hypothesis test, and should not be the major goal of your statistical analysis or even your research study.

• Still, we rely heavily on the outcome of hypothesis testing in the conclusions of our analysis. So, it is important to understand the mechansims underlying the hypothesis-testing procedure.

• Conceptually, what are the factors that may influence your study to be statistically significant or not?

• From a practical perspective, you may want to know how large the sample size you need to obtain for your research question, and how to estimate the minimum sample size for your study.

• Hence, we need power analysis.

1.1 What are alpha (α), beta (β), and statistical power?

In hypothesis testing,

• alpha (α) – the probability of committing a Type I error in hypothesis testing (i.e., rejecting the null hypothesis when it's true).

• beta (β) – the probability of committing a Type II error in hypothesis testing (i.e., failing to reject the null hypothesis when it's false).

Power analysis involves four elements: (1) Statistical Power, (2) Sample Size, N, (3) Effect Size, and (4) Significant Level (alpha, usually set as .05).

Statistical power is a function of sample size, alpha, and effect size. There are multiple reasons to carry out a power analysis:

• Sample Size Determination: calculate a minimum N when both the alpha, effect size, and power are decided. We usually do it before conducting the study (A priori power analysis)

• Power Estimation: calculate the power when both the alpha, sample size, and effect size are provided. We usually do it when the study is conducted (Post hoc power analysis)

• Effect Size Calculation: calculate the minimum, detectable, and hypothetical effect size when both alpha, sample size, and power are provided. (sensitivity power analysis)

Statistical significance criterion - alpha (α)

• When α is set at 0.05, the probability of the null hypothesis being true needs to be less than 0.05 in order to be rejected. In other words, the alternative hypothesis implying an observed effect needs to be greater than 0.95 to be accepted which will reduce the power.

• If you want to increase the power of the test, you may adopt a less conservative α (setting α to a greater value, like 0.05 to 0.10). However, if you increase your power by having a more liberal alpha level, you are also increasing the chance of having Type I error at the same time. While setting a more conservative criterion (e.g. α = 0.05 to 0.01) will reduce the power, but, at the same time, reduce the Type I error rate and increase the robustness of your test results.

Sample size

• The amount of sampling error in the test results depends on the sample size. Increasing sample size can reduce sampling error and increase the probability to detect a true effect.

• Increasing the sample size also increases the statistical power of a test.

The effect size of interest

• Effect size is about the strength of an effect that is of your interest. It can be small or large.

• A greater effect size will increase the power- in order words, a stronger effect will be more likely to be correctly detected.

Speaking of effect sizes, they can be grouped into two families: d family (strength of difference) and r family (strength of association)

• d family effect size is about the strength of difference,

  • ranging from 0 to infinity (|0.2| = small, |0.5| = medium, |0.8| = large, by Cohen's 1988 interpretation)

  • manifested by the standardized mean difference between two values.

  • calculated by the mean difference between two values divided by the standard deviations (SD)

• Commonly known as the Cohen’s d, that is, the standardized mean difference of an effect

• if the sample size is too small (N < 50), a bias-corrected effect size, Hedges’ g (by multiplying Cohen's d by a correcting factor), will be preferable

• r family is about the strength of association. The most commonly known effect size is the Pearson correlation coefficient, r.

• r can range from -1 (perfectly negative relationship) to 0 (no relationship) to 1 (perfectly positive relationship), yet -1 or 1 rarely occur in the real world.

• r can also be used to describe the proportion of variance. for example, a correlation (r) of 0.5 indicates 25% (r²) of the variance is explained by the predictor. In multiple regression analysis, R² is the effect size indicating the variance of the dependent variable is explained by a group of predictors.

• Other effect sizes in the r effect sizes family include Eta-squared ( η² ) and Omega-squared (ω²), but we won't be discussing them in-depth.

3.1 Install and run jpower

In jamovi, you can perform a power analysis with the module jpower. The jpower - Power Analysis for Common Research Designs is a module to compute power for three types of t-test (one-sample, dependent sample, and independent sample t-test). To use jpower, you will have to install jpower in jamovi.

To install jpower, please follow the steps:

• Install the latest version of jamovi and run it.

• Go to Modules menu on the upper right corner, and go to the jamovi library.

• Go to the Available tab, look for jpower.

• Install jpower.

You can now click the jpower icon and select an analysis for your power analysis.

Note: the minimally-interesting effect size (delta, δ) means the minimum standardised mean difference you would like to detect.

• Recently, a new claims that girls are more likely to get better grades than boys in school. You would like to test if this is actually true and are planning to conduct a study on the gender difference in academic performance. But you are not so sure about the sample size. So you conducted a power analysis to determine the sample size for your study in an independent t-test (female group and male group).

• You decided to go with a medium effect size of 0.5, and a statistical power of 80% (0.8), which is within the acceptable range.

• You may want to conduct research of other experimental designs, for example, factorial ANOVA, correlation, etc. However, power analysis within jamovi is only available for t-tests only at the moment. For conducting studies of various designs, you may use another tool for power analysis - G*Power.

• G*Power is an alternative tool to compute statistical power analyses for many different t-tests, F tests, χ2 tests, z tests and some exact tests. If you are also interested in knowing more about other power analysis and G*Power, you can check out the G*Power website:
  https://www.psychologie.hhu.de/arbeitsgruppen/allgemeine-psychologie-und-arbeitspsychologie/gpower.html