T-Tests

Purpose:

Compare the means between two groups. There are three kinds of t-tests: an independent samples t-test (for between-group manipulation), paired samples t-test (for within-group manipulation), and one-sample t-tests (for one sample being compared to the population).

Note: Between-group manipulation (between-subjects study design) means that each group received a different IV treatment condition (level). Within-group manipulation (within-subjects study design) means that every participant experiences every IV treatment condition (level).

Context Used:

• There must be only ONE CATEGORICAL IV, with only TWO LEVELS.

• Additionally, there must be only ONE QUANTITATIVE DV.

Note: Categorical variables are variables where the data represents groups with no real numerical value (e.g., gender, car brand, hair color, eduction level). Quantitative variables are variables where the data represents amounts with a real numerical value (e.g., height, weight, age, time, speed, distance).

Do NOT use t-test if: Your IV has more than two levels → use ANOVA, your DV is categorical → use chi-square, you have multiple DVs → use MANOVA, you have covariates you need to control for → use ANCOVA or regression.

Before running a test, three assumptions should be met:

1. Normality: The DV should be approximately normally distributed within each group. In Jamovi, check this under "Assumption Checks" → "Normality test (Shapiro-Wilk)." If p < .05, normality is violated, and should use a non-parametric test instead (e.g., Mann-Whitney U or WIlcoxon Signed-Rank Test).

2. Independence: Observations must be independent of one another (each participant contributes one score). 

3. Homogeneity of variances (independent samples only): The two groups should have similar variances. Jamovi tests this with Levene's test. If p < .05, select "Welch's" correction under "Variances."

Jamovi Walkthrough:

Independent Samples T-test:

1. Click "T-Tests"

2. Click "Independent Samples T-test"

3. Enter the dependent variable into "Dependent Variables" box

4. Enter the categorical independent variable in "Group Variable" box

5. Check "Effect size" under "Additional Statistics"

Paired Samples T-test:

1. Click "T-Tests"

2. Click "Paired Samples T-Test"

3. Enter the two IV level scores into the "Paired Variables" box

4. Check "Effect size" under "Additional Statistics"

One-Sample T-test:

1. Click "T-Tests"

2. Click "One Sample T-Test"

3. Enter the sample's DV scores of the sample and population into the "Dependent Variables" box

4. Check "Effect size" under "Additional Statistics"

Output Interpretation:

p-value: The probability of detecting a meaningful relationship/difference when there is none. We are typically looking for a small value (p < .05).

If p < .05, reject the null hypothesis. There IS a difference.

If p > .05, accept the null hypothesis. There is NO difference.

t-value: The number of estimated standard error of the mean (SEM) units the sample is from the population.

If t-value is small, there is more similarity between the two sample sets.

If t-value is large, a large difference exists between the two sample sets.

If t-value is negative, it has no bearing on the significance of the difference. It only indicates a reversed directionality of the effect.

SEM: Average deviation of sample means from the population mean. A measure of sampling error

If SEM is small, sample means are similar to the population mean, with little sampling error (This is what we want).

If SEM is large sample means are variable.

df: Values in a study that have the freedom to vary and are essential for assessing the importance and validity of the null hypothesis. 

Effect size: A measure of the magnitude of the mean difference. Bigger numbers mean a bigger difference between groups in terms of standard deviation. 

If Cohen's d = <0.20, very small effect size.

        If Cohen's d = 0.20-.49, small effect size.

If Cohen's d = 0.50-0.79, medium effect size

If Cohen's d = 0.80+, large effect size.

Confidence Interval (CI): Refers to the confidence interval around the mean difference, lower and upper bounds. If the 95% CI does not include zero, the mean difference is statistically significant. If it includes zero, the difference is not significant at p < .05.

APA Format:

Appropriate data visualization: Bar graphs (with error bars).

Sample table: https://apastyle.apa.org/style-grammar-guidelines/tables-figures/sample-tables#tests

Sample write-up:

A paired-sample t-test compared the average [DV] by a [Condition 1] (M =, SD =) and [Condition 2] (M = , SD =). There was a significant difference in [DV] , t(df )=X , p < .05, d=  , CI [ ,] . These results suggest that [DV] increases/decreases from [Condition 1] to [Condition 2]. 

Note: plugin the appropriate test used, means, standard deviations, whether the test was significant, t-value, df, p-level, the strength of the relationship (d) and confidence interval, and a conclusion.