12.0 Bivariate C2N Statistics

Measuring Differences Between Group Means

When we look at comparing numerical and categorical data, we typically want to understand if there is a statistically meaningful difference between groups defined by the categorical variable. We can accomplish this using t-tests and ANOVA. Both do effectively the same thing: they test whether the means of different groups are significantly different from each other. We use a t-test when comparing two groups, and ANOVA when comparing three or more groups.

For instance, recent research has indicated that women are outpacing men in college completion rates.¹ It is highly likely that this research was conducted using t-tests to compare the means of college completion rates between men and women.

T-tests are great because they only have two variables, so we learn both which category is larger and whether the difference is statistically significant. We can do this because with two groups, we get both the directionality and magnitude of the difference in a single test. If the t-statistic is positive, we know the first group has a larger mean; if negative, the second group does. We can then use the p-value to determine if this difference is statistically significant.

ANOVAs are a bit different. They compare three or more groups, so they cannot tell us whether or not one specific group is larger than the other. Instead, they tell us if there is a difference in the mean groups with the f-statistic and the p-value tells us if the f-statistic is significant.

But we cannot be done just by looking at the t-statistic or f-statistic along with the p-value. We also need to consider if the t-statistic or the f-statistic is large enough to be meaningful. These are called the critical-t and the critical-f values, respectively. The establish the absolute value the t-statistic or f-statistic must exceed in order to be considered statistically significant.

In other words, compared to the correlation tests from last chapter, we need to go beyond the p-value to establish significance. We need to consider the actual test statistic (t or f) and compare it to a critical value to determine if the observed difference is meaningful. In addition, the p-value must still be below our significance threshold (commonly α = 0.05) to confirm that the difference is statistically significant.

Finally, after doing all this, we still will not know which groups are different from each other when using ANOVA. For that, we need to do a post-hoc test like Tukey’s HSD, which we will cover in section 12.3.

• t-tests: Used to compare means between 2 groups
  • Provide directionality (which group is larger)
    • Positive t-statistic: First group mean is larger
    • Negative t-statistic: Second group mean is larger
  • p-value indicates statistical significance
  • critical-t value establishes meaningfulness of the difference
• ANOVA (Analysis of Variance): Used to compare means across 3 or more groups
  • F-statistic indicates if there is a difference among group means
  • p-value indicates statistical significance
  • critical-f value establishes meaningfulness of the difference
  • Does not indicate which groups differ; requires post-hoc tests (e.g., Tukey’s HSD)

Footnotes

1. “U.S. women are outpacing men in college completion, including in every major racial and ethnic group,” Pew Research Center. Link here.