16 Week 6: Two-Sample Tests for Independent Means

16.1 From One Group to Two

Chapter 15 compared a single sample against a fixed benchmark, a known number decided in advance. Often the more interesting business question compares two independent groups to each other, with neither mean known in advance: do customers acquired through one marketing channel spend more, on average, than customers from another? Does one store layout produce faster checkout times than another? These are two-sample tests for independent groups: “independent” because the two samples are separate sets of observations with no natural pairing between them (contrast this with paired comparisons, covered separately).

16.2 Setting Up the Comparison

Let $\mu_1$ and $\mu_2$ denote the true means of the two populations. The hypotheses are usually written in terms of their difference: \[ \begin{align*} &H_0: \mu_1 = \mu_2 \\ &H_a: \mu_1 \neq \mu_2 \;\; (\text{or } \mu_1 > \mu_2, \text{ or } \mu_1 < \mu_2) \end{align*} \] This test requires two independent random samples (the two groups don’t influence or overlap with each other), and each population approximately normal, or large enough sample sizes for the Central Limit Theorem to apply.

16.3 Two Approaches: Pooled vs. Welch

There are two versions of the two-sample t-test, and the difference between them comes down to one assumption: do the two populations have the same variance?

Student’s (Pooled) t-Test

If the two populations are assumed to have equal variance, we combine both samples’ variability into a single pooled variance estimate: \[ s_p^2 = \frac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n_1+n_2-2} \] and the test statistic is \[ t = \frac{\bar{x}_1 - \bar{x}_2}{s_p\sqrt{1/n_1 + 1/n_2}} \] which follows a $t$-distribution with $n_1+n_2-2$ degrees of freedom.

Welch’s t-Test (Unequal Variances)

If the two populations’ variances can’t be assumed equal, Welch’s t-test avoids pooling and instead uses each sample’s own variance directly: \[ t = \frac{\bar{x}_1 - \bar{x}_2}{\sqrt{s_1^2/n_1 + s_2^2/n_2}} \] The degrees of freedom for this version come out as a non-integer, computed from a more complicated formula that software handles for you; there’s no need to compute it by hand.

Which one should you use?

In practice, default to Welch’s t-test unless you have strong prior reason to believe the two populations’ variances are truly equal. Two group variances are rarely identical in real business data, and Welch’s test performs about as well as the pooled test when variances genuinely are equal, but performs much better when they aren’t. This is also R’s default behavior, t.test() uses Welch’s test unless you explicitly request the pooled version.

Example 6.5: Order value by marketing channel

A retailer wants to know whether customers acquired through email marketing spend a different average order amount than customers acquired through social media ads.

Channel	$n$	$\bar{x}$	$s$
Email	40	$82	$20
Social media	45	$68	$22

The two groups’ sample standard deviations are reasonably close ($20 vs. $22), but we’ll still use Welch’s test by default, per the guidance above. \[ H_0: \mu_{\text{email}} = \mu_{\text{social}} \qquad H_a: \mu_{\text{email}} \neq \mu_{\text{social}} \] The test statistic is \[ t = \frac{82 - 68}{\sqrt{20^2/40 + 22^2/45}} = \frac{14}{\sqrt{10.00 + 10.76}} = \frac{14}{4.56} \approx 3.073 \] Using software, the Welch degrees of freedom come out to approximately 83, and the two-sided p-value for $t \approx 3.073$ is approximately 0.003. Since $0.003 < 0.05$, we reject $H_0$: this sample provides strong evidence that the true mean order value differs between the two channels, with email customers spending more on average.

16.4 Confidence Interval for the Difference

Alongside (or instead of) the test, it’s often more useful to report a confidence interval for $\mu_1 - \mu_2$, since it shows the range of plausible differences, not just whether zero is ruled out: \[ (\bar{x}_1 - \bar{x}_2) \;\pm\; t_{\alpha/2, \, df}\sqrt{s_1^2/n_1 + s_2^2/n_2} \]

Example 6.6: Confidence interval for the channel difference

Continuing Example 6.5, with $df \approx 83$, the 95% critical value is $t_{0.025,83} \approx 1.989$. The margin of error is \[ 1.989 \times 4.56 \approx 9.07 \] The 95% confidence interval for $\mu_{\text{email}} - \mu_{\text{social}}$ is \[ 14 \pm 9.07, \quad \text{or} \quad (4.93, \, 23.07) \] This interval excludes 0, consistent with rejecting $H_0$ above: the data suggest email customers’ true average order value is somewhere between about $4.93 and $23.07 higher than social media customers’, not just numerically different in this one sample.

16.5 Computing Two-Sample Tests in R and Excel

Example 6.7: The channel comparison in R and Excel

If you have the raw order values for each group, R’s t.test() runs Welch’s test by default:

t.test(email_orders, social_orders)

To instead run the pooled (equal-variances) version, add var.equal = TRUE:

t.test(email_orders, social_orders, var.equal = TRUE)

Working from just the summary statistics, as in Example 6.5:

x1 <- 82; s1 <- 20; n1 <- 40
x2 <- 68; s2 <- 22; n2 <- 45

se <- sqrt(s1^2/n1 + s2^2/n2)
t_stat <- (x1 - x2) / se
t_stat

[1] 3.072988

In Excel, T.TEST() computes the p-value directly from two ranges of raw data (its third argument, 1, requests a two-tailed test; the fourth, 3, requests the unequal-variances (Welch) version, 2 requests the pooled version):

=T.TEST(email_range, social_range, 2, 3)

16.6 The Two-Sample Test for Proportions

Everything so far in this section compared two groups’ means. Just as often, the outcome of interest is binary, whether a visitor converts, whether a loan defaults, whether a customer churns, and we want to compare two groups’ proportions instead. Let $p_1$ and $p_2$ be the true proportions in the two populations, estimated by $\hat{p}_1$ and $\hat{p}_2$ from independent random samples of size $n_1$ and $n_2$. \[ \begin{align*} &H_0: p_1 = p_2\\ &H_a: p_1 \neq p_2 \;\; (\text{or } p_1 > p_2, \text{ or } p_1 < p_2) \end{align*} \] This test requires the success-failure conditions to hold in each group separately ($n_1\hat{p}_1 \ge 15$, $n_1(1-\hat{p}_1) \ge 15$, and likewise for group 2), along with independence both within and between the two samples.

Under $H_0$, both groups share one common (but unknown) proportion, so we estimate it with a pooled proportion, combining both samples’ successes and totals: \[ p = \frac{x_1 + x_2}{n_1 + n_2} \] where $x_1$ and $x_2$ are the number of successes in each group. The test statistic is \[ z = \frac{\hat{p}_1 - \hat{p}_2}{\sqrt{p(1-p)\left(\dfrac{1}{n_1} + \dfrac{1}{n_2}\right)}} \]

Example 6.8: Conversion rate by marketing channel

Continuing the channel comparison, the retailer also wants to know whether the two channels convert visitors to customers at different rates. Out of 500 email-referred visitors, 75 converted; out of 600 social-media-referred visitors, 60 converted. \[ \hat{p}_{\text{email}} = \frac{75}{500} = 0.15 \qquad \hat{p}_{\text{social}} = \frac{60}{600} = 0.10 \] Both groups comfortably satisfy the success-failure conditions ($n\hat{p}$ and $n(1-\hat{p})$ are all well above 15). The pooled proportion is \[ p = \frac{75+60}{500+600} = \frac{135}{1100} \approx 0.1227 \] \[ H_0: p_{\text{email}} = p_{\text{social}} \qquad H_a: p_{\text{email}} \neq p_{\text{social}} \] The test statistic is \[ z = \frac{0.15 - 0.10}{\sqrt{0.1227(0.8773)\left(\frac{1}{500}+\frac{1}{600}\right)}} = \frac{0.05}{0.0199} \approx 2.517 \] The two-sided p-value is $2 \times P(Z>2.517) \approx 0.012$. Since $0.012 < 0.05$, we reject $H_0$: the data provide evidence that the true conversion rate differs between the two channels, with email showing a higher rate.

Confidence Interval for the Difference in Proportions

As with means, a confidence interval for $p_1 - p_2$ conveys more than the test alone, an estimated size for the difference, not just whether zero is ruled out. Notably, the confidence interval does not use the pooled proportion, since it isn’t assuming $p_1=p_2$ in the first place; each group’s own sample proportion is used instead: \[ (\hat{p}_1 - \hat{p}_2) \;\pm\; z_{\alpha/2}\sqrt{\frac{\hat{p}_1(1-\hat{p}_1)}{n_1} + \frac{\hat{p}_2(1-\hat{p}_2)}{n_2}} \]

Example 6.9: Confidence interval for the conversion-rate difference

Continuing Example 6.8, the (unpooled) standard error is \[ \sqrt{\frac{0.15(0.85)}{500} + \frac{0.10(0.90)}{600}} \approx 0.0201 \] The 95% confidence interval for $p_{\text{email}} - p_{\text{social}}$ is \[ (0.15-0.10) \pm 1.96(0.0201), \quad \text{or} \quad (0.011, \, 0.089) \] This interval excludes 0, matching the rejection above: the true conversion-rate advantage for email over social media is estimated to be somewhere between about 1.1 and 8.9 percentage points.

Computing the Two-Sample Proportion Test in R and Excel

Example 6.10: The conversion-rate comparison in R and Excel

R’s prop.test() runs this test directly from the raw counts:

prop.test(x = c(75, 60), n = c(500, 600), correct = FALSE)


    2-sample test for equality of proportions without continuity correction

data:  c(75, 60) out of c(500, 600)
X-squared = 6.3328, df = 1, p-value = 0.01185
alternative hypothesis: two.sided
95 percent confidence interval:
 0.01055649 0.08944351
sample estimates:
prop 1 prop 2 
  0.15   0.10

Working from the formulas directly:

x1 <- 75; n1 <- 500
x2 <- 60; n2 <- 600
p1 <- x1/n1; p2 <- x2/n2
p_pool <- (x1 + x2) / (n1 + n2)

z_stat <- (p1 - p2) / sqrt(p_pool * (1 - p_pool) * (1/n1 + 1/n2))
p_value <- 2 * (1 - pnorm(abs(z_stat)))

c(z_stat = z_stat, p_value = p_value)

    z_stat    p_value 
2.51649709 0.01185279

In Excel, there’s no single built-in function for a two-sample proportion test, so the formula is built directly from its pieces:

=2 * (1 - NORM.S.DIST(ABS((0.15-0.10) / SQRT(0.1227*(1-0.1227)*(1/500+1/600))), TRUE))

16.7 Recap

Keyword	Definition
Two-sample independent test	A hypothesis test comparing the means of two separate, unrelated groups.
Pooled (Student’s) t-test	Assumes equal population variances; combines both samples into a pooled variance estimate, $df = n_1+n_2-2$.
Welch’s t-test	Does not assume equal variances; uses each sample’s variance separately, with an approximate (non-integer) degrees of freedom.
Confidence interval for $\mu_1-\mu_2$	$(\bar{x}_1-\bar{x}_2) \pm t_{\alpha/2,\,df}\sqrt{s_1^2/n_1 + s_2^2/n_2}$.
Two-sample proportion test	Tests $H_0: p_1=p_2$ using a pooled proportion estimate in the test statistic.
Pooled proportion	$p = (x_1+x_2)/(n_1+n_2)$; used only in the test statistic, not the confidence interval.
Confidence interval for $p_1-p_2$	$(\hat{p}_1-\hat{p}_2) \pm z_{\alpha/2}\sqrt{\hat{p}_1(1-\hat{p}_1)/n_1 + \hat{p}_2(1-\hat{p}_2)/n_2}$, using the unpooled sample proportions.

16.8 Check Your Understanding

Problems

A company compares average daily sales at stores using two different floor layouts. Layout A: $n_1=25$, $\bar{x}_1 = \$4{,}200$, $s_1=\$600$. Layout B: $n_2=28$, $\bar{x}_2 = \$3{,}950$, $s_2=\$550$. Using Welch’s test, compute the test statistic for $H_0: \mu_A = \mu_B$.
Why might a business analyst prefer to default to Welch’s test rather than the pooled test, even when the two sample standard deviations look fairly similar?
A 95% confidence interval for $\mu_1 - \mu_2$ is $(1.2, \, 8.7)$. What does this tell you about whether a two-sided test of $H_0: \mu_1=\mu_2$ at $\alpha=0.05$ would be rejected? Which group appears to have the higher mean?
Explain, in your own words, why the pooled t-test’s degrees of freedom ($n_1+n_2-2$) is a simple whole number, while Welch’s degrees of freedom typically is not.
A subscription app tests two onboarding flows. Flow A: 40 of 250 new users cancel within 30 days. Flow B: 54 of 260 new users cancel within 30 days. Test at $\alpha=0.05$ whether the true cancellation rates differ, and construct a 95% confidence interval for the difference.
Explain why the pooled proportion is used in the two-sample proportion test statistic, but not in the confidence interval for $p_1-p_2$.

Solutions

$\text{SE} = \sqrt{600^2/25 + 550^2/28} = \sqrt{14400 + 10803.6} = \sqrt{25203.6} \approx 158.76$. $t = (4200-3950)/158.76 = 250/158.76 \approx 1.575$.
Because in real business data, two groups rarely have exactly equal variances, and Welch’s test performs nearly as well as the pooled test when variances happen to be equal, but noticeably better when they aren’t. Defaulting to Welch’s avoids having to verify an assumption that’s hard to check precisely and is often not exactly true anyway.
Since the interval $(1.2, 8.7)$ does not contain 0, a two-sided test of $H_0: \mu_1=\mu_2$ at $\alpha=0.05$ would be rejected. Because the entire interval is positive, group 1 appears to have the higher mean.
The pooled test assumes both populations share a single common variance, so both samples’ information can be combined (“pooled”) into one estimate in a way that keeps the degrees-of-freedom bookkeeping simple, exactly $n_1+n_2-2$. Welch’s test does not assume a common variance, so it weights each sample’s contribution according to its own variability, and that weighting produces a more complex formula whose result generally isn’t a whole number.
$\hat{p}_A = 40/250 = 0.16$, $\hat{p}_B = 54/260 \approx 0.208$. Pooled proportion: $p = (40+54)/(250+260) = 94/510 \approx 0.184$. Test statistic: $z = (0.16-0.208)/\sqrt{0.184(0.816)(1/250+1/260)} \approx -0.0477/0.0344 \approx -1.389$. Two-sided p-value $\approx 0.165$. Since $0.165 > 0.05$, we fail to reject $H_0$: the data don’t provide strong evidence that the two flows’ cancellation rates differ. The (unpooled) 95% confidence interval is $(0.16-0.208) \pm 1.96\sqrt{0.16(0.84)/250 + 0.208(0.792)/260} \approx -0.0477 \pm 0.0671$, or $(-0.115, \, 0.019)$, which contains 0, consistent with failing to reject.
Under $H_0$, both groups are assumed to share one common, unknown proportion, so pooling both samples’ data gives the best available estimate of that shared value for computing the standard error used in the test. A confidence interval, by contrast, doesn’t assume the two proportions are equal, it’s trying to estimate how different they actually are, so it uses each group’s own sample proportion to estimate the standard error rather than assuming they’re the same.