Correlation and the coefficient of determination are two common ways to summarize the strength of a linear relationship. Both are connected to the least squares line, but they answer slightly different questions.
In this chapter, we will focus on:
The claim is often made that the crime rate and the unemployment rate are “highly correlated.” Another popular belief is that IQ and academic performance are correlated. Some people even claim that the Dow Jones Industrial Average and the length of fashionable skirts are correlated.
The term correlation implies a relationship or association between two variables. In regression, we are especially interested in whether that relationship is approximately linear.
For data \((x_i,y_i)\), \(i=1,\ldots,n\), we want a measure of how strongly \(x\) and \(y\) move together in a linear way.
Recall the quantities \(SS_{xx}\), \(SS_{yy}\), and \(SS_{xy}\).
Recall that \[ \begin{align*} SS_{xx} &= \sum (x_i-\bar{x})^2\\ SS_{yy} &= \sum (y_i-\bar{y})^2\\ SS_{xy} &= \sum (x_i-\bar{x})(y_i-\bar{y}). \end{align*} \]
\(SS_{xx}\) and \(SS_{yy}\) measure the variability of \(x\) and \(y\) separately. They describe how much each variable varies around its own mean.
\(SS_{xy}\) measures how \(x\) and \(y\) vary together.
Example 5.1
Consider the data from Table 2.1. We can find \(SS_{xx}\), \(SS_{yy}\), and \(SS_{xy}\) in R.
library(tidyverse)
x <- c(1, 2, 2.75, 4, 6, 7, 8, 10)
y <- c(2, 1.4, 1.6, 1.25, 1, 0.5, 0.5, 0.4)
dat <- tibble(x, y)
ybar <- mean(y)
xbar <- mean(x)
ggplot(data = dat, aes(x = x, y = y)) +
geom_point() +
xlim(0, 10) +
ylim(0, 2) +
geom_hline(yintercept = ybar, col = "red") +
geom_vline(xintercept = xbar, col = "red")
The vertical red line marks \(\bar{x}\), and the horizontal red line marks \(\bar{y}\).
| x | y | dev_x_squared | dev_y_squared | dev_xy |
|---|---|---|---|---|
| 1.00 | 2.00 | 16.759 | 0.844 | -3.761 |
| 2.00 | 1.40 | 9.571 | 0.102 | -0.986 |
| 2.75 | 1.60 | 5.493 | 0.269 | -1.216 |
| 4.00 | 1.25 | 1.196 | 0.028 | -0.185 |
| 6.00 | 1.00 | 0.821 | 0.007 | -0.074 |
| 7.00 | 0.50 | 3.634 | 0.338 | -1.108 |
| 8.00 | 0.50 | 8.446 | 0.338 | -1.689 |
| 10.00 | 0.40 | 24.071 | 0.464 | -3.342 |
In the output, dev_x_squared represents \((x_i-\bar{x})^2\), dev_y_squared represents \((y_i-\bar{y})^2\), and dev_xy represents \((x_i-\bar{x})(y_i-\bar{y})\).
We can calculate the sums:
# SS_xx
sum(dev_x^2)[1] 69.99219# SS_yy
sum(dev_y^2)[1] 2.389687# SS_xy
sum(dev_xy)[1] -12.36094For these data, \(SS_{xy}\) is negative. This happens because smaller values of \(x\) tend to occur with larger values of \(y\), and larger values of \(x\) tend to occur with smaller values of \(y\). That is the numerical fingerprint of a negative linear association.
Example 5.2
For another example, consider the trees dataset.
In R, the datasets package includes several built-in datasets. One of them is trees.
Girth Height Volume
1 8.3 70 10.3
2 8.6 65 10.3
3 8.8 63 10.2
4 10.5 72 16.4
5 10.7 81 18.8
6 10.8 83 19.7There are 31 observations in this dataset. The variables are:
Girth: tree diameter, measured in inches at 54 inches above the ground,Height: tree height, measured in feet,Volume: timber volume, measured in cubic feet.Suppose we want to predict Volume from Girth.
Again, we plot the data with red lines representing \(\bar{x}\) and \(\bar{y}\).
library(datasets)
library(tidyverse)
xbar <- mean(trees$Girth)
ybar <- mean(trees$Volume)
ggplot(data = trees, aes(x = Girth, y = Volume)) +
geom_point() +
geom_hline(yintercept = ybar, col = "red") +
geom_vline(xintercept = xbar, col = "red")
| x | y | dev_x_squared | dev_y_squared | dev_xy |
|---|---|---|---|---|
| 8.3 | 10.3 | 24.487 | 394.855 | 98.329 |
| 8.6 | 10.3 | 21.608 | 394.855 | 92.368 |
| 8.8 | 10.2 | 19.788 | 398.840 | 88.839 |
| 10.5 | 16.4 | 7.554 | 189.640 | 37.848 |
| 10.7 | 18.8 | 6.494 | 129.299 | 28.978 |
| 10.8 | 19.7 | 5.995 | 109.641 | 25.637 |
| 11.0 | 15.6 | 5.055 | 212.313 | 32.761 |
| 11.0 | 18.2 | 5.055 | 143.304 | 26.915 |
| 11.1 | 22.6 | 4.616 | 57.320 | 16.265 |
| 11.2 | 19.9 | 4.196 | 105.493 | 21.039 |
| 11.3 | 24.2 | 3.796 | 35.652 | 11.634 |
| 11.4 | 21.0 | 3.417 | 84.107 | 16.951 |
| 11.4 | 21.4 | 3.417 | 76.930 | 16.212 |
| 11.7 | 21.3 | 2.398 | 78.694 | 13.736 |
| 12.0 | 19.1 | 1.558 | 122.566 | 13.821 |
| 12.9 | 22.2 | 0.121 | 63.536 | 2.777 |
| 12.9 | 33.8 | 0.121 | 13.170 | -1.264 |
| 13.3 | 27.4 | 0.003 | 7.678 | -0.143 |
| 13.7 | 25.7 | 0.204 | 19.990 | -2.019 |
| 13.8 | 24.9 | 0.304 | 27.783 | -2.908 |
| 14.0 | 34.5 | 0.565 | 18.741 | 3.254 |
| 14.2 | 31.7 | 0.906 | 2.338 | 1.455 |
| 14.5 | 36.3 | 1.567 | 37.565 | 7.671 |
| 16.0 | 38.3 | 7.571 | 66.081 | 22.368 |
| 16.3 | 42.6 | 9.312 | 154.481 | 37.929 |
| 17.3 | 55.4 | 16.416 | 636.504 | 102.218 |
| 17.5 | 55.7 | 18.076 | 651.731 | 108.540 |
| 17.9 | 58.3 | 21.638 | 791.242 | 130.845 |
| 18.0 | 51.5 | 22.578 | 454.928 | 101.347 |
| 18.0 | 51.0 | 22.578 | 433.849 | 98.971 |
| 20.6 | 77.0 | 54.046 | 2192.958 | 344.269 |
# SS_xx
sum(dev_x^2)[1] 295.4374# SS_yy
sum(dev_y^2)[1] 8106.084# SS_xy
sum(dev_xy)[1] 1496.644In this example, most observations have positive values of \((x-\bar{x})(y-\bar{y})\). This occurs when both values are below their means or both values are above their means.
There are a few observations with negative values of \((x-\bar{x})(y-\bar{y})\), but the positive terms dominate, so \(SS_{xy}\) is positive overall. Therefore, Volume tends to increase as Girth increases.
If \(SS_{xy}\) is positive, then \(y\) tends to increase as \(x\) increases. If \(SS_{xy}\) is negative, then \(y\) tends to decrease as \(x\) increases. If \(SS_{xy}\) is zero or close to zero, then there is little evidence of a linear association.
We first note that \(SS_{xy}\) cannot be greater in absolute value than \[ \sqrt{SS_{xx}SS_{yy}}. \]
We will not prove this here, but it is a direct application of the Cauchy-Schwarz inequality.
We define the linear correlation coefficient as \[ \begin{align} r=\frac{SS_{xy}}{\sqrt{SS_{xx}SS_{yy}}}. \end{align} \tag{5.1}\]
\(r\) is also called the Pearson correlation coefficient.
We note that \[ -1\le r \le 1. \]
If \(r=0\), then there is no linear relationship between \(x\) and \(y\).
If \(r\) is positive, then the slope of the linear relationship is positive. If \(r\) is negative, then the slope of the linear relationship is negative.
The closer \(r\) is to 1 or -1, the stronger the linear relationship is between \(x\) and \(y\).
A correlation near 0 means there is little evidence of a linear relationship. It does not necessarily mean there is no relationship at all.
For example, a curved relationship can have a correlation near 0 if the upward and downward parts of the pattern cancel each other out.
The scatterplot below has a very strong relationship, but the relationship is curved rather than linear.
curve_x <- seq(-3, 3, length.out = 150)
curve_y <- curve_x^2
curve_dat <- tibble(curve_x, curve_y)
curve_r <- cor(curve_x, curve_y)
ggplot(curve_dat, aes(x = curve_x, y = curve_y)) +
geom_point() +
geom_smooth(method = "lm", se = FALSE, color = "red") +
labs(
x = "x",
y = "y",
subtitle = paste("Correlation r =", round(curve_r, 3))
)
In this constructed example, the correlation is exactly 0 because the left side and right side of the curve balance each other out. But the plot clearly shows a strong relationship.
This is why we should never use correlation as a replacement for looking at the scatterplot. Correlation measures linear association, not all possible forms of association.
The best way to grasp correlation is to see examples. In Figure 5.1, scatterplots of 200 observations are shown with a least squares line.
Note how the value of \(r\) relates to how tightly the points cluster around the line and to whether the relationship is positive or negative.
The correlation coefficient, \(r\), quantifies the strength and direction of the linear relationship between two variables. Unlike the slope, the correlation coefficient is scaleless. Its value always falls between -1 and 1, regardless of the units used for \(x\) and \(y\).
The calculation of \(r\) uses the same data used to fit the least squares line. Since both \(r\) and \(b_1\) provide information about the usefulness of the linear model, it is not surprising that their computational formulas are related.
Suppose we measure height in inches and weight in pounds. The slope from a regression of weight on height is measured in pounds per inch.
If we convert height from inches to centimeters, the fitted slope changes because the unit of \(x\) changed. A 1-unit increase in centimeters is not the same as a 1-unit increase in inches.
The correlation, however, does not change. Correlation has no units, so changing inches to centimeters or pounds to kilograms does not affect \(r\).
This is why correlation is useful for summarizing the strength of a linear association, while the slope is better for interpreting the expected change in \(y\) for a 1-unit increase in \(x\).
A high positive or negative value of \(r\) does not mean that changes in \(x\) cause changes in \(y\).
Correlation can be created by causation, but it can also be created by lurking variables, common trends, selection effects, or coincidence. A correlation by itself only supports a statement about linear association.
The correlation \(r\) is computed from observed data, usually from a sample. Thus, \(r\) is the sample correlation coefficient.
The population correlation coefficient is denoted by \(\rho\). A common hypothesis test is \[\begin{align*} H_0 &: \rho = 0\\ H_a &: \rho \ne 0. \end{align*}\]
Recall the hypothesis test for the slope in Section 4.6.
If we test \[\begin{align*} H_0 &: \beta_1=0\\ H_a &: \beta_1\ne 0, \end{align*}\] then this is equivalent to testing1 \[\begin{align*} H_0 &: \rho=0\\ H_a &: \rho\ne 0, \end{align*}\] because both hypotheses test whether there is a linear relationship between \(x\) and \(y\).
Using Equation 2.5, \(b_1\) can be rewritten as \[ \begin{align} b_1 &=\frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sum (x_i-\bar{x})^2}\\ &=\frac{SS_{xy}}{SS_{xx}}\\ &=r\frac{s_y}{s_x}, \end{align} \tag{5.2}\] where \(s_y\) and \(s_x\) are the sample standard deviations of \(y\) and \(x\), respectively.
Starting with \(b_1=SS_{xy}/SS_{xx}\) and \(r=SS_{xy}/\sqrt{SS_{xx}SS_{yy}}\), we get \[\begin{align*} b_1 &=\frac{SS_{xy}}{SS_{xx}}\\ &=\frac{r\sqrt{SS_{xx}SS_{yy}}}{SS_{xx}}\\ &=r\frac{\sqrt{\frac{SS_{xx}}{n-1}\frac{SS_{yy}}{n-1}}}{\frac{SS_{xx}}{n-1}}\\ &=r\frac{s_xs_y}{s_x^2}\\ &=r\frac{s_y}{s_x}. \end{align*}\]
The test statistic for testing the population correlation is \[ \begin{align} t=\frac{r\sqrt{n-2}}{\sqrt{1-r^2}}. \end{align} \tag{5.3}\]
If \(H_0\) is true, then \(t\) has a Student’s \(t\) distribution with \(n-2\) degrees of freedom.
In simple linear regression, testing whether \(\rho=0\) is equivalent to testing whether \(\beta_1=0\).
For the trees data, suppose we compute a strong positive correlation between Girth and Volume. The corresponding slope test asks whether the population slope for predicting Volume from Girth is 0.
These tests agree because, in simple linear regression, a nonzero linear correlation corresponds to a nonzero slope. The slope is usually more useful to report because it tells us the expected change in the response for a 1-unit increase in the predictor.
The only real difference between the least squares slope \(b_1\) and the correlation coefficient \(r\) is measurement scale.2
Therefore, the information they provide about the utility of the least squares model is partly redundant. However, the slope \(b_1\) gives additional information about the amount of increase or decrease in \(y\) for every 1-unit increase in \(x\).
For this reason, the slope is usually recommended for making inferences about the existence of a positive or negative linear relationship between two variables.
The second measure of how well the model fits the data involves measuring the amount of variability in \(y\) explained by the model using \(x\).
We start by examining the variability of the response variable \(y\). One way to measure the total variability in \(y\) is \[ SS_{yy}=\sum (y_i-\bar{y})^2. \]
Note that \(SS_{yy}\) does not include the model or \(x\). It is just a measure of how \(y\) deviates from its mean \(\bar{y}\).
We also have the variability of the points around the fitted line. We measure this with the sum of squared errors: \[ SSE=\sum (y_i-\hat{y}_i)^2. \]
SSE does include \(x\) because the fitted value \(\hat{y}\) is a function of \(x\).
Here are a few key points about these sums of squares:
Suppose \(x\) is hours studied and \(y\) is test score.
If study time does not help predict test score, then the prediction errors from the fitted line will not be much smaller than the deviations from the overall mean score. In that case, \(SSE\) will be close to \(SS_{yy}\).
If study time is a strong predictor, then the fitted line will reduce prediction error substantially. In that case, \(SSE\) will be much smaller than \(SS_{yy}\).
If every point falls exactly on the fitted line, then every residual is 0 and \(SSE=0\).
We want to explain as much of the variation in \(y\) as possible. To quantify how much of that variation is explained by using a linear regression model with \(x\), we calculate \[ \begin{align} SSR=SS_{yy}-SSE. \end{align} \tag{5.4}\]
SSR is called the sum of squares regression.
We calculate the proportion of variation in \(y\) explained by the regression model using \[ \begin{align} r^2=\frac{SSR}{SS_{yy}}. \end{align} \tag{5.5}\]
In simple linear regression, this quantity is equal to the square of the sample correlation coefficient \(r\).
\(r^2\) is called the coefficient of determination. Some software reports it as \(R^2\).
Practical interpretation:
About \(100(r^2)\%\) of the sample variation in \(y\), measured by the total sum of squares of deviations of the sample \(y\) values around their mean \(\bar{y}\), can be explained by using \(x\) to predict \(y\) in the straight-line model.
An \(r^2\) value of 0.80 does not mean the model predicts 80% of observations correctly.
It means 80% of the sample variation in the response variable is explained by the linear model using the predictor variable.
We can find the coefficient of determination using the summary() function with an lm object.
Call:
lm(formula = Volume ~ Girth, data = trees)
Residuals:
Min 1Q Median 3Q Max
-8.065 -3.107 0.152 3.495 9.587
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -36.9435 3.3651 -10.98 7.62e-12 ***
Girth 5.0659 0.2474 20.48 < 2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 4.252 on 29 degrees of freedom
Multiple R-squared: 0.9353, Adjusted R-squared: 0.9331
F-statistic: 419.4 on 1 and 29 DF, p-value: < 2.2e-16For this model, \(r^2=0.9353\). Therefore, about 93.53% of the variability in tree volume can be explained by the linear model using girth to predict volume.
If we want to find the correlation coefficient directly, we can use the cor() function.
cor(trees$Girth, trees$Volume)[1] 0.9671194So the correlation between Girth and Volume is approximately 0.9671. Squaring this value gives approximately 0.9353, which matches the \(r^2\) value from the regression model.
Suppose a regression model has \(r^2=0.18\).
This means 18% of the sample variation in \(y\) is explained by the linear model using \(x\). The remaining 82% is unexplained by this model.
That does not automatically mean the model is useless. In some fields, especially when predicting human behavior, health outcomes, or economic outcomes, an \(r^2\) of 0.18 may still be meaningful. The usefulness of an \(r^2\) value depends on the context, the purpose of the model, and the amount of natural variability in the response.
Each summary is useful, but each answers a different question.
Suppose a researcher fits a simple linear regression model relating weekly exercise time, \(x\), to resting heart rate, \(y\).
Try to decide which summary is most useful in each situation: the slope, the correlation, or \(r^2\).
| Situation | Best summary | Why |
|---|---|---|
| The researcher wants to describe the expected change in resting heart rate for each additional hour of exercise. | Slope | The slope gives the expected change in \(y\) for a 1-unit increase in \(x\). |
| The researcher wants a unitless description of the strength and direction of the linear association. | Correlation | Correlation has no units and always falls between -1 and 1. |
| The researcher wants to describe how much variation in resting heart rate is explained by exercise time. | \(r^2\) | \(r^2\) gives the proportion of sample variation in \(y\) explained by the linear model. |
| The researcher wants to compare the strength of association across two studies that used different measurement units. | Correlation | Correlation is not affected by changing units, while the slope is. |
In practice, regression reports often include more than one of these summaries. The slope is usually best for interpretation in context, while \(r^2\) helps describe overall fit.
This chapter connected the correlation coefficient, the least squares slope, and the coefficient of determination.
| Idea | Meaning |
|---|---|
| \(SS_{xx}\) | Total variation in the predictor values around \(\bar{x}\). |
| \(SS_{yy}\) | Total variation in the response values around \(\bar{y}\). |
| \(SS_{xy}\) | A measure of how \(x\) and \(y\) vary together. |
| Correlation coefficient \(r\) | A unitless measure of the direction and strength of the linear relationship between \(x\) and \(y\). |
| Population correlation \(\rho\) | The population version of the correlation coefficient. |
| Slope-correlation connection | In simple linear regression, \(b_1=r(s_y/s_x)\). |
| Correlation test | In simple linear regression, testing \(\rho=0\) is equivalent to testing \(\beta_1=0\). |
| SSE | Unexplained variation around the fitted regression line. |
| SSR | Explained variation, calculated as \(SS_{yy}-SSE\). |
| Variation decomposition | In simple linear regression, \(SS_{yy}=SSR+SSE\). |
| \(r^2\) | The proportion of sample variation in \(y\) explained by the linear regression model using \(x\). |
What does the sign of \(SS_{xy}\) tell us about the direction of a linear association?
Why does correlation have to be between -1 and 1?
What does it mean for correlation to be unitless? Why is this different from the slope?
If \(r\) is close to 0, what can we conclude? What should we avoid concluding?
Why is a high correlation not enough to conclude that \(x\) causes \(y\)?
In simple linear regression, why are the tests \(H_0:\rho=0\) and \(H_0:\beta_1=0\) equivalent?
What does \(r^2=0.72\) mean in words?
Why is it incorrect to say that \(r^2=0.72\) means the model predicts 72% of observations correctly?
If two variables have a strong nonlinear relationship, must the correlation be large in absolute value? Explain.
Why might the slope be more useful than correlation when reporting the result of a regression analysis?
In the relationship \(SS_{yy}=SSR+SSE\), what do the three pieces represent?
Suppose you want to report the expected change in \(y\) for a 1-unit increase in \(x\). Should you report the slope, the correlation, or \(r^2\)? Explain.
Direction of association. If \(SS_{xy}\) is positive, then larger values of \(x\) tend to occur with larger values of \(y\). If \(SS_{xy}\) is negative, then larger values of \(x\) tend to occur with smaller values of \(y\).
Standardized co-variation. Correlation divides \(SS_{xy}\) by the largest possible magnitude it could have, \(\sqrt{SS_{xx}SS_{yy}}\). This standardization forces \(r\) to fall between -1 and 1.
No measurement units. Correlation does not depend on the units used to measure \(x\) and \(y\). The slope does depend on units because it measures the change in \(y\) for a 1-unit increase in \(x\).
Little linear association. If \(r\) is close to 0, there is little evidence of a linear relationship. We should avoid saying there is no relationship at all because the relationship could be nonlinear.
Association is not causation. A high correlation can occur because of causation, but it can also occur because of lurking variables, common trends, or coincidence. Correlation alone does not establish a cause-and-effect relationship.
Both test linear association. In simple linear regression, the slope and correlation have the same sign, and one is zero exactly when the other is zero. Therefore, testing whether \(\rho=0\) is equivalent to testing whether \(\beta_1=0\).
Variation explained. An \(r^2\) value of 0.72 means that 72% of the sample variation in \(y\) is explained by the linear regression model using \(x\).
Not accuracy classification. \(r^2\) is about variation explained, not the percentage of observations predicted correctly. Regression predictions are numerical, and the model can still make errors even when \(r^2\) is high.
Not necessarily. Correlation measures linear association. A strong curved relationship can have a small correlation if the positive and negative linear trends cancel out.
Slope gives a contextual effect size. Correlation summarizes strength and direction, but the slope tells us how much the predicted response changes for a 1-unit increase in the predictor. That makes the slope more directly interpretable in context.
Total, explained, and unexplained variation. \(SS_{yy}\) is the total variation in the response around its mean. SSR is the part explained by the regression model. SSE is the part left unexplained around the fitted line.
Report the slope. The slope directly answers this question because it measures the expected change in the response variable for a 1-unit increase in the predictor. Correlation describes strength and direction, and \(r^2\) describes variation explained, but neither gives the expected change in response units.