“If all you have is a hammer, everything looks like a nail.” - Bernard Baruch
In previous chapters, the response variable was quantitative. Logistic regression is used when the response is categorical, especially when the response has two possible outcomes such as yes/no, success/failure, or present/absent.
The main idea is simple but important: when \(Y\) is binary, the mean response is a probability. Logistic regression models that probability while keeping fitted values between 0 and 1.
23.1 Binary Response Variable
In a variety of regression applications, the response variable of interest has only two possible qualitative outcomes, and therefore can be represented by a binary indicator variable taking on values 0 and 1.
A binary response variable is said to involve binary responses or dichotomous responses.
We consider first the meaning of the response function when the outcome variable is binary, and then we take up some special problems that arise with this type of response variable.
23.1.1 Response Function Meaning
Consider the simple linear regression model \[
\begin{align*}
y_{i} & =\beta_{0}+\beta_{1}x_{i}+\varepsilon_{i}\qquad y_{i}=0,1
\end{align*}
\] where the outcome \(y_{i}\) is binary taking on the value of either 0 or 1.
The expected response $E$ has a special meaning in this case.
Since \(E\left[ \varepsilon_{i}\right] =0\), we have \[
\begin{align*}
E\left[ y_{i}\right] & =\beta_{0}+\beta_{1}x_{i}
\end{align*}
\]
Consider \(y_{i}\) to be a Bernoulli random variable for which we can state the probability distribution as follows \[
\begin{align*}
y_{i} & \quad\textbf{ Probability}\\
1 & \quad P\left(y_{i}=1\right)={\pi_{i}}\\
0 & \quad P\left(y_{i}=0\right)={1-\pi_{i}}
\end{align*}
\]
Thus, \(\pi_{i}\) is the probability that \(y_{i}=1\) and \(1-\pi_{i}\) is the probability that \(y_{i}=0\)
By the definition of a discrete random variable, we have \[
{\begin{align*}
E\left[ y_{i}\right] & =1\left(\pi_{i}\right)+0\left(1-\pi_{i}\right)\\
& =\pi_{i}\\
& =P\left(y_{i}=1\right)
\end{align*}}
\]
The mean response \(E\{y_i\}\) as given by the response function is therefore simply the probability that \(y_i = 1\) when the level of the predictor variable is \(x_i\).
This interpretation of the mean response applies whether the response function is a simple linear one, as here, or a complex multiple regression one.
The mean response, when the outcome variable is a 0, 1 indicator variable, always represents the probability that \(Y = 1\) for the given levels of the predictor variables.
Example 23.1
ExampleThe mean of a binary response is a probability
Suppose \(Y\) indicates whether a student passes an exam:
\(Y = 1\) means the student passes,
\(Y = 0\) means the student does not pass.
If the probability of passing is \(\pi = 0.80\), then
\[
E(Y) = 1(0.80) + 0(0.20) = 0.80.
\]
For a binary response, the expected value is not an average outcome such as “0.80 students.” It is the probability that the outcome equals 1.
23.1.2 Problems when response is binary
Special problems arise, unfortunately, when the response variable is an indicator variable. We consider three of these now, using a simple linear regression model as an illustration.
Nonnormal error terms: For a binary 0, 1 response variable, each error term can take on only two values: \[
\begin{align*}
\text{When }y_{i}=1: & \quad\varepsilon_{i}=1-\beta_{0}-\beta_{1}x_{i}\\
\text{When }y_{i}=0: & \quad\varepsilon_{i}=0-\beta_{0}-\beta_{1}x_{i}
\end{align*}
\]
Clearly, the normal error regression model, which assumes that the \(\varepsilon_i\) are normally distributed, is not appropriate.
Nonconstant error variance: Another problem with the error terms is that they do not have equal variances when the response variable is an indicator variable.
To see this, note that \[
\begin{align*}
Var[\varepsilon_i] = \pi_i(1-\pi_i)
\end{align*}
\]
Note that \(Var\{\varepsilon_i\}\) depends on \(\pi_i\), which usually depends on \(x_i\). Hence, the error variances will differ at different levels of \(X\), and ordinary least squares will no longer be optimal.
Constraints on the response function: Since the response function represents probabilities when the outcome variable is a 0, 1 indicator variable, the mean responses should be constrained as follows: \[
0\le E\{Y\}=\pi \le 1
\]
The difficulties created by the need for the restriction on the response function are the most serious.
One could use weighted least squares to handle the problem of unequal error variances.
In addition, with large sample sizes the method of least squares provides estimators that are asymptotically normal under quite general conditions, even if the distribution of the error terms is far from normal.
However, the constraint on the mean responses to fall between 0 and 1 frequently will rule out a linear response function.
NoteThe biggest problem is the probability scale
For binary responses, nonnormality and nonconstant variance matter. But the most visible problem is that a straight-line model can predict values below 0 or above 1.
Logistic regression solves this by modeling a transformed probability with a linear predictor while keeping the fitted probability itself between 0 and 1.
23.2 Sigmoidal Response Functions
In this section, we will examine two response functions for modeling binary responses.
These functions are bounded between 0 and 1, have a characteristic sigmoidal- or S-shape, and approach 0 and 1 asymptotically.
These functions arise naturally when the binary response variable results from a zero-one recoding (or dichotomization) of an underlying continuous response variable, and they are often appropriate for discrete binary responses as well.
23.2.1 Probit Mean Response Function
Consider a health researcher studying the effect of a mother’s use of alcohol (\(x\) -an index of degree of alcohol use during pregnancy) on the duration of her pregnancy (\(y^C\)).
Here we use the superscript \(c\) to emphasize that the response variable, pregnancy duration, is a continuous response.
This can be represented by a simple linear regression model: \[
y^C_i = \beta_0^c+\beta_1^c x_i +\varepsilon_i^c
\] and we will assume that \(\varepsilon^c_i\) is normally distributed with mean zero and variance \(\sigma^2_c\).
If the continuous response variable, pregnancy duration, were available, we might proceed with the usual simple linear regression analysis. However, in this instance, researchers coded each pregnancy duration as preterm or full term using the following rule: \[
\begin{align*}
y_{i} & =\begin{cases}
1 & \text{ if }y_{i}^{c}\le38\text{ weeks (preterm)}\\
0 & \text{ if }y_{i}^{c}>38\text{ weeks (full term)}
\end{cases}
\end{align*}
\]
It follows then that \[
\begin{align*}
P\left(y_{i}=1\right)=\pi_{i} & =P\left(y_{i}^{c}\le38\right)\\
& =P\left(\beta_{0}^{c}+\beta_{1}^{c}x_{i}+\varepsilon_{i}^{c}\le38\right)\\
& =P\left(\varepsilon_{i}^{c}\le38-\beta_{0}^{c}-\beta_{1}^{c}x_{i}\right)\\
& =P\left(\frac{\varepsilon_{i}^{c}}{\sigma_{c}}\le\frac{38-\beta_{0}^{c}}{\sigma_{c}}-\frac{\beta_{1}^{c}}{\sigma_{c}}x_{i}\right)\\
& =P\left(Z\le\beta_{0}^{*}+\beta_{1}^{*}x_{i}\right)
\end{align*}
\] where \[
\begin{align*}
\beta_{0}^{*} & =\frac{38-\beta_{0}^{c}}{\sigma_{c}}\\
\beta_{1}^{*} & =-\frac{\beta_{1}^{c}}{\sigma_{c}}\\
Z & =\frac{\varepsilon_{i}^{c}}{\sigma_{c}}.
\end{align*}
\]
Note that \(Z\) follows a standard normal distribution.
If we let \(P\left(Z\le z\right)=\Phi\left(z\right)\), we have \[
\begin{align*}
P\left(y_{i}=1\right) & =\Phi\left(\beta_{0}^{*}+\beta_{1}^{*}x_{i}\right)
\end{align*}
\]
From this we have what is known as the probit mean response function \[
\begin{align*}
E\left[ y_{i}\right] & =\pi_{i}=\Phi\left(\beta_{0}^{*}+\beta_{1}^{*}x_{i}\right)
\end{align*}
\]
The inverse function \(\Phi^{-1}\) of the standard normal cumulative distribution function is sometimes called the probit transformation.
We solve for the linear predictor \(\beta_{0}^{*}+\beta_{1}^{*}x_{i}\) by applying the probit transformation to both sides of the expression: \[
\begin{align*}
\Phi^{-1}\left(\pi_{i}\right) & =\pi_{i}^{\prime}=\beta_{0}^{*}+\beta_{1}^{*}x_{i}
\end{align*}
\]
The resulting expression \(\pi_{i}^{\prime}=\beta_{0}^{*}+\beta_{1}^{*}x_{i}\) is called the probit response function, or more generally, the linear predictor.
Example 23.2
ExampleHow probit coefficients affect the S-shaped curve
The probit mean response function is
\[
\pi_i = \Phi(\beta_0^*+\beta_1^*x_i).
\]
The intercept shifts the curve left or right, while the slope controls how quickly the probability changes.
library(tidyverse)probit_scenario_levels<-c("Same intercept, steeper slope","Different intercept, same slope")probit_grid<-expand_grid( x =seq(-4, 4, length.out =200), scenario =probit_scenario_levels)|>mutate( scenario =factor(scenario, levels =probit_scenario_levels), beta_0 =if_else(scenario=="Same intercept, steeper slope", 0, 1), beta_1 =if_else(scenario=="Same intercept, steeper slope", 1.5, 1), probability =pnorm(beta_0+beta_1*x))ggplot(probit_grid, aes(x =x, y =probability))+geom_line(linewidth =1, color ="#0072B2")+facet_wrap(~scenario)+labs( x ="x", y =expression(pi), title ="Probit response curves")+ylim(0, 1)+theme_minimal()
Both panels keep the fitted probability between 0 and 1. The curve changes on the probability scale, but the probit transformation changes linearly with \(x\).
23.3 Logistic Mean Response Function
We have seen that the assumption of normally distributed errors for the underlying continuous response variable led to the use of the standard normal cumulative distribution function, \(\Phi\) to model \(\pi_i\).
An alternative error distribution that is very similar to the normal distribution is the logistic distribution.
Plots of the standard normal density function and the logistic density function, each with mean zero and variance one, are very similar. The logistic distribution has slightly heavier tails.
Example 23.3
ExampleNormal and logistic densities are similar
The following plot compares the standard normal density to a rescaled logistic density with mean 0 and variance 1.
density_grid<-tibble(x =seq(-4, 4, length.out =400))|>mutate( normal =dnorm(x), logistic =dlogis(x, scale =sqrt(3)/pi))|>pivot_longer( cols =c(normal, logistic), names_to ="distribution", values_to ="density")ggplot(density_grid, aes(x =x, y =density, color =distribution))+geom_line(linewidth =1)+labs( x ="value", y ="density", color ="Distribution", title ="Normal and logistic densities")+theme_minimal()
This similarity is one reason probit and logistic regression often produce similar fitted probabilities.
Note the cumulative distribution function of a logistic random variable \(\varepsilon_{L}\) having mean 0 and standard deviation \(\sigma=\pi/\sqrt{3}\) is: \[
\begin{align*}
F_{L}\left(\varepsilon_{L}\right) & =\frac{\exp\left(\varepsilon_{L}\right)}{1+\exp\left(\varepsilon_{L}\right)}
\end{align*}
\]
Suppose now that \(\varepsilon_{i}^{c}\) has a logistic distribution with mean 0 and standard deviation \(\sigma_{c}\). Then we have \[
\begin{align*}
P\left(y_{i}=1\right) & =P\left(\frac{\varepsilon_{i}^{c}}{\sigma_{c}}\le\beta_{0}^{*}+\beta_{1}^{*}x_{i}\right)
\end{align*}
\] where \(\varepsilon_{i}^{c}/\sigma_{c}\) follows a logistic distribution with mean zero and standard deviation one.
Multiplying both sides of the inequality inside the probability statement on the right by \(\pi/\sqrt{3}\) gives us \[
\begin{align*}
P\left(y_{i}=1\right)=\pi_{i} & =P\left(\frac{\pi}{\sqrt{3}}\frac{\varepsilon_{i}^{c}}{\sigma_{c}}\le\frac{\pi}{\sqrt{3}}\beta_{0}^{*}+\frac{\pi}{\sqrt{3}}\beta_{1}^{*}x_{i}\right)\\
& =P\left(\varepsilon_{L}\le\beta_{0}+\beta_{1}x_{i}\right)\\
& =F_{L}\left(\beta_{0}+\beta_{1}x_{i}\right)\\
& =\frac{\exp\left(\beta_{0}+\beta_{1}x_{i}\right)}{1+\exp\left(\beta_{0}+\beta_{1}x_{i}\right)}
\end{align*}
\] where \[
\begin{align*}
\beta_{0} & =\frac{\pi}{\sqrt{3}}\beta_{0}^{*}\\
\beta_{1} & =\frac{\pi}{\sqrt{3}}\beta_{1}^{*}
\end{align*}
\] denote the logistic regression parameters.
To summarize, the logistic mean response function is \[
\begin{align*}
E\left[ y_{i}\right] & =\pi_{i}\\
& =F_{L}\left(\beta_{0}+\beta_{1}x_{i}\right)\\
& =\frac{\exp\left(\beta_{0}+\beta_{1}x_{i}\right)}{1+\exp\left(\beta_{0}+\beta_{1}x_{i}\right)}\\
& =\frac{1}{1+\exp\left(-\beta_{0}-\beta_{1}x_{i}\right)}
\end{align*}
\]
Applying the inverse of the cumulative distribution function \(F_{L}\) gives \[
\begin{align*}
F_{L}^{-1}\left(\pi_{i}\right) & =\beta_{0}+\beta_{1}x_{i}=\pi_{i}^{\prime}
\end{align*}
\]
\(F_{L}^{-1}\left(\pi_{i}\right)\) is called the logit transformation and is given by \[
\begin{align*}
F_{L}^{-1}\left(\pi_{i}\right) & =\log\left(\frac{\pi_{i}}{1-\pi_{i}}\right)
\end{align*}
\] where the ratio \(\pi_{i}/\left(1-\pi_{i}\right)\) is called the odds.
Example 23.4
ExampleHow logistic coefficients affect the probability curve
The slope \(\beta_1\) determines whether the probability increases or decreases as \(x\) increases. Its magnitude controls how steeply the transition occurs.
logistic_slope_levels<-c("Negative slope","Small positive slope","Large positive slope")logistic_grid<-expand_grid( x =seq(-4, 4, length.out =200), slope =logistic_slope_levels)|>mutate( slope =factor(slope, levels =logistic_slope_levels), beta_0 =0, beta_1 =case_when(slope=="Negative slope"~-1,slope=="Small positive slope"~0.7,TRUE~2), probability =plogis(beta_0+beta_1*x))ggplot(logistic_grid, aes(x =x, y =probability))+geom_line(linewidth =1, color ="#0072B2")+facet_wrap(~slope)+labs( x ="x", y =expression(pi), title ="Logistic response curves")+ylim(0, 1)+theme_minimal()
The fitted relationship is linear on the logit scale, but it is S-shaped on the probability scale.
Example 23.5
ExampleProbabilities, odds, and logits
The logit transformation converts a probability to log-odds:
\[
\log\left(\frac{\pi}{1-\pi}\right).
\]
The table below shows how probabilities, odds, and logits are connected.
When \(\pi = 0.50\), the odds are 1 and the logit is 0. Probabilities below 0.50 have negative logits, and probabilities above 0.50 have positive logits.
23.4 Interpretation of the Coefficients
The interpretation of the estimated regression coefficient \(\hat{\beta}_1\) in the fitted logistic response function is not the straightforward interpretation of the slope in a linear regression model.
The reason is that the effect of a unit increase in \(x\) varies for the logistic regression model according to the location of the starting point on the \(x\) scale.
An interpretation of \(\hat{\beta}_1\) is found in the property of the fitted logistic function that the estimated odds \[
\frac{\hat{\pi}}{1-\hat{\pi}}
\] are multiplied by \[
\exp(\hat{\beta}_1)
\] for any unit increase in \(x\).
To see this, we consider the value of the fitted logit response function at \(X = x_j\); \[
\hat{\pi}^\prime (x_j)={\hat{\beta}_0+\hat{\beta}_1 x_j}
\]
The notation \(\hat{\pi}^\prime (x_j)\) indicates specifically the \(x\) level associated with the fitted value.
We also consider the value of the fitted logit response function at \(X = x_j + 1\); \[
\hat{\pi}^\prime (x_j+1)={\hat{\beta}_0+\hat{\beta}_1( x_j+1)}
\] The difference between the two fitted values is simply \[
{\begin{align*}
\hat{\pi}^\prime (x_j+1)-\hat{\pi}^\prime (x_j)& = \hat{\beta}_0+\hat{\beta}_1( x_j+1) - \hat{\beta}_0-\hat{\beta}_1 x_j \\
& = \hat{\beta}_1
\end{align*}}
\]
Now \(\hat{\pi}^\prime (x_j)\) is the logarithm of the estimated odds when \(X=x_j\); we shall denote it by \(\ln(\text{odds}_1)\).
Similarly, \(\hat{\pi}^\prime (x_j+1)\) is the logarithm of the estimated odds when \(X=x_j+1\); we shall denote it by \(\ln(\text{odds}_2)\).
Hence, the difference between the two fitted logit response values can be expressed as follows: \[
\begin{align*}
\ln(\text{odds}_2)-\ln(\text{odds}_1) = \ln\left(\frac{\text{odds}_2}{\text{odds}_1}\right) = \hat{\beta}_1
\end{align*}
\]
Taking antilogs (exponentials) of each side, we see that the estimated ratio of the odds, called the odds ratio, \(\hat{OR}\), is \[
\hat{OR} = \frac{\text{odds}_2}{\text{odds}_1}=\exp(\hat{\beta}_1)
\]
NoteA logistic slope is not a probability slope
In simple linear regression, a slope describes an additive change in the mean response. In logistic regression, \(\beta_1\) describes an additive change in log-odds.
After exponentiating, \(\exp(\beta_1)\) describes a multiplicative change in odds, not a direct additive change in probability.
Example 23.6
ExampleFitting a logistic regression model in R
In the mtcars data, the variable am equals 1 for manual transmission and 0 for automatic transmission. Suppose we model the probability that a car has manual transmission using weight, wt, measured in thousands of pounds.
The coefficient of wt is negative, so heavier cars are estimated to have lower odds of manual transmission. The odds ratio \(\exp(\hat{\beta}_1)\) gives the multiplicative change in the odds for each additional 1000 pounds.
We can also compute fitted probabilities for two car weights.
The probability decreases as weight increases, but the size of the probability change depends on where we are on the S-shaped curve.
23.5 Recap
In this chapter, we introduced logistic regression as a model for binary response variables.
Idea
Meaning
Binary response
A response variable with two possible outcomes, often coded 0 and 1.
Bernoulli response
A random variable that equals 1 with probability \(\pi\) and 0 with probability \(1-\pi\).
Mean of a binary response
For \(Y \in \{0,1\}\), \(E(Y)=P(Y=1)=\pi\).
Linear probability model
A linear model applied directly to a binary response; it can produce fitted values outside \([0,1]\).
Nonconstant variance
For a Bernoulli response, \(Var(Y)=\pi(1-\pi)\), so the variance depends on the probability.
Sigmoidal response function
An S-shaped function that stays between 0 and 1.
Probit model
A binary-response model using the standard normal cumulative distribution function, \(\Phi\).
Logistic model
A binary-response model using the logistic cumulative distribution function.
Linear predictor
The quantity \(\eta_i=\beta_0+\beta_1x_i\), which is linear in the model parameters.
Logit transformation
The transformation \(\log\left(\frac{\pi}{1-\pi}\right)\).
Odds
The ratio \(\frac{\pi}{1-\pi}\), comparing the probability of success to the probability of failure.
Odds ratio
A ratio of two odds; in simple logistic regression, \(\exp(\beta_1)\) is the odds ratio for a one-unit increase in \(x\).
Coefficient interpretation
Logistic coefficients are additive on the log-odds scale and multiplicative on the odds scale.
Fitted probability
A model-based estimate of \(P(Y=1)\) for given predictor values.
23.6 Check your understanding
NoteProblems
Why does the mean of a binary response equal a probability?
What are the two possible values of a Bernoulli response coded as 0 and 1?
Why can ordinary least squares be awkward for binary response data?
What is the most serious limitation of using a straight-line response function for probabilities?
What does it mean for a response function to be sigmoidal?
How are the probit and logistic mean response functions similar?
What is the linear predictor in simple logistic regression?
What is the logit transformation?
What are odds, in words?
If a probability is 0.50, what are the odds and logit?
Why is a logistic regression slope not interpreted like a linear regression slope?
What does \(\exp(\beta_1)\) represent in simple logistic regression?
If \(\exp(\beta_1)>1\), what does that imply about the odds as \(x\) increases?
If \(\exp(\beta_1)<1\), what does that imply about the odds as \(x\) increases?
Why can the effect of \(x\) on probability depend on the starting value of \(x\)?
What does type = "response" return when using predict() with a logistic regression model in R?
TipSolutions
A binary response can only be 0 or 1. Its expected value is \(1\cdot P(Y=1)+0\cdot P(Y=0)=P(Y=1)\).
The response equals either 0 or 1. Usually 1 represents the event or success of interest, and 0 represents its absence.
Binary data violate several usual linear-model assumptions. The errors are not normally distributed, the variance is not constant, and fitted values may fall outside the probability range.
Probabilities must stay between 0 and 1. A straight line can go below 0 or above 1, which produces impossible fitted probabilities.
It is S-shaped and bounded. A sigmoidal function approaches 0 and 1 gradually and never leaves the probability range.
Both map a linear predictor to a probability. Probit uses the standard normal cumulative distribution function, while logistic regression uses the logistic cumulative distribution function.
The linear predictor is \(\eta_i=\beta_0+\beta_1x_i\). It is the part of the model that is linear in the coefficients.
The logit is the log of the odds. It is \(\log\left(\frac{\pi}{1-\pi}\right)\).
Odds compare success to failure. If the probability of success is \(\pi\), then the odds are \(\pi/(1-\pi)\).
The odds are 1 and the logit is 0. A probability of 0.50 means success and failure are equally likely.
The slope is on the log-odds scale. It describes the additive change in log-odds for a one-unit increase in \(x\), not an additive change in probability.
It is an odds ratio. It gives the multiplicative change in the odds for a one-unit increase in \(x\).
The odds increase. An odds ratio greater than 1 means the odds of \(Y=1\) are multiplied upward as \(x\) increases.
The odds decrease. An odds ratio less than 1 means the odds of \(Y=1\) are multiplied downward as \(x\) increases.
The logistic curve is not a straight line on the probability scale. The same change in \(x\) can produce a small probability change near 0 or 1 and a larger change near the middle of the curve.
It returns fitted probabilities. With family = binomial(link = "logit"), type = "response" converts predictions back to the probability scale.
