1 Introduction to Regression Analysis

“Math is the logic of certainty; statistics is the logic of uncertainty.” - Joe Blitzstein

1.1 The Probabilistic Model

Most students who take an intro stats course are familiar with the idea of a random variable. Students are usually introduced to random variables using the notation \(X\).

In the technical sense, capital \(X\) should denote the random variable (that is, the function itself), while lowercase \(x\) denotes the values of that random variable. We will be loose on that convention, so you will see the lowercase used almost extensively even when discussing the random variable itself.

Review: Random Variable

A random variable is a function that assigns a numeric value to the outcomes in the sample space.

In regression, the random variable of interest is usually denoted as \(y\).

We want to predict or model (explain) this variable. Thus, we call this the response (or dependent) variable.

If we have measurements of this random variable, then we can express each value \(y\) as the mean value of \(y\) plus some random error.

That is, we can model the variable as \[ y = E(y) + \varepsilon \tag{1.1}\]

where \[\begin{align*} y = &\text{ dependent variable}\\ E(y) =& \text{ mean (or expected) value of } y\\ \varepsilon =& \text{ random error} \end{align*}\]

Mean plus error

The basic regression idea can be summarized as:

flowchart LR
  A["Mean response E(y)<br/>(systematic part)"] --> C["Observed response y"]
  B["Random error epsilon<br/>(individual deviation)"] --> C

The mean response is the part we try to explain with a model. The error term represents the leftover variation from one observation to another.

This model is referred to as a probabilistic model for \(y\). The term “probabilistic” is used because, under certain assumptions, we can make probability-based statements about the extent of the difference between \(y\) and \(E(y)\).

We might assert that the error term, \[ \varepsilon = y - E(y) \] follows a normal distribution.

Example 1.1: Separating the typical value from random error

Suppose \(y\) represents a student’s exam score. If a student studies 10 hours, the mean exam score for all students who study 10 hours might be \[ E(y) = 82. \]

That does not mean every student who studies 10 hours earns exactly 82. One student may earn 88, another may earn 79, and another may earn 83. The difference between an individual student’s score and the mean score for students with the same study time is the random error: \[ \varepsilon = y - E(y). \]

For the student who earns 88, \[ \varepsilon = 88 - 82 = 6. \]

Regression models try to estimate the systematic part, \(E(y)\), while acknowledging that individual observations still vary around that mean.

In practice, we will use sample data to estimate the parameters of the probabilistic model—specifically, the mean \(E(y)\) and the random error \(\varepsilon\).

We will later discuss a common assumption in regression: that the mean error is zero.

In other words, \[ E(\varepsilon) = 0 \]

Given this assumption, our best estimate of \(\varepsilon\) is zero. Therefore, we only need to estimate \(E(y)\).

The simplest method of estimating \(E(y)\) is to use the sample mean of \(y\) which we will denote as \[\begin{align*} \bar y= \frac{1}{n}\sum_{i=1}^n y_i \end{align*}\]

If we desired to predict a value of \(y\), then our best prediction would be just the sample mean: \[\begin{align*} \hat y = \bar y \end{align*}\] where \(\hat y\) denotes a predicted value of \(y\).

This would be the case with univariate data (we only have one variable in our data: \(y\)).

Unfortunately, this simple model does not take into consideration a number of variables, called predictor or explanatory variables, that may help predict the response variable.

These variables are sometimes called independent variables, but in regression it is often clearer to call them predictors because they are not always independent of each other or controlled by the researcher.

When there is only one predictor, a common starting point is the simple linear regression mean function \[ E(y) = \beta_0 + \beta_1x. \] In this model, \(E(y)\) changes linearly as \(x\) changes. The full probabilistic version keeps the same mean structure but adds random error: \[ y = \beta_0 + \beta_1x + \varepsilon. \]

Where this is heading

Much of the course will focus on choosing a useful form for \(E(y)\), estimating the unknown \(\beta\) parameters from data, and checking whether the error term behaves reasonably. The notation may look abstract now, but the core idea is simple: use predictors to model the mean response, then account for the fact that individual observations vary around that mean.

The process of identifying the mathematical model that describes the relationship between \(y\) and a set of predictor variables, and that best fits the data, is known as regression analysis.

1.2 Overview of Regression Analysis

We will denote the predictor variables as \[\begin{align*} x_1, x_2, \ldots, x_k \end{align*}\] where \(k\) is the number of predictor variables.

The goal of regression analysis is to create a prediction equation that accurately relates \(y\) to predictor variables, allowing us to predict \(y\) for given values of \(x_1, x_2, \ldots, x_k\) with minimal prediction error.

When predicting \(y\), we also need a measure of the reliability of our prediction, indicating how large the prediction error might be.

These elements form the core of regression analysis.

Beyond predicting \(y\), a regression model can also estimate the mean value of \(y\) for specific values of \(x_1, x_2, \ldots, x_k\) and explore the relationship between \(y\) and one or more predictor variables.

Example 1.2: Three regression goals

Suppose a university is studying the relationship between weekly study time (\(x\)) and final exam score (\(y\)).

Prediction: A student plans to study 12 hours. What final exam score should we predict for that individual student?
Estimation: Among all students who study 12 hours, what is the mean final exam score?
Explanation: Is there evidence that exam scores tend to increase as study time increases?

These questions are related, but they are not identical. Prediction focuses on an individual outcome, estimation focuses on a mean response, and explanation focuses on understanding the relationship between variables.

The process of regression analysis typically involves six key steps:

Hypothesize the form of the model for \(E(y)\).
Collect sample data.
Estimate the model’s unknown parameters using the sample data.
Define the probability distribution of the random error term, estimate any unknown parameters, and validate the assumptions made about this distribution.
Statistically assess the model’s usefulness.
If the model is effective, use it for prediction, estimation, and other purposes.

1.3 Collecting the Data for Regression

The first step listed above will be discussed later.

Once you’ve proposed a model for \(E(y)\), the next step is to gather sample data to estimate the model.

This means collecting data on both the response variable \(y\) and the predictor variables \(x_1, x_2, \ldots, x_k\) for each observation in your sample. In regression analysis, the sample includes data on multiple variables: \[ y, x_1, x_2, \ldots, x_k \] This is known as multivariate data.

Regression data can be either observational or experimental:

For observational data no control is exerted over the predictor variables (\(x\)’s).

For experimental data the predictor variables are controlled or manipulated.

Example 1.3: Observational regression data for GPA

Suppose you want to model a student’s annual GPA (\(y\)). One approach is to randomly select a sample of \(n=100\) students and record their GPA along with the values of each predictor variable.

Data for the first three students in the sample are shown in Table 1.1.

Table 1.1: Values of the response variable and predictor variables for the first three students.

	Student 1	Student 2	Student 3
Annual GPA \(y\)	3.8	2.7	3.5
Study Hours per Week, \(x_1\)	15	5	10
Class Attendance, \(x_2\) (days)	30	20	25
Extracurriculars, \(x_3\)	2	1	3
Age, \(x_4\) (years)	21	19	22
Employed, \(x_5\) (1 if yes, 0 if no)	0	1	0
Lives On Campus, \(x_6\) (1 if yes, 0 if no)	1	0	1

In this example, the \(x\) values, like study hours, class attendance, and extracurricular activities, are not predetermined before observing GPA \(y\); thus, the \(x\) values are uncontrolled. Therefore, the sample data are observational.

This type of study can help us describe relationships and make predictions. However, because the \(x\) values were not assigned by the researcher, we should be cautious about making cause-and-effect claims from the regression alone.

Association does not imply causation

Regression is excellent for describing relationships and making predictions, but the study design matters. If the data are observational, a relationship between \(x\) and \(y\) may reflect confounding variables, selection effects, or other background differences rather than a direct cause-and-effect relationship.

For example, students with higher attendance may also differ in motivation, preparation, work schedules, or access to support. Those factors can also affect GPA, so an attendance-GPA relationship by itself does not prove that changing attendance alone would change GPA.

Determining Sample Size for Regression with Observational Data

When applying regression to observational data, the required sample size for estimating the mean \(E(y)\) depends on three key factors:

Estimated population standard deviation
Confidence level
Desired margin of error (half-width of the confidence interval)

However, unlike the univariate case, \(E(y)\) is modeled as a function of multiple predictor variables, which adds complexity. The sample size must be large enough to estimate and test all parameters in the model.

To ensure a sufficient sample size, a common guideline is to select a sample size \(n\) that is at least 10 times the number of parameters in the model.

For instance, if a university registrar’s office uses the following model for the annual GPA \(y\) of a current student:

\[E(y) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_6 x_6\]

where \(x_1, x_2, \dots, x_6\) are defined in Table 1.1, the model includes six \(\beta\) parameters (excluding \(\beta_0\)). Therefore, they should include at least:

\[ 10 \times 6 = 60 \]

students in the sample.

Experimental Data

The second type of data in regression, experimental data, is generated through designed experiments where the predictor variables are set in advance (i.e., controlled) before observing the value of \(y\).

Example 1.4: Experimental regression data for plant growth

Consider a scenario where a researcher wants to study the effect of two predictor variables—say, fertilizer amount \(x_1\) and irrigation level \(x_2\)—on the growth rate \(y\) of plants. The researcher could choose three levels of fertilizer (10g, 20g, and 30g) and three levels of irrigation (1L, 2L, and 3L) and measure the growth rate in one plant for each of the \(3\times 3=9\) fertilizer–irrigation combinations (see Table 1.2 below).

Table 1.2: Values of the response variable and two predictor variables for the growth rate of plants.

Fertilizer, \(x_1\)	Irrigation, \(x_2\)	Growth Rate, \(y\)
10g	1L	5.2
10g	2L	6.1
10g	3L	5.8
20g	1L	7.0
20g	2L	7.5
20g	3L	7.3
30g	1L	8.4
30g	2L	8.7
30g	3L	8.1

In this experiment, the settings of the predictor variables are controlled, in contrast to the uncontrolled nature of observational data, like in the real estate sales example.

Because the researcher assigns the fertilizer and irrigation levels before observing plant growth, this design gives stronger evidence about the effect of those variables on the response.

In many studies, it is often not possible to control the values of the \(x\)’s, so most data collected for regression are observational.

So, why do we differentiate between these two types of data? We will learn that inferences from regression studies based on observational data have more limitations than those based on experimental data. Specifically, establishing a cause-and-effect relationship between variables is much more challenging with observational data than with experimental data.

1.4 Recap

Keyword	Definition
Random variable	A rule that assigns a numerical value to each outcome in a sample space.
Response variable	The variable being predicted, modeled, or explained; usually denoted by \(y\).
Expected value	The mean or typical value of a random variable; in regression, \(E(y)\) represents the mean response.
Random error	The difference between an observed value and its mean value, written as \(\varepsilon = y - E(y)\).
Probabilistic model	A model that represents an observation as a systematic mean component plus random error.
Predictor variable	A variable used to help explain or predict the response; also called an independent or explanatory variable.
Simple linear regression	A regression model with one predictor where the mean response is modeled as \(E(y)=\beta_0+\beta_1x\).
Regression analysis	The process of building and evaluating a model that relates a response variable to one or more predictors.
Prediction equation	An estimated equation used to predict \(y\) from values of the predictor variables.
Multivariate data	Data containing measurements on two or more variables for each observation.
Observational data	Data collected without controlling or assigning the predictor variables.
Experimental data	Data collected from a study in which the predictor variables are controlled or assigned before observing the response.
Association	A relationship or pattern between variables; an association alone does not prove that one variable causes another.

1.5 Check your understanding

Problems

In the model \(y = E(y) + \varepsilon\), why do we need both \(E(y)\) and \(\varepsilon\)? What would be missing if we only described the mean response?
Explain the difference between a response variable and a predictor variable using your own example.
Why is the sample mean \(\bar y\) a reasonable prediction when we only have measurements of \(y\) and no predictor variables?
A study finds that students who attend more classes tend to have higher GPAs. Why should we be careful about saying that attendance causes higher GPA if the data are observational?
Why do experimental data generally provide stronger evidence for cause-and-effect relationships than observational data?
In the simple linear regression mean function \(E(y)=\beta_0+\beta_1x\), what is being modeled: each individual value of \(y\), or the mean value of \(y\) for a given value of \(x\)? Explain why that distinction matters.

Solutions

Mean plus individual variation. The expected value \(E(y)\) describes the typical or systematic part of the response, while \(\varepsilon\) represents the random deviation of an individual observation from that typical value. If we only described the mean response, we would ignore the fact that real observations vary.
Response vs. predictor. The response variable is the outcome we want to model or predict, while predictor variables are used to explain or predict that outcome. For example, if we model house price using square footage, number of bedrooms, and age of the house, then house price is the response and the others are predictors.
Using the mean without predictors. If no predictor variables are available, every observation receives the same prediction. The sample mean is a natural choice because it summarizes the center of the observed \(y\) values and balances positive and negative deviations around that center.
Association is not causation. In observational data, students choose or experience their own attendance patterns. Other variables, such as motivation, course difficulty, work schedule, or prior preparation, may be related to both attendance and GPA. Regression can describe the association, but the study design alone may not justify a causal conclusion.
Control supports causal claims. In an experiment, the researcher assigns or controls the predictor values before observing the response. This helps isolate the effect of the predictor from other competing explanations. Observational studies often contain uncontrolled differences among individuals or units, which makes causal interpretation more limited.
Mean response, not every individual response. The expression \(E(y)=\beta_0+\beta_1x\) models the mean value of \(y\) for observations with a given value of \(x\). Individual observations may fall above or below that mean because of random error. This distinction matters because regression describes the typical response pattern while still allowing individual outcomes to vary.