8  Web Scraping with Rvest

“You’re playing worse everyday and right now you’re playing like it’s next month.” - Herb Brooks (Coach of the 1980 U.S. Olympic Hokey team.)

The 1980 U.S. Olympic Hockey team defeats the heavy favorite Soviet Union in the semifinal game on February 22, 1980. Known as the “Miracle on Ice,” the win propels the U.S. to the gold medal.

In sports analytics, data is often not available in a neat, downloadable format but is instead embedded within web pages across the internet. Web scraping provides a method to extract this data directly from websites, enabling analysts to gather information such as player statistics, game outcomes, or rankings. This section introduces web scraping using the rvest package in R, a tool within the tidyverse ecosystem that aligns with the data manipulation and modeling techniques covered in previous sections (e.g., “Data Handling” and “Tidymodels”).

We will explore the fundamentals of web scraping, address ethical considerations and data reliability, and demonstrate the process with examples from hockey and tennis websites.

8.1 Introduction to Web Scraping

Web scraping is the automated process of retrieving data from websites by fetching their HTML content and extracting specific pieces of information. In the context of sports analytics, web scraping is particularly valuable because much of the data—such as detailed player performance metrics or historical game records—is presented on web pages rather than in structured databases or APIs. The process involves two main steps: (1) accessing the web page’s HTML code and (2) parsing that code to isolate the desired data, such as tables, text, or lists.

The rvest package simplifies this process by providing functions to read HTML, select elements, and convert the extracted data into R data frames, which can then be manipulated using tidyverse tools. For example, a sports analyst might scrape a table of hockey player statistics from a website and use it to fit a regression model, building on skills from the “Regression” and “Tidymodels” sections.

8.2 Why Use Rvest?

The rvest package is an ideal choice for this course because it integrates seamlessly with the tidyverse framework you’ve already encountered. Functions like read_html() and html_table() work naturally with the pipe operator (|>), allowing scraped data to flow directly into dplyr operations for cleaning and analysis. This consistency reduces the learning curve and leverages your existing knowledge of data handling and modeling in R.

8.3 Ethical Considerations

Web scraping, while powerful, raises important ethical issues that must be addressed to ensure responsible use. Consider the following principles:

  • Website Terms of Service: Many websites specify in their terms of service whether scraping is permitted. Before scraping, review these terms or the site’s robots.txt file (e.g., http://example.com/robots.txt), which outlines allowable access by automated tools. Violating these terms could have legal implications.

  • Server Impact: Sending numerous requests to a website in a short time can strain its server, potentially disrupting service for other users. To avoid this, limit the frequency of requests—introduce delays if scraping multiple pages—and test your code on small samples first.

  • Data Privacy: Ensure that the data you scrape does not include personal or sensitive information (e.g., email addresses or birth dates) without explicit consent. In sports analytics, this is rarely an issue with public statistics, but it’s a critical consideration in broader contexts.

Ethical scraping respects the website’s resources and policies, ensuring that your data collection does not harm the source or its users.

8.4 Data Reliability

Data scraped from websites may not always be as reliable as data from curated datasets. Key concerns include:

  • Timeliness: Web content changes frequently. A table of player statistics scraped today might be outdated tomorrow if the website updates its records.

  • Accuracy: Websites may contain errors, such as typos in player names or incorrect numerical entries, which can affect subsequent analyses.

  • Consistency: Data presentation may vary across pages or over time, complicating efforts to combine scraped datasets.

To mitigate these issues, verify scraped data against alternative sources when feasible (e.g., official league records) and document the date of scraping for reference. Cleaning and validation steps, using techniques from the “Data Handling” section, are essential to prepare the data for reliable statistical modeling.

8.5 Basic Concepts of HTML

Web scraping requires a rudimentary understanding of HTML (HyperText Markup Language), the structure behind web pages. HTML organizes content into elements, defined by tags enclosed in angle brackets, such as <p> for paragraphs or <table> for tables. Tags often come in pairs, with an opening tag (e.g., <div>) and a closing tag (e.g., </div>), containing the content between them.

Elements can have attributes, such as id or class, which uniquely identify or categorize them. For example:

<table id="stats">
  <tr><td>Player</td><td>Goals</td></tr>
  <tr><td>Wayne Gretzky</td><td>894</td></tr>
</table>

Here, the <table> element has an id attribute "stats", and nested <tr> (table row) and <td> (table data) tags define its structure. The hierarchical nature of HTML—elements nested within elements—allows rvest to target specific data using these tags and attributes.

8.6 Using Rvest: A Step-by-Step Guide

Let’s outline the process of web scraping with rvest, which we’ll apply in the examples below.

8.6.1 Step 1: Load the Package

Begin by loading rvest, assuming it’s installed.

8.6.2 Step 2: Fetch the Web Page

Use read_html() to retrieve the HTML content of a web page, specified by its URL:

url = "http://example.com"
page = read_html(url)

This creates an object containing the page’s HTML structure.

8.6.3 Step 3: Select Elements

Identify the HTML elements containing your data using CSS selectors, which reference tags, classes (prefixed with .), or IDs (prefixed with #). For instance, .player selects elements with class="player", and #stats targets an element with id="stats". The function html_nodes() retrieves all matching elements, while html_node() selects the first:

elements = page |> html_nodes(".player")

Tools like SelectorGadget (a browser extension) can assist in finding selectors by highlighting elements on the page.

8.6.4 Step 4: Extract Data

Extract content from selected elements. Use html_text() for text within elements or html_table() for tables, which returns a data frame:

text_data = elements |> html_text()
table_data = page |> html_node("table") |> html_table()

8.6.5 Step 5: Clean the Data

Raw scraped data often requires cleaning. Apply dplyr functions (from the “Data Handling” section) to refine it:

library(dplyr)
clean_data = table_data |> 
  janitor::clean_names() |>  # Standardize column names
  filter(row_number() > 1)    # Remove header rows if repeated

The janitor package, often used with tidyverse, simplifies name standardization.

8.7 Example: Scraping Hockey Data

Consider scraping NHL player statistics from Hockey-Reference.com, a publicly accessible site with detailed sports data. We’ll target the 2023 skaters table at https://www.hockey-reference.com/leagues/NHL_2023_skaters.html.

8.7.1 Inspect the Page

Using your browser’s developer tools (right-click > “Inspect”), examine the page source. The skaters table has an id="player_stats". This is our target.

8.7.2 Scrape and Clean the Data

Here’s the complete code:

library(janitor)

# Specify the URL
url = "https://www.hockey-reference.com/leagues/NHL_2023_skaters.html"
page = read_html(url)

# Extract the table
table = page |> 
  html_node("#player_stats") |> 
  html_table()

# Clean the data
clean_data = table |> 
  #the first row is actually the row names
  row_to_names(row_number = 1) |>  
  # Convert column names to lowercase, remove spaces
  clean_names() |>
   # Select columns: goals (g), assists (a), points (pts)
  select(player, g, a, pts)

8.7.3 Verify the Result

The clean_data object is a data frame with columns like player, g (goals), a (assists), and pts (points), ready for analysis—perhaps a regression model predicting points from goals and assists, using techniques from the “Regression” section.

Ethical Note: Hockey-Reference.com allows limited scraping for personal use, per its terms, but always confirm current policies before extensive scraping.