20 Basic programming II: iteration and functional programming

Objectives

Recognize how iteration differs in R. In many languages you must explicitly loop over a vector’s elements. R is different: doubling a numeric vector is as simple as 2 * x, because the operation is vectorized. You’ve already used implicit iteration through facet_wrap()/facet_grid(), group_by() + summarize(), and the unnest_*() functions. This session covers more general tools for explicit iteration.
Modify multiple columns with across(). across() applies one or more functions to a set of columns inside summarize() or mutate(). Learn to select columns with .cols, supply functions via .fns, control output names with .names, and recognize the one setting that silently overwrites your original columns if you forget it.
Read and combine many files with purrr. purrr::map() iterates over a vector or list, applying a function to each element and returning a list. Learn the three-step pattern for importing a directory of files: list paths with list.files(), map a reading function over them, and combine the results with purrr::list_rbind(), including how it quietly handles a list with gaps in it.
Handle failures and heterogeneous data. When iterating over many inputs, some will fail. Use purrr::possibly() to wrap a function so failures return a default (such as NULL) instead of stopping the whole iteration, and know exactly what happens to those defaults once you try to recombine everything.
Save multiple outputs. The same iteration patterns apply in reverse: writing multiple data frames or plots to disk with walk2()/iwalk().

Notes

library(tidyverse)

Introduction: implicit and explicit iteration

Iteration is everywhere in R, but much of it is invisible. Doubling a vector is a single vectorized expression, 2 * x, not a loop, and tools you’ve already used, faceting, group-wise summarizing, unnesting a list-column, all repeat the same action across many subsets or elements without you ever writing an explicit loop yourself. Functional programming tools make the remaining, less automatic cases just as concise. This session covers three common ones: modifying multiple columns, reading multiple files, and saving multiple outputs.

Modifying multiple columns with `across()`

across(), used inside summarize() or mutate(), applies a function (or several) to a set of columns at once, so you never have to write out the same summary call once per column. It takes three key arguments. .cols selects which columns to operate on, using the same tidy-select syntax as select(), including helpers like where(is.numeric). .fns supplies the function or functions to apply: a single bare function name (no parentheses; see Example 19.1 for what happens if you add them), an anonymous function (\(x) ...) when you need extra arguments, or a named list when you want to apply several functions at once, with the names becoming part of the output column names. .names controls those output names directly, using a glue specification like "{.fn}_{.col}"; inside mutate() specifically, leaving .names out means the results silently overwrite the original columns rather than creating new ones (see Example 19.2).

# summarize every numeric column with two functions at once, custom names
iris |>
  summarize(
    across(
      where(is.numeric),
      list(mean = \(x) mean(x, na.rm = TRUE), sd = \(x) sd(x, na.rm = TRUE)),
      .names = "{.fn}_{.col}"
    )
  )

  mean_Sepal.Length sd_Sepal.Length mean_Sepal.Width sd_Sepal.Width
1          5.843333       0.8280661         3.057333      0.4358663
  mean_Petal.Length sd_Petal.Length mean_Petal.Width sd_Petal.Width
1             3.758        1.765298         1.199333      0.7622377

# replace missing values with zero, keeping the originals alongside
df_miss <- tibble(a = c(1, NA, 3), b = c(4, 5, NA))
df_miss |> mutate(across(a:b, \(x) coalesce(x, 0), .names = "{.col}_na0"))

# A tibble: 3 × 4
      a     b a_na0 b_na0
  <dbl> <dbl> <dbl> <dbl>
1     1     4     1     4
2    NA     5     0     5
3     3    NA     3     0

if_any() and if_all() extend the same idea to filtering: filter(if_any(where(is.na))) keeps rows with at least one missing value among the selected columns, while filter(if_all(where(is.na))) keeps only rows where every selected column is missing. And because .cols is tidy-select, a helper function that embraces its column argument (as in Session 19) can accept a user-supplied set of columns and hand it straight to across().

Reading multiple files with `purrr::map()`

Combining data spread across many files follows a reliable three-step pattern: list the files, read each one, then stack the results. Here it is end to end, first writing one file per species so there’s something real to read back:

csv_dir <- tempfile("species_csv_")
dir.create(csv_dir)

palmerpenguins::penguins |>
  group_by(species) |>
  group_walk(~ write_csv(.x, file.path(csv_dir, paste0(.y$species, ".csv"))))

list.files(csv_dir)

[1] "Adelie.csv"    "Chinstrap.csv" "Gentoo.csv"

paths <- list.files(csv_dir, pattern = "[.]csv$", full.names = TRUE)

penguins_combined <- paths |>
  set_names(\(p) str_remove(basename(p), "[.]csv$")) |>   # name each path by its file name
  map(read_csv) |>                                         # read each file into a tibble
  list_rbind(names_to = "species_file")                    # stack them, keeping the source

penguins_combined |> count(species_file)

# A tibble: 3 × 2
  species_file     n
  <chr>        <int>
1 Adelie         152
2 Chinstrap       68
3 Gentoo         124

list.files() collects the paths, restricting to .csv files with a regular-expression pattern and returning full, readable paths with full.names = TRUE. map() applies the reading function to each path in turn, returning a list of tibbles. list_rbind() stacks that list into one tibble; names_to turns whatever name each list element had (here, the species, extracted from the file name) into an ordinary column, which is exactly how information hiding in a file name makes it into your data. When the files aren’t perfectly uniform in structure, inspecting each one’s columns and types before binding (a small helper that reports each file’s column names and types, mapped over the same list of paths) catches a structural mismatch before it becomes a confusing binding error.

Handling failures

map() stops at the first error, which means one unreadable file can prevent every other file in the batch from being read at all. purrr::possibly() wraps a function so that a failure returns a specified default, commonly NULL, instead of stopping everything:

bad_paths <- c(paths, file.path(csv_dir, "does_not_exist.csv"))

safe_read <- possibly(\(p) read_csv(p, show_col_types = FALSE), otherwise = NULL)
results <- map(bad_paths, safe_read)

failed <- map_lgl(results, is.null)
bad_paths[failed]        # exactly the paths that failed, ready to investigate

[1] "C:\\Users\\JOSHUA~1\\AppData\\Local\\Temp\\RtmpM1bKIm\\species_csv_76803f3656eb/does_not_exist.csv"

list_rbind(results) |> nrow()   # every successfully read row, recombined anyway

[1] 344

This pattern lets a batch job process every file that can be read while cleanly flagging the ones that can’t, rather than an all-or-nothing failure (see Example 19.3 for exactly how list_rbind() handles the NULLs left behind by the failures).

Saving multiple outputs

The same tools work in reverse for writing multiple outputs. walk2() iterates over two parallel vectors or lists purely for a side effect (here, writing a file), discarding whatever the function returns, which is exactly right for something like write_csv() that you call only for what it does, not what it returns:

out_dir <- tempfile("summaries_")
dir.create(out_dir)

summaries <- list(
  by_species = penguins_combined |> count(species_file),
  by_island = palmerpenguins::penguins |> count(island)
)

out_paths <- file.path(out_dir, paste0(names(summaries), ".csv"))
walk2(summaries, out_paths, write_csv)

list.files(out_dir)

[1] "by_island.csv"  "by_species.csv"

The same pattern extends to saving a series of plots: build a named list of ggplot objects, then iwalk() (or walk2() with matching file names) over the list, calling ggsave() once per plot.

Fringe cases and common pitfalls

Example 19.1

Calling a function inside across() instead of naming it produces a real, specific error.

iris |> summarize(across(where(is.numeric), mean()))

Error in `summarize()`:
ℹ In argument: `across(where(is.numeric), mean())`.
Caused by error in `mean.default()`:
! argument "x" is missing, with no default

across() expects a function it can call once per column, so the second argument should be the bare function name, mean, not the result of calling it, mean(). Writing mean() calls mean immediately, with no arguments at all, which fails right away with “argument ‘x’ is missing, with no default,” before across() even gets a chance to do anything. If you see that particular error coming from inside an across() call, check for a stray pair of parentheses on the function you meant to pass by name.

Example 19.2

across() inside mutate(), without .names, silently overwrites your original columns.

df <- tibble(x = c(1, 2, 3), y = c(10, 20, 30))

df |> mutate(across(where(is.numeric), \(v) v - mean(v)))          # x and y are gone

# A tibble: 3 × 2
      x     y
  <dbl> <dbl>
1    -1   -10
2     0     0
3     1    10

df |> mutate(across(where(is.numeric), \(v) v - mean(v), .names = "{.col}_centered"))

# A tibble: 3 × 4
      x     y x_centered y_centered
  <dbl> <dbl>      <dbl>      <dbl>
1     1    10         -1        -10
2     2    20          0          0
3     3    30          1         10

Inside summarize(), across() always produces new, separate output columns, since summarize() doesn’t have “original” row-level columns to collide with. Inside mutate(), though, a transformed column named exactly the same as the column it came from replaces that column outright, with no warning that the original values are gone. There’s no error because this is a completely valid thing to want to do; the danger is only in not intending it. Supplying .names (as in the second call above) keeps the originals and adds the transformed versions alongside them instead.

Example 19.3

list_rbind() drops NULL list elements without saying a word.

partial_results <- list(tibble(x = 1), NULL, tibble(x = 2), NULL)
length(partial_results)     # 4 elements went in...

[1] 4

list_rbind(partial_results) # ...and only the 2 real ones come out, silently

# A tibble: 2 × 1
      x
  <dbl>
1     1
2     2

This is exactly the behavior the possibly() pattern from this session depends on: every failed read becomes a NULL, and list_rbind() quietly skips every NULL when stacking the rest into one tibble, with no message telling you how many were dropped. That silence is convenient when you already know some inputs might fail and you’ve separately checked which ones did (as in the map_lgl(results, is.null) line above), and it’s a trap if you haven’t: a batch job that “worked” and returned a smaller-than-expected tibble may have silently lost several inputs along the way. Always compare the length of your input list to the row count (or list length, if you’re not immediately combining) of what came out.

Example 19.4

across(everything()) after group_by() skips the grouping column on its own, without being asked to.

grouped <- tibble(g = c("a", "a", "b"), x = 1:3, y = 4:6) |> group_by(g)
grouped |> summarize(across(everything(), mean))

# A tibble: 2 × 3
  g         x     y
  <chr> <dbl> <dbl>
1 a       1.5   4.5
2 b       3     6

g never gets passed to mean() at all, even though everything() would ordinarily include it; dplyr automatically excludes the active grouping columns from across(everything(), ...), since they’re already preserved as the grouping structure of the output and averaging a character column would make no sense (and would error) anyway. This is a helpful default rather than a trap, but it’s worth knowing about explicitly: if you ever do want to apply a function to a grouping column too, you need to name it directly rather than relying on everything() to reach it.

Recap

Term	Definition
`across(.cols, .fns, .names)`	Applies one or more functions to a tidy-selected set of columns inside `summarize()`/`mutate()`.
Bare function name	`across()` expects a function to call (`mean`), not the result of calling it (`mean()`).
`.names` inside `mutate()`	Without it, `across()`’s output silently replaces the original columns instead of adding new ones.
`if_any()` / `if_all()`	Filter helpers combining a condition across several columns with OR / AND logic.
`list.files()`	Lists file paths in a directory, filterable by a regex `pattern`, with `full.names = TRUE` for ready-to-use paths.
`map()`	Applies a function to every element of a vector or list, returning a list.
`list_rbind()`	Stacks a list of data frames into one tibble; silently drops any `NULL` elements.
`possibly()`	Wraps a function so a failure returns a default value (often `NULL`) instead of stopping the whole iteration.
`walk()` / `walk2()` / `iwalk()`	Like `map()`, but for a function called purely for its side effect (such as writing a file), discarding the return value.
Grouping-column exclusion	`across(everything(), ...)` automatically skips the active `group_by()` columns.

Check your understanding

Problems

What is the difference between passing mean and mean() as the second argument to across()? What error would the second one produce?
You run df |> mutate(across(where(is.numeric), \(x) x * 2)) and notice your original numeric columns are gone, replaced by their doubled values. What caused this, and how would you keep both?
You use possibly() to read 20 files and combine the results with list_rbind(), and get back a tibble built from only 17 files’ worth of rows, with no error or warning. What is the most likely explanation, and how would you confirm it?
Why doesn’t group_by(region) |> summarize(across(everything(), mean)) try to average the region column, even though everything() would normally include it?
What is the three-step pattern for reading and combining many files, and which function in that pattern is responsible for turning a list of tibbles into one?

Solutions

mean (no parentheses) hands across() the function itself, ready to be called once per column, which is what across() expects. mean() calls mean immediately with no arguments and fails right away with "argument 'x' is missing, with no default", before across() ever gets to use it.
Inside mutate(), across()’s output columns default to the same names as the input columns unless you specify .names, so the doubled values silently overwrote the originals instead of creating new columns. Adding .names = "{.col}_doubled" (or any other pattern that doesn’t collide with the original names) keeps both versions.
possibly() almost certainly returned NULL for the 3 files that failed to read, and list_rbind() silently drops any NULL elements when combining a list, with no warning about how many (if any) were dropped. Running map_lgl(results, is.null) (or comparing length(results) to the number of rows or tibbles actually combined) would confirm exactly which files were skipped.
dplyr automatically excludes the active grouping columns from across(everything(), ...), since a grouping column is already preserved as part of the output’s structure and computing something like a mean on a character grouping column wouldn’t be meaningful (and would error) in the first place.
List the files with list.files() (optionally filtering by pattern and using full.names = TRUE), read each one with map() applied to a reading function like read_csv(), and combine the resulting list of tibbles into one with list_rbind(), which is the function that actually performs the stacking.