Beginner Intermediate 11 min read

Chapter 50: R & the Tidyverse

R has occupied a unique position in data science for decades — born from statistical computing, it treats data manipulation, statistical modeling, and visualization as first-class concerns rather than afterthoughts. While Python dominates general-purpose ML engineering, R remains the language of choice for biostatistics, econometrics, clinical trials, and rigorous exploratory analysis. More importantly, the Tidyverse — a coherent collection of packages sharing a common grammar and philosophy — makes R's idioms remarkably readable and composable once you learn the patterns.

This chapter is written for engineers and analysts who already know Python. Rather than treating R as an alien language, we map its concepts onto familiar Python equivalents: a vector is like a NumPy array; a data frame is like a pandas DataFrame; a tibble is a stricter, friendlier DataFrame; dplyr is pandas' method-chaining API done right; ggplot2 is a grammar-of-graphics alternative to Matplotlib/Seaborn that lets you build charts by declaring what you want rather than scripting how to draw it. The pipe operator (|> or %>%) is the glue that makes long analysis pipelines read like prose.

By the end of this chapter you will be able to wrangle, reshape, iterate over, visualize, and model tabular data entirely within the Tidyverse, produce publication-quality figures, and embed your work in reproducible R Markdown documents. The goal is not to make you abandon Python — it is to give you a second powerful tool and the vocabulary to read and contribute to the enormous body of statistical software written in R.

Libraries covered: dplyrggplot2tidyverse

Learning Objectives

  • Understand R's core data structures — vectors, lists, factors, data frames, and tibbles — and map them to Python/pandas equivalents.
  • Use the pipe operator and core dplyr verbs (filter, select, mutate, summarise, group<em>by, arrange, join) to build readable data-transformation pipelines.
  • Reshape data between wide and long formats with tidyr's pivot</em>longer and pivot_wider, and handle missing values idiomatically.
  • Apply functional iteration patterns with purrr's map family in place of explicit loops, and understand when to use them over vectorized operations.
  • Build layered, publication-quality visualizations with ggplot2 using the grammar-of-graphics model.
  • Fit, summarize, and compare statistical models using base R and broom, and understand the tidy model output format.
  • Author reproducible analyses in R Markdown, knitting to HTML, PDF, and Word outputs with embedded code, results, and narrative.

50.1 R Fundamentals: Vectors, Data Frames, and Tibbles Beginner

R's Data Model

Everything in R is a vector. Unlike Python, where a scalar 42 is an int object, in R 42 is a numeric vector of length 1. This single idea explains most of R's surprising behaviors.

R
x <- c(1, 2, 3, 4, 5)   # c() "combines" scalars into a vector
class(x)                  # "numeric"
length(x)                 # 5
x[2]                      # 2  (R is 1-indexed!)
x[c(1, 3)]                # 1 3
x > 3                     # FALSE FALSE FALSE TRUE TRUE  (logical vector)
x[x > 3]                  # 4 5  (logical subsetting)

The &lt;- assignment arrow is idiomatic R; = works too but is typically reserved for function arguments.

Atomic Vector Types

R has six atomic types. The ones you will use daily:

  • logicalTRUE / FALSE (Python: bool)
  • integer1L, 2L (the L suffix marks integer literals)
  • double3.14 (Python: float; this is the default numeric type)
  • character&quot;hello&quot; (Python: str)
  • factor — categorical variable with fixed levels (Python: pd.Categorical)

Factors deserve special mention because they appear constantly in statistical output and plotting:

R
status <- factor(c("low", "high", "med", "high", "low"),
                 levels = c("low", "med", "high"))
table(status)   # counts per level, in level order
nlevels(status) # 3

forcats (part of the Tidyverse) provides helpers like fct<em>reorder, fct</em>lump, and fct<em>relevel for manipulating factors — essential before feeding categoricals to ggplot2.

Lists

A list is R's heterogeneous container — the Python dict / list hybrid:

R
person <- list(name = "Ada", age = 36, scores = c(95, 87, 92))
person$name       # "Ada"      ($ is like Python .attribute)
person[["age"]]   # 36        (double brackets extract the value)
person["age"]     # a list of length 1 (single brackets keep the list wrapper)

Data Frames and Tibbles

A data frame is a named list of equal-length vectors — directly analogous to a pandas DataFrame. A tibble (tbl</em>df) is the Tidyverse's improved version:

  • Never converts strings to factors automatically (unlike base data.frame).
  • Prints only the first 10 rows and truncates wide columns.
  • Gives informative errors on non-existent column access instead of silently returning NULL.

R
library(tibble)

df <- tibble(
  name   = c("Alice", "Bob", "Carol"),
  score  = c(88, 95, 72),
  passed = score >= 75          # you can reference earlier columns!
)
df
# A tibble: 3 × 3
#   name  score passed
#   <chr> <dbl> <lgl>
# 1 Alice    88 TRUE
# 2 Bob      95 TRUE
# 3 Carol    72 FALSE

Notice the column type tags (&lt;chr&gt;, &lt;dbl&gt;, &lt;lgl&gt;) printed beneath headers — a convenience absent from base data frames.

Python ↔ R Quick Reference

  • pd.DataFrame({...})tibble(...)
  • df[&#039;col&#039;]df$col or df[[&#039;col&#039;]]
  • df.shapedim(df) or nrow(df) / ncol(df)
  • df.dtypesstr(df) or glimpse(df) (from dplyr)
  • df.head(10)head(df, 10) or just df (tibble truncates automatically)
  • df.describe()summary(df)
  • Reading Data

readr (Tidyverse) reads rectangular text files faster than base R and returns tibbles:

R
library(readr)

students <- read_csv("students.csv")          # comma-separated
exams    <- read_tsv("exams.tsv")              # tab-separated
config   <- read_csv("data.csv",
             col_types = cols(
               id    = col_integer(),
               score = col_double(),
               date  = col_date(format = "%Y-%m-%d")
             ))

read<em>csv (with underscore) is readr's function; read.csv (with dot) is the slower base-R version. Prefer readr.

Missing Values

R uses NA (not NaN) as the universal missing-value sentinel. NA is typed: NA</em>real<em>, NA</em>integer<em>, NA</em>character_. Key behaviors:

R
is.na(NA)          # TRUE
sum(c(1, NA, 3))   # NA  (NA is contagious)
sum(c(1, NA, 3), na.rm = TRUE)  # 4

The na.rm = TRUE pattern recurs across virtually all R summary functions.

Code Examples

Exploring a Built-in Dataset with glimpse and summary

R
library(tibble)
library(dplyr)

# R ships with many practice datasets; as_tibble() wraps them nicely
data(mtcars)
cars <- as_tibble(mtcars, rownames = "model")

glimpse(cars)   # compact transposed view
summary(cars)   # descriptive statistics per column
Output
Rows: 32
Columns: 12
$ model <chr> "Mazda RX4", "Mazda RX4 Wag", ...
$ mpg   <dbl> 21.0, 21.0, 22.8, 21.4, ...
...

Factor Manipulation with forcats

R
library(forcats)
library(dplyr)

survey <- tibble(
  response = c("Agree", "Disagree", "Neutral", "Agree",
               "Strongly Agree", "Disagree", "Agree")
)

survey <- survey |>
  mutate(response = fct_relevel(response,
    "Strongly Disagree", "Disagree", "Neutral",
    "Agree", "Strongly Agree"))

levels(survey$response)
count(survey, response)
Output
[1] "Strongly Disagree" "Disagree" "Neutral" "Agree" "Strongly Agree"
# A tibble: 4 x 2
  response       n
  <fct>      <int>
1 Disagree       2
2 Neutral        1
3 Agree          3
4 Strongly Agree 1

50.2 Data Transformation with dplyr and the Pipe Beginner

The Pipe Operator

R 4.1 introduced the native pipe |&gt;. The Tidyverse also provides %&gt;% (from magrittr). Both pass the left-hand result as the first argument to the right-hand function:

R
# These three are equivalent:
result <- summarise(group_by(filter(df, score > 50), group), mean_score = mean(score))

result <- df |> filter(score > 50) |> group_by(group) |> summarise(mean_score = mean(score))

# Formatted readably:
result <- df |>
  filter(score > 50) |>
  group_by(group) |>
  summarise(mean_score = mean(score))

The pipe transforms nested function calls into a left-to-right pipeline — the same pattern as pandas method chaining but more general because it works across any functions.

The Core dplyr Verbs

dplyr provides a small, consistent vocabulary for data transformation. Each verb takes a data frame as its first argument and returns a data frame — which is why they compose so naturally through the pipe.

filter — row selection

Equivalent to df[df[&#039;col&#039;] &gt; x] in pandas or WHERE in SQL:

R
library(dplyr)
data(nycflights13::flights, package = "nycflights13")

flights |>
  filter(month == 1, day == 1, dep_delay > 60)

# Multiple conditions: comma = AND, | = OR
flights |>
  filter(carrier %in% c("UA", "AA") | dep_delay < 0)

select — column selection

R
flights |> select(year, month, day, dep_delay, arr_delay)

# Helpers: starts_with, ends_with, contains, matches, where
flights |> select(starts_with("dep"), ends_with("delay"))
flights |> select(where(is.numeric))

# Rename while selecting:
flights |> select(departure_delay = dep_delay, arrival_delay = arr_delay)

mutate — adding or modifying columns

Equivalent to df[&#039;new<em>col&#039;] = ... but inside a pipeline, and you can chain multiple mutations referencing earlier ones:

R
flights |>
  mutate(
    gain        = dep_delay - arr_delay,
    gain_per_hr = gain / (air_time / 60),
    on_time     = arr_delay <= 0
  ) |>
  select(flight, gain, gain_per_hr, on_time)

summarise + groupby

Equivalent to pandas groupby().agg():

R
flights |>
  group_by(carrier) |>
  summarise(
    n_flights    = n(),
    mean_delay   = mean(arr_delay, na.rm = TRUE),
    pct_on_time  = mean(arr_delay <= 0, na.rm = TRUE)
  ) |>
  arrange(desc(mean_delay))

n() counts rows within each group. mean() on a logical vector computes the proportion of TRUE values — an extremely common idiom.

arrange — row ordering

R
flights |> arrange(year, month, day, desc(dep_delay))

Joins

dplyr's join family mirrors SQL (INNER JOIN, LEFT JOIN, etc.):

  • inner<em>join(x, y, by = &quot;key&quot;) — keep rows with matches in both
  • left</em>join(x, y, by = &quot;key&quot;) — keep all rows in x
  • anti<em>join(x, y, by = &quot;key&quot;) — keep rows in x with NO match in y
  • semi</em>join(x, y, by = &quot;key&quot;) — keep rows in x WITH a match, but don't add y's columns

R
airlines <- nycflights13::airlines
flights |>
  left_join(airlines, by = "carrier") |>
  select(flight, name, dep_delay) |>
  rename(airline = name)

For multi-column keys: by = c(&quot;origin&quot; = &quot;faa&quot;) when names differ between tables.

Window Functions and Grouped Mutate

A powerful dplyr pattern: use group<em>by() before mutate() instead of summarise() to compute group-level statistics while retaining individual rows:

R
flights |>
  group_by(carrier) |>
  mutate(
    delay_rank   = min_rank(desc(dep_delay)),  # rank within carrier
    mean_carrier = mean(dep_delay, na.rm = TRUE),
    above_avg    = dep_delay > mean_carrier
  ) |>
  ungroup()   # always ungroup when done to prevent downstream surprises

Common window functions: row</em>number(), min<em>rank(), dense</em>rank(), lag(), lead(), cumsum(), cummean().

Useful Helper: across()

across() applies a function to multiple columns without copy-pasting:

R
flights |>
  group_by(carrier) |>
  summarise(across(ends_with("delay"), mean, na.rm = TRUE))

# Apply multiple functions at once:
flights |>
  summarise(across(where(is.numeric),
                   list(mean = mean, sd = sd),
                   na.rm = TRUE))

This replaces the older summarise<em>at / summarise</em>if / summarise_all functions that you may see in older code.

Common Pitfalls

  • Forgetting ungroup(): grouping persists through subsequent verbs in the same pipeline. Always ungroup() when the grouped computation is done.
  • NA in filter: filter(x == NA) silently drops everything. Use is.na(x) or filter(!is.na(x)).
  • = vs ==: Inside filter(), use == for equality. = is valid R there but assigns, not compares.
  • Overwriting with mutate: mutate(col = ...) overwrites col in place — use a new name if you need both.

Code Examples

Grouped Summary with Multiple Aggregations

R
library(dplyr)
library(nycflights13)

daily_summary <- flights |>
  group_by(year, month, day) |>
  summarise(
    n_flights        = n(),
    pct_cancelled    = mean(is.na(dep_time)),
    mean_dep_delay   = mean(dep_delay, na.rm = TRUE),
    median_dep_delay = median(dep_delay, na.rm = TRUE),
    .groups = "drop"   # drop grouping after summarise
  )

print(daily_summary, n = 5)
Output
# A tibble: 365 x 6
   year month   day n_flights pct_cancelled mean_dep_delay
  <int> <int> <int>     <int>         <dbl>          <dbl>
1  2013     1     1       842        0.0154           11.5
...

across() for Bulk Column Transformations

R
library(dplyr)

set.seed(42)
df <- tibble(
  x = c(1, 2, NA, 4),
  y = c(5, NA, 7, 8),
  z = c(9, 10, 11, NA),
  grp = c("a", "a", "b", "b")
)

# Replace NAs with column mean, grouped
df |>
  group_by(grp) |>
  mutate(across(where(is.numeric),
                ~ ifelse(is.na(.), mean(., na.rm = TRUE), .))) |>
  ungroup()
Output
# A tibble: 4 x 4
      x     y     z grp  
  <dbl> <dbl> <dbl> <chr>
1     1     5     9 a    
2     2     5    10 a    
3     3.5   7    11 b    
4     4     8    10 b

Anti-join for Set Difference

R
library(dplyr)

all_students  <- tibble(id = 1:6, name = c("A","B","C","D","E","F"))
submitted     <- tibble(id = c(1, 3, 4, 6))

# Who hasn't submitted?
anti_join(all_students, submitted, by = "id")
Output
# A tibble: 2 x 2
     id name 
  <int> <chr>
1     2 B    
2     5 E

50.3 Reshaping Data with tidyr and Handling Strings Intermediate

Tidy Data Principles

Hadley Wickham's tidy data framework defines a standard structure that most Tidyverse functions expect:

  1. Each variable forms a column.
  2. Each observation forms a row.
  3. Each type of observational unit forms a table.

Most real-world data is messy. The two most common violations are wide data (multiple observations per row, encoded as separate columns) and long data that conflates different variables into a single column.

R
# Wide — one row per student, one column per exam
wide <- tibble(
  student = c("Alice", "Bob"),
  exam1   = c(88, 75),
  exam2   = c(92, 81),
  exam3   = c(79, 95)
)

# Tidy — one row per (student, exam) pair
long <- tidyr::pivot_longer(wide,
  cols      = starts_with("exam"),
  names_to  = "exam",
  values_to = "score"
)

pivotlonger and pivotwider

pivot<em>longer collapses many columns into key-value pairs (equivalent to pd.melt):

R
library(tidyr)

# names_prefix strips a common prefix from the new key column
long <- wide |>
  pivot_longer(
    cols         = exam1:exam3,
    names_to     = "exam",
    names_prefix = "exam",          # strip "exam" -> "1", "2", "3"
    values_to    = "score"
  ) |>
  mutate(exam = as.integer(exam))

pivot</em>wider is the inverse (equivalent to pd.pivot or pd.unstack):

R
# Reconstruct wide format
long |>
  pivot_wider(
    names_from  = exam,
    values_from = score,
    names_prefix = "exam"
  )

Advanced pivot: multiple value columns

When each measurement has multiple associated values, pivot<em>longer can handle them simultaneously via names</em>to = c(&quot;var&quot;, &quot;time&quot;) with names<em>sep or names</em>pattern:

R
df <- tibble(
  id      = 1:3,
  bp_t1   = c(120, 130, 115),
  bp_t2   = c(118, 128, 120),
  hr_t1   = c(70, 80, 65),
  hr_t2   = c(72, 78, 68)
)

df |>
  pivot_longer(
    cols         = -id,
    names_to     = c("metric", "time"),
    names_sep    = "_",
    values_to    = "value"
  )

Handling Nested and Missing Data

complete() makes implicit missing values explicit — crucial before plotting time series:

R
sales <- tibble(
  month   = c(1, 2, 4),   # March is missing!
  revenue = c(100, 150, 130)
)

sales |> complete(month = 1:4, fill = list(revenue = 0))
# Now month 3 appears with revenue = 0

fill() carries values forward or backward (LOCF / NOCB):

R
tibble(month = 1:4, category = c("A", NA, NA, "B")) |>
  fill(category, .direction = "down")   # propagate "A" downward

separate() and unite() split and join columns:

R
tibble(date_id = c("2024-01", "2024-02")) |>
  separate(date_id, into = c("year", "month"), sep = "-", convert = TRUE)

String Manipulation with stringr

stringr provides a consistent interface wrapping PCRE regex. All functions start with str_:

R
library(stringr)

x <- c("  Hello, World!  ", "foo@bar.com", "2024-06-04")

str_trim(x)                       # strip whitespace
str_to_lower(x)                   # lowercase
str_detect(x, "@")                # logical: contains @?
str_extract(x, "\\d{4}-\\d{2}-\\d{2}")  # extract ISO date
str_replace_all(x, "[aeiou]", "*")      # replace vowels
str_split(x[1], ",")             # split on comma -> list

For dates, lubridate is the go-to package:

R
library(lubridate)

ymd("2024-06-04")        # parse ISO date -> Date object
dmy("04/06/2024")        # day-month-year
now()                    # current datetime with timezone
ymd("2024-06-04") + days(30)
floor_date(now(), "month")       # first of current month

Nesting Data Frames

One of R's most powerful patterns is the list-column: a tibble column that contains lists of objects (including other tibbles). nest() groups rows into sub-tibbles:

R
library(dplyr)
library(tidyr)

by_species <- iris |>
  as_tibble() |>
  group_by(Species) |>
  nest()   # 'data' column contains a tibble per species

by_species
# A tibble: 3 x 2
#   Species    data
#   <fct>      <list>
# 1 setosa     <tibble [50 x 4]>
# 2 versicolor <tibble [50 x 4]>
# 3 virginica  <tibble [50 x 4]>

This nest-map-unnest workflow is the Tidyverse approach to split-apply-combine and is explored further in the purrr section.

Code Examples

Reshaping WHO Tuberculosis Data

R
library(tidyr)
library(dplyr)

# who is a built-in messy dataset with TB cases
data(who, package = "tidyr")

who_tidy <- who |>
  pivot_longer(
    cols         = new_sp_m014:newrel_f65,
    names_to     = "key",
    values_to    = "cases",
    values_drop_na = TRUE
  ) |>
  mutate(
    key = str_replace(key, "newrel", "new_rel")
  ) |>
  separate(key, c("new", "type", "sexage"), sep = "_") |>
  separate(sexage, c("sex", "age"), sep = 1) |>
  select(-new, -iso2, -iso3)

head(who_tidy, 4)
Output
# A tibble: 4 x 6
  country      year type  sex   age   cases
  <chr>       <int> <chr> <chr> <chr> <int>
1 Afghanistan  1997 sp    m     014      0
2 Afghanistan  1997 sp    m     1524    10
...

String Parsing Pipeline for Log Data

R
library(stringr)
library(dplyr)
library(tibble)

logs <- tibble(
  entry = c(
    "2024-06-01 ERROR: connection refused (retry 3)",
    "2024-06-01 INFO: request completed in 120ms",
    "2024-06-02 WARN: memory usage at 87%"
  )
)

logs |>
  mutate(
    date    = str_extract(entry, "^\\d{4}-\\d{2}-\\d{2}"),
    level   = str_extract(entry, "(ERROR|WARN|INFO)"),
    message = str_remove(entry, "^.*?(?:ERROR|WARN|INFO): ")
  )
Output
# A tibble: 3 x 4
  entry                                     date       level message
  <chr>                                     <chr>      <chr> <chr>
1 2024-06-01 ERROR: connection refused...   2024-06-01 ERROR connection refused (retry 3)
...

50.4 Functional Iteration with purrr and List-Column Workflows Intermediate

Why purrr Instead of Loops?

Explicit for loops work in R but produce verbose, error-prone code and encourage thinking in steps rather than transformations. The purrr package provides a family of map<em>* functions — R's equivalent to Python's map(), itertools, and list comprehensions — that express "do this to each element" concisely and safely.

The key design principle: map always returns a list. Type-stable variants (map</em>dbl, map<em>chr, map</em>int, map<em>lgl) return atomic vectors and raise an error if any element has the wrong type — catching bugs that loops silently ignore.

R
library(purrr)

numbers <- list(a = 1:5, b = 10:15, c = 100:110)

map(numbers, mean)            # returns a list of means
map_dbl(numbers, mean)        # returns a named numeric vector
map_int(numbers, length)      # returns a named integer vector
map_chr(numbers, ~ paste0("n=", length(.x)))  # named character vector

The ~ (tilde) syntax creates an anonymous function (lambda). .x refers to the current element; .y is available in two-argument variants (map2).

The map Family

  • map(.x, .f) — apply .f to each element of .x
  • map2(.x, .y, .f) — apply .f to paired elements of .x and .y
  • pmap(.l, .f) — apply .f to parallel elements of a list of vectors
  • walk(.x, .f) — like map but called for side effects (printing, writing files); returns .x invisibly
  • imap(.x, .f) — like map but passes the name/index as the second argument

R
# map2: compute weighted means
values  <- list(c(1,2,3), c(4,5,6))
weights <- list(c(1,1,2), c(2,2,1))
map2_dbl(values, weights, weighted.mean)
# [1] 2.25 4.60

# pmap: operate on multiple parallel columns as a list
params <- tibble(
  x    = c(1, 2, 3),
  mean = c(0, 5, 10),
  sd   = c(1, 2, 3)
)
pmap_dbl(params, ~ dnorm(..1, ..2, ..3))  # density at x given mean, sd

Predicate and Extraction Helpers

R
numbers <- list(1, "two", 3, "four", 5)

keep(numbers, is.numeric)         # list(1, 3, 5)
discard(numbers, is.numeric)      # list("two", "four")
every(numbers, is.numeric)        # FALSE
some(numbers, is.numeric)         # TRUE
detect(numbers, ~ .x > 2)        # 3  (first match)

# Deep extraction with map + pluck:
response <- list(
  list(user = list(name = "Alice", age = 30)),
  list(user = list(name = "Bob",   age = 25))
)
map_chr(response, ~ pluck(.x, "user", "name"))  # c("Alice", "Bob")

The Nest-Map-Unnest Pattern

The most powerful Tidyverse workflow combines tidyr::nest(), purrr::map(), and tidyr::unnest() to fit models or run computations per group without explicit loops:

R
library(dplyr)
library(tidyr)
library(purrr)
library(broom)

# Fit a linear model per species in the iris dataset
models <- iris |>
  as_tibble() |>
  group_by(Species) |>
  nest() |>
  mutate(
    fit     = map(data, ~ lm(Sepal.Length ~ Petal.Length, data = .x)),
    tidied  = map(fit, tidy),     # broom::tidy extracts coefficients
    glanced = map(fit, glance)    # broom::glance extracts model stats
  )

# Extract all coefficients into one tidy data frame:
models |>
  select(Species, tidied) |>
  unnest(tidied)

This pattern scales to hundreds of groups without changing a single line of code. The same structure works for cross-validation splits, bootstrap samples, or any per-group computation.

reduce and accumulate

reduce() folds a list into a single value, applying a binary function left-to-right — equivalent to Python's functools.reduce:

R
reduce(list(1, 2, 3, 4), `+`)         # 10
reduce(list(df1, df2, df3), left_join, by = "id")  # join a list of frames!
accumulate(1:5, `+`)                   # running totals: 1 3 6 10 15

The reduce(frames, left</em>join, by = &quot;id&quot;) pattern elegantly joins an arbitrary number of tables without nesting left_join calls.

Error Handling in Iterations

When one element fails, the entire map fails. Use safely() or possibly() wrappers to handle errors gracefully:

R
safe_log <- safely(log, otherwise = NA_real_)
results  <- map(list(10, -1, 0, 100), safe_log)

# Extract results and errors separately:
map_dbl(results, ~ .x$result %||% NA)
map_chr(results, ~ if (!is.null(.x$error)) .x$error$message else "ok")

possibly() is simpler when you only want the fallback value:

R
possibly_log <- possibly(log, otherwise = NA_real_)
map_dbl(list(10, -1, 0, 100), possibly_log)
# [1]  2.302585        NA      -Inf  4.605170

Code Examples

Fitting and Extracting Models Per Group

R
library(dplyr)
library(tidyr)
library(purrr)
library(broom)

results <- mtcars |>
  as_tibble(rownames = "model") |>
  group_by(cyl) |>
  nest() |>
  mutate(
    fit     = map(data, ~ lm(mpg ~ wt + hp, data = .x)),
    r2      = map_dbl(fit, ~ glance(.x)$r.squared),
    n       = map_int(data, nrow)
  ) |>
  arrange(desc(r2))

results |> select(cyl, n, r2)
Output
# A tibble: 3 x 3
    cyl     n    r2
  <dbl> <int> <dbl>
1     4    11 0.893
2     6     7 0.879
3     8    14 0.736

Reading Multiple Files with map_dfr

R
library(purrr)
library(readr)
library(dplyr)

# Simulate multiple CSV files then read them all at once
tmp <- tempdir()
for (yr in 2021:2023) {
  write_csv(
    tibble(year = yr, value = rnorm(5, mean = yr * 10)),
    file.path(tmp, paste0("data_", yr, ".csv"))
  )
}

files <- list.files(tmp, pattern = "data_.*\.csv", full.names = TRUE)
all_data <- map_dfr(files, read_csv, .id = "source")
nrow(all_data)  # 15 rows (5 per file)
Output
15

Joining a Dynamic List of Data Frames

R
library(purrr)
library(dplyr)

# Build a list of lookup tables programmatically
tables <- list(
  demographics = tibble(id = 1:4, age  = c(25, 30, 35, 40)),
  scores       = tibble(id = 1:4, score = c(88, 92, 79, 85)),
  labels       = tibble(id = 1:4, group = c("A","B","A","B"))
)

reduce(tables, left_join, by = "id")
Output
# A tibble: 4 x 4
     id   age score group
  <int> <dbl> <dbl> <chr>
1     1    25    88 A
2     2    30    92 B
3     3    35    79 A
4     4    40    85 B

50.5 Visualization with ggplot2 Intermediate

The Grammar of Graphics

ggplot2 implements Leland Wilkinson's Grammar of Graphics — a principled decomposition of a chart into independent components that can be combined freely. This is fundamentally different from Matplotlib's procedural API or Seaborn's chart-type functions.

Every ggplot2 plot is built from these layers:

  • Data — a data frame (always tidy format)
  • Aesthetics (aes()) — mappings from data columns to visual properties (x, y, color, size, shape, fill, alpha, linetype)
  • Geometries (geom<em><em>) — the visual marks (points, lines, bars, boxes, ribbons...)
  • Scales (scale</em></em>) — control how data values map to visual properties
  • Facets (facet<em><em>) — small multiples splitting the plot by a variable
  • Statistics (stat</em></em>) — computed transformations (binning, smoothing, summarizing)
  • Coordinates (coord<em><em>) — the coordinate system (Cartesian, polar, flipped, map)
  • Themes (theme</em></em>) — non-data ink (fonts, backgrounds, grid lines)
  • Building a Plot Layer by Layer

R
library(ggplot2)

# Start with data + mapping, add geom
ggplot(mtcars, aes(x = wt, y = mpg)) +
  geom_point()

# Add a smooth trend line with confidence band
ggplot(mtcars, aes(x = wt, y = mpg)) +
  geom_point(aes(color = factor(cyl)), size = 2) +
  geom_smooth(method = "lm", se = TRUE, color = "steelblue") +
  scale_color_brewer(palette = "Set1", name = "Cylinders") +
  labs(
    title    = "Fuel Efficiency vs. Weight",
    subtitle = "Linear fit by cylinder count",
    x        = "Weight (1000 lbs)",
    y        = "Miles per Gallon",
    caption  = "Source: Motor Trend, 1974"
  ) +
  theme_minimal(base_size = 13)

Notice: + rather than |&gt; — ggplot2 uses + to add layers because it predates the pipe and its API is additive, not sequential.

Common Geoms

  • geom<em>point() — scatter plot
  • geom</em>line() — line chart (order matters: sorts by x)
  • geom<em>bar(stat = &quot;count&quot;) / geom</em>col() — bar chart (col takes pre-computed heights)
  • geom<em>histogram(bins = 30) — histogram
  • geom</em>boxplot() — box-and-whisker
  • geom<em>violin() — violin plot (richer than boxplot for multimodal distributions)
  • geom</em>tile() — heatmap
  • geom<em>ribbon(aes(ymin, ymax)) — confidence bands
  • geom</em>text() / geom<em>label() — text annotations
  • Faceting for Small Multiples

Faceting is one of ggplot2's strongest features — it splits a plot by a categorical variable, keeping scales consistent:

R
ggplot(nycflights13::flights |> dplyr::sample_n(5000),
       aes(x = dep_delay, y = arr_delay)) +
  geom_point(alpha = 0.3, size = 0.8) +
  geom_abline(slope = 1, intercept = 0, color = "red", linetype = "dashed") +
  facet_wrap(~ carrier, ncol = 4) +
  coord_cartesian(xlim = c(-30, 120), ylim = c(-60, 120)) +
  theme_light()

facet</em>grid(rows ~ cols) creates a grid facet; facet<em>wrap(~ var) wraps into a specified number of columns.

Statistical Transformations

Many geoms implicitly apply a stat. You can override or use stats directly:

R
# stat_summary computes mean + CI and draws error bars
ggplot(iris, aes(x = Species, y = Sepal.Length)) +
  stat_summary(fun = mean, geom = "point", size = 4) +
  stat_summary(fun.data = mean_cl_boot, geom = "errorbar", width = 0.2)

Scales and Color

Scales control the mapping from data to aesthetics and are where most customization happens:

R
ggplot(economics, aes(x = date, y = unemploy / pop)) +
  geom_line(color = "#2c7bb6") +
  scale_x_date(date_labels = "%Y", date_breaks = "5 years") +
  scale_y_continuous(labels = scales::percent_format(accuracy = 0.1)) +
  labs(y = "Unemployment Rate", x = NULL)

Color scale guide:

  • scale</em>color<em>brewer / scale</em>fill<em>brewer — ColorBrewer palettes (good for categorical)
  • scale</em>color<em>viridis</em>c / <em>d — perceptually uniform, colorblind-safe
  • scale</em>color<em>gradient(low, high) — two-color continuous gradient
  • scale</em>color_gradient2(low, mid, high, midpoint) — diverging
  • Themes

R
# Built-in themes
theme_gray()     # default
theme_bw()       # white background, grey grid
theme_minimal()  # minimal chrome
theme_classic()  # no grid, like base R

# Fine-grained control via theme()
p + theme(
  axis.text.x   = element_text(angle = 45, hjust = 1),
  legend.position = "bottom",
  plot.title    = element_text(face = "bold", size = 16)
)

Saving Plots

R
ggsave("my_plot.png", plot = last_plot(), width = 8, height = 5, dpi = 300)
ggsave("my_plot.pdf", width = 8, height = 5)  # vector for publications

ggsave automatically infers format from the file extension and respects dpi for raster output.

Code Examples

Layered Plot: Distributions by Group

R
library(ggplot2)
library(dplyr)

set.seed(99)
df <- tibble(
  group = rep(c("Control", "Treatment"), each = 200),
  value = c(rnorm(200, 50, 10), rnorm(200, 55, 12))
)

ggplot(df, aes(x = value, fill = group)) +
  geom_histogram(aes(y = after_stat(density)), bins = 25,
                 alpha = 0.5, position = "identity") +
  geom_density(aes(color = group), linewidth = 1, fill = NA) +
  scale_fill_manual(values  = c("#E69F00", "#56B4E9")) +
  scale_color_manual(values = c("#E69F00", "#56B4E9")) +
  labs(title = "Distribution of Outcome by Group",
       x = "Outcome", y = "Density",
       fill = "Group", color = "Group") +
  theme_minimal()
Output
# Produces a layered histogram + density plot with two overlapping distributions

Heatmap of Correlation Matrix

R
library(ggplot2)
library(dplyr)
library(tidyr)

cor_data <- cor(mtcars) |>
  as.data.frame() |>
  tibble::rownames_to_column("var1") |>
  pivot_longer(-var1, names_to = "var2", values_to = "r")

ggplot(cor_data, aes(x = var1, y = var2, fill = r)) +
  geom_tile(color = "white") +
  scale_fill_gradient2(low = "#d73027", mid = "white", high = "#1a9850",
                       midpoint = 0, limits = c(-1, 1), name = "r") +
  theme_minimal() +
  theme(axis.text.x = element_text(angle = 45, hjust = 1)) +
  labs(title = "Correlation Matrix: mtcars", x = NULL, y = NULL)
Output
# Produces a symmetric correlation heatmap with diverging color scale

Line Chart with Annotations

R
library(ggplot2)
library(dplyr)
library(lubridate)

economics_long <- economics |>
  mutate(
    recession_start = date >= ymd("2007-12-01") & date <= ymd("2009-06-01")
  )

ggplot(economics_long, aes(x = date, y = unemploy / 1000)) +
  geom_rect(data = ~filter(.x, recession_start),
            aes(xmin = date, xmax = date + months(1),
                ymin = -Inf, ymax = Inf),
            fill = "salmon", alpha = 0.02, inherit.aes = FALSE) +
  geom_line(color = "#2166ac", linewidth = 0.8) +
  annotate("text", x = ymd("2008-09-01"), y = 14,
           label = "2008 Recession", color = "salmon", fontface = "bold") +
  scale_x_date(date_breaks = "10 years", date_labels = "%Y") +
  labs(title = "US Unemployment 1967–2015",
       x = NULL, y = "Unemployed (millions)")
Output
# Produces a time series line chart with recession period shading

50.6 Statistical Modeling and Reproducible Research with R Markdown Intermediate

Statistical Modeling in R

R's formula syntax y ~ x1 + x2 is a first-class language construct — the same expression that specifies a model structure is used consistently across lm, glm, lmer, survival::coxph, and hundreds of other modeling functions.

Linear Models

R
data(mtcars)
fit <- lm(mpg ~ wt + hp + factor(cyl), data = mtcars)
summary(fit)

summary(fit) prints coefficients, standard errors, t-statistics, p-values, and $R^2$. The formula factor(cyl) creates dummy variables on the fly.

Formula operators:

  • y ~ x1 + x2 — main effects
  • y ~ x1 * x2 — main effects plus interaction (expands to x1 + x2 + x1:x2)
  • y ~ x1:x2 — interaction only
  • y ~ . — all columns in data except y
  • y ~ . - x1 — all except y and x1
  • y ~ poly(x, 2) — orthogonal polynomial
  • y ~ I(x^2)I() escapes arithmetic inside formulas
  • Generalized Linear Models

R
# Logistic regression
birth_fit <- glm(low ~ age + lwt + smoke,
                 data   = MASS::birthwt,
                 family = binomial(link = "logit"))

# Poisson regression for counts
count_fit <- glm(n ~ offset(log(exposure)) + group,
                 family = poisson())

Tidy Model Output with broom

Base R's summary() output is S3 objects that are hard to work with programmatically. broom converts model objects into tidy tibbles:

R
library(broom)

tidy(fit)     # coefficients table as tibble
glance(fit)   # one-row model summary (R2, AIC, BIC, F-stat)
augment(fit)  # original data + fitted values, residuals, leverage

augment() is particularly powerful for diagnostics:

R
augment(fit) |>
  ggplot(aes(x = .fitted, y = .resid)) +
  geom_point(alpha = 0.5) +
  geom_hline(yintercept = 0, linetype = "dashed", color = "red") +
  labs(title = "Residuals vs Fitted", x = "Fitted Values", y = "Residuals")

Comparing Models

R
fit_base  <- lm(mpg ~ wt,          data = mtcars)
fit_full  <- lm(mpg ~ wt + hp + cyl, data = mtcars)

# Likelihood ratio test (nested models)
anova(fit_base, fit_full)

# AIC comparison
map_dfr(list(base = fit_base, full = fit_full),
        ~ glance(.x) |> select(r.squared, AIC, BIC),
        .id = "model")

R Markdown for Reproducible Research

R Markdown (.Rmd) documents weave R code, output, and narrative prose into a single source file that renders to HTML, PDF, Word, slides, or dashboards. This is the R equivalent of a Jupyter notebook, but with tighter integration of the document structure and more options for polished output.

Anatomy of an R Markdown Document

An .Rmd file has three parts:

1. YAML header (between --- delimiters):

YAML
---
title: "Analysis of Flight Delays"
author: "Your Name"
date: "`r Sys.Date()`"
output:
  html_document:
    toc: true
    toc_float: true
    theme: flatly
    code_folding: hide
---

2. Markdown prose — standard Markdown text with R expressions inline: ` r nrow(df) renders the value in the output.</p> <p><strong>3. Code chunks</strong>:</p> <p>

{r setup, include=FALSE} knitr::opts_chunk$set(echo = TRUE, warning = FALSE, message = FALSE, fig.width = 8, fig.height = 5, dpi = 150) library(tidyverse)
</p> <p>Chunk options control output: echo = FALSE hides code, eval = FALSE shows code without running it, cache = TRUE caches slow computations, fig.cap = "My figure"` adds a caption.

Inline R Expressions

Inline code makes prose self-updating — when data changes, the numbers in the text re-render automatically:

The dataset contains `r nrow(flights)` flights operated by
`r n_distinct(flights$carrier)` carriers. The mean departure
delay was `r round(mean(flights$dep_delay, na.rm = TRUE), 1)` minutes.

Rendering

R
rmarkdown::render("analysis.Rmd")               # HTML by default
rmarkdown::render("analysis.Rmd",
                  output_format = "pdf_document")  # requires LaTeX
rmarkdown::render("analysis.Rmd",
                  output_format = "word_document")

Or use the Knit button in RStudio. From the command line:

BASH
Rscript -e "rmarkdown::render('analysis.Rmd')"

Parameterized Reports

Declare parameters in the YAML header and pass different values to generate multiple reports from one template:

YAML
---
params:
  carrier: "UA"
  min_year: 2013
---

R
# In the body, reference params$carrier and params$min_year
filtered <- flights |>
  filter(carrier == params$carrier,
         year >= params$min_year)

R
# Render a report per carrier:
purrr::walk(
  c("UA", "AA", "DL"),
  ~ rmarkdown::render("report.Rmd",
                      params = list(carrier = .x),
                      output_file = paste0("report_", .x, ".html"))
)

Parameterized reports are a production-ready pattern for automated reporting pipelines.

Code Examples

Full Modeling Workflow with broom

R
library(dplyr)
library(broom)
library(ggplot2)

# Fit and compare two models
fit1 <- lm(Sepal.Length ~ Petal.Length,              data = iris)
fit2 <- lm(Sepal.Length ~ Petal.Length + Species,    data = iris)
fit3 <- lm(Sepal.Length ~ Petal.Length * Species,    data = iris)

# Compare with AIC
purrr::map_dfr(
  list(additive  = fit1,
       species   = fit2,
       interaction = fit3),
  glance,
  .id = "model"
) |>
  select(model, r.squared, adj.r.squared, AIC, BIC) |>
  arrange(AIC)

# Diagnostic plot for best model
augment(fit3) |>
  ggplot(aes(x = .fitted, y = .resid, color = Species)) +
  geom_point(alpha = 0.6) +
  geom_hline(yintercept = 0, linetype = "dashed") +
  facet_wrap(~Species) +
  theme_minimal() +
  labs(title = "Residuals by Species", x = "Fitted", y = "Residual")
Output
# A tibble: 3 x 5
  model       r.squared adj.r.squared   AIC   BIC
  <chr>           <dbl>         <dbl> <dbl> <dbl>
1 interaction     0.867         0.861  111.  131.
2 species         0.837         0.833  127.  144.
3 additive        0.760         0.758  188.  197.

Minimal R Markdown Document

R
# Save this content as 'analysis.Rmd' and render with rmarkdown::render()

content <- '
---
title: "Flights Analysis"
author: "Data Scientist"
date: "`r Sys.Date()`"
output: html_document
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE, message = FALSE)
library(tidyverse)
library(nycflights13)
```

## Overview

This report analyses the `r nrow(flights)` flights in the nycflights13 dataset.

## Delay Distribution

```{r delay-plot, fig.cap="Distribution of departure delays"}
flights |>
  filter(dep_delay > 0) |>
  ggplot(aes(x = dep_delay)) +
  geom_histogram(bins = 50, fill = "steelblue", color = "white") +
  scale_x_log10() +
  labs(x = "Departure Delay (minutes, log scale)", y = "Count")
```
'

writeLines(content, "/tmp/analysis.Rmd")
rmarkdown::render("/tmp/analysis.Rmd", quiet = TRUE)
message("Report written to /tmp/analysis.html")
Output
Report written to /tmp/analysis.html

Logistic Regression with tidy Output

R
library(dplyr)
library(broom)

# birthwt: baby birthweight data from MASS
data(birthwt, package = "MASS")

fit <- glm(low ~ age + lwt + smoke + ht,
           data   = birthwt,
           family = binomial())

# Tidy coefficients as odds ratios with CI
tidy(fit, exponentiate = TRUE, conf.int = TRUE) |>
  filter(term != "(Intercept)") |>
  mutate(across(where(is.double), ~ round(.x, 3))) |>
  arrange(p.value)
Output
# A tibble: 4 x 7
  term  estimate std.error statistic p.value conf.low conf.high
  <chr>    <dbl>     <dbl>     <dbl>   <dbl>    <dbl>     <dbl>
1 ht       6.25      0.703      2.57  0.0101    1.605    26.9
2 smoke    3.05      0.436      2.56  0.0105    1.30      7.36
3 lwt      0.986     0.007     -1.96  0.0502    0.972     1.00
4 age      0.965     0.034     -1.03  0.302     0.904     1.03