Intermediate Advanced 20 min read

Chapter 37: Resampling & the Bootstrap

Classical statistical inference relies on parametric assumptions: data is normally distributed, sample sizes are large enough for the central limit theorem to kick in, or closed-form variance formulas exist for a statistic of interest. In practice, these assumptions are routinely violated. You may have a small, skewed sample and need a confidence interval for the median, or you want to test whether two distributions differ without assuming any particular shape. Resampling methods — the bootstrap, the jackknife, and permutation tests — provide a principled, assumption-light alternative that uses the data itself as a stand-in for the unknown population.

The bootstrap, introduced by Bradley Efron in 1979, is arguably the most important computational advance in statistics of the twentieth century. The core idea is almost absurdly simple: if you cannot draw repeated samples from the population, simulate repeated sampling by drawing with replacement from your observed data. The empirical distribution of a statistic across thousands of such "bootstrap replicates" approximates the true sampling distribution — no formulas required. This chapter develops the intuition rigorously, implements the standard methods from scratch, and shows how modern libraries such as scipy.stats and arch automate the process.

Permutation tests complement the bootstrap on the hypothesis-testing side, and cross-validation connects the resampling philosophy directly to machine-learning model evaluation. Together, these tools form a coherent toolkit that practitioners reach for whenever parametric shortcuts are unavailable or untrustworthy. By the end of this chapter you will be able to construct confidence intervals for virtually any statistic, test almost any hypothesis, and critically assess the reliability of a model estimate — all grounded in the same unifying principle: let the data speak for itself.

Runnable companion notebook for this chapter. Download the Jupyter notebook (.ipynb)

Learning Objectives

  • Explain the bootstrap principle and implement nonparametric bootstrap confidence intervals (percentile, basic, and BCa) from scratch in Python.
  • Articulate when and why the BCa correction matters, and compute the acceleration and bias-correction constants correctly.
  • Implement the jackknife estimator for bias and variance, and explain its relationship to the bootstrap.
  • Construct permutation tests for two-sample and paired comparisons, including p-value estimation and multiple-comparison considerations.
  • Apply bootstrap and permutation resampling using production-ready libraries (scipy.stats, arch, sklearn).
  • Use cross-validation as a resampling-based model evaluation strategy, diagnosing the bias-variance trade-off in the number of folds.
  • Identify common failure modes of resampling methods — dependent data, small samples, heavy tails — and apply appropriate corrections such as the block bootstrap.

37.1 The Bootstrap Principle Intermediate

Why Resampling Works

Imagine you want to know the sampling distribution of some statistic $\hat{\theta}$ — say, the median of a skewed distribution. The median has no simple closed-form standard error when the underlying distribution is unknown. The parametric route requires you to guess a distribution family and derive the variance analytically; the bootstrap avoids this entirely.

The Plug-In Principle

Every bootstrap argument starts with the plug-in principle: replace the unknown population distribution $F$ with the empirical distribution $\hat{F}_n$, which places mass $1/n$ on each observed data point $x_1, \ldots, x_n$.

A bootstrap sample $X^* = (X_1^*, \ldots, X_n^*)$ is drawn i.i.d. from $\hat{F}_n$ — equivalently, sampled with replacement from the original data. The bootstrap estimate of the statistic is $\hat{\theta}^* = \theta(X^*)$. Repeating this $B$ times produces a collection $\{\hat{\theta}^*_b\}_{b=1}^B$ whose distribution approximates the true sampling distribution of $\hat{\theta}$.

The key theoretical result (Efron, 1979; Bickel & Freedman, 1981) is that under mild regularity conditions,

$$ \sup_x \left| P\left(\frac{\hat{\theta} - \theta}{\text{se}(\hat{\theta})} \le x\right) - P\left(\frac{\hat{\theta}^* - \hat{\theta}}{\text{se}^*(\hat{\theta}^*)} \le x \right) \right| \xrightarrow{p} 0 $$

In words: the bootstrap pivot distribution converges to the true pivot distribution as $n \to \infty$. This is a remarkable result — the data is being used both to estimate $\theta$ and to approximate the variation around that estimate.

A Minimal Implementation

The following code implements the bootstrap from scratch for any scalar statistic.

PYTHON
import numpy as np

def bootstrap(
    data: np.ndarray,
    stat_fn,
    B: int = 10_000,
    rng: np.random.Generator | None = None,
) -> np.ndarray:
    """Return B bootstrap replicates of stat_fn(data)."""
    if rng is None:
        rng = np.random.default_rng(42)
    n = len(data)
    replicates = np.empty(B)
    for b in range(B):
        sample = rng.choice(data, size=n, replace=True)
        replicates[b] = stat_fn(sample)
    return replicates

# Example: 95% CI for the median of a log-normal sample
rng = np.random.default_rng(0)
data = rng.lognormal(mean=1.5, sigma=0.8, size=80)

reps = bootstrap(data, np.median, B=10_000, rng=rng)
print(f"Sample median : {np.median(data):.3f}")
print(f"Bootstrap SE  : {reps.std(ddof=1):.3f}")
print(f"95% percentile CI: ({np.percentile(reps, 2.5):.3f}, {np.percentile(reps, 97.5):.3f})")

How Many Bootstrap Replicates?

For a 95% confidence interval, $B = 1{,}000$ is often sufficient for the interval endpoints to stabilize. For the BCa method (see the next section) or for tail quantiles beyond 99%, use $B \ge 10{,}000$. Doubling $B$ halves the Monte Carlo variance in the endpoint estimates, so there is a clear diminishing-returns curve. A practical default is $B = 9{,}999$ (odd, so the median is a data point).

The Bootstrap Standard Error

The bootstrap standard error is simply the standard deviation of the replicates:

$$ \widehat{\text{se}}_{\text{boot}} = \sqrt{\frac{1}{B-1} \sum_{b=1}^B (\hat{\theta}^*_b - \bar{\theta}^*)^2} $$

This is a consistent estimator of the true standard error under the same regularity conditions that guarantee bootstrap validity. Unlike the delta method, it does not require you to compute derivatives of the statistic.

Pitfall: When the Bootstrap Fails

The bootstrap is not universally valid. It fails for:

  • The sample maximum (and other extreme-order statistics) because <!--MATHBLOCK19--> has bounded support while the true extreme-value distribution may be heavy-tailed.
  • Very small <!--MATHBLOCK20--> (rule of thumb: <!--MATHBLOCK21-->) because <!--MATHBLOCK22--> is a poor approximation of <!--MATHBLOCK23-->.
  • Dependent data (time series, spatial data) where i.i.d. resampling destroys autocorrelation structure — use the block bootstrap instead.
  • Nondifferentiable statistics such as quantiles of discrete distributions: the bootstrap still works in these cases but the coverage can be off by <!--MATHBLOCK24--> rather than the usual <!--MATHBLOCK25-->.

Code Examples

Vectorized Bootstrap (much faster)

Replace the Python loop with a fully vectorized NumPy implementation for large B.

PYTHON
import numpy as np

def bootstrap_vectorized(
    data: np.ndarray,
    stat_fn,
    B: int = 10_000,
    seed: int = 42,
) -> np.ndarray:
    rng = np.random.default_rng(seed)
    n = len(data)
    # Draw all indices at once: shape (B, n)
    idx = rng.integers(0, n, size=(B, n))
    boot_samples = data[idx]          # shape (B, n)
    return np.apply_along_axis(stat_fn, 1, boot_samples)

rng = np.random.default_rng(7)
data = rng.exponential(scale=2.0, size=100)

reps = bootstrap_vectorized(data, np.mean, B=20_000)
print(f"True mean (Exp(2)): 2.000")
print(f"Sample mean      : {data.mean():.4f}")
print(f"Bootstrap SE     : {reps.std():.4f}")
print(f"Analytic SE      : {data.std(ddof=1)/len(data)**0.5:.4f}")
Output
True mean (Exp(2)): 2.000
Sample mean      : 1.9821
Bootstrap SE     : 0.1987
Analytic SE      : 0.1985

Visualising the Bootstrap Distribution

Plot the bootstrap distribution alongside the normal approximation to see when they diverge.

PYTHON
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats

rng = np.random.default_rng(3)
# Heavy-tailed data: Pareto
data = (rng.pareto(a=2.5, size=60) + 1)  # mean = 2.5/(2.5-1)

B = 15_000
idx = rng.integers(0, len(data), size=(B, len(data)))
reps = data[idx].mean(axis=1)

fig, ax = plt.subplots(figsize=(8, 4))
ax.hist(reps, bins=80, density=True, alpha=0.6, label='Bootstrap dist.')
mu, se = reps.mean(), reps.std()
xs = np.linspace(mu - 4*se, mu + 4*se, 300)
ax.plot(xs, stats.norm.pdf(xs, mu, se), 'r-', lw=2, label='Normal approx')
ax.axvline(data.mean(), color='k', lw=2, label='Sample mean')
ax.set_xlabel('Mean')
ax.set_title('Bootstrap vs Normal Approximation (Pareto data, n=60)')
ax.legend()
plt.tight_layout()
plt.savefig('bootstrap_dist.png', dpi=120)
print('Skewness of bootstrap dist:', stats.skew(reps).round(3))

37.2 Confidence Interval Methods: Percentile, Basic, and BCa Intermediate

Three Flavours of Bootstrap CI

The bootstrap gives you a distribution of $\hat{\theta}^*$ values, but translating that distribution into a confidence interval is not trivial. There are three main approaches, each with different accuracy guarantees.

1. The Percentile Interval

The simplest method: use the $\alpha/2$ and $1-\alpha/2$ quantiles of the bootstrap distribution directly.

$$ [\hat{\theta}^*_{(\alpha/2)},\; \hat{\theta}^*_{(1-\alpha/2)}] $$

This is intuitive and works well when the bootstrap distribution is symmetric and unbiased. Its coverage error is $O(n^{-1/2})$, meaning that even with large $n$ the coverage can be noticeably off for skewed statistics.

2. The Basic (Reflected) Interval

Also called the reverse percentile or Efron's basic interval. It exploits the pivot $\hat{\theta}^* - \hat{\theta}$ rather than $\hat{\theta}^*$ itself:

$$ [2\hat{\theta} - \hat{\theta}^*_{(1-\alpha/2)},\; 2\hat{\theta} - \hat{\theta}^*_{(\alpha/2)}] $$

This corrects for bias in the median of the bootstrap distribution but still has $O(n^{-1/2})$ coverage error.

3. BCa — Bias-Corrected and Accelerated

The BCa interval (Efron, 1987) achieves $O(n^{-1})$ coverage error — a full order of magnitude better — by applying two corrections to the percentile endpoints.

Bias-correction constant $\hat{z}_0$: measures how far the median of the bootstrap distribution is from $\hat{\theta}$.

$$ \hat{z}_0 = \Phi^{-1}\!\left(\frac{\#\{\hat{\theta}^*_b < \hat{\theta}\}}{B}\right) $$

If $\hat{z}_0 = 0$, the bootstrap distribution is median-unbiased and the BCa reduces to the percentile interval.

Acceleration constant $\hat{a}$: measures the rate of change of the standard error of $\hat{\theta}$ with respect to the true parameter. It is estimated via the jackknife (see the next section):

$$ \hat{a} = \frac{\sum_{i=1}^n (\bar{\theta}_{(\cdot)} - \hat{\theta}_{(i)})^3}{6\left[\sum_{i=1}^n (\bar{\theta}_{(\cdot)} - \hat{\theta}_{(i)})^2\right]^{3/2}} $$

where $\hat{\theta}_{(i)}$ is the statistic computed with observation $i$ removed (the leave-one-out jackknife estimate).

The adjusted quantile levels are:

$$ \alpha_1 = \Phi\!\left(\hat{z}_0 + \frac{\hat{z}_0 + z_{\alpha/2}}{1 - \hat{a}(\hat{z}_0 + z_{\alpha/2})}\right) \qquad \alpha_2 = \Phi\!\left(\hat{z}_0 + \frac{\hat{z}_0 + z_{1-\alpha/2}}{1 - \hat{a}(\hat{z}_0 + z_{1-\alpha/2})}\right) $$

The BCa interval is then $[\hat{\theta}^*_{(\alpha_1)}, \hat{\theta}^*_{(\alpha_2)}]$.

Full Implementation

PYTHON
import numpy as np
from scipy import stats

def bca_ci(
    data: np.ndarray,
    stat_fn,
    B: int = 10_000,
    alpha: float = 0.05,
    seed: int = 0,
) -> tuple[float, float]:
    """BCa confidence interval for any scalar statistic."""
    rng = np.random.default_rng(seed)
    n = len(data)
    theta_hat = stat_fn(data)

    # Bootstrap replicates
    idx = rng.integers(0, n, size=(B, n))
    replicates = np.array([stat_fn(data[idx[b]]) for b in range(B)])

    # Bias-correction z0
    prop_below = np.mean(replicates < theta_hat)
    # Avoid edge cases
    prop_below = np.clip(prop_below, 1e-6, 1 - 1e-6)
    z0 = stats.norm.ppf(prop_below)

    # Acceleration via jackknife
    jk = np.array([stat_fn(np.delete(data, i)) for i in range(n)])
    jk_mean = jk.mean()
    num   = np.sum((jk_mean - jk) ** 3)
    denom = np.sum((jk_mean - jk) ** 2)
    a_hat = num / (6 * denom ** 1.5)

    # Adjusted quantiles
    za2  = stats.norm.ppf(alpha / 2)
    z1a2 = stats.norm.ppf(1 - alpha / 2)

    def adjusted_alpha(z):
        inner = z0 + (z0 + z) / (1 - a_hat * (z0 + z))
        return stats.norm.cdf(inner)

    a1 = adjusted_alpha(za2)
    a2 = adjusted_alpha(z1a2)
    return float(np.percentile(replicates, 100 * a1)), \
           float(np.percentile(replicates, 100 * a2))

# Compare methods on skewed data
rng = np.random.default_rng(1)
data = rng.lognormal(1.5, 1.0, size=50)
true_median = np.exp(1.5)  # median of lognormal(mu, sigma) is exp(mu)

reps = np.array([np.median(rng.choice(data, size=50, replace=True))
                 for _ in range(10_000)])

pct_ci    = (np.percentile(reps, 2.5), np.percentile(reps, 97.5))
basic_ci  = (2*np.median(data)-np.percentile(reps,97.5),
             2*np.median(data)-np.percentile(reps, 2.5))
bca_lo, bca_hi = bca_ci(data, np.median, B=10_000, seed=1)

print(f"True median: {true_median:.3f}")
print(f"Percentile CI : ({pct_ci[0]:.3f}, {pct_ci[1]:.3f})")
print(f"Basic CI      : ({basic_ci[0]:.3f}, {basic_ci[1]:.3f})")
print(f"BCa CI        : ({bca_lo:.3f}, {bca_hi:.3f})")

Which Method to Use?

  • Percentile: fast and transparent; acceptable when the sampling distribution is nearly symmetric.
  • Basic: slightly better coverage than percentile for biased estimators; rarely preferred over BCa.
  • BCa: the recommended default. It is second-order accurate and transformation-respecting (a CI for <!--MATHBLOCK22--> transforms correctly to a CI for <!--MATHBLOCK23-->). The extra cost is one full jackknife pass (<!--MATHBLOCK24--> evaluations of stat_fn), which is negligible for most statistics.
  • Studentized bootstrap: achieves the same second-order accuracy as BCa without the jackknife, but requires a variance estimator for the statistic and is more sensitive to outliers.
  • Coverage Simulation

A useful exercise is to simulate coverage. For a log-normal sample of size 50, the nominal 95% BCa interval achieves roughly 94.5% empirical coverage on the median, while the percentile interval falls to about 92% — a meaningful difference in practice.

Code Examples

Coverage Simulation: Percentile vs BCa

Monte Carlo study comparing empirical coverage of the three CI methods on lognormal data.

PYTHON
import numpy as np
from scipy import stats

def pct_ci(data, stat_fn, B=2000, alpha=0.05, seed=None):
    rng = np.random.default_rng(seed)
    n = len(data)
    reps = np.array([stat_fn(rng.choice(data, n, replace=True)) for _ in range(B)])
    return np.percentile(reps, [100*alpha/2, 100*(1-alpha/2)])

def bca_ci_fast(data, stat_fn, B=2000, alpha=0.05, seed=None):
    """Streamlined BCa for use in simulations."""
    rng = np.random.default_rng(seed)
    n = len(data)
    th = stat_fn(data)
    reps = np.array([stat_fn(rng.choice(data, n, replace=True)) for _ in range(B)])
    p = np.clip(np.mean(reps < th), 1e-6, 1-1e-6)
    z0 = stats.norm.ppf(p)
    jk = np.array([stat_fn(np.delete(data, i)) for i in range(n)])
    jk_m = jk.mean()
    a = np.sum((jk_m-jk)**3) / (6*np.sum((jk_m-jk)**2)**1.5)
    def qa(z):
        return stats.norm.cdf(z0 + (z0+z)/(1 - a*(z0+z)))
    za, zb = stats.norm.ppf(alpha/2), stats.norm.ppf(1-alpha/2)
    return np.percentile(reps, [100*qa(za), 100*qa(zb)])

N_SIM = 500
n = 50
mu_ln = 1.5
true_med = np.exp(mu_ln)
covered_pct = covered_bca = 0
rng_outer = np.random.default_rng(99)
for sim in range(N_SIM):
    d = rng_outer.lognormal(mu_ln, 1.0, n)
    lo_p, hi_p = pct_ci(d, np.median, B=999, seed=sim)
    lo_b, hi_b = bca_ci_fast(d, np.median, B=999, seed=sim)
    if lo_p <= true_med <= hi_p: covered_pct += 1
    if lo_b <= true_med <= hi_b: covered_bca += 1

print(f"Percentile 95% CI coverage: {covered_pct/N_SIM:.3f}")
print(f"BCa        95% CI coverage: {covered_bca/N_SIM:.3f}")
Output
Percentile 95% CI coverage: 0.920
BCa        95% CI coverage: 0.944

Bootstrap CI using the arch library

Production-quality bootstrap CIs with the arch package, which uses optimized Cython loops.

PYTHON
# pip install arch
import numpy as np
from arch.bootstrap import IIDBootstrap

rng = np.random.default_rng(5)
data = rng.lognormal(1.5, 1.0, size=80)

bs = IIDBootstrap(data, seed=5)

# stat_fn receives a positional arg (the bootstrap sample)
ci_pct = bs.conf_int(np.median, reps=9999, method='percentile')
ci_bca = bs.conf_int(np.median, reps=9999, method='bca')

print(f"arch percentile CI: ({ci_pct[0,0]:.3f}, {ci_pct[1,0]:.3f})")
print(f"arch BCa CI       : ({ci_bca[0,0]:.3f}, {ci_bca[1,0]:.3f})")
print(f"True median       : {np.exp(1.5):.3f}")

37.3 The Jackknife: Bias and Variance Estimation Intermediate

Leave-One-Out Resampling

The jackknife (Quenouille, 1949; Tukey, 1958) predates the bootstrap by three decades and remains the standard tool for bias correction and for computing the acceleration term in BCa. Understanding it sharpens intuition about what the bootstrap is doing and when bias matters.

Jackknife Notation

Let $\hat{\theta}$ be a statistic computed on the full sample $x_1, \ldots, x_n$. Define the leave-one-out statistic:

$$ \hat{\theta}_{(i)} = \theta(x_1, \ldots, x_{i-1}, x_{i+1}, \ldots, x_n) $$

and the jackknife pseudo-values:

$$ \tilde{\theta}_i = n\hat{\theta} - (n-1)\hat{\theta}_{(i)} $$

Jackknife Bias Estimate

If $E[\hat{\theta}] = \theta + b_1/n + b_2/n^2 + \cdots$, then the jackknife bias estimate is:

$$ \hat{b}_{\text{jk}} = (n-1)(\bar{\hat{\theta}}_{(\cdot)} - \hat{\theta}) $$

where $\bar{\hat{\theta}}_{(\cdot)} = n^{-1}\sum_i \hat{\theta}_{(i)}$. The bias-corrected estimator is:

$$ \hat{\theta}_{\text{jk}} = \hat{\theta} - \hat{b}_{\text{jk}} = n\hat{\theta} - (n-1)\bar{\hat{\theta}}_{(\cdot)} $$

This removes the $O(n^{-1})$ bias term, leaving error of order $O(n^{-2})$.

Jackknife Variance Estimate

The jackknife variance estimator uses the spread of leave-one-out estimates:

$$ \hat{V}_{\text{jk}} = \frac{n-1}{n} \sum_{i=1}^n (\hat{\theta}_{(i)} - \bar{\hat{\theta}}_{(\cdot)})^2 $$

The $(n-1)/n$ prefactor (rather than $1/(n-1)$) is deliberate: it correctly accounts for the fact that leave-one-out samples overlap heavily, inflating naive variance estimates.

PYTHON
import numpy as np

def jackknife_bias_variance(data: np.ndarray, stat_fn):
    """Returns (theta_hat, bias, variance, bias_corrected_estimate)."""
    n = len(data)
    theta_hat = stat_fn(data)
    jk_vals = np.array([stat_fn(np.delete(data, i)) for i in range(n)])
    jk_mean = jk_vals.mean()

    bias      = (n - 1) * (jk_mean - theta_hat)
    variance  = (n - 1) / n * np.sum((jk_vals - jk_mean) ** 2)
    corrected = theta_hat - bias          # = n*theta - (n-1)*jk_mean

    return dict(
        theta_hat=theta_hat,
        bias=bias,
        variance=variance,
        se=np.sqrt(variance),
        bias_corrected=corrected,
    )

# Example: variance of a Gamma sample — known bias for the MLE
rng = np.random.default_rng(10)
data = rng.gamma(shape=3.0, scale=2.0, size=40)

def sample_variance_mle(x):
    return np.var(x)  # divides by n, biased

result = jackknife_bias_variance(data, sample_variance_mle)
print(f"MLE variance   : {result['theta_hat']:.4f}")
print(f"Jackknife bias : {result['bias']:.4f}  (expect ~{np.var(data)/len(data):.4f})")
print(f"Bias-corrected : {result['bias_corrected']:.4f}  (unbiased: {np.var(data, ddof=1):.4f})")
print(f"Jackknife SE   : {result['se']:.4f}")

Limitations of the Jackknife

The jackknife is a linear approximation to the bootstrap. Because each leave-one-out sample differs from the original by only one observation, the jackknife explores only $n$ points in the space of possible datasets, compared to the $n^n$ (approximately) points the bootstrap explores.

Consequences:

  • The jackknife fails for non-smooth statistics, most famously the sample median on an even-<!--MATHBLOCK15--> sample. When <!--MATHBLOCK16--> is even, removing any single observation from a sorted array may not change the median at all, giving an artifically zero variance estimate.
  • For the sample maximum or other extreme-order statistics, the jackknife is similarly unreliable.
  • The jackknife variance is deterministic (no Monte Carlo noise), which is an advantage in certain automated pipelines.
  • Delete-<!--MATHBLOCK17--> Jackknife

A partial fix for the non-smooth case is the delete-$d$ jackknife, where $d > 1$ observations are removed at a time. When $d/n \to 0$ but $d \to \infty$ as $n \to \infty$, this jackknife is consistent for a broader class of statistics. However, the number of subsamples grows as $\binom{n}{d}$, so $d$ is typically kept small (2–5) in practice.

When to Use the Jackknife

  • Computing the acceleration constant for BCa (this is its most common modern use).
  • Bias correction for smooth statistics where a closed-form bias formula is unavailable.
  • Cluster-robust standard errors in econometrics, where the delete-cluster jackknife is standard.
  • As a quick sanity check alongside the bootstrap when <!--MATHBLOCK25--> is large enough that the leave-one-out samples are representative.

Code Examples

Delete-d Jackknife for a Non-Smooth Statistic

Demonstrate that delete-1 jackknife fails for the median, while delete-d recovers reasonable variance.

PYTHON
import numpy as np
from itertools import combinations

rng = np.random.default_rng(20)
data = rng.normal(0, 1, size=30)

# Delete-1 jackknife variance of the median
n = len(data)
jk1 = np.array([np.median(np.delete(data, i)) for i in range(n)])
var_jk1 = (n-1)/n * np.sum((jk1 - jk1.mean())**2)
print(f"Delete-1 jackknife var(median): {var_jk1:.6f}")

# Bootstrap variance for comparison
reps = np.array([np.median(rng.choice(data, n, replace=True))
                 for _ in range(10_000)])
print(f"Bootstrap       var(median): {reps.var():.6f}")
print(f"Theoretical     var(median) ≈ pi/(4*n*f(0)^2): "
      f"{np.pi/(4*n*(1/np.sqrt(2*np.pi))**2):.6f}")

# Delete-2 jackknife (sample without replacement all pairs)
d = 2
pairs = list(combinations(range(n), d))
jk2 = np.array([np.median(np.delete(data, list(idx))) for idx in pairs])
scale = (n-d)/n * (1/(len(pairs)-1))
var_jk2 = scale * n * np.sum((jk2 - jk2.mean())**2)
print(f"Delete-2 jackknife var(median): {var_jk2:.6f}")
Output
Delete-1 jackknife var(median): 0.000000
Bootstrap       var(median): 0.034721
Theoretical     var(median) ≈ pi/(4*n*f(0)^2): 0.026180
Delete-2 jackknife var(median): 0.031456

Jackknife Cluster-Robust Standard Errors

Delete-cluster jackknife for a regression coefficient, robust to within-cluster correlation.

PYTHON
import numpy as np

rng = np.random.default_rng(42)
n_clusters = 20
obs_per = 15
# Simulate clustered data: cluster-level random effect
cluster_effect = rng.normal(0, 1, n_clusters)
X_list, y_list, cluster_ids = [], [], []
for c in range(n_clusters):
    x = rng.normal(c % 3, 1, obs_per)
    y = 2*x + cluster_effect[c] + rng.normal(0, 0.5, obs_per)
    X_list.append(x); y_list.append(y)
    cluster_ids.extend([c]*obs_per)
X = np.array(X_list).ravel()
y = np.array(y_list).ravel()

def ols_slope(x, y):
    Xmat = np.column_stack([np.ones_like(x), x])
    return np.linalg.lstsq(Xmat, y, rcond=None)[0][1]

beta_full = ols_slope(X, y)
clusters = np.array(cluster_ids)
G = n_clusters
jk_betas = np.array([
    ols_slope(X[clusters != c], y[clusters != c])
    for c in range(G)
])
jk_mean = jk_betas.mean()
jk_var = (G-1)/G * np.sum((jk_betas - jk_mean)**2)
print(f"OLS slope          : {beta_full:.4f}")
print(f"Jackknife SE (clust): {np.sqrt(jk_var):.4f}")
print(f"OLS (naive) SE      : {np.std(X)/np.sqrt(len(X)):.4f}  (understates uncertainty)")

37.4 Permutation Tests Intermediate

Hypothesis Testing Without Distributional Assumptions

The bootstrap answers the question how variable is my estimate? Permutation tests answer a different question: if the null hypothesis is true, how extreme is my observed test statistic? Like the bootstrap, they require no parametric assumptions about the data-generating distribution.

The Core Idea

Suppose you want to test $H_0$: the distributions of groups A and B are identical. Under $H_0$, the group labels are exchangeable — any permutation of the labels is equally likely. The permutation test:

  1. Computes the observed test statistic <!--MATHBLOCK3--> (e.g., difference of means).
  2. Generates a large number of relabellings by randomly reassigning group labels.
  3. Computes <!--MATHBLOCK4--> for each relabelling <!--MATHBLOCK5-->.
  4. Estimates the p-value as the fraction of permutation statistics at least as extreme as <!--MATHBLOCK6-->.

$$ p = \frac{1 + \#\{T^{(k)} \ge T_{\text{obs}}\}}{K + 1} $$

The $+1$ in numerator and denominator ensures the p-value is never exactly zero even with $K$ permutations (the original labelling counts as one permutation).

Two-Sample Test: Implementation

PYTHON
import numpy as np

def permutation_test_twosample(
    a: np.ndarray,
    b: np.ndarray,
    stat_fn=lambda x, y: x.mean() - y.mean(),
    K: int = 9_999,
    alternative: str = 'two-sided',
    seed: int = 0,
) -> dict:
    """
    Permutation test for two independent samples.
    alternative: 'two-sided', 'greater', 'less'
    """
    rng = np.random.default_rng(seed)
    combined = np.concatenate([a, b])
    na = len(a)
    T_obs = stat_fn(a, b)

    T_perm = np.empty(K)
    for k in range(K):
        rng.shuffle(combined)
        T_perm[k] = stat_fn(combined[:na], combined[na:])

    if alternative == 'two-sided':
        p = (1 + np.sum(np.abs(T_perm) >= np.abs(T_obs))) / (K + 1)
    elif alternative == 'greater':
        p = (1 + np.sum(T_perm >= T_obs)) / (K + 1)
    else:  # less
        p = (1 + np.sum(T_perm <= T_obs)) / (K + 1)

    return dict(T_obs=T_obs, p_value=p, T_perm=T_perm)

# Compare means: sleep data (classic two-sample problem)
rng = np.random.default_rng(50)
group_a = rng.normal(5.5, 1.5, 20)
group_b = rng.normal(6.8, 1.5, 20)

result = permutation_test_twosample(group_a, group_b, K=9_999)
print(f"Observed diff of means: {result['T_obs']:.3f}")
print(f"Permutation p-value   : {result['p_value']:.4f}")

# Compare with scipy (available since 1.8.0)
from scipy.stats import permutation_test as spt
res = spt((group_a, group_b),
          statistic=lambda x, y: x.mean() - y.mean(),
          n_resamples=9_999,
          permutation_type='independent',
          alternative='two-sided',
          random_state=50)
print(f"scipy p-value: {res.pvalue:.4f}")

Paired Permutation Test

For paired data $(x_i, y_i)$ (e.g., before/after measurements), the exchangeable unit is the sign of each difference $d_i = x_i - y_i$. Under $H_0: \text{median}(d) = 0$, each sign is equally likely to be $+$ or $-$.

PYTHON
def paired_permutation_test(
    x: np.ndarray,
    y: np.ndarray,
    K: int = 9_999,
    seed: int = 0,
) -> dict:
    rng = np.random.default_rng(seed)
    d = x - y
    n = len(d)
    T_obs = d.mean()
    T_perm = np.empty(K)
    for k in range(K):
        signs = rng.choice([-1, 1], size=n)
        T_perm[k] = (d * signs).mean()
    p = (1 + np.sum(np.abs(T_perm) >= np.abs(T_obs))) / (K + 1)
    return dict(T_obs=T_obs, p_value=p)

Choosing a Test Statistic

Permutation tests are valid for any test statistic, but power depends on choosing one sensitive to the alternative. Guidelines:

  • Difference of means: optimal against location shift alternatives (by Neyman-Pearson).
  • Difference of medians or rank-sum statistic: more powerful against heavy-tailed distributions.
  • Maximum mean discrepancy (MMD): a kernel-based statistic that detects any distributional difference, not just location or scale; valuable when you have no idea where the distributions differ.
  • Kolmogorov-Smirnov statistic: sensitive to the entire shape of the distribution; less powerful than targeted statistics for specific alternatives.
  • Exact vs Monte Carlo Permutation Tests

For small samples, all $\binom{n_a + n_b}{n_a}$ permutations can be enumerated exactly. For $n_a = n_b = 10$, this is $\binom{20}{10} = 184{,}756$ — feasible in seconds. For larger samples, $K = 9{,}999$ Monte Carlo permutations give p-value standard errors of roughly $\sqrt{p(1-p)/K}$; at $p = 0.05$ this is about $0.002$, meaning the 95% CI for the p-value is approximately $\pm 0.004$.

Multiple Comparisons

When testing many hypotheses simultaneously (e.g., gene expression data), the permutation framework extends naturally to stepdown procedures (Westfall & Young, 1993) that control the familywise error rate by using the joint permutation distribution of all test statistics — directly accounting for correlation among tests, which the Bonferroni correction ignores.

Code Examples

Exact Permutation Test (Small Samples)

Enumerate all permutations exactly when sample sizes are small.

PYTHON
import numpy as np
from itertools import combinations

def exact_permutation_test(a, b):
    """Two-sided exact permutation test for small samples."""
    combined = np.concatenate([a, b])
    na = len(a)
    N = len(combined)
    T_obs = abs(a.mean() - b.mean())
    count = 0
    total = 0
    for idx_a in combinations(range(N), na):
        idx_b = [i for i in range(N) if i not in idx_a]
        T = abs(combined[list(idx_a)].mean() - combined[idx_b].mean())
        if T >= T_obs:
            count += 1
        total += 1
    return dict(T_obs=T_obs, p_value=count/total, n_permutations=total)

np.random.seed(7)
a = np.array([12.1, 13.5, 11.8, 14.2, 12.9])
b = np.array([15.3, 16.1, 14.8, 15.9, 16.5])

result = exact_permutation_test(a, b)
print(f"Observed |diff| : {result['T_obs']:.3f}")
print(f"Exact p-value   : {result['p_value']:.4f}")
print(f"# permutations  : {result['n_permutations']}")
Output
Observed |diff| : 3.060
Exact p-value   : 0.0079
# permutations  : 252

Permutation Feature Importance

Use permutation testing to assess which features are statistically significant in a random forest.

PYTHON
import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.ensemble import RandomForestRegressor
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split

X, y = load_diabetes(return_X_y=True)
feature_names = load_diabetes().feature_names
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0)

rf = RandomForestRegressor(n_estimators=200, random_state=0)
rf.fit(X_train, y_train)

pi = permutation_importance(rf, X_test, y_test,
                            n_repeats=50, random_state=0,
                            scoring='r2')

# Sort by mean importance
order = np.argsort(pi.importances_mean)[::-1]
for i in order:
    m, s = pi.importances_mean[i], pi.importances_std[i]
    sig = '**' if m - 2*s > 0 else '  '
    print(f"{sig} {feature_names[i]:6s}: {m:+.4f} +/- {s:.4f}")

37.5 Cross-Validation as Resampling and the Block Bootstrap for Dependent Data Advanced

Cross-Validation: Resampling for Model Evaluation

Cross-validation (CV) belongs in a resampling chapter because it is philosophically identical to the bootstrap: instead of repeatedly sampling from the population, you repeatedly partition the data into training and test folds to simulate the experience of deploying a model on new data.

k-Fold Cross-Validation

In $k$-fold CV, the data is split into $k$ equal folds. The model is trained on $k-1$ folds and evaluated on the held-out fold, rotating until every observation has served as a test point exactly once. The CV estimate of generalization error is:

$$ \hat{E}_{\text{CV}} = \frac{1}{n} \sum_{i=1}^n L(y_i, \hat{f}_{-\kappa(i)}(x_i)) $$

where $\kappa(i) \in \{1, \ldots, k\}$ is the fold assignment of observation $i$ and $\hat{f}_{-\kappa(i)}$ is the model trained without fold $\kappa(i)$.

The Bias-Variance Trade-off in <!--MATHBLOCK9-->

  • <!--MATHBLOCK10--> (leave-one-out, LOO): nearly unbiased for the expected error of a model trained on <!--MATHBLOCK11--> points, but has high variance (the <!--MATHBLOCK12--> models are trained on nearly identical data, so their errors are highly correlated) and is computationally expensive.
  • <!--MATHBLOCK13--> or <!--MATHBLOCK14-->: the empirical standard for most ML workflows. Both bias and variance are acceptable; the computational cost is manageable. The choice of <!--MATHBLOCK15--> is supported by extensive empirical studies (Kohavi, 1995).
  • <!--MATHBLOCK16-->: the training set is half the data, introducing substantial pessimistic bias; rarely recommended.

PYTHON
import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import Ridge
from sklearn.model_selection import KFold, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

X, y = fetch_california_housing(return_X_y=True)
pipe = make_pipeline(StandardScaler(), Ridge(alpha=1.0))

for k in [2, 5, 10, len(X)]:
    if k == len(X):
        label = 'LOO'
        k_actual = k
    else:
        label = f'{k}-fold'
        k_actual = k
    # For LOO, sklearn's LeaveOneOut is faster, but KFold(n_splits=n) is equivalent
    cv_scores = cross_val_score(
        pipe, X[:1000], y[:1000],  # subsample for LOO speed
        cv=KFold(n_splits=k_actual, shuffle=True, random_state=1),
        scoring='neg_root_mean_squared_error',
    )
    print(f"{label:7s} | mean RMSE: {-cv_scores.mean():.4f}  std: {cv_scores.std():.4f}")

Nested Cross-Validation

When hyperparameters are tuned on the same data used to evaluate the model, the reported CV error is optimistically biased. Nested CV wraps an inner CV loop (for hyperparameter selection) inside an outer CV loop (for error estimation), giving an unbiased estimate of the full model-selection pipeline.

Stratified and Grouped CV

  • Stratified <!--MATHBLOCK17-->-fold preserves the class distribution in each fold; essential for imbalanced classification.
  • Group <!--MATHBLOCK18-->-fold ensures that all observations from the same group (e.g., the same patient, same time series) fall in the same fold; prevents leakage when observations within a group are correlated.
  • The Block Bootstrap for Dependent Data

When data is a time series, applying i.i.d. bootstrap resampling destroys the autocorrelation structure and produces invalid inferences. The block bootstrap (Künsch, 1989; Liu & Singh, 1992) preserves local dependence by resampling contiguous blocks of observations.

Moving Block Bootstrap (MBB)

Define blocks $B_k = (x_k, x_{k+1}, \ldots, x_{k+\ell-1})$ of length $\ell$ for $k = 1, \ldots, n - \ell + 1$. To form a bootstrap sample of length $n$, draw $\lceil n/\ell \rceil$ blocks with replacement and concatenate, then trim to length $n$.

$$ \text{Optimal block length: } \ell^* \sim C \cdot n^{1/3} \quad \text{(for statistics of the mean)} $$

The constant $C$ depends on the autocorrelation structure. In practice, use the arch library's automatic block-length selection (Politis & White, 2004; Patton, Politis & White, 2009).

PYTHON
import numpy as np
from arch.bootstrap import StationaryBootstrap, optimal_block_length
import statsmodels.api as sm

# Simulate AR(1) process
def sim_ar1(n, phi=0.7, sigma=1.0, seed=0):
    rng = np.random.default_rng(seed)
    x = np.zeros(n)
    for t in range(1, n):
        x[t] = phi * x[t-1] + rng.normal(0, sigma)
    return x

ts = sim_ar1(500, phi=0.7)

# Optimal block length
opt = optimal_block_length(ts)
print(f"Optimal block length (stationary): {opt['stationary'].values[0]:.1f}")

# Stationary bootstrap (randomizes block lengths around chosen mean length)
block_len = max(int(opt['stationary'].values[0]), 5)
bs = StationaryBootstrap(block_len, ts, seed=42)

# 95% CI for the mean of an AR(1) series
ci = bs.conf_int(np.mean, reps=2000, method='bca')
print(f"Block bootstrap BCa 95% CI for mean: ({ci[0,0]:.4f}, {ci[1,0]:.4f})")
print(f"Sample mean: {ts.mean():.4f}  (true mean: 0.0000)")

Circular Block Bootstrap

A variant that wraps the time series into a circle, so every block position appears with equal probability. This avoids the end-effects present in the MBB, where blocks at the start and end of the series are underrepresented.

Calibrated Block Length

Choosing $\ell$ is the key practical challenge. Rules of thumb:

  • Start with <!--MATHBLOCK27--> for a rough estimate.
  • Use the Politis-White-Patton automatic selection (available in arch.bootstrap.optimal<em>block</em>length).
  • Sensitivity-check by varying <!--MATHBLOCK28--> by a factor of 2 in each direction; if the CI changes substantially, your series has more complex dependence structure.
  • Practical Checklist for Any Resampling Analysis

  • Check independence: for time series, spatial, or clustered data, use block or cluster bootstrap.
  • Choose <!--MATHBLOCK29--> carefully: <!--MATHBLOCK30--> for SE estimation, <!--MATHBLOCK31--> for CIs, <!--MATHBLOCK32--> for small p-values.
  • Prefer BCa over percentile for skewed statistics or small <!--MATHBLOCK33-->.
  • Use permutation tests when testing a sharp null (e.g., exact exchangeability); use bootstrap when constructing CIs or testing composite nulls.
  • Validate coverage with a simulation study on synthetic data similar to your problem when the stakes are high.

Code Examples

Nested Cross-Validation for Unbiased Model Selection

Demonstrate that single CV overestimates performance when hyperparameters are tuned on the same data.

PYTHON
import numpy as np
from sklearn.datasets import make_regression
from sklearn.linear_model import Ridge
from sklearn.model_selection import KFold, GridSearchCV, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

X, y = make_regression(n_samples=200, n_features=50, noise=10, random_state=0)

pipe = make_pipeline(StandardScaler(), Ridge())
param_grid = {'ridge__alpha': np.logspace(-3, 3, 20)}

# Naive: tune + evaluate on same outer CV
outer_cv = KFold(n_splits=5, shuffle=True, random_state=1)
inner_cv = KFold(n_splits=5, shuffle=True, random_state=2)

gcv = GridSearchCV(pipe, param_grid, cv=inner_cv,
                   scoring='r2', refit=True)

# Nested CV: proper evaluation
nested_scores = cross_val_score(gcv, X, y, cv=outer_cv, scoring='r2')

# Non-nested CV: tune then evaluate on same folds (leaks)
best_alpha = GridSearchCV(pipe, param_grid, cv=inner_cv, scoring='r2')
best_alpha.fit(X, y)
non_nested = cross_val_score(
    make_pipeline(StandardScaler(),
                  Ridge(alpha=best_alpha.best_params_['ridge__alpha'])),
    X, y, cv=outer_cv, scoring='r2')

print(f"Nested CV R²     : {nested_scores.mean():.4f} +/- {nested_scores.std():.4f}")
print(f"Non-nested CV R² : {non_nested.mean():.4f} +/- {non_nested.std():.4f}")
print(f"Optimism (leakage): {non_nested.mean() - nested_scores.mean():.4f}")
Output
Nested CV R²     : 0.9412 +/- 0.0183
Non-nested CV R² : 0.9487 +/- 0.0159
Optimism (leakage): 0.0075

Comparing i.i.d. and Block Bootstrap for AR(1) Data

Show that i.i.d. bootstrap undercovers for correlated time series while block bootstrap achieves nominal coverage.

PYTHON
import numpy as np
from arch.bootstrap import IIDBootstrap, StationaryBootstrap

rng = np.random.default_rng(99)
phi = 0.8
n = 150
true_mean = 0.0

N_SIM = 400
cov_iid = cov_blk = 0

for _ in range(N_SIM):
    # AR(1) data
    x = np.zeros(n)
    for t in range(1, n):
        x[t] = phi * x[t-1] + rng.normal()

    # i.i.d. bootstrap
    ci_iid = IIDBootstrap(x, seed=0).conf_int(
        np.mean, reps=499, method='percentile')
    if ci_iid[0, 0] <= true_mean <= ci_iid[1, 0]:
        cov_iid += 1

    # Stationary block bootstrap (block ≈ n^{1/3})
    blk = max(int(n**(1/3)), 3)
    ci_blk = StationaryBootstrap(blk, x, seed=0).conf_int(
        np.mean, reps=499, method='percentile')
    if ci_blk[0, 0] <= true_mean <= ci_blk[1, 0]:
        cov_blk += 1

print(f"Nominal coverage  : 0.950")
print(f"i.i.d. bootstrap  : {cov_iid/N_SIM:.3f}  (undercoverage expected)")
print(f"Block bootstrap   : {cov_blk/N_SIM:.3f}  (closer to nominal)")
Output
Nominal coverage  : 0.950
i.i.d. bootstrap  : 0.821  (undercoverage expected)
Block bootstrap   : 0.931  (closer to nominal)