Reference

Appendix G: Statistics & Probability Formula Reference

Descriptive Statistics

Measures of Central Tendency

Sample Mean

$$ \bar{x} = \frac{1}{n} \sum_{i=1}^{n} x_i $$

Weighted Mean

$$ \bar{x}_w = \frac{\sum_{i=1}^{n} w_i x_i}{\sum_{i=1}^{n} w_i} $$

Sample Median — the middle value when data is sorted. For even $n$, average the two middle values.

Sample Mode — the most frequently occurring value.

Geometric Mean (useful for growth rates, ratios)

$$ \bar{x}_g = \left( \prod_{i=1}^{n} x_i \right)^{1/n} = \exp\!\left(\frac{1}{n}\sum_{i=1}^n \ln x_i\right) $$

Measures of Spread

Sample Variance (unbiased, Bessel's correction)

$$ s^2 = \frac{1}{n-1} \sum_{i=1}^{n} (x_i - \bar{x})^2 $$

Population Variance

$$ \sigma^2 = \frac{1}{N} \sum_{i=1}^{N} (x_i - \mu)^2 $$

Standard Deviation — $s = \sqrt{s^2}$, same units as the data.

Coefficient of Variation

$$ CV = \frac{s}{\bar{x}} $$

Interquartile Range — $IQR = Q_3 - Q_1$, robust to outliers.

Standard Error of the Mean

$$ SE_{\bar{x}} = \frac{s}{\sqrt{n}} $$

Shape

Sample Skewness

$$ g_1 = \frac{\frac{1}{n}\sum(x_i-\bar{x})^3}{s^3} $$

Sample Excess Kurtosis

$$ g_2 = \frac{\frac{1}{n}\sum(x_i-\bar{x})^4}{s^4} - 3 $$

A normal distribution has $g_1 = 0$, $g_2 = 0$.

Covariance and Correlation

Sample Covariance

$$ s_{xy} = \frac{1}{n-1}\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y}) $$

Pearson Correlation Coefficient

$$ r = \frac{s_{xy}}{s_x s_y} = \frac{\sum(x_i-\bar{x})(y_i-\bar{y})}{\sqrt{\sum(x_i-\bar{x})^2 \sum(y_i-\bar{y})^2}} $$

$r \in [-1, 1]$. Measures linear association only.

Spearman Rank Correlation — apply Pearson's formula to the ranks of $x$ and $y$. Robust to monotone nonlinearity and outliers.

PYTHON
import numpy as np
from scipy import stats

x = np.array([2.1, 3.4, 5.0, 1.2, 4.7])
y = np.array([1.8, 3.1, 4.9, 1.5, 4.2])

print(f"mean={np.mean(x):.3f}, std={np.std(x, ddof=1):.3f}")
print(f"pearson r={np.corrcoef(x, y)[0,1]:.4f}")
print(f"spearman r={stats.spearmanr(x, y).statistic:.4f}")

Probability Rules

Complement Rule — $P(A^c) = 1 - P(A)$

Addition Rule — $P(A \cup B) = P(A) + P(B) - P(A \cap B)$

Multiplication Rule — $P(A \cap B) = P(A)\,P(B \mid A)$

Conditional Probability

$$ P(A \mid B) = \frac{P(A \cap B)}{P(B)}, \quad P(B) > 0 $$

Independence — $A \perp B$ iff $P(A \cap B) = P(A)P(B)$

Law of Total Probability (partition $\{B_i\}$)

$$ P(A) = \sum_i P(A \mid B_i)\,P(B_i) $$

Bayes' Theorem

$$ P(B_i \mid A) = \frac{P(A \mid B_i)\,P(B_i)}{\sum_j P(A \mid B_j)\,P(B_j)} $$

In data science this is central to Naive Bayes classifiers, Bayesian updating, and posterior inference. $P(B_i)$ is the prior, $P(A \mid B_i)$ is the likelihood, and $P(B_i \mid A)$ is the posterior.

Expectation and Variance Rules

For random variables $X$, $Y$ and constants $a$, $b$:

$$ E[aX + b] = aE[X] + b $$

$$ E[X + Y] = E[X] + E[Y] \quad \text{(always)} $$

$$ \text{Var}(aX + b) = a^2\,\text{Var}(X) $$

$$ \text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y) + 2\,\text{Cov}(X,Y) $$

$$ \text{Var}(X) = E[X^2] - (E[X])^2 $$

If $X \perp Y$: $\text{Cov}(X,Y)=0$, so $\text{Var}(X+Y)=\text{Var}(X)+\text{Var}(Y)$.

Law of Total Expectation

$$ E[X] = E\bigl[E[X \mid Y]\bigr] $$

Law of Total Variance

$$ \text{Var}(X) = E\bigl[\text{Var}(X\mid Y)\bigr] + \text{Var}\bigl(E[X\mid Y]\bigr) $$

Common Probability Distributions

Discrete Distributions

Bernoulli($p$)

  • PMF: <!--MATHBLOCK88-->, <!--MATHBLOCK89-->
  • Mean: <!--MATHBLOCK90--> — Variance: <!--MATHBLOCK91-->

Binomial($n$, $p$)

$$ P(X=k) = \binom{n}{k} p^k (1-p)^{n-k}, \quad k=0,1,\ldots,n $$

  • Mean: <!--MATHBLOCK94--> — Variance: <!--MATHBLOCK95-->

Geometric($p$) — number of trials until first success

$$ P(X=k) = (1-p)^{k-1}p, \quad k = 1,2,\ldots $$

  • Mean: <!--MATHBLOCK97--> — Variance: <!--MATHBLOCK98-->

Poisson($\lambda$) — counts of rare events

$$ P(X=k) = \frac{\lambda^k e^{-\lambda}}{k!}, \quad k=0,1,2,\ldots $$

  • Mean: <!--MATHBLOCK100--> — Variance: <!--MATHBLOCK101-->

Binomial($n$,$p$) $\to$ Poisson($\lambda=np$) when $n\to\infty$, $p\to 0$, $np$ fixed.

Continuous Distributions

Uniform($a$, $b$)

$$ f(x) = \frac{1}{b-a}, \quad x \in [a,b] $$

  • Mean: <!--MATHBLOCK111--> — Variance: <!--MATHBLOCK112-->

Normal($\mu$, $\sigma^2$)

$$ f(x) = \frac{1}{\sigma\sqrt{2\pi}} \exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right) $$

  • Mean: <!--MATHBLOCK115--> — Variance: <!--MATHBLOCK116-->
  • Standard Normal: <!--MATHBLOCK117-->
  • 68-95-99.7 rule: <!--MATHBLOCK118-->, <!--MATHBLOCK119-->, <!--MATHBLOCK120--> cover roughly 68%, 95%, 99.7% of the distribution.

Exponential($\lambda$)

$$ f(x) = \lambda e^{-\lambda x}, \quad x \geq 0 $$

  • Mean: <!--MATHBLOCK122--> — Variance: <!--MATHBLOCK123-->
  • Memoryless: <!--MATHBLOCK124-->

Gamma($\alpha$, $\beta$) — sum of $\alpha$ independent Exp($\beta$) variables

$$ f(x) = \frac{\beta^\alpha}{\Gamma(\alpha)} x^{\alpha-1} e^{-\beta x}, \quad x>0 $$

  • Mean: <!--MATHBLOCK129--> — Variance: <!--MATHBLOCK130-->

Beta($\alpha$, $\beta$) — models probabilities, conjugate prior to Binomial

$$ f(x) = \frac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha,\beta)}, \quad x\in(0,1) $$

  • Mean: <!--MATHBLOCK133--> — Variance: <!--MATHBLOCK134-->

Chi-Squared($k$) — $\chi^2_k = \sum_{i=1}^k Z_i^2$, $Z_i \sim N(0,1)$ i.i.d.

  • Mean: <!--MATHBLOCK138--> — Variance: <!--MATHBLOCK139-->
  • Used in goodness-of-fit tests and sampling distributions of <!--MATHBLOCK140-->.

Student's $t$($\nu$) — $t = Z/\sqrt{\chi^2_\nu/\nu}$

  • Mean: <!--MATHBLOCK144--> (<!--MATHBLOCK145-->) — Variance: <!--MATHBLOCK146--> (<!--MATHBLOCK147-->)
  • Heavier tails than Normal; <!--MATHBLOCK148--> as <!--MATHBLOCK149-->

$F$($d_1$, $d_2$) — ratio of two chi-squared variables; appears in ANOVA and regression $F$-tests.

PYTHON
from scipy import stats
import matplotlib.pyplot as plt
import numpy as np

# Quick CDF / PPF lookups — bread and butter in practice
print(stats.norm.ppf(0.975))          # 1.96 (z critical value)
print(stats.t.ppf(0.975, df=29))      # t critical for n=30
print(stats.chi2.ppf(0.95, df=5))     # chi-sq critical value
print(stats.binom.pmf(k=3, n=10, p=0.3))   # P(X=3)
print(stats.poisson.cdf(k=4, mu=3))        # P(X<=4) with lambda=3

Sampling Distributions and the Central Limit Theorem

Sampling Distribution of $\bar{X}$

For i.i.d. $X_i \sim (\mu, \sigma^2)$:

$$ E[\bar{X}] = \mu, \qquad \text{Var}(\bar{X}) = \frac{\sigma^2}{n} $$

Central Limit Theorem (CLT)

$$ \frac{\bar{X} - \mu}{\sigma/\sqrt{n}} \xrightarrow{d} N(0,1) \quad \text{as } n \to \infty $$

This holds for any distribution with finite mean and variance. In practice, $n \geq 30$ is often sufficient unless the population is highly skewed.

Sampling Distribution of $s^2$

$$ \frac{(n-1)s^2}{\sigma^2} \sim \chi^2_{n-1} $$

Point Estimation

Method of Moments — set sample moments equal to population moments and solve for parameters. For $\bar{x}$ and $s^2$, this gives $\hat{\mu}=\bar{x}$, $\hat{\sigma}^2=s^2$.

Maximum Likelihood Estimator (MLE) — maximize the log-likelihood:

$$ \hat{\theta}_{MLE} = \arg\max_\theta \sum_{i=1}^n \ln f(x_i;\theta) $$

  • Normal MLE: <!--MATHBLOCK162-->, <!--MATHBLOCK163--> (biased).
  • Binomial MLE: <!--MATHBLOCK164--> (sample proportion).
  • Poisson MLE: <!--MATHBLOCK165-->.

Bias, Variance, MSE of an Estimator

$$ \text{Bias}(\hat{\theta}) = E[\hat{\theta}] - \theta $$

$$ \text{MSE}(\hat{\theta}) = \text{Var}(\hat{\theta}) + \text{Bias}(\hat{\theta})^2 $$

Confidence Intervals

All formulas below assume two-sided intervals at confidence level $1-\alpha$.

Mean, $\sigma$ known (z-interval)

$$ \bar{x} \pm z_{\alpha/2} \frac{\sigma}{\sqrt{n}} $$

Mean, $\sigma$ unknown (t-interval)

$$ \bar{x} \pm t_{\alpha/2,\, n-1} \frac{s}{\sqrt{n}} $$

Proportion (large $n$, Wald interval)

$$ \hat{p} \pm z_{\alpha/2} \sqrt{\frac{\hat{p}(1-\hat{p})}{n}} $$

Difference of two means (independent, equal variance)

$$ (\bar{x}_1 - \bar{x}_2) \pm t_{\alpha/2,\, n_1+n_2-2} \; s_p \sqrt{\frac{1}{n_1}+\frac{1}{n_2}} $$

where the pooled standard deviation is:

$$ s_p = \sqrt{\frac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n_1+n_2-2}} $$

Variance ($\chi^2$ interval)

$$ \left(\frac{(n-1)s^2}{\chi^2_{\alpha/2,\,n-1}},\; \frac{(n-1)s^2}{\chi^2_{1-\alpha/2,\,n-1}}\right) $$

PYTHON
import scipy.stats as stats
import numpy as np

data = np.random.default_rng(42).normal(loc=5, scale=2, size=40)
n, xbar, s = len(data), np.mean(data), np.std(data, ddof=1)

# 95% t-interval
ci = stats.t.interval(0.95, df=n-1, loc=xbar, scale=s/np.sqrt(n))
print(f"95% CI for mu: ({ci[0]:.3f}, {ci[1]:.3f})")

Hypothesis Testing

Framework

  1. State <!--MATHBLOCK171--> (null) and <!--MATHBLOCK172--> (alternative).
  2. Choose significance level <!--MATHBLOCK173--> (typically 0.05).
  3. Compute test statistic from data.
  4. Find p-value = <!--MATHBLOCK174-->.
  5. Reject <!--MATHBLOCK175--> if p-value <!--MATHBLOCK176-->.

Type I Error — reject $H_0$ when true: probability $= \alpha$. Type II Error — fail to reject $H_0$ when false: probability $= \beta$. Power $= 1 - \beta$.

Common Test Statistics

One-sample z-test (known $\sigma$)

$$ z = \frac{\bar{x} - \mu_0}{\sigma/\sqrt{n}} $$

One-sample t-test (unknown $\sigma$)

$$ t = \frac{\bar{x} - \mu_0}{s/\sqrt{n}} \sim t_{n-1} \text{ under } H_0 $$

Two-sample t-test (independent groups, pooled)

$$ t = \frac{\bar{x}_1 - \bar{x}_2}{s_p\sqrt{1/n_1 + 1/n_2}} \sim t_{n_1+n_2-2} $$

Paired t-test — compute $d_i = x_{1i} - x_{2i}$ and apply one-sample t-test to $d_i$:

$$ t = \frac{\bar{d}}{s_d/\sqrt{n}} \sim t_{n-1} $$

Proportion z-test

$$ z = \frac{\hat{p} - p_0}{\sqrt{p_0(1-p_0)/n}} $$

Chi-Square Goodness-of-Fit

$$ \chi^2 = \sum_{i=1}^k \frac{(O_i - E_i)^2}{E_i} \sim \chi^2_{k-1} \text{ under } H_0 $$

Chi-Square Test of Independence (contingency table with $r$ rows, $c$ cols)

$$ \chi^2 = \sum_{i,j} \frac{(O_{ij} - E_{ij})^2}{E_{ij}}, \quad E_{ij} = \frac{R_i C_j}{n} $$

degrees of freedom $= (r-1)(c-1)$.

F-test for equality of variances

$$ F = \frac{s_1^2}{s_2^2} \sim F_{n_1-1,\, n_2-1} \text{ under } H_0: \sigma_1^2 = \sigma_2^2 $$

One-Way ANOVA F-statistic

$$ F = \frac{MS_{between}}{MS_{within}} = \frac{SS_B/(k-1)}{SS_W/(N-k)} $$

where $k$ = number of groups, $N$ = total observations.

PYTHON
from scipy import stats
import numpy as np

rng = np.random.default_rng(0)
a = rng.normal(5, 1, 30)
b = rng.normal(5.5, 1, 30)

t_stat, p_val = stats.ttest_ind(a, b, equal_var=True)
print(f"Two-sample t: t={t_stat:.3f}, p={p_val:.4f}")

# Chi-square test of independence
observed = np.array([[20, 30], [15, 35]])
chi2, p, dof, expected = stats.chi2_contingency(observed)
print(f"Chi-sq={chi2:.3f}, p={p:.4f}, df={dof}")

Simple Linear Regression

Model: $Y_i = \beta_0 + \beta_1 x_i + \varepsilon_i$, where $\varepsilon_i \sim N(0, \sigma^2)$ i.i.d.

OLS Estimates

$$ \hat{\beta}_1 = \frac{\sum(x_i - \bar{x})(y_i - \bar{y})}{\sum(x_i - \bar{x})^2} = \frac{s_{xy}}{s_x^2} = r \frac{s_y}{s_x} $$

$$ \hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x} $$

Residual Standard Error

$$ \hat{\sigma} = s_e = \sqrt{\frac{SSE}{n-2}}, \qquad SSE = \sum(y_i - \hat{y}_i)^2 $$

Standard Errors of Coefficients

$$ SE(\hat{\beta}_1) = \frac{s_e}{\sqrt{\sum(x_i-\bar{x})^2}}, \qquad SE(\hat{\beta}_0) = s_e\sqrt{\frac{1}{n} + \frac{\bar{x}^2}{\sum(x_i-\bar{x})^2}} $$

t-test for slope ($H_0: \beta_1 = 0$)

$$ t = \frac{\hat{\beta}_1}{SE(\hat{\beta}_1)} \sim t_{n-2} $$

Coefficient of Determination

$$ R^2 = 1 - \frac{SSE}{SST} = r^2, \qquad SST = \sum(y_i - \bar{y})^2 $$

Adjusted $R^2$ (penalizes extra predictors in multiple regression)

$$ R^2_{adj} = 1 - \frac{(1-R^2)(n-1)}{n-p-1} $$

where $p$ = number of predictors.

Multiple Linear Regression (Matrix Form)

Model: $\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}$

OLS solution

$$ \hat{\boldsymbol{\beta}} = (\mathbf{X}^\top \mathbf{X})^{-1} \mathbf{X}^\top \mathbf{y} $$

Covariance of estimates

$$ \text{Cov}(\hat{\boldsymbol{\beta}}) = \sigma^2 (\mathbf{X}^\top \mathbf{X})^{-1} $$

Overall F-test ($H_0$: all slope coefficients $= 0$)

$$ F = \frac{(SST - SSE)/p}{SSE/(n-p-1)} \sim F_{p,\,n-p-1} $$

PYTHON
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
import scipy.stats as stats

rng = np.random.default_rng(7)
x = rng.uniform(0, 10, 50)
y = 3 + 1.5 * x + rng.normal(0, 1.5, 50)

X = x.reshape(-1, 1)
model = LinearRegression().fit(X, y)
yhat = model.predict(X)
n, p = len(y), 1

SSE = np.sum((y - yhat)**2)
SST = np.sum((y - np.mean(y))**2)
se = np.sqrt(SSE / (n - p - 1))
print(f"beta_0={model.intercept_:.3f}, beta_1={model.coef_[0]:.3f}")
print(f"R^2={r2_score(y, yhat):.4f},  sigma_hat={se:.3f}")

Key Identities and Inequalities

Markov's Inequality (non-negative $X$, $a > 0$)

$$ P(X \geq a) \leq \frac{E[X]}{a} $$

Chebyshev's Inequality

$$ P(|X - \mu| \geq k\sigma) \leq \frac{1}{k^2} $$

Jensen's Inequality (convex $\phi$)

$$ \phi(E[X]) \leq E[\phi(X)] $$

This immediately shows $E[X]^2 \leq E[X^2]$ and justifies that the log of an average exceeds the average of the log.

Bayesian Conjugate Pairs

Common prior-likelihood pairs where the posterior stays in the same family:

  • Beta–Binomial: prior <!--MATHBLOCK203-->, observe <!--MATHBLOCK204--> successes in <!--MATHBLOCK205--> trials, posterior <!--MATHBLOCK206-->.
  • Gamma–Poisson: prior <!--MATHBLOCK207-->, observe <!--MATHBLOCK208--> counts with sum <!--MATHBLOCK209-->, posterior <!--MATHBLOCK210-->.
  • Normal–Normal: prior <!--MATHBLOCK211-->, observe <!--MATHBLOCK212--> samples with known <!--MATHBLOCK213-->, posterior mean is a precision-weighted average of prior mean and sample mean.

Bayesian point estimates

  • MAP (Maximum A Posteriori): mode of the posterior.
  • Posterior mean: <!--MATHBLOCK214-->, minimizes squared-error loss.
  • Posterior median: minimizes absolute-error loss.

PYTHON
import numpy as np
from scipy import stats

# Beta-Binomial updating
alpha_prior, beta_prior = 2, 2   # prior: Beta(2,2)
k, n = 7, 10                     # 7 successes in 10 trials
alpha_post = alpha_prior + k
beta_post  = beta_prior + (n - k)

posterior = stats.beta(alpha_post, beta_post)
print(f"Posterior: Beta({alpha_post},{beta_post})")
print(f"Posterior mean={posterior.mean():.3f}, 95% CI={posterior.interval(0.95)}")

Quick-Reference: Critical Values

For the standard Normal $Z \sim N(0,1)$:

  • <!--MATHBLOCK216--> (one-tailed 90%)
  • <!--MATHBLOCK217--> (one-tailed 95%)
  • <!--MATHBLOCK218--> (two-tailed 95%)
  • <!--MATHBLOCK219--> (two-tailed 99%)

For $t$ with $\nu$ degrees of freedom at $\alpha/2 = 0.025$ (two-tailed 95%):

  • <!--MATHBLOCK223-->: <!--MATHBLOCK224-->
  • <!--MATHBLOCK225-->: <!--MATHBLOCK226-->
  • <!--MATHBLOCK227-->: <!--MATHBLOCK228-->
  • <!--MATHBLOCK229-->: <!--MATHBLOCK230-->
  • <!--MATHBLOCK231-->: <!--MATHBLOCK232-->

PYTHON
from scipy import stats

# Reproduce the table above programmatically
for nu in [10, 20, 30, 60, 120]:
    t_crit = stats.t.ppf(0.975, df=nu)
    print(f"df={nu:4d}  t*={t_crit:.4f}")

Common Pitfalls

  • Confusing standard deviation and standard error. <!--MATHBLOCK233--> describes the spread of individual observations; <!--MATHBLOCK234--> describes the spread of the sample mean.
  • Using z when you should use t. Whenever <!--MATHBLOCK235--> is estimated from data (almost always), use the <!--MATHBLOCK236-->-distribution. For large <!--MATHBLOCK237--> the difference is negligible.
  • Forgetting Bessel's correction. Dividing by <!--MATHBLOCK238--> gives a biased estimator of <!--MATHBLOCK239-->; divide by <!--MATHBLOCK240--> for the unbiased estimator.
  • Interpreting <!--MATHBLOCK241--> as causal. High <!--MATHBLOCK242--> only means the predictors explain variance in <!--MATHBLOCK243-->, not that they cause changes in <!--MATHBLOCK244-->.
  • p-value misinterpretation. The p-value is not the probability that <!--MATHBLOCK245--> is true. It is <!--MATHBLOCK246-->.
  • Multiple comparisons inflation. Running <!--MATHBLOCK247--> tests at <!--MATHBLOCK248--> inflates the familywise error rate. Apply Bonferroni (<!--MATHBLOCK249-->) or Benjamini-Hochberg FDR correction.