Beginner Intermediate 19 min read

Chapter 51: Reproducible Research & Notebooks

Reproducibility is the cornerstone of credible data science. A result that cannot be independently verified — or that you yourself cannot reproduce six months later — provides little scientific or engineering value. Yet the data science workflow is notoriously hostile to reproducibility: notebooks accumulate hidden state, package versions drift silently, random seeds go unset, and data files are modified in place. This chapter confronts those hazards directly, giving you the tools and habits to produce analyses that are fully reproducible, shareable, and auditable.

We begin with Jupyter notebooks, the de-facto interactive medium for data exploration, and examine both their genuine strengths and their well-documented failure modes. From there we move to environment management — the bedrock of reproducibility — covering Python virtual environments, conda, and dependency pinning strategies that ensure your code runs identically on every machine and at every point in time. We then address project structure, version control for notebooks and data assets, automated notebook execution with Papermill and nbconvert, and the broader discipline of literate programming. By the chapter's end you will have a complete, professional-grade reproducibility toolkit.

The payoff is not merely academic. Engineering teams that invest in reproducibility ship faster because debugging is cheaper; researchers who publish reproducible analyses receive more citations and fewer retractions; and every practitioner benefits from being able to revisit their own work without the sinking feeling of "I have no idea what environment I was using when I wrote this." Reproducibility is not overhead — it is the professional standard.

Runnable companion notebook for this chapter. Download the Jupyter notebook (.ipynb)

Libraries covered: DVC Matplotlib NumPy Pandas Scikit-learn

Learning Objectives

Set up and manage isolated Python environments with venv and conda, and pin dependencies so analyses are exactly reproducible across machines and time.
Identify and avoid the most common Jupyter notebook pitfalls — hidden state, out-of-order execution, and untested cells — and apply best practices for notebook hygiene.
Structure a data science project directory so that raw data, processed data, notebooks, source code, and reports are cleanly separated and self-documenting.
Use Papermill to parameterize and execute notebooks programmatically, and nbconvert to render notebooks to HTML, PDF, and other formats for distribution.
Control all sources of randomness (NumPy, Python, TensorFlow/PyTorch seeds) and record environment metadata so that numerical results are exactly or statistically reproducible.
Apply version control strategies appropriate to notebooks (nbstripout, Jupytext) and to data assets (DVC, checksums), keeping repositories clean and auditable.
Practice literate programming principles — interweaving prose, equations, and code — to produce self-contained, narrative-driven analyses.

51.1 Jupyter Notebooks: Strengths, Pitfalls, and Best Practices Beginner

What Notebooks Are Good At

Jupyter notebooks combine executable code, rich output, and narrative prose in a single document. This makes them ideal for exploratory data analysis, teaching, and communicating results to stakeholders. A well-written notebook tells a story: it explains why each step is taken, not merely what is computed.

The notebook file format (*.ipynb) is JSON under the hood. Each cell has a type (code, markdown, or raw), source text, and a list of outputs. The kernel maintains a live Python (or R, Julia, …) process; executing a cell sends its source to the kernel and captures stdout, stderr, rich MIME outputs (images, HTML, LaTeX), and the execution count.

The Hidden-State Problem

The single greatest reproducibility hazard in notebooks is out-of-order execution. The kernel maintains a global namespace. If you run cell 5 before cell 3, then edit cell 3, then re-run cell 5, the results depend on execution history that is not visible in the document. A collaborator who opens the notebook and runs all cells top-to-bottom may get completely different results.

The canonical failure looks like this:

PYTHON

# Cell 1 — run first
x = 10

# Cell 3 — accidentally run before cell 2
result = x * multiplier   # Works only if multiplier exists from a previous session

# Cell 2 — run after cell 3
multiplier = 5

When you share this notebook, the recipient who runs it top-to-bottom gets a NameError. You never saw that error because your kernel already had multiplier defined from an earlier run.

Rule 1: Restart & Run All before committing or sharing a notebook. Kernel → Restart & Run All should complete without errors. This is non-negotiable for any notebook that leaves your machine.

Cell Organization and Notebook Hygiene

Keep notebooks linear

Organize cells so execution flows strictly from top to bottom. If you find yourself needing to re-run an earlier cell after running later ones, that is a design smell — factor the dependency into a shared function.

One notebook, one narrative

A notebook should answer one question or tell one story. If it exceeds roughly 400 lines of code, break it apart. Extract reusable logic into .py modules in a src/ directory and import them; this makes code testable and keeps notebooks focused on narrative.

PYTHON

# src/features.py  — plain Python module, version-controlled, unit-testable
def compute_rolling_average(series, window=7):
    """Return the rolling mean with min_periods=1."""
    return series.rolling(window, min_periods=1).mean()

PYTHON

# In the notebook
import importlib, sys
sys.path.insert(0, "../src")   # or install the package in editable mode
import features
import importlib; importlib.reload(features)  # during active development

df["rolling_avg"] = features.compute_rolling_average(df["value"])

Use `%autoreload` during development

PYTHON

%load_ext autoreload
%autoreload 2   # reload all modules before each cell execution

This prevents the stale-import trap where you edit a .py file but the notebook still uses the old version.

Markdown cells are first-class

Every significant block of computation should be preceded by a markdown cell explaining what it does and why. This is the essence of literate programming: the notebook is a document that happens to contain executable code, not a script with comments.

Inline math uses single dollars: $\mu = \frac{1}{n}\sum_{i=1}^n x_i$ .

Block math uses double dollars on their own line:

\text{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(\hat{y}_i - y_i)^2}

Outputs and Version Control

Notebook outputs (plots, tables, print statements) are stored in the JSON file. This creates two problems for version control:

Diff noise: a single rerun changes dozens of JSON lines even if the code is identical.
Binary bleed: embedded base64-encoded PNG images bloat the repository.

The solution is to strip outputs before committing using nbstripout:

BASH

pip install nbstripout
nbstripout --install   # installs a git filter for *.ipynb in this repo

After this one-time setup, git diff and git add operate on output-free notebooks. Outputs are regenerated on demand by running the notebook.

Code Examples

Install and configure nbstripout as a git filter

Strips notebook outputs before every git add, keeping diffs clean and the repo lean.

BASH

pip install nbstripout

# From inside the git repository:
nbstripout --install

# Verify the filter is registered:
cat .git/config | grep filter -A3

Output

[filter "nbstripout"]
	clean = nbstripout
	smudge = cat
	required = true

Programmatically validate that a notebook runs top-to-bottom without errors

A utility function suitable for CI pipelines: raises on the first cell that errors out.

PYTHON

import nbformat
from nbconvert.preprocessors import ExecutePreprocessor
from pathlib import Path

def assert_notebook_runs(path: str, timeout: int = 600) -> None:
    """Execute a notebook and raise if any cell raises an exception."""
    nb = nbformat.read(path, as_version=4)
    ep = ExecutePreprocessor(timeout=timeout, kernel_name="python3")
    ep.preprocess(nb, {"metadata": {"path": str(Path(path).parent)}})
    print(f"Notebook {path} executed successfully ({len(nb.cells)} cells).")

assert_notebook_runs("notebooks/01_eda.ipynb")

Output

Notebook notebooks/01_eda.ipynb executed successfully (34 cells).

51.2 Virtual Environments, Dependency Pinning, and Environment Management Beginner

Why Isolation Matters

Without isolated environments every project on your machine shares a single pool of packages. Installing scikit-learn 1.4 for one project silently breaks another that depended on scikit-learn 1.1 API. Three months later, a pip install --upgrade destroys reproducibility entirely. Environment isolation is not a luxury — it is the minimum entry fee for reproducible work.

Python `venv` — Lightweight and Built-In

venv ships with Python 3.3+ and creates a self-contained directory containing a Python interpreter copy and a private site-packages tree.

BASH

# Create environment
python -m venv .venv

# Activate (Linux/macOS)
source .venv/bin/activate

# Activate (Windows PowerShell)
.venv\Scripts\Activate.ps1

# Verify isolation
which python   # should point to .venv/bin/python

# Install dependencies
pip install jupyter pandas scikit-learn matplotlib

# Deactivate
deactivate

The .venv/ directory should be listed in .gitignore — never commit it. Instead, commit a requirements file.

Dependency Pinning with pip

There are two levels of requirements files:

Abstract requirements (requirements.in): direct dependencies only, possibly with loose constraints.
Pinned lockfile (requirements.txt): every package including transitive dependencies, pinned to exact versions and hashes.

pip-tools manages both:

BASH

pip install pip-tools

# requirements.in — authored by hand
cat requirements.in
# jupyter>=7
# pandas>=2.0
# scikit-learn>=1.4
# matplotlib

# Resolve and pin everything:
pip-compile requirements.in   # produces requirements.txt

# Install from the lockfile:
pip-sync requirements.txt

The generated requirements.txt contains lines like:

pandas==2.2.1 \
    --hash=sha256:a3ab818... \
    # via -r requirements.in

Hash-pinned installs fail loudly if a package is tampered with or if PyPI serves a different artifact — crucial for supply-chain security.

Conda — When You Need Non-Python Dependencies

Conda manages Python interpreters, C libraries, CUDA toolkits, and R packages in addition to Python packages. It is the preferred choice for scientific computing stacks that depend on compiled extensions.

BASH

# Create a new conda environment
conda create -n myproject python=3.11 -y
conda activate myproject

# Install mixed conda + pip packages
conda install -c conda-forge pandas scikit-learn matplotlib -y
pip install papermill jupytext

# Export the exact environment
conda env export > environment.yml

# Recreate from export on another machine
conda env create -f environment.yml

A committed environment.yml gives collaborators a one-command setup. Note that conda env export includes build hashes; use conda env export --no-builds for cross-platform portability (Linux vs macOS).

Poetry — Modern Dependency Management

Poetry combines environment creation, dependency resolution, and packaging in a single tool. Its pyproject.toml and poetry.lock are analogous to pip-tools' requirements.in and requirements.txt but richer:

BASH

pip install poetry

# Initialise a new project
poetry new myproject
cd myproject

# Add dependencies
poetry add pandas scikit-learn matplotlib
poetry add --group dev jupyter pytest

# Install (creates .venv automatically)
poetry install

# Activate shell
poetry shell

The poetry.lock file should always be committed. It records exact versions and SHA256 hashes for every package, making installs fully deterministic.

Registering the Kernel with Jupyter

A common pitfall: you create a virtual environment and install Jupyter, but when you launch JupyterLab from outside the environment, the kernel uses the wrong Python. The fix is to register a named kernel:

BASH

# Inside the activated environment:
pip install ipykernel
python -m ipykernel install --user --name myproject --display-name "Python (myproject)"

Now JupyterLab shows "Python (myproject)" as a kernel option regardless of how Jupyter was launched. Always verify import sys; sys.executable at the top of your notebook to confirm the expected interpreter is running.

Capturing the Full Environment Snapshot

Even with pinned files, it is good practice to record the actual installed environment inside the notebook itself:

PYTHON

import subprocess, sys

def snapshot_environment():
    result = subprocess.run(
        [sys.executable, "-m", "pip", "freeze"],
        capture_output=True, text=True
    )
    for line in result.stdout.splitlines()[:10]:
        print(line)
    print(f"Python {sys.version}")

snapshot_environment()

This output is stored in the notebook's executed form, giving future readers an exact record of what was installed at analysis time.

Code Examples

Full venv + pip-tools workflow from scratch

End-to-end script to create a reproducible, kernel-registered Python environment.

BASH

# 1. Create and activate
python -m venv .venv
source .venv/bin/activate   # or .venv\Scripts\Activate.ps1 on Windows

# 2. Upgrade pip and install pip-tools
pip install --upgrade pip pip-tools

# 3. Write abstract requirements
cat > requirements.in << 'EOF'
jupyter>=7
pandas>=2.1
numpy>=1.26
scikit-learn>=1.4
matplotlib>=3.8
pip-tools
EOF

# 4. Compile to locked requirements.txt
pip-compile requirements.in

# 5. Install exactly
pip-sync requirements.txt

# 6. Register kernel
pip install ipykernel
python -m ipykernel install --user --name project_env --display-name "Python (project_env)"

echo "Environment ready."

Output

Environment ready.

Conda environment export and recreation

Demonstrates portable conda environment export and recreation.

BASH

# Export with build strings stripped for cross-platform use
conda env export --no-builds -n myproject > environment.yml

# Show first 20 lines of the export
head -20 environment.yml

# On a new machine:
conda env create -f environment.yml
conda activate myproject
python -c "import pandas; print(pandas.__version__)"

Output

2.2.1

51.3 Project Structure and Version Control for Data Science Intermediate

Why Structure Matters

A reproducible analysis is not just one that runs — it is one that a new collaborator (or your future self) can understand, navigate, and extend without a guided tour. A clear directory structure encodes conventions that eliminate decision fatigue and make project state legible at a glance.

A Reference Directory Layout

The following structure is inspired by Cookiecutter Data Science and widely adopted in industry:

myproject/
├── .venv/                  # virtual environment — NEVER committed
├── data/
│   ├── raw/                # immutable original data — read-only
│   ├── interim/            # intermediate transformations
│   └── processed/          # final, analysis-ready datasets
├── notebooks/
│   ├── 01_eda.ipynb
│   ├── 02_feature_engineering.ipynb
│   └── 03_model_evaluation.ipynb
├── src/
│   └── myproject/
│       ├── __init__.py
│       ├── data.py         # data loading / cleaning functions
│       ├── features.py
│       └── models.py
├── tests/
│   └── test_features.py
├── reports/
│   └── figures/
├── requirements.in
├── requirements.txt        # pinned lockfile
├── pyproject.toml
├── .gitignore
└── README.md

Key principles:

data/raw/ is treated as read-only. Scripts never write to it. If your pipeline must modify incoming data, it copies to data/interim/.
Notebooks are numbered to indicate execution order. This is a convention, not a guarantee — Restart & Run All is still required for each.
Business logic lives in src/, not in notebooks. Notebooks are the user interface of the analysis, not the engine.

The `.gitignore` Contract

A thoughtful .gitignore is part of the project structure:

BASH

# .gitignore
.venv/
__pycache__/
*.pyc
.ipynb_checkpoints/
data/raw/          # large or sensitive — use DVC instead
data/processed/    # reproducibly derived; see DVC section
*.egg-info/
.env               # never commit secrets

Version Control for Notebooks with Jupytext

nbstripout strips outputs before committing, but the JSON cell structure still produces noisy diffs. Jupytext solves this more completely by maintaining a plain-text mirror of the notebook — a .py file using percent (# %%) or light format — which is what gets committed to version control.

BASH

pip install jupytext

# Pair a notebook with a Python script mirror
jupytext --set-formats ipynb,py:percent notebooks/01_eda.ipynb

# Now editing either file syncs the other:
jupytext --sync notebooks/01_eda.ipynb

The paired .py file looks like:

PYTHON

# %% [markdown]
# ## Exploratory Data Analysis
# This section loads and profiles the raw dataset.

# %%
import pandas as pd
df = pd.read_parquet("../data/raw/sales.parquet")
df.head()

# %% [markdown]
# ### Missing Value Audit

# %%
df.isnull().sum()

This file is clean Python, diffs beautifully, and can be linted and reviewed like any source file. The .ipynb file is regenerated on demand with jupytext --sync.

Version Control for Data with DVC

Git is not designed for large binary files. Data Version Control (DVC) extends git with content-addressed storage of data files in a remote (S3, GCS, Azure Blob, SSH, etc.) while tracking lightweight .dvc pointer files in git.

BASH

pip install dvc dvc-s3   # or dvc-gs, dvc-azure, etc.

# Initialise DVC alongside git
git init
dvc init

# Track a raw data file
dvc add data/raw/sales.parquet
git add data/raw/sales.parquet.dvc .gitignore
git commit -m "Track raw sales data with DVC"

# Configure a remote and push
dvc remote add -d myremote s3://my-bucket/dvc-store
dvc push

# On another machine, pull the data
git clone https://github.com/me/myproject
dvc pull

The .dvc pointer file records the MD5 hash of the data file. Checking out an older git commit and running dvc checkout restores the exact data version that existed at that point in history — a complete audit trail.

Checksums for Lightweight Data Integrity

For small datasets not warranting full DVC, store a checksum alongside the data and assert it at load time:

PYTHON

import hashlib
from pathlib import Path

def md5(path: str) -> str:
    h = hashlib.md5()
    for chunk in Path(path).open("rb"):
        h.update(chunk)
    return h.hexdigest()

EXPECTED_HASH = "d41d8cd98f00b204e9800998ecf8427e"  # replace with real hash
assert md5("data/raw/sales.parquet") == EXPECTED_HASH, (
    "Raw data checksum mismatch — file may have been modified!"
)

Place this assertion at the top of every notebook that uses the file. It costs microseconds and catches accidental overwrites instantly.

Code Examples

Scaffold a project directory structure

Creates the standard data science project directory layout in one script.

BASH

#!/usr/bin/env bash
# Run from the desired parent directory
PROJECT=myproject

mkdir -p $PROJECT/{data/{raw,interim,processed},notebooks,src/$PROJECT,tests,reports/figures}
touch $PROJECT/src/$PROJECT/__init__.py
touch $PROJECT/src/$PROJECT/{data,features,models}.py
touch $PROJECT/{requirements.in,README.md,pyproject.toml}

cat > $PROJECT/.gitignore << 'EOF'
.venv/
__pycache__/
*.pyc
.ipynb_checkpoints/
data/raw/
data/processed/
*.egg-info/
.env
EOF

echo "Project scaffold created at ./$PROJECT"

Output

Project scaffold created at ./myproject

Set up Jupytext pairing for all notebooks in a directory

Converts an existing notebook directory to Jupytext-managed text mirrors for clean version control.

BASH

pip install jupytext

# Pair every notebook in notebooks/ with a percent-format Python mirror
for nb in notebooks/*.ipynb; do
    jupytext --set-formats ipynb,py:percent "$nb"
done

# Add the .py mirrors to git, keep .ipynb in .gitignore
git add notebooks/*.py
echo 'notebooks/*.ipynb' >> .gitignore
git add .gitignore
git commit -m "Switch to Jupytext percent-format notebook mirrors"

Output

[main abc1234] Switch to Jupytext percent-format notebook mirrors
 5 files changed, 120 insertions(+)

DVC pipeline stage definition (dvc.yaml equivalent in Python API)

Generates a three-stage DVC pipeline definition for data preparation, feature engineering, and model training.

PYTHON

# Alternatively, define pipeline stages programmatically
# This script generates dvc.yaml for a multi-stage pipeline
import yaml

pipeline = {
    "stages": {
        "prepare": {
            "cmd": "python src/myproject/data.py",
            "deps": ["data/raw/sales.parquet", "src/myproject/data.py"],
            "outs": ["data/interim/sales_cleaned.parquet"],
        },
        "featurize": {
            "cmd": "python src/myproject/features.py",
            "deps": ["data/interim/sales_cleaned.parquet", "src/myproject/features.py"],
            "outs": ["data/processed/features.parquet"],
        },
        "train": {
            "cmd": "python src/myproject/models.py",
            "deps": ["data/processed/features.parquet", "src/myproject/models.py"],
            "outs": ["models/model.pkl"],
            "metrics": [{"metrics/scores.json": {"cache": False}}],
        },
    }
}

with open("dvc.yaml", "w") as f:
    yaml.dump(pipeline, f, default_flow_style=False)

print("dvc.yaml written — run: dvc repro")

Output

dvc.yaml written — run: dvc repro

51.4 Seeding, Determinism, and Capturing Environment Metadata Intermediate

Sources of Non-Determinism

A reproducible analysis must produce identical (or statistically equivalent) results when re-run. Non-determinism enters from several sources:

Random number generation: random shuffles, train/test splits, weight initializations, bootstrap samples.
Algorithm non-determinism: some BLAS/LAPACK routines use non-deterministic threading; certain GPU kernels are non-deterministic by design.
OS and hardware: floating-point fused multiply-add (FMA) behavior varies; multithreaded sort may produce different tie-breaking order.
External data drift: APIs, web scrapes, or database queries may return different results over time.

Seeding Random Number Generators

For pure Python and NumPy, set seeds at the top of every notebook or script:

PYTHON

import random
import numpy as np

RANDOM_SEED = 42

random.seed(RANDOM_SEED)
np.random.seed(RANDOM_SEED)  # legacy global RNG

# Preferred: use a Generator instance (NumPy 1.17+)
rng = np.random.default_rng(RANDOM_SEED)
samples = rng.standard_normal(1000)

Passing rng as an explicit parameter to functions is better than relying on the global seed because it is composable and testable.

For scikit-learn, pass random_state to every estimator and splitter that accepts it:

PYTHON

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=RANDOM_SEED
)

model = RandomForestClassifier(n_estimators=100, random_state=RANDOM_SEED)
model.fit(X_train, y_train)

For TensorFlow and PyTorch:

PYTHON

import os
os.environ["PYTHONHASHSEED"] = str(RANDOM_SEED)

# TensorFlow
import tensorflow as tf
tf.random.set_seed(RANDOM_SEED)

# PyTorch
import torch
torch.manual_seed(RANDOM_SEED)
torch.cuda.manual_seed_all(RANDOM_SEED)
# For fully deterministic GPU ops (may reduce performance):
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

Numerical Reproducibility and Its Limits

Even with identical seeds, floating-point results can differ across:

Different processor architectures (x86 vs ARM)
Different BLAS implementations (OpenBLAS vs MKL)
Different numbers of threads (due to non-associativity of floating-point addition)

The practical approach is statistical reproducibility: results should be within expected sampling variance, and assertions should use approximate equality:

PYTHON

import numpy as np

expected_auc = 0.8732
actual_auc   = evaluate_model()

# Allow for platform-level floating-point variation
np.testing.assert_allclose(actual_auc, expected_auc, rtol=1e-3,
    err_msg="AUC deviated beyond expected tolerance — check seeds!")

Capturing Environment Metadata

Seeding ensures identical results on the same platform; environment metadata ensures that differences between platforms can be diagnosed. Capture it programmatically:

PYTHON

import sys, platform, datetime
import importlib.metadata

def environment_report(packages=("numpy", "pandas", "scikit-learn", "matplotlib")):
    print(f"Timestamp  : {datetime.datetime.now(datetime.timezone.utc).isoformat()}")
    print(f"Python     : {sys.version}")
    print(f"Platform   : {platform.platform()}")
    print(f"Processor  : {platform.processor()}")
    for pkg in packages:
        try:
            ver = importlib.metadata.version(pkg)
            print(f"  {pkg:<20} {ver}")
        except importlib.metadata.PackageNotFoundError:
            print(f"  {pkg:<20} NOT INSTALLED")

environment_report()

Run this as the first code cell of every analysis notebook. Its output is stored in the executed notebook and provides a permanent record.

Using `watermark` for One-Line Environment Reporting

The watermark IPython extension produces a compact summary:

PYTHON

%pip install watermark -q
%load_ext watermark
%watermark -v -p numpy,pandas,scikit-learn,matplotlib -m -g

Typical output:

Python implementation: CPython
Python version       : 3.11.8
IPython version      : 8.22.1

numpy      : 1.26.4
pandas     : 2.2.1
scikit-learn: 1.4.1
matplotlib : 3.8.3

Compiler    : GCC 11.4.0
OS          : Linux
Release     : 5.15.0-1051-aws
Machine     : x86_64
Git hash    : a3f9b12

The git hash ties the report to a specific commit, closing the loop between code and environment.

Code Examples

Global seed context manager for temporarily overriding the seed

A context manager that seeds all RNGs, runs code, then restores prior state — useful in testing or calling non-seed-aware third-party functions.

PYTHON

import contextlib
import random
import numpy as np

@contextlib.contextmanager
def seeded(seed: int):
    """Temporarily fix all RNG seeds, then restore previous state."""
    py_state = random.getstate()
    np_state = np.random.get_state()
    try:
        random.seed(seed)
        np.random.seed(seed)
        yield
    finally:
        random.setstate(py_state)
        np.random.set_state(np_state)

# Usage — useful in tests or when calling third-party code
with seeded(42):
    sample = np.random.choice(1000, size=10, replace=False)
    print(sample)

Output

[102 435 860 270 106 937 344 609 198  77]

Assert a model's metrics are within tolerance across runs

Demonstrates seeded training plus an approximate equality assertion that serves as a reproducibility regression test.

PYTHON

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import cross_val_score

SEED = 42
np.random.seed(SEED)

X, y = make_classification(n_samples=2000, n_features=20,
                            random_state=SEED)

model = GradientBoostingClassifier(n_estimators=100, random_state=SEED)
scores = cross_val_score(model, X, y, cv=5, scoring="roc_auc")
mean_auc = scores.mean()

print(f"CV AUC scores : {scores.round(4)}")
print(f"Mean AUC      : {mean_auc:.4f}")

# Regression guard — fails if results drift beyond 0.5%
np.testing.assert_allclose(mean_auc, 0.961, rtol=5e-3,
    err_msg="AUC regression detected — investigate environment changes")

Output

CV AUC scores : [0.963 0.958 0.962 0.960 0.961]
Mean AUC      : 0.9608

51.5 Parameterizing and Automating Notebooks with Papermill and nbconvert Intermediate

From Interactive to Automated

Jupyter notebooks start as interactive tools, but mature analyses need to be run automatically: on a schedule, with different parameters, as part of a CI pipeline, or to produce reports for multiple segments. Papermill and nbconvert are the two standard tools for this transition.

nbconvert — Rendering Notebooks to Other Formats

nbconvert converts executed (or unexecuted) notebooks to HTML, PDF, Markdown, script, or slides. It is bundled with Jupyter.

BASH

# Convert to HTML (with outputs)
jupyter nbconvert --to html --execute notebooks/01_eda.ipynb --output reports/01_eda.html

# Convert to PDF via LaTeX (requires a LaTeX installation)
jupyter nbconvert --to pdf --execute notebooks/01_eda.ipynb

# Export as a plain Python script (useful for code review)
jupyter nbconvert --to script notebooks/01_eda.ipynb

# Execute only, save executed notebook in place
jupyter nbconvert --to notebook --execute notebooks/01_eda.ipynb \
    --output notebooks/01_eda_executed.ipynb

Controlling Execution with Tags

Tag a cell raises-exception to allow expected errors without failing the conversion; tag a cell skip-execution to exclude it from automated runs. Tags are set in JupyterLab via the right-hand Property Inspector or programmatically via nbformat.

Papermill — Parameterized Notebook Execution

Papermill injects parameter values into a notebook at a designated parameters cell, then executes it and writes a new output notebook. This turns a notebook into a reusable template.

Step 1: Tag the parameters cell

In JupyterLab, create a cell with your default parameters and add the tag parameters to it:

PYTHON

# parameters  <-- this cell has the 'parameters' tag
start_date = "2024-01-01"
end_date   = "2024-12-31"
region     = "us-east"
alpha      = 0.05

Papermill injects a new cell immediately after this one, overriding the defaults with supplied values.

Step 2: Execute with Papermill

PYTHON

import papermill as pm

pm.execute_notebook(
    input_path="notebooks/02_regional_analysis.ipynb",
    output_path="reports/us-west_2025.ipynb",
    parameters={
        "start_date": "2025-01-01",
        "end_date":   "2025-06-30",
        "region":     "us-west",
        "alpha":      0.01,
    },
    kernel_name="python3",
)
print("Notebook executed successfully.")

Or from the command line:

BASH

papermill notebooks/02_regional_analysis.ipynb \
    reports/us-west_2025.ipynb \
    -p start_date 2025-01-01 \
    -p end_date   2025-06-30 \
    -p region     us-west \
    -p alpha      0.01

Step 3: Batch execution across parameter grids

PYTHON

import papermill as pm
from itertools import product
from pathlib import Path

regions = ["us-east", "us-west", "eu-central"]
years   = [2023, 2024, 2025]

Path("reports").mkdir(exist_ok=True)

for region, year in product(regions, years):
    out = f"reports/analysis_{region}_{year}.ipynb"
    pm.execute_notebook(
        input_path="notebooks/02_regional_analysis.ipynb",
        output_path=out,
        parameters={"region": region, "start_date": f"{year}-01-01",
                    "end_date": f"{year}-12-31"},
    )
    print(f"  Completed: {out}")

This pattern — a template notebook plus a parameter sweep — replaces dozens of nearly-identical notebooks with a single, maintainable source of truth.

Combining Papermill and nbconvert for a Report Pipeline

The full pattern is: execute with Papermill (produces output .ipynb with results), then render with nbconvert (produces HTML/PDF for stakeholders).

PYTHON

import papermill as pm
import subprocess
from pathlib import Path

def run_and_render(template: str, params: dict, report_stem: str) -> str:
    """Execute a notebook with params, render to HTML, return HTML path."""
    ipynb_out = f"reports/{report_stem}.ipynb"
    html_out  = f"reports/{report_stem}.html"

    pm.execute_notebook(template, ipynb_out, parameters=params)
    subprocess.run(
        ["jupyter", "nbconvert", "--to", "html", "--no-input",
         ipynb_out, "--output", html_out],
        check=True
    )
    return html_out

path = run_and_render(
    template="notebooks/02_regional_analysis.ipynb",
    params={"region": "eu-central", "start_date": "2025-01-01", "end_date": "2025-12-31"},
    report_stem="eu_central_2025"
)
print(f"Report available at: {path}")

The --no-input flag hides code cells in the HTML output, producing a clean stakeholder report. Code is still version-controlled in the template notebook; only rendered output is distributed.

Error Handling in Automated Pipelines

Papermill raises PapermillExecutionError when a notebook cell raises an exception. Catch it to get structured error information:

PYTHON

import papermill as pm
from papermill.exceptions import PapermillExecutionError

try:
    pm.execute_notebook("notebooks/fragile.ipynb", "reports/fragile_out.ipynb",
                        parameters={"data_path": "/data/missing.csv"})
except PapermillExecutionError as e:
    print(f"Notebook failed at cell {e.exec_count}: {e.ename}: {e.evalue}")
    # Log, alert, or re-raise as appropriate
    raise

Code Examples

nbconvert command-line cheat sheet

Common nbconvert invocations for different audience and use-case combinations.

BASH

# Render to HTML without code cells (stakeholder report)
jupyter nbconvert --to html --no-input --execute notebooks/analysis.ipynb \
    --output reports/analysis_report.html

# Render to HTML WITH code cells (developer reference)
jupyter nbconvert --to html --execute notebooks/analysis.ipynb \
    --output reports/analysis_with_code.html

# Export as a clean Python script for code review
jupyter nbconvert --to script notebooks/analysis.ipynb \
    --output src/analysis

# Execute notebook and save with outputs (without rendering)
jupyter nbconvert --to notebook --execute \
    --ExecutePreprocessor.timeout=600 \
    notebooks/long_running.ipynb \
    --output notebooks/long_running_executed.ipynb

Output

[NbConvertApp] Converting notebook notebooks/analysis.ipynb to html
[NbConvertApp] Writing 287488 bytes to reports/analysis_report.html

Read Papermill output notebook and extract scalar metrics

Extracts structured JSON metrics from cells tagged 'metrics' in a Papermill-executed output notebook — enables programmatic result aggregation across many notebook runs.

PYTHON

import nbformat
import json
from pathlib import Path

def extract_metrics(notebook_path: str) -> dict:

    """Extract the last JSON output of a cell tagged 'metrics' from a Papermill-executed notebook."""
    nb = nbformat.read(notebook_path, as_version=4)
    metrics = {}
    for cell in nb.cells:
        if "metrics" in cell.get("metadata", {}).get("tags", []):
            for output in cell.get("outputs", []):
                if output.output_type == "stream":
                    try:
                        metrics.update(json.loads(output.text))
                    except json.JSONDecodeError:
                        pass
    return metrics

# Example usage after a papermill run
results = extract_metrics("reports/us_east_2025.ipynb")
print(results)

Output

{'auc': 0.923, 'precision': 0.881, 'recall': 0.897, 'region': 'us-east', 'year': 2025}

51.6 Literate Programming, pre-commit Hooks, and the Reproducibility Checklist Intermediate

Literate Programming in Practice

Donald Knuth's original vision of literate programming — programs written primarily for human readers, with code as a secondary concern — maps naturally to data science notebooks. A notebook is a primary document that explains a question, works through the analysis, and interprets results, with code as the mechanistic detail.

Practically, this means:

Start with a title and abstract markdown cell that states the question being answered and the key finding.
Use section headings (##, ###) to structure the narrative as you would a written report.
Interpret every significant output immediately below it with a markdown cell. Do not leave plots or tables to speak for themselves.
Prefer named constants over magic numbers: CONFIDENCE<em>LEVEL = 0.95 is self-documenting; 0.95 scattered through code is not.
Write the conclusion first (as a markdown placeholder) to clarify the analytical goal before writing code.

Example narrative structure

## 3. Feature Importance

We use permutation importance rather than MDI (mean decrease in impurity)
because MDI is biased toward high-cardinality features (Strobl et al., 2007).

[code cell: compute permutation importance]

The top-5 features account for 73% of cumulative importance (chart above).
`prior_purchases` dominates, consistent with the customer lifetime value
hypothesis stated in Section 1.

This level of documentation means a reader who cannot run the notebook can still follow the analysis. It also forces you to articulate why each step makes sense — the single most effective guard against analytical errors.

Automating Quality Checks with pre-commit

pre-commit is a framework that runs a configurable set of checks every time you run git commit. For data science projects, a well-configured .pre-commit-config.yaml enforces notebook hygiene automatically:

YAML

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.6.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
      - id: check-json
      - id: detect-private-key     # blocks accidental secret commits
      - id: check-added-large-files
        args: ["--maxkb=5000"]

  - repo: https://github.com/kynan/nbstripout
    rev: 0.7.1
    hooks:
      - id: nbstripout

  - repo: https://github.com/psf/black
    rev: 24.4.2
    hooks:
      - id: black
        language_version: python3

  - repo: https://github.com/PyCQA/flake8
    rev: 7.0.0
    hooks:
      - id: flake8
        args: ["--max-line-length=99"]

BASH

pip install pre-commit
pre-commit install          # registers git hooks
pre-commit run --all-files  # run manually against everything

With this configuration, a developer who accidentally tries to commit a notebook with outputs will have nbstripout strip them automatically before the commit proceeds. A file containing a private key will be blocked outright.

The Reproducibility Checklist

Before publishing or handing off any analysis, verify the following:

Environment — A requirements.txt (pinned) or environment.yml is committed and up-to-date. The environment can be recreated from scratch with a single command.
Seeds — All random seeds are set at the top of every script and notebook. random</em>state parameters are passed to every stochastic scikit-learn object.
Restart & Run All — The notebook runs without errors from a fresh kernel. This was the last thing tested before committing.
Data integrity — Raw data files have checksums or are tracked by DVC. The pipeline never overwrites data/raw/.
No absolute paths — All file paths are relative to the project root or constructed from a config variable. No /home/alice/projects/... paths.
No hardcoded credentials — API keys, passwords, and tokens are loaded from environment variables or a .env file that is .gitignored.
Outputs stripped — Notebook outputs are stripped by nbstripout before committing (verify with git show HEAD -- notebooks/*.ipynb | head -40).
Metadata recorded — The first cell of every executed notebook captures Python version, key package versions, timestamp, and git hash.
Tested logic — Functions extracted to src/ have unit tests in tests/ that pass under pytest.

Closing the Loop: a Minimal CI Pipeline

A GitHub Actions workflow that enforces all of the above:

YAML

# .github/workflows/reproducibility.yml
name: Reproducibility Check
on: [push, pull_request]
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: {python-version: "3.11"}
      - name: Install dependencies
        run: pip install -r requirements.txt
      - name: Run pre-commit
        run: |
          pip install pre-commit
          pre-commit run --all-files
      - name: Execute notebooks
        run: |
          for nb in notebooks/*.ipynb; do
            jupyter nbconvert --to notebook --execute "$nb" \
              --ExecutePreprocessor.timeout=300
          done
      - name: Run tests
        run: pytest tests/ -v

This pipeline runs on every push, confirms the environment installs cleanly, enforces pre-commit hooks, executes every notebook top-to-bottom, and runs the test suite. A passing green check is strong evidence of reproducibility.

Code Examples

Install pre-commit and verify all hooks pass

Installs and exercises pre-commit hooks across the entire repository.

BASH

pip install pre-commit

# Install git hooks
pre-commit install

# Run against all files (useful before first commit or after adding hooks)
pre-commit run --all-files

# Update all hook versions to latest
pre-commit autoupdate

# Skip hooks for a single commit (use sparingly, document why)
git commit --no-verify -m "WIP: skip hooks for draft"

Output

Trim Trailing Whitespace.................................................Passed
Fix End of Files.........................................................Passed
Check Yaml...............................................................Passed
nbstripout...............................................................Passed
black....................................................................Passed
flake8...................................................................Passed

Notebook-level reproducibility header (paste as cell 1 in every notebook)

A self-contained reproducibility header that seeds all RNGs and prints a timestamped environment snapshot. Copy into cell 1 of every notebook.

PYTHON

# Cell 1 of every notebook — reproducibility header
import sys, platform, datetime, subprocess, importlib.metadata

PROJECT_PACKAGES = ["numpy", "pandas", "scikit-learn", "matplotlib", "papermill"]
RANDOM_SEED = 42

# --- Seed everything ---
import random, numpy as np
random.seed(RANDOM_SEED)
np.random.seed(RANDOM_SEED)

# --- Environment snapshot ---
try:
    git_hash = subprocess.check_output(
        ["git", "rev-parse", "--short", "HEAD"], text=True
    ).strip()
except Exception:
    git_hash = "(not a git repo)"

print(f"{'='*60}")
print(f"Run timestamp : {datetime.datetime.now(datetime.timezone.utc).isoformat()}")
print(f"Python        : {sys.version.split()[0]}")
print(f"Platform      : {platform.platform()}")
print(f"Git commit    : {git_hash}")
print(f"Random seed   : {RANDOM_SEED}")
print(f"{'='*60}")
for pkg in PROJECT_PACKAGES:
    try:
        print(f"  {pkg:<20} {importlib.metadata.version(pkg)}")
    except importlib.metadata.PackageNotFoundError:
        print(f"  {pkg:<20} NOT FOUND")

Output

============================================================
Run timestamp : 2025-09-15T14:23:07.123456+00:00
Python        : 3.11.8
Platform      : Linux-5.15.0-1051-aws-x86_64
Git commit    : a3f9b12
Random seed   : 42
============================================================
  numpy                1.26.4
  pandas               2.2.1
  scikit-learn         1.4.1
  matplotlib           3.8.3
  papermill            2.6.0

Programmatic reproducibility audit across all notebooks in a directory

Scans all notebooks for uncommitted outputs and hardcoded absolute file paths — suitable for use in CI or as a pre-push check.

PYTHON

import nbformat
from pathlib import Path

def audit_notebooks(nb_dir: str) -> None:
    """Check that notebooks have no outputs and do not use absolute paths."""
    issues = []
    for path in sorted(Path(nb_dir).glob("*.ipynb")):
        nb = nbformat.read(path, as_version=4)
        has_output = any(
            cell.get("outputs") or cell.get("execution_count")
            for cell in nb.cells if cell.cell_type == "code"
        )
        abs_paths = [
            (i+1, line.strip())
            for i, cell in enumerate(nb.cells)
            if cell.cell_type == "code"
            for line in cell.source.splitlines()
            if line.strip().startswith(("/home", "/Users", "C:\\", "C:/"))
        ]
        if has_output:
            issues.append(f"  {path.name}: has uncommitted outputs (run nbstripout)")
        for cell_num, line in abs_paths:
            issues.append(f"  {path.name} cell ~{cell_num}: absolute path: {line}")
    
    if issues:
        print("Reproducibility issues found:")
        for issue in issues:
            print(issue)
    else:
        print(f"All {len(list(Path(nb_dir).glob('*.ipynb')))} notebooks passed audit.")

audit_notebooks("notebooks")

Output

All 4 notebooks passed audit.

Learning Objectives

51.1 Jupyter Notebooks: Strengths, Pitfalls, and Best Practices Beginner

What Notebooks Are Good At

The Hidden-State Problem

Cell Organization and Notebook Hygiene

Keep notebooks linear

One notebook, one narrative

Use %autoreload during development

Markdown cells are first-class

Outputs and Version Control

Code Examples

51.2 Virtual Environments, Dependency Pinning, and Environment Management Beginner

Why Isolation Matters

Python venv — Lightweight and Built-In

Dependency Pinning with pip

Conda — When You Need Non-Python Dependencies

Poetry — Modern Dependency Management

Registering the Kernel with Jupyter

Capturing the Full Environment Snapshot

Code Examples

51.3 Project Structure and Version Control for Data Science Intermediate

Why Structure Matters

A Reference Directory Layout

The .gitignore Contract

Version Control for Notebooks with Jupytext

Version Control for Data with DVC

Checksums for Lightweight Data Integrity

Code Examples

51.4 Seeding, Determinism, and Capturing Environment Metadata Intermediate

Sources of Non-Determinism

Seeding Random Number Generators

Numerical Reproducibility and Its Limits

Capturing Environment Metadata

Using watermark for One-Line Environment Reporting

Code Examples

51.5 Parameterizing and Automating Notebooks with Papermill and nbconvert Intermediate

From Interactive to Automated

nbconvert — Rendering Notebooks to Other Formats

Controlling Execution with Tags

Papermill — Parameterized Notebook Execution

Step 1: Tag the parameters cell

Step 2: Execute with Papermill

Step 3: Batch execution across parameter grids

Combining Papermill and nbconvert for a Report Pipeline

Error Handling in Automated Pipelines

Code Examples

51.6 Literate Programming, pre-commit Hooks, and the Reproducibility Checklist Intermediate

Literate Programming in Practice

Example narrative structure

Automating Quality Checks with pre-commit

The Reproducibility Checklist

Closing the Loop: a Minimal CI Pipeline

Code Examples

Use `%autoreload` during development

Python `venv` — Lightweight and Built-In

The `.gitignore` Contract

Using `watermark` for One-Line Environment Reporting