From Sequences to Supervised Learning
The first and most consequential design decision in ML forecasting is how to flatten a time series into a feature matrix. The core transformation is straightforward: given a series $y_1, y_2, \ldots, y_T$, we create a row for each time step $t$ whose feature vector contains past values (lags), statistics computed over rolling windows, and deterministic calendar variables. The target is the value we want to predict, $y_{t+h}$, where $h$ is the forecast horizon.
Formally, denote by $\mathcal{F}(t)$ the feature map:
where $\bar{y}_{t,w}$ is the rolling mean over a window of width $w$ ending at $t-1$, and $\sigma_{t,w}$ is the corresponding standard deviation. The critical constraint — easily violated — is that every feature must be computable from information available strictly before time $t$. Rolling windows must be shifted (lagged) so the window never overlaps the prediction target.
Lag Selection
Choosing which lags to include is part science, part domain knowledge. Plot the partial autocorrelation function (PACF) to identify statistically significant lags. For a daily retail series, lags 1, 7, 14, 28, and 365 are natural starting points. For financial returns, lag 1 often contains the only meaningful autocorrelation. Including too many lags increases dimensionality and can cause overfitting; gradient boosted trees mitigate this somewhat through feature importance pruning, but excessive noise features still hurt.
import pandas as pd
import numpy as np
def make_lag_features(series: pd.Series, lags: list[int]) -> pd.DataFrame:
"""Create lag features. Series must have a DatetimeIndex."""
df = pd.DataFrame({"y": series})
for lag in lags:
df[f"lag_{lag}"] = series.shift(lag)
return df
def make_rolling_features(
series: pd.Series,
windows: list[int],
lag: int = 1, # shift to avoid leakage
) -> pd.DataFrame:
"""Rolling mean and std, shifted by `lag` to prevent leakage."""
df = pd.DataFrame()
for w in windows:
rolled = series.shift(lag).rolling(w)
df[f"rmean_{w}"] = rolled.mean()
df[f"rstd_{w}"] = rolled.std()
return df
def make_calendar_features(index: pd.DatetimeIndex) -> pd.DataFrame:
df = pd.DataFrame(index=index)
df["dayofweek"] = index.dayofweek # 0=Mon
df["month"] = index.month
df["dayofyear"] = index.dayofyear
df["weekofyear"] = index.isocalendar().week.astype(int)
# Cyclical encoding avoids the discontinuity at year / week boundaries
df["dow_sin"] = np.sin(2 * np.pi * df["dayofweek"] / 7)
df["dow_cos"] = np.cos(2 * np.pi * df["dayofweek"] / 7)
df["month_sin"] = np.sin(2 * np.pi * (df["month"] - 1) / 12)
df["month_cos"] = np.cos(2 * np.pi * (df["month"] - 1) / 12)
return dfCyclical Encoding
Day-of-week encoded as an integer (0–6) misleads tree models into thinking Friday (4) is "close" to Saturday (5) but "far" from Sunday (6), breaking the circular topology of the week. Sine/cosine projection onto the unit circle resolves this: Monday and Sunday become equidistant from each other.
The Leakage Checklist
Before training, verify each feature column:
- Rolling statistics: Was
.shift(1)applied before.rolling(w).mean()? If not, the window includes <!--MATHBLOCK11--> itself. - Target-derived features: Never use the test-set target to compute means or normalisation constants (target encoding leakage).
- External features: A "tomorrow's temperature" column is a future covariate — valid only if you genuinely have it at forecast time (e.g., a weather forecast made at time <!--MATHBLOCK12-->).
fillnaorder: Filling NaN lag values with future information (forward-fill applied before the train/test split) is a subtle but common mistake.
Assembling the Training Matrix
from sklearn.pipeline import Pipeline
def build_supervised(
series: pd.Series,
lags: list[int],
windows: list[int],
horizon: int = 1,
) -> tuple[pd.DataFrame, pd.Series]:
"""Build (X, y) for single-step direct forecasting."""
lag_df = make_lag_features(series, lags)
roll_df = make_rolling_features(series, windows)
cal_df = make_calendar_features(series.index)
X = pd.concat([lag_df.drop(columns="y"), roll_df, cal_df], axis=1)
y = series.shift(-horizon) # target is h steps ahead
mask = X.notna().all(axis=1) & y.notna()
return X[mask], y[mask]This function is the workhorse. Note that series.shift(-horizon) shifts the target forward by horizon — the feature row at time $t$ predicts $y_{t+h}$. After dropping rows with any NaN (caused by the initial lags), we have a clean training matrix ready for any sklearn-compatible estimator.