Why Select Features?
More features are not always better. Irrelevant or redundant features can:
- Increase training and inference time.
- Introduce noise that harms generalization (the curse of dimensionality is especially acute for distance-based methods).
- Make models harder to interpret and audit.
- Increase storage and serialization costs in production.
Feature selection answers the question: given a candidate set of features, which subset should we use? The three major families — filter, wrapper, and embedded — trade off computational cost against fidelity to the model being trained.
Filter Methods
Filter methods score features independently of any model, using a statistical measure of relevance to the target.
Variance Threshold
Removes features whose variance falls below a threshold — a cheap way to discard near-constant columns:
from sklearn.feature_selection import VarianceThreshold
import numpy as np
X = np.array([[0, 1, 0.5], [0, 0, 0.6], [0, 1, 0.4]])
sel = VarianceThreshold(threshold=0.01)
print(sel.fit_transform(X)) # drops column 0 (all zeros)
Mutual information (MI) measures the nonlinear statistical dependence between a feature $X_j$ and target $Y$:
$$I(X_j; Y) = \sum_{x,y} p(x,y) \log \frac{p(x,y)}{p(x)p(y)}$$
Unlike correlation, MI detects non-linear relationships and interactions. SelectKBest with mutual<em>info</em>classif or mutual<em>info</em>regression selects the top $k$ features by MI score:
from sklearn.feature_selection import SelectKBest, mutual_info_regression
from sklearn.datasets import fetch_california_housing
import pandas as pd
data = fetch_california_housing()
X, y = data.data, data.target
features = data.feature_names
sel = SelectKBest(mutual_info_regression, k=4)
sel.fit(X, y)
scores = pd.Series(sel.scores_, index=features).sort_values(ascending=False)
print(scores.round(3))
print("\nSelected:", [features[i] for i in sel.get_support(indices=True)])
Correlation-Based Filtering
Remove features that are highly correlated with each other (redundancy) or have very low correlation with the target (irrelevance). A practical implementation:
import pandas as pd
import numpy as np
def drop_correlated(df, threshold=0.95):
corr = df.corr().abs()
upper = corr.where(np.triu(np.ones(corr.shape), k=1).astype(bool))
to_drop = [col for col in upper.columns if any(upper[col] > threshold)]
return df.drop(columns=to_drop), to_drop
Wrapper Methods
Wrapper methods evaluate feature subsets by actually training a model and measuring validation performance. They are model-aware but expensive.
Recursive Feature Elimination (RFE)
RFE fits the model, removes the feature(s) with the lowest importance, and repeats until the desired number of features is reached:
from sklearn.feature_selection import RFE
from sklearn.linear_model import Ridge
from sklearn.datasets import fetch_california_housing
data = fetch_california_housing()
X, y = data.data, data.target
rfe = RFE(estimator=Ridge(), n_features_to_select=4, step=1)
rfe.fit(X, y)
print(dict(zip(data.feature_names, rfe.ranking_)))
print("Selected:", [f for f, s in zip(data.feature_names, rfe.support_) if s])
RFECV extends this by choosing the optimal feature count via cross-validation, avoiding the need to specify n<em>features</em>to<em>select manually.
Embedded Methods
Embedded methods perform feature selection as part of model training. They are typically more computationally efficient than wrappers because the model is trained once.
Lasso (L1 Regularization)
The Lasso penalty shrinks coefficients of uninformative features exactly to zero:
$$\min_{\beta} \|y - X\beta\|_2^2 + \lambda \|\beta\|_1$$
from sklearn.linear_model import LassoCV
from sklearn.feature_selection import SelectFromModel
from sklearn.datasets import fetch_california_housing
import numpy as np
data = fetch_california_housing()
X, y = data.data, data.target
lasso_cv = LassoCV(cv=5, random_state=0).fit(X, y)
sel = SelectFromModel(lasso_cv, prefit=True)
X_reduced = sel.transform(X)
print(f"Original features: {X.shape[1]}, After Lasso selection: {X_reduced.shape[1]}")
print("Non-zero coefs:", np.sum(lasso_cv.coef_ != 0))
Tree-Based Feature Importance
Gradient boosted trees and random forests compute feature importances as the total impurity reduction attributed to each feature across all splits. SelectFromModel with a tree estimator is a fast, non-linear alternative to Lasso:
from sklearn.ensemble import RandomForestRegressor
from sklearn.feature_selection import SelectFromModel
from sklearn.datasets import fetch_california_housing
import pandas as pd
data = fetch_california_housing()
X, y = data.data, data.target
rf = RandomForestRegressor(n_estimators=100, random_state=0, n_jobs=-1)
rf.fit(X, y)
importances = pd.Series(rf.feature_importances_, index=data.feature_names)
print(importances.sort_values(ascending=False).round(4))
Pitfall: The default impurity-based importance is biased toward high-cardinality continuous features. Use permutation</em>importance for a model-agnostic, bias-free estimate at the cost of additional computation.
Feature Stores: Operationalizing Feature Engineering
A feature store is a centralized repository that stores, versions, and serves computed features for both model training and online inference. It solves three problems at scale:
- Training-serving skew: Features are computed once and stored; the exact same values are used in training and served at prediction time.
- Reuse: Features computed for one model are available to any other model, avoiding duplicate computation.
- Point-in-time correctness: When training on historical data, a feature store ensures only features available before each label's timestamp are used — preventing temporal leakage.
Popular options include Feast (open-source), Tecton, Hopsworks, and Vertex AI Feature Store. Even without a formal feature store, the same principles apply: keep feature computation code versioned and separate from model code, and never recompute features at serving time using data not available at the prediction timestamp.
Code Examples
RFECV: Automatic Feature Count Selection
Uses RFECV with Ridge regression to automatically determine the optimal number of features on the California housing dataset.
import numpy as np
import matplotlib.pyplot as plt
from sklearn.feature_selection import RFECV
from sklearn.linear_model import Ridge
from sklearn.datasets import fetch_california_housing
from sklearn.preprocessing import StandardScaler
data = fetch_california_housing()
X = StandardScaler().fit_transform(data.data)
y = data.target
rfecv = RFECV(
estimator=Ridge(),
step=1,
cv=5,
scoring="r2",
min_features_to_select=1,
n_jobs=-1,
)
rfecv.fit(X, y)
print(f"Optimal feature count: {rfecv.n_features_}")
print(f"Selected features: {[data.feature_names[i] for i in rfecv.get_support(indices=True)]}")
plt.plot(range(1, len(rfecv.cv_results_['mean_test_score']) + 1),
rfecv.cv_results_['mean_test_score'])
plt.xlabel("Number of features")
plt.ylabel("CV R²")
plt.title("RFECV: Performance vs Feature Count")
plt.axvline(rfecv.n_features_, color="r", linestyle="--", label=f"Optimal = {rfecv.n_features_}")
plt.legend()
plt.show()
OutputOptimal feature count: 7
Selected features: ['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms', 'Population', 'AveOccup', 'Latitude']
Permutation Importance vs. Impurity Importance
Compares impurity-based and permutation-based importances to illustrate bias toward high-cardinality features in the impurity method.
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(7)
n = 1000
# Two informative binary features, one high-cardinality uninformative feature
X = pd.DataFrame({
"useful_A": rng.integers(0, 2, size=n),
"useful_B": rng.integers(0, 2, size=n),
"noise_hicardinal": rng.uniform(0, 1, size=n), # continuous noise
})
y = (X["useful_A"] ^ X["useful_B"]).values
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
rf = RandomForestClassifier(n_estimators=200, random_state=0).fit(X_tr, y_tr)
impurity = pd.Series(rf.feature_importances_, index=X.columns)
perm = permutation_importance(rf, X_te, y_te, n_repeats=30, random_state=0)
perm_means = pd.Series(perm.importances_mean, index=X.columns)
print("Impurity importance:")
print(impurity.sort_values(ascending=False).round(4))
print("\nPermutation importance:")
print(perm_means.sort_values(ascending=False).round(4))
OutputImpurity importance:
noise_hicardinal 0.4521
useful_A 0.2751
useful_B 0.2728
Permutation importance:
useful_A 0.2183
useful_B 0.2171
noise_hicardinal -0.0011
SelectFromModel with Lasso in a Pipeline
Embeds Lasso-based feature selection inside a Pipeline so selection is re-fit on each CV fold.
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LassoCV, Ridge
from sklearn.feature_selection import SelectFromModel
from sklearn.model_selection import cross_val_score
from sklearn.datasets import fetch_california_housing
data = fetch_california_housing()
X, y = data.data, data.target
pipe = Pipeline([
("scale", StandardScaler()),
("lasso_sel", SelectFromModel(LassoCV(cv=3, random_state=0))),
("ridge", Ridge()),
])
scores = cross_val_score(pipe, X, y, cv=5, scoring="r2")
print(f"CV R²: {scores.round(3)}")
print(f"Mean: {scores.mean():.3f}")
OutputCV R²: [0.614 0.601 0.589 0.633 0.617]
Mean: 0.611