Feature Engineering and Preprocessing Pipelines

On tabular business data, the quality of the features usually matters more than the choice of algorithm. A gradient-boosted model fed well-designed features (time since last purchase, ratio of spend to income, a properly encoded merchant category) routinely beats a fancier model fed raw columns. Feature engineering is the craft of turning raw records into numbers that expose the signal, and preprocessing is the discipline of doing it identically, and without leakage, at training time and in production. This lesson covers both.

Intuition: translating for the model

A model only sees numbers in a matrix. Everything you know about the domain has to be translated into that matrix, and every model has its own "reading habits." Linear models read each feature as a straight-line effect and care about scale. Distance-based models (k-NN, SVMs with RBF kernels, k-means) read Euclidean geometry, so a feature measured in thousands drowns one measured in fractions. Tree models read only orderings and thresholds, so they shrug off scaling but cannot easily see a ratio or a difference between two columns unless you compute it for them. Good feature engineering is writing for your reader.

Numeric features

Scaling and normalization

• Standardization (z-score): $x' = (x - \mu)/\sigma$, where $\mu$ and $\sigma$ are the training mean and standard deviation. Result: mean 0, standard deviation 1. The default for linear models, SVMs, k-NN, PCA and neural networks.
• Min-max scaling: $x' = (x - x_{\min})/(x_{\max} - x_{\min})$, mapping training data to $[0,1]$. Useful for bounded inputs such as pixel intensities; fragile with outliers.
• Robust scaling: $x' = (x - \text{median})/\text{IQR}$, where IQR is the interquartile range. Resistant to outliers.
• Log and power transforms: $x' = \log(1 + x)$ for right-skewed non-negative quantities (income, counts, prices); Yeo-Johnson or Box-Cox (PowerTransformer) to make distributions more Gaussian.
• Quantile/rank transform: maps values to their quantiles; very robust but discards distances.

Normalization is overloaded: in scikit-learn, Normalizer rescales each row to unit norm (useful for text vectors and embeddings), which is different from scaling columns.

Worked example. Training incomes (thousands): 30, 40, 50, 60, 220. Mean $= 80$, population standard deviation $= \sqrt{(50^2+40^2+30^2+20^2+140^2)/5} = \sqrt{5000} \approx 70.7$. Standardized: $-0.71, -0.57, -0.42, -0.28, 1.98$. The single high earner squeezes everyone else into a narrow band. After $\log(1+x)$, the values become $3.43, 3.71, 3.93, 4.11, 5.40$, much more even. Which model needs which: tree ensembles need none of this; regularized linear models and neural networks need it badly, because the penalty and the optimizer both assume features are on comparable scales.

Outliers

First ask whether an outlier is an error (a typo, a sensor glitch, a placeholder like 99999) or a real extreme (a whale customer). Fix or null errors. For real extremes, options are clipping to percentiles (winsorizing, for example at the 1st and 99th), a log transform, robust scaling, or robust losses (Huber, MAE). Never delete outliers from the test set to make scores look better; production data will contain them.

Binning and crosses

Binning (KBinsDiscretizer) turns a numeric feature into intervals, letting a linear model represent non-linear effects at the cost of resolution. A feature cross combines features so a model can capture interactions it could not otherwise represent: a product $x_1 x_2$, a ratio (debt/income), a difference (days between signup and first purchase), or a categorical cross ("city × device"). Linear models need crosses explicitly; trees can find interactions but still benefit from well-chosen ratios, which they can only approximate with many splits.

Categorical features

One-hot encoding

Replace a category with $K$ levels by $K$ binary indicator columns (or $K-1$, dropping one level, to avoid perfect collinearity with an intercept in unregularized linear models). One-hot is exact and leak-free but grows with cardinality: a ZIP-code column with 30,000 levels creates 30,000 sparse columns, most with only a few training examples. Practical controls in scikit-learn's OneHotEncoder: handle_unknown="ignore" so unseen categories at prediction time become all-zeros instead of crashing, and min_frequency / max_categories to group rare levels into an "infrequent" bucket.

Ordinal encoding maps categories to integers. It is appropriate when an order exists (small < medium < large) and acceptable for tree models even without one, but misleading for linear models when no order exists.

Target encoding and its leakage risk

Target encoding (also called mean encoding) replaces each category with the average target value for that category. To avoid trusting tiny categories, blend with the global mean $\bar y$ using a smoothing strength $m$:

$\text{enc}(c) = \frac{n_c \, \bar y_c + m \, \bar y}{n_c + m},$

where $n_c$ is the number of training rows with category $c$ and $\bar y_c$ their mean target.

Worked example. Churn rate overall is $\bar y = 0.20$. City "Springfield" has $n_c = 3$ customers, 2 of whom churned, so $\bar y_c = 0.667$. With $m = 10$: $\text{enc} = (3 \times 0.667 + 10 \times 0.20)/(3 + 10) = (2 + 2)/13 \approx 0.31$. A city with 3,000 customers and the same raw rate would get almost exactly 0.667.

The leakage. If you compute the encoding on the training set and then train on those same rows, each row's own label is baked into its feature. A category seen once gets an encoding equal to its own label, a perfect "predictor" during training that is worthless on new data. The model learns to trust the encoded column far too much and validation scores look great until production. The fix is cross-fitting: split the training data into folds and encode each fold's rows using statistics computed only from the other folds. scikit-learn's TargetEncoder does this automatically inside fit_transform, while transform on new data uses encodings learned from the full training set. This is why you must call fit_transform (not fit then transform) on training data, and why target encoding must sit inside the cross-validated pipeline, never be precomputed on the whole dataset.

Other options for high-cardinality categories: frequency/count encoding (no target used, so no target leakage), hashing (FeatureHasher, fixed width, collisions), and learned embeddings in neural networks.

Missing values

Missingness is often informative: a blank "previous loan amount" usually means "no previous loan." Strategies:

• Simple imputation: median for numeric (robust to skew), most frequent or a constant "missing" category for categoricals.
• Missingness indicators: add a binary column flagging that the value was missing (SimpleImputer(add_indicator=True)), so the model can learn from the fact of missingness.
• Model-based imputation: KNNImputer or IterativeImputer predict missing values from other features; more accurate in some settings, slower, and still must be fitted on training data only.
• Native handling: histogram gradient boosting in scikit-learn, XGBoost and LightGBM learn which branch missing values should take, so imputation is optional.

Understand why data is missing. If it is missing completely at random, simple methods work. If missingness depends on the value itself (high earners skip the income question), imputation with the median biases the feature, and the indicator becomes important.

Date, time and text features

Dates and times

A raw timestamp is nearly useless to most models. Extract what matters: year, month, day of week, hour, is-weekend, is-holiday, and especially elapsed times (days since signup, hours since last login, age of account). For cyclical quantities, encode with sine and cosine so that the end of the cycle is next to the start:

$\text{hour}_{\sin} = \sin\!\left(\frac{2\pi h}{24}\right), \qquad \text{hour}_{\cos} = \cos\!\left(\frac{2\pi h}{24}\right).$

With this encoding, 23:00 and 01:00 are close (as they are in reality), whereas the raw integers 23 and 1 are far apart. Tree models can often use the raw hour fine; linear and distance-based models need the cyclical form. For time-dependent data, compute aggregates (rolling 7-day spend) using only information available before the prediction time, or you leak the future.

Text: bag-of-words and TF-IDF

Bag-of-words represents a document by counts of each vocabulary term, ignoring order. TF-IDF reweights the counts so that words common across all documents (like "the") count less. scikit-learn's default (smooth_idf=True) uses

$\text{idf}(t) = \ln\frac{1 + N}{1 + \text{df}(t)} + 1, \qquad \text{tfidf}(t,d) = \text{tf}(t,d) \cdot \text{idf}(t),$

where $N$ is the number of documents, $\text{df}(t)$ the number containing term $t$, and $\text{tf}(t,d)$ the raw count of $t$ in $d$. Each document vector is then scaled to unit L2 norm.

Worked example. Three documents: d1 "cheap flights cheap hotels", d2 "cheap flights to paris", d3 "paris hotels reviews". $N = 3$.

• "cheap", "flights", "hotels", "paris" each appear in 2 documents: $\text{idf} = \ln(4/3) + 1 \approx 1.288$.
• "to" and "reviews" appear in 1 document: $\text{idf} = \ln(4/2) + 1 \approx 1.693$.

For d1: cheap $= 2 \times 1.288 = 2.575$, flights $= 1.288$, hotels $= 1.288$. The L2 norm is $\sqrt{2.575^2 + 1.288^2 + 1.288^2} = \sqrt{9.949} \approx 3.154$. Normalized: cheap $\approx 0.816$, flights $\approx 0.408$, hotels $\approx 0.408$, others 0. Useful options include ngram_range=(1,2) for word pairs, min_df to drop rare terms, and sublinear_tf=True to use $1 + \ln(\text{tf})$. TF-IDF with a linear model remains a strong, fast baseline for text classification; dense embeddings from pretrained language models (covered in the NLP and RAG lessons) usually win on semantic tasks.

Feature selection

More features are not always better: irrelevant features add variance and cost, and redundant ones complicate interpretation. Three families:

• Filter methods: score each feature independently (variance threshold, correlation, mutual information, chi-squared) and keep the top ones. Fast, but blind to interactions.
• Wrapper methods: search over subsets using model performance, e.g. recursive feature elimination (RFECV) or sequential selection. Expensive, and prone to overfitting the validation data.
• Embedded methods: selection as a side effect of training, e.g. L1 (lasso) regularization driving coefficients to zero, or tree-based importances.

Prefer permutation importance on validation data over impurity-based tree importances, which are biased toward high-cardinality features. As with every learned step, selection must happen inside cross-validation; selecting features on all the data and then cross-validating is a classic leak that can make pure noise look predictive.

Pipelines and ColumnTransformer

Every transformation above learns parameters (means, categories, IDF weights, encodings) from data, so it must be fitted on training data only and applied unchanged to validation, test and production data. scikit-learn's Pipeline chains steps so that fit fits each step on the training data and predict applies them in order; ColumnTransformer routes different columns to different preprocessing.

import numpy as np, pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder, TargetEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

rng = np.random.default_rng(0); n = 2000
df = pd.DataFrame({
    "income": rng.lognormal(10, 0.5, n), "age": rng.integers(18, 80, n).astype(float),
    "plan": rng.choice(["basic", "plus", "pro"], n), "city": rng.choice([f"c{i}" for i in range(300)], n)})
df.loc[rng.random(n) < 0.1, "income"] = np.nan
y = (rng.random(n) < 0.2 + 0.1 * (df["plan"] == "basic")).astype(int)

num = Pipeline([("impute", SimpleImputer(strategy="median", add_indicator=True)),
                ("scale", StandardScaler())])
pre = ColumnTransformer([
    ("num", num, ["income", "age"]),
    ("onehot", OneHotEncoder(handle_unknown="ignore"), ["plan"]),
    ("target", TargetEncoder(), ["city"])])          # cross-fitted internally
model = Pipeline([("pre", pre), ("clf", LogisticRegression(max_iter=1000))])
print(cross_val_score(model, df, y, cv=5, scoring="roc_auc").mean())

Because the whole pipeline is passed to cross_val_score, the imputer's medians, the scaler's statistics, the one-hot categories and the target encodings are re-learned inside every training fold. The same fitted object can be saved and served, guaranteeing identical preprocessing in production. Grid search can tune preprocessing too, using names like pre__target__smooth.

Handling imbalanced data

When one class is rare (fraud, churn, defects), a model minimizing average loss can ignore it. Options:

• Change the metric and threshold first. Often the model ranks well and only the default 0.5 threshold is wrong; see the evaluation lesson.
• Class weights: scale each example's loss by a class weight. scikit-learn's class_weight="balanced" sets $w_c = \frac{n}{K \, n_c}$ for $K$ classes. With 950 negatives and 50 positives: $w_0 = 1000/(2 \times 950) \approx 0.53$ and $w_1 = 1000/(2 \times 50) = 10$, so each positive counts about 19 times as much as a negative.
• Random undersampling of the majority class: fast and simple, discards data.
• Random oversampling of the minority class: duplicates examples, risks overfitting them.
• SMOTE (Chawla et al., 2002) creates synthetic minority examples by interpolating between a minority point and one of its minority-class nearest neighbours. Available in the imbalanced-learn package, whose own Pipeline applies resampling only during fit.

Two rules matter more than the choice of method. First, resample only the training folds, never validation or test data, which must reflect the real class ratio; oversampling before splitting puts copies of the same example on both sides and inflates scores. Second, weighting and resampling distort predicted probabilities (the model now believes positives are more common), so recalibrate or adjust thresholds if you use the probabilities. With strong modern learners such as gradient boosting, class weights plus threshold tuning frequently perform as well as synthetic oversampling, so try the simple approach first.

Common pitfalls

• Fitting preprocessing on the full dataset before splitting (scalers, imputers, encoders, feature selection, PCA): a subtle but real leak.
• Target encoding without cross-fitting, or computing encodings on data that includes the validation fold.
• Future leakage in features: using information not available at prediction time, such as "total refunds this month" for a prediction made mid-month.
• Crashing on unseen categories in production because the encoder was not configured to ignore or bucket them.
• Resampling before the split, or evaluating on a resampled test set.
• Dropping rows with missing values by default, which can discard most of the data and bias the rest.
• Scaling for trees (harmless, wasted effort) while forgetting to scale for linear models, SVMs and neural networks (harmful).

In practice

• Start with a baseline pipeline: median imputation plus indicators, standard scaling, one-hot for low-cardinality and target or frequency encoding for high-cardinality categoricals, and a gradient-boosting or regularized linear model. Measure, then add engineered features one hypothesis at a time.
• Get domain features from domain experts: ratios, recency/frequency/monetary aggregates, and elapsed times are the workhorses of tabular ML.
• Keep all preprocessing in a single serialized pipeline (or a feature store with point-in-time correct joins) so training and serving cannot diverge; training/serving skew is one of the most common production bugs.
• Track data and feature versions alongside models, covered in the MLOps lessons.
• For text, images and audio, feature engineering has largely been replaced by learned representations; for tabular data it remains a core skill.