Normalization & Scaling

Features on different scales pull a model toward the loud columns, so the work is to put them on a shared scale without inventing signal. This page compares normalization and scaling choices, including when to standardize versus normalize.

The same notes are in BATCH NORMALIZATION, Data Normalization/Augmentation, KNN, LDA - Linear discriminant analysis, and Regression.

The problem is easiest to see on real columns. A comparison of normalization / scaling techniques in sklearn compares the effect of different scalers on data with outliers, using two California Housing features, median income in a block and average house occupancy, that have very different scales and contain some very large outliers. Ben Alex Keen's Feature Scaling with scikit-learn is Another great explanation on sklearn and (general) scaling - normal, min max, etc. Jason Brownlee's Normalization/standardize features does the same in Weka, starting from the point that raw data often has attributes with varying scales, one in kilograms and another a count, and that carefully choosing a rescaling method can often give a boost in performance.

Because data has varying scales, the choice is between two targets. Normalize to a range of 0 to 1 when the algorithm you are using does not make assumptions about the distribution of your data, such as k-nearest neighbors and artificial neural networks. Standardize to a mean of 0 and a std of 1 when the algorithm assumes a Gaussian dist, such as linear regression, logistic regression and linear discriminant analysis (LR, LogR, LDA).

Generally, it is a good idea to standardize data that has a Gaussian (bell curve) distribution and normalize otherwise. In general terms, we should test 0,1 or -1,1 empirically and possibly match the range to the NN gates/activation function etc.