Feature Scaling and Normalization

Overview
Feature scaling and normalization are preprocessing techniques used in Feature Engineering to standardize the range or distribution of features. These techniques ensure that features are on a similar scale, allowing machine learning models to perform optimally.

Why Feature Scaling and Normalization?

• Prevent Biased Learning: Some machine learning algorithms are sensitive to the scale of features. Without scaling, features with larger magnitudes may dominate the learning process, biasing the model towards those features.
• Improve Convergence: Scaling can help accelerate the convergence of gradient-based optimization algorithms, such as gradient descent.
• Ensure Fair Comparisons: Scaling features allows for fair comparisons between different units or measurement scales, avoiding skewed interpretations of feature importance.

Common Techniques for Feature Scaling and Normalization

Min-Max Scaling (Normalization)

• Rescales features to a specific range, typically between 0 and 1.
• Formula: X_scaled = (X — X_min) / (X_max — X_min), where X is the original feature, X_scaled is the scaled feature, X_min is the minimum value of X, and X_max is the maximum value of X.
• Suitable for features with a bounded range and no significant outliers.

Dataset before Min-Max Scaling:

Dataset after Min-Max Scaling:

In this example, min-max scaling rescales the values of each feature to a specific range (typically between 0 and 1). The minimum value in the original dataset becomes 0, and the maximum value becomes 1. The other values are scaled proportionally within this range.

Standardization (Z-score Scaling)

• Transforms features to have zero mean and unit variance.
• Formula: X_scaled = (X — X_mean) / X_std, where X is the original feature, X_scaled is the scaled feature, X_mean is the mean of X, and X_std is the standard deviation of X.
• Suitable for features with unknown or non-normal distributions.

Dataset before Standardization:

Dataset after Standardization:

In this example, standardization (Z-score scaling) transforms the values of each feature to have a mean of 0 and a standard deviation of 1. The original values are shifted and rescaled according to the mean and standard deviation of the feature.

Robust Scaling

• Rescales features by subtracting the median and dividing by the interquartile range (IQR).
• Formula: X_scaled = (X — X_median) / IQR, where X is the original feature, X_scaled is the scaled feature, X_median is the median of X, and IQR is the interquartile range.
• Robust to the presence of outliers, making it suitable for features with outliers or skewed distributions.

Dataset before Robust Scaling:

Dataset after Robust Scaling:

In this example, robust scaling rescales the values of each feature by removing the median and dividing by the interquartile range (IQR). Robust scaling is less sensitive to outliers compared to other scaling techniques.

Log Transformation

• Applies the logarithm function to the features, which can help normalize skewed distributions.
• Particularly useful for features with highly skewed or long-tailed distributions.

Dataset before Log Transformation:

Dataset after Log Transformation:

In this example, the log transformation is applied to each value in the dataset by taking the logarithm with base 10. The resulting values represent the logarithmic scale of the original data.

Considerations and Best Practices

• Feature Scaling: Apply feature scaling before feeding the data into the machine learning model, except for algorithms that are scale-invariant, such as decision trees.
• Maintain Consistency: Ensure that the scaling is performed consistently across training, validation, and testing datasets to avoid introducing bias.
• Avoid Data Leakage: Scale features based on the training data statistics to prevent information leakage from the test set.
• Domain Knowledge: Consider the specific characteristics of the problem domain when selecting the appropriate scaling technique.
• Evaluate Impact: Assess the impact of scaling on the model’s performance through cross-validation or other evaluation methods.
• Remember, the choice of feature scaling technique depends on the nature of the data and the requirements of the machine learning algorithm being used.