Feature Importance Explained

Source

What is Feature importance ?

It assigns the score of input features based on their importance to predict the output. More the features will be responsible to predict the output more will be their score. We can use it in both classification and regression problem.Suppose you have a bucket of 10 fruits out of which you would like to pick mango, lychee,orange so these fruits will be important for you, same way feature importance works in machine learning. In this blog we will understand various feature importance methods.let’s get started…….

1. Permutation Feature Importance :

It is Best for those algorithm which natively does not support feature importance . It calculate relative importance score independent of model used.It is one of the best technique to do feature selection.lets’ understand it ;

Step 1 : - It randomly take one feature and shuffles the variable present in that feature and does prediction .

Step 2 :- In this step it finds the loss using loss function and check the variability between predicted and actual output.

Step 3:- Returns the variable of feature into original order or undo reshuffle.

Step 4 :- Does the above three procedure with all the features present in dataset.

Step 5 :- Final important features will be calculated by comparing individual score with mean importance score.

This feature selection model to overcome from over fitting which is most common among tree based feature selection technique.

from sklearn.datasets import make_classification
from sklearn.neighbors import KNeighborsClassifier
from sklearn.inspection import permutation_importance
from matplotlib import pyplot
# define dataset
X, y = make_classification(n_samples=1000, n_features=10, n_informative=5, random_state=1)
# define the model
model = KNeighborsClassifier()
# fit the model
model.fit(X, y)
# perform permutation importance
results = permutation_importance(model, X, y, scoring='accuracy')
# get importance
importance = results.importances_mean
importance=np.sort(importance)
# summarize feature importance
for i,v in enumerate(importance):
    print('Feature: {}  Score: {}' .format(i,v))
# plot feature importance
pyplot.bar([x for x in range(len(importance))] ,importance)
pyplot.show()

Permutation importance

2. Coefficient as feature importance :

In case of linear model (Logistic Regression,Linear Regression, Regularization) we generally find coefficient to predict the output.let’s understand it by code.

a. Feature importance for classification problem in linear model

import pandas as pd
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
import matplotlib.pyplot as plt
import seaborn as sns
X, y = make_classification(n_samples=1000, n_features=10, n_informative=5, n_redundant=5, random_state=1)
model=LogisticRegression()
model.fit(X,y)
importance=model.coef_[0]
importance=np.sort(importance)
importance
[out]>> aarray([-0.64301454, -0.51785423, -0.46189527, -0.4060204 , -0.11978098,0.03771881,  0.16319742,  0.18431777,  0.26539871,  0.4849665 ])

Printing the all the important feature in ascending order

for index,val in enumerate(importance):
    print("Feature : {} has score  : {} ".format(index,val))

Figure 1

#plotting the features and their score in ascending order
sns.set_style("darkgrid")
plt.bar([i for i in range (len(importance))],importance)
plt.show()

Figure 2

b. Feature importance of regression problem in linear model

import pandas as pd
import numpy as np
from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression
import matplotlib.pyplot as plt
import seaborn as sns
X, y = make_regression(n_samples=1000, n_features=10, n_informative=5, random_state=1)
model=LinearRegression()
model.fit(X,y)
importance=model.coef_
importance=np.sort(importance)
#plotting the features and their score in ascending order
sns.set_style("darkgrid")
plt.bar([i for i in range (len(importance))],importance)
plt.show()

Figure 3

3 .Decision Tree as Feature Importance :

Decision tree uses CART technique to find out important features present in it.All the algorithm which is based on Decision tree uses similar technique to find out the important feature.

#decision tree for feature importance on a regression problem
from sklearn.datasets import make_regression
from sklearn.tree import DecisionTreeRegressor
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
X, y = make_regression(n_samples=1000, n_features=10, n_informative=5, random_state=1)
# define the model
model = DecisionTreeRegressor()
# fit the model
model.fit(X, y)
# get importance
importance = model.feature_importances_
importance=np.sort(importance)
# summarize feature importance
for i,v in enumerate(importance):
    print('Feature: {}, Score: {}'.format(i,v))
# plot feature importance
plt.bar([x for x in range(len(importance))], importance)
plt.show()

Figure 4

The way we have find the important feature in Decision tree same technique is used to find the feature importance in Random Forest and Xgboost.

Why Feature importance is so important ?

Feature importance gives us better interpretability of data.

let’s understand Interpret-ability;

Suppose you have a dataset of hospital now owner want to know which kind of symptomatic people will again come to hospital.How each disease(feature) make them profit.What is the sentiment of people about treatment in this hospital these all are known as interpretability. It also helps us to find most important feature for prediction.

Conclusion :-

This is all from my side if you have any suggestion please comment below.