I've been looking for the most unbiased algorithm to find out the feature importances in random forests if there are correlations among the input features.
Besides the most commonly preferred methodologies; gini-impurity reduction, drop-column importance and permutation importance, I found an algorithm called conditional permutation importance, in the given article: (https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-9-307#Sec8)
The steps for calculating the conditional permutation importance are given in the article like this:
- In each tree compute the oob-prediction accuracy before the permutation
- For all variables Z to be conditioned on: Extract the cutpoints that split this variable in the current tree and create a grid by means of bisecting the sample space in each cutpoint.
- Within this grid permute the values of X j and compute the oob-prediction accuracy after permutation
- The difference between the prediction accuracy before and after the permutation accuracy again gives the importance of X j for one tree. The importance of X j for the forest is again computed as an average over all trees.
For the first step, I'm having difficulties to reach oob scores of each tree as the default oob_score is calculated for all trees in the forest in scikit's methods. However, since I can still reach single trees as decision trees, I tried test inputs in these trees instead of oob samples but the kernel kept dying...
Sample RF Classiffier
clf=RandomForestClassifier(n_estimators=200,max_depth=3,oob_score = True) forest = clf.fit(train_inputs_arr, train_targets_arr)
Reaching single decision tree properties
For the second step, I'm having difficulty to understand what is meant by "creating a gird by means of bisecting the sample space at each cutpoint", and didn't really understand if I should determine the cutpoints of the selected Xj or for the other variables Z to be conditioned on.
Additionally, I'm also sharing the permutation importance method structure that I previously used, It simply permutes every feature calculates how the oob score decreases for each feature after permutation and the highest decrease in the oob score means higher feature importance. I wanted to modify this structure but I'm theoretically stuck at this point. What I really want to learn is any implementation of this algorithm on python.
def permutation_importances(rf, x_tr, y_train): rf.fit(x_tr,y_train) baseline = rf.oob_score_ imp =  for col in x_tr.columns: rf_ = rf save = x_tr[col] x_tr.loc[:,col] = np.random.permutation(save) rf_.fit(x_tr, y_train) m = rf_.oob_score_ x_tr.loc[:,col] = save imp.append(baseline - m)