# Random Forest - Conditional Permutation Importance

I've been looking for the most unbiased algorithm to find out the feature importances in random forests if there are correlations among the input features.

Besides the most commonly preferred methodologies; gini-impurity reduction, drop-column importance and permutation importance, I found an algorithm called conditional permutation importance, in the given article: (https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-9-307#Sec8)

The steps for calculating the conditional permutation importance are given in the article like this:

1. In each tree compute the oob-prediction accuracy before the permutation
2. For all variables Z to be conditioned on: Extract the cutpoints that split this variable in the current tree and create a grid by means of bisecting the sample space in each cutpoint.
3. Within this grid permute the values of X j and compute the oob-prediction accuracy after permutation
4. The difference between the prediction accuracy before and after the permutation accuracy again gives the importance of X j for one tree. The importance of X j for the forest is again computed as an average over all trees.

For the first step, I'm having difficulties to reach oob scores of each tree as the default oob_score is calculated for all trees in the forest in scikit's methods. However, since I can still reach single trees as decision trees, I tried test inputs in these trees instead of oob samples but the kernel kept dying...

• Sample RF Classiffier

clf=RandomForestClassifier(n_estimators=200,max_depth=3,oob_score = True) forest = clf.fit(train_inputs_arr, train_targets_arr)

• Reaching single decision tree properties

forest.tree_.predict(test_inputs)

forest.tree_.predict(test_inputs)


For the second step, I'm having difficulty to understand what is meant by "creating a gird by means of bisecting the sample space at each cutpoint", and didn't really understand if I should determine the cutpoints of the selected Xj or for the other variables Z to be conditioned on.

Additionally, I'm also sharing the permutation importance method structure that I previously used, It simply permutes every feature calculates how the oob score decreases for each feature after permutation and the highest decrease in the oob score means higher feature importance. I wanted to modify this structure but I'm theoretically stuck at this point. What I really want to learn is any implementation of this algorithm on python.

def permutation_importances(rf, x_tr, y_train):
rf.fit(x_tr,y_train)
baseline = rf.oob_score_
imp = []
for col in x_tr.columns:
rf_ = rf
save = x_tr[col]
x_tr.loc[:,col] = np.random.permutation(save)
rf_.fit(x_tr, y_train)
m = rf_.oob_score_
x_tr.loc[:,col] = save
imp.append(baseline - m)

• Can you clarify what your question is? Normally we prefer that a post have a single question. I see 3 or 4 things in your post that it looks like you might be hoping for an answer to. Might you be able to pick one, and edit your post about that? – D.W. Mar 19 '19 at 16:38
• I'm sorry for the obscurity, in the end, I'd like to learn how to implement this algorithm on python – Orkun Tunay Mar 19 '19 at 20:47
• I think a useful way to make use of this site is to try to implement it, and then if you run into something specific that is unclear, ask a question about that. If you run into multiple things, consider posting them separately as separate questions. That helps keep your question focused. Note that coding questions and Python-specific questions are off-topic here, but understanding how the algorithm works is on-topic. – D.W. Mar 19 '19 at 21:13
• Since your question is about a very specific paper, have you tried emailing the first author at carolin.strobl@*** as provided on the website? – John L. Mar 23 '19 at 22:38