I've been looking for the most unbiased algorithm to find out the feature importances in random forests if there are correlations among the input features.

Besides the most commonly preferred methodologies; gini-impurity reduction, drop-column importance and permutation importance, I found an algorithm called conditional permutation importance, in the given article: (https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-9-307#Sec8)

The steps for calculating the conditional permutation importance are given in the article like this:

  1. In each tree compute the oob-prediction accuracy before the permutation
  2. For all variables Z to be conditioned on: Extract the cutpoints that split this variable in the current tree and create a grid by means of bisecting the sample space in each cutpoint.
  3. Within this grid permute the values of X j and compute the oob-prediction accuracy after permutation
  4. The difference between the prediction accuracy before and after the permutation accuracy again gives the importance of X j for one tree. The importance of X j for the forest is again computed as an average over all trees.

For the first step, I'm having difficulties to reach oob scores of each tree as the default oob_score is calculated for all trees in the forest in scikit's methods. However, since I can still reach single trees as decision trees, I tried test inputs in these trees instead of oob samples but the kernel kept dying...

  • Sample RF Classiffier

    clf=RandomForestClassifier(n_estimators=200,max_depth=3,oob_score = True) forest = clf.fit(train_inputs_arr, train_targets_arr)

  • Reaching single decision tree properties



For the second step, I'm having difficulty to understand what is meant by "creating a gird by means of bisecting the sample space at each cutpoint", and didn't really understand if I should determine the cutpoints of the selected Xj or for the other variables Z to be conditioned on.

Additionally, I'm also sharing the permutation importance method structure that I previously used, It simply permutes every feature calculates how the oob score decreases for each feature after permutation and the highest decrease in the oob score means higher feature importance. I wanted to modify this structure but I'm theoretically stuck at this point. What I really want to learn is any implementation of this algorithm on python.

def permutation_importances(rf, x_tr, y_train):
baseline = rf.oob_score_
imp = []
for col in x_tr.columns:
    rf_ = rf
    save = x_tr[col]
    x_tr.loc[:,col] = np.random.permutation(save)
    rf_.fit(x_tr, y_train)
    m = rf_.oob_score_
    x_tr.loc[:,col] = save
    imp.append(baseline - m)
  • $\begingroup$ Can you clarify what your question is? Normally we prefer that a post have a single question. I see 3 or 4 things in your post that it looks like you might be hoping for an answer to. Might you be able to pick one, and edit your post about that? $\endgroup$ – D.W. Mar 19 '19 at 16:38
  • $\begingroup$ I'm sorry for the obscurity, in the end, I'd like to learn how to implement this algorithm on python $\endgroup$ – Orkun Tunay Mar 19 '19 at 20:47
  • 2
    $\begingroup$ I think a useful way to make use of this site is to try to implement it, and then if you run into something specific that is unclear, ask a question about that. If you run into multiple things, consider posting them separately as separate questions. That helps keep your question focused. Note that coding questions and Python-specific questions are off-topic here, but understanding how the algorithm works is on-topic. $\endgroup$ – D.W. Mar 19 '19 at 21:13
  • $\begingroup$ Since your question is about a very specific paper, have you tried emailing the first author at carolin.strobl@*** as provided on the website? $\endgroup$ – John L. Mar 23 '19 at 22:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.