I trained an XGBoost model that classifies 50 different classes, therefore it generates a lot of boosters (trees). Also, I wrote a script that turns trained model to C++ code and it works really fast because every tree is simply nested if-else statement. What I noticed is that many trees are similar and I was wondering if there is a way to merge all of those trees into one tree or maybe to reduce number of trees by merging some of them? Maybe even convert everything to some different data structure?
This is what dump looks like:
booster: 0:['::'] yes=2,no=1 1:['=>'] yes=4,no=3 3:['es'] yes=8,no=7 7:['ef'] yes=10,no=9 9:leaf=-0.0339584 10:leaf=0.0612883 8:leaf=-0.0484913 4:leaf=0.0358804 2:[';\s'] yes=6,no=5 5:leaf=0.373993 6:leaf=-0.0376609 booster: 0:['</'] yes=2,no=1 1:['h\n'] yes=4,no=3 3:[':v'] yes=8,no=7 7:leaf=-0.0487989 8:leaf=0.0200258 4:['_t'] yes=10,no=9 9:leaf=-0.03375 10:leaf=0.172536 2:['*\s'] yes=6,no=5 5:leaf=0.354577 6:leaf=0.0165644 // etc.. boosters go on and on...
Let me explain how it works by going through booster. First we check if sample we're classyfing contains '::'; if yes we jump to branch 2. There we check if sample contains ';\s' and if yes, we jump to branch 6. Branch 6 is leaf so we just take value -0.0376609 and we add it to probability that sample is class '0'.
We do the same for booster and it gives us probability that sample is of class '1'. It goes on and on for all 50 classes and then the process is repeated for many iterations that gives us 50 * num_iterations of trees. It comes up around 10k of trees.
Anyone has any idea if there is a better way of representing these trees so our classification works even faster?