Weighted Average of Multi-Output Neural Networks

Question

[1] discusses how to construct an ensemble of neural networks by giving each network a certain weight $\alpha_i$: \begin{equation} f_\mathrm{GEM}(\boldsymbol{x}) = \sum_{i=1}^N \alpha_i f_i(\boldsymbol{x}) \end{equation} where each $\alpha_i$ is obtained using a correlation matrix which is defined in the following way: \begin{equation} C_{ij} = E[m_i(\boldsymbol{x}) m_j(\boldsymbol{x})] \end{equation} where $m_i(\boldsymbol{x}) \equiv f(\boldsymbol{x}) - f_i(\boldsymbol{x})$, with $f(\boldsymbol{x})$ being the target function and $f_i(\boldsymbol{x})$ being the output.

I was wondering how should one compute $C_{ij}$ when the neural network has multiple outputs. Would one firstly calculate $\boldsymbol{m}_i(\boldsymbol{x}) \equiv \boldsymbol{f}(\boldsymbol{x}) - \boldsymbol{f}_i(\boldsymbol{x})$ for all examples and then compute $C_{ij} = E[\boldsymbol{m}^T_i(\boldsymbol{x}) \boldsymbol{m}_j(\boldsymbol{x})]$? I tried doing that (with MNIST data) but the accuracy was not better than when taking a simple average of all the networks in the ensemble. I would appreciate any help on this.

[1] M. P. Perrone and L. N. Cooper, "When networks disagree: Ensemble methods for hybrid neural networks," in Artificial Neural Networks for Speech and Vision. Chapman and Hall, 1993, pp. 126-142.

Answer 1

Background: The simplest method of ensembling is to take an unweighted average of the outputs of the ensembles. In general, if you want to use a simple ensemble of neural networks for classification, there are two reasonable options:

1. Average the logits from each model, then apply softmax once to the averaged estimate.

2. Each model applies softmax to get that model's confidence score, then average the confidence scores from each model.

Both are reasonable approaches. Both can be used with multi-class classification: you independently average all the logits or confidence scores for each class separately. You could try both to see which works better. Personally, I would expect option #2 might work better, but the best test is to try them.

Perrone and Cooper's method: The framework you summarize generalizes the simple method, from an unweighted average to a weighted average, where the weights are calculated in a particular way. In principle, that theoretical framework can be used for any univariate function, i.e., any function that outputs a real number, as long as we know what the "correct" output is for each instance in the training set.

In the context of multi-class classification with neural networks, you'll only be able to use Perrone and Cooper's method with option #2, i.e., applying it to the confidence scores from the models. You can still use it in a multi-class classification setting straightforwardly, by applying it separately to each class: to get the confidence score of the ensemble for class $c$, take a weighted average of the confidence scores of each model in the ensemble, with weights derived based on the formulas you listed. Basically, for 10-class classification, you treat this as 10 separate univariate functions (each class is treated independently).

Caveats: That paper is from 1993. I don't know if it is the best way to create an ensemble of neural networks. For one thing, they appear to be using MSE error as their loss. Early work on neural networks used MSE loss, but since then we have learned that cross-entropy loss typically works a lot better for classification tasks. I am not sure that their method of computing the weights will "play well" with cross-entropy loss. So, it's not clear to me that their method will perform better than a simple ensemble (unweighted average) of models trained with cross-entropy loss. I encourage you to do some experiments before relying too heavily on their approach.