# Weighted Average of Neural Networks with Cross Entropy Cost Function

What is the best way to create an ensemble of neural networks utilising weighted averaging when these networks were trained to minimise cross entropy error function?

The literature I found (e.g. [1]) seems to be talking only about weighted averages of neural networks that employ mean square error (MSE) cost function. Anything other than that seems to require a modification of the training algorithm. Is there an analytic extension of [1]'s generalized ensemble method (GEM) algorithm to compute neural networks' weightings when cross entropy error function is used? Or some entirely different algorithm would be required? Or maybe it can only be done by numerically estimating those optimal weightings?

[1] M. P. Perrone and L. N. Cooper, "When networks disagree: Ensemble methods for hybrid neural networks," in Artificial Neural Networks for Speech and Vision. Chapman and Hall, 1993, pp. 126-142.

• May I ask why you insist on weighted averaging specifically? – D.W. Dec 15 '19 at 5:35

Disclaimer: In most situations, I would expect that a single neural network will be at least as good as an ensemble, so there's typically probably not much reason to bother with an ensemble. In particular, if we compare an ensemble of $$n$$ small neural networks to a single big neural network that is $$n$$ times as large, I would expect that typically the big neural network will perform at least well as the ensemble. So in most situations you're probably better off skipping the ensemble and training a big neural net. That said, here's how you can train an ensemble.

Unweighted average: The simplest way to train an ensemble is to use an unweighted average of the softmax outputs of the individual networks. Let $$F_1,\dots,F_n$$ denote the networks in the ensemble, $$F_i(x)$$ denote the vector of softmax outputs of the $$i$$th network on input $$i$$, and $$F_i(x)_c$$ denote the softmax output for class $$c$$ from $$F_i(x)$$. Define $$F$$ by

$$F(x) = (F_1(x) + \dots + F_n(x))/n,$$

i.e.,

$$F(x)_c = (F_1(x)_c + \dots + F_n(x)_c)/n.$$

This is the unweighted average of the individual networks. You can train $$F$$ in two ways: either train each network individually, or train the whole ensemble end-to-end.

• To train the ensemble end-to-end, you treat $$F$$ as a big model whose parameters are all of the parameters of all of the networks, and you optimize cross-entropy loss applied to the output of $$F$$.

• Alternatively, you can train each network individually. Train $$F_1$$ using cross-entropy loss on the output of $$F_1$$; then train $$F_2$$; and so on.

In principle, end-to-end training should yield performance at least as good as training each network individually. However, training each network individually is simpler and requires less memory on the GPU.

Weighted average: We can generalize these ideas to train an ensemble that is a weighted average of the outputs of the individual neural networks. Define $$F$$ by

$$F(x)_c = {\alpha_{1c} F_1(x)_c + \dots + \alpha_{nc} F_n(x)_c \over \alpha_{1c} + \dots + \alpha_{nc}},$$

where the $$\alpha_{ic}$$ are the weights. There are two ways to train such an ensemble:

• You can train it end-to-end. You have a big model for $$F$$, whose trainable parameters are the parameters of the individual networks as well as the weights $$\alpha_{ic}$$. Then, apply an optimizer to minimize cross-entropy loss applied to the output of $$F$$.

• Alternatively, you can train each network individually, then fix the networks and train the weights $$\alpha_{ic}$$ subsequently. Once you've trained the individual networks, compute new feature vectors $$v_{ij} = F_i(x_j)$$, where $$x_j$$ is the $$j$$th element of the training set. Now you can throw away the $$x$$'s and work only with the $$v_{ij}$$'s, as we have $$F(x_j)_c = {\alpha_{1c} v_{1jc} + \dots + \alpha_{nc} v_{njc} \over \alpha_{1c} + \dots + \alpha_{nc}}.$$ Finally, user an optimizer to learn the parameters $$\alpha$$ that minimize the cross-entropy loss on $$F(x_j)$$'s, using the above equation to relate this to the (trainable) $$\alpha$$'s and the (fixed) $$v$$'s. Any stochastic gradient descent based optimization method should work fine. You might want to initialize all the $$\alpha$$'s to $$1/n$$, and renormalize the $$\alpha$$'s after each iteration to prevent them from growing without bound.

The same tradeoffs between end-to-end training vs training each network individually apply.

I'm not expecting an ensemble based on a weighted average to do significantly better than an unweighted average, in most cases; and I'm not expecting an ensemble to do significantly better than a single large network, in most cases. But there could always be exceptions, and hopefully this gives you a reasonable way to train such ensembles.