Is Weighted Averages the Best Method to Aggregate Information?

Question

I'm working on a recommendation system. My system uses user's past rating data, to predict future ratings.

I designed mathematical methods for generating recommendation algorithms that allows me to generate an unlimited number of recommendation algorithms (given a large enough set of relevant ratings).

I intend to aggregate the results from all this algorithms to compute a final expected rating of the user. So I thought of taking the weighted averages of the expected ratings of each algorithm to use to calculate a final expected rating.

Basically I either:
1. Assign weights based on the amount of information (ratings collected).
2. Assign weights based on the accuracy of the algorithm used.

The algorithms arrive at their answer via different means, and it is possible that two may give the same answer, but they may take into account different information and/or use different inferential techniques on that information.

So if an accuracy of 1 means there's no deviation between the algorithm's expected ratings, and the user's actual ratings, and an accuracy of 0 means there's maximum deviation between an algorithm's expected ratings and user's actual ratings, we may get $\{Al_1, Al_2, Al_3\}$ with accuracies of $(0.9, 0.85, 0.92)$.

I have a minimum information and/or accuracy threshold that I would use to filter the algorithms to assign weights to (I have not yet figured out what this threshold would be, and as it may vary on a per user basis, I want to leave determining the threshold to an ML system.

My question is this:

Is weighted averages the Best methods for me to aggregate the data provided by my various algorithms/infer from it?
If not, what other methods are more apt for the problem at hand?

Answer 1

A weighted average might or might not be the best method. This will depend on (a) the nature of the algorithms, and (b) what objective function you are trying to minimize.

Let me give you a sample set of assumptions. Suppose we treat the predictions from the recommendation algorithms as independent random variables (e.g., because each recommendation algorithm was trained independently). Also, suppose we wish to minimize the variance of the final prediction. In other words, our loss function is the $L_2$ loss: if our prediction was $Y$, and the correct value was $\tilde{Y}$, then the loss (penalty) is $(Y-\tilde{Y})^2$.

Then in this case we can find the optimal set of weights. In particular, in this case, if we restrict ourselves to linear combinations, the optimal set of weights is inversely proportional to the variance of each estimator. Suppose $X_1,\dots,X_n$ are the predictions from our $n$ recommendation algorithms. Suppose that we have measured the accuracy of each recommendation algorithm by measuring its $L_2$ loss on a validation set; in other words, we know the variance $\text{Var}(X_i)$ for each $i$. Here $\text{Var}(X_i)$ is a measure of the accuracy of the $i$th recommendation algorithm: the larger the variance, the larger the expected $L_2$ loss, and thus the less accurate the prediction is. Then the optimal linear weighting is to select a final prediction $Y$ by

$$Y = c_1 X_1 + \dots + c_n X_n,$$

where we choose the weights

$$c_i = {a_i \over a_1 + \dots + a_n}$$

where

$$a_i = {1 \over \text{Var}(X_i)}.$$

Is a linear combination optimal (as opposed to other, nonlinear methods of combining the predictions)? In this situation, if we additionally know that the errors made by each recommendation algorithm have a Gaussian distribution, then yes, a linear combination is optimal. If the errors have a different distribution, then some other combination might be better.

As you can see, this will depend on the loss function we select, and on other assumptions. That said, in practice, a linear combination (i.e., a weighted average) is often a reasonable choice.

There's lots written about this sort of thing in the statistics literature. For instance, the above discussion is just a restatement of the fact that the average is the optimal estimator of the mean of a Gaussian distribution (i.e., it minimizes the mean squared error). See, e.g., https://en.wikipedia.org/wiki/Estimator, https://en.wikipedia.org/wiki/Mean_squared_error, https://stats.stackexchange.com/q/81571/2921, https://stats.stackexchange.com/q/97765/2921, https://stats.stackexchange.com/q/48864/2921, https://math.stackexchange.com/q/9032/14578.