As a side-project, I had the idea to write some kind of an algorithm that would evaluate all participations in our weekly Pub quiz, to then calculate the average strength of the participants.
This is the information that I have available:
- The final score of the group
- All members of the group for a particular week
- As such, also the team size
- As such, also all the past participations of every single person
This is how I attempted to solve it:
The first, and obviously the simplest solution that I had, was thinking of the information that I have as a system of linear equations:
$$ 0 * x_1 + 0 * x_2 + 0 * x_3 + \dots + 0 * x_n = S $$
The idea was that every row represents a game, every participant gets an ID somewhere between 1 and n and every person who participated on that specific week gets a factor 1 in the corresponding row with the rest getting a zero. after solving all the linear equations, the value of each $ x_n $ would represent how strong the person with ID $ n $ is, and the $ S $ would be the final score of the group for that week. (In the case where it's impossible to solve it, use the difference between expected answers and calculated answer as fitness to be minimized). Obviously this had a ton of flaws, the biggest one being that it would predict insane scores for bigger groups, since all the strengths are added together. This seemed like an easy fix with adding:
$$ (0 * x_1 + 0 * x_2 + 0 * x_3 + \dots + 0 * x_n) * (1 - w * log(P)) = S $$
with $ P $ being the number of players. This already worked a lot better, the idea was that the overlap of knowledge goes up logarithmically multiplied by some weight that will get calculated in the future. (can then be rewritten as $ S / (1 - w * log(P)) $ once w is set to some constant). While this "worked" for a couple of months, I am spotting more and more flaws that seem to be unsolvable without scrapping the whole system of linear equations stuff. The reason why I believe ML to not be a good solution for this, and the same reason why I think this isn't purely a data science question, is because with one datapoint being added per week, that's 52 data points per year, which I believe is way too low for any type of actual machine learning.
- The biggest flaw that I see in my system is that for a player $ x_a $ with a low weight it is often that:
$$ (x_1 + \dots + x_n) * (1 - w * log(P)) > (x_1 + \dots + x_n + x_a) * (1 - w * log(P + 1)) $$
which to me seems absolutely absurd. (Adding an extra person to the group leads to lower predicted score than without that person)
- Second biggest flaw I see that the probabilities are missing from the equation. Assume 4 Players get 20 points on a bad week, then get 30 points on the next week with a 5th player who is participating for the first time. The solution that I get every time is to set the 5th player strength to 10 and make the 30 point game have no influence on the other 4 players. That's definitely not the solution I am looking for though, as I would expect all games to have at least some influence on the strength of all participants.
The question is, are there any real life problems that are similar to this? Does this type or class of problem already have a name?