3
$\begingroup$

I have developed an algorithm which recommends geographical locations to users based on popular trends and their own interests. The dataset is created by my organization. So the user selects a few categories and based on his interest and rating by other people he is presented with recommended places for him.

How do you evaluate or use metrics for such system given that ground truth doesn't makes sense in this case ?

EDIT: I meant that how to evaluate the accuracy or quality of results in such case especially for published work.

EDIT 2 (as per the request for details in the comments)

Details of the system

  1. The user indicates his preferences from a predefined set of tags (cultural, mountains etc)

  2. Users can also rate different places on a scale of 1-5.

  3. Users profile (geographical location, places already visited etc are already stored in his profile)

  4. Based on user's choice and other heuristics(rating etc), a set of places is recommended by the system.

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ Googling for "Evaluating recommendation systems" gives quite a few relevant results. Two top ones for me are research.microsoft.com/pubs/115396/evaluationmetrics.tr.pdf and machinelearning.wustl.edu/mlpapers/paper_files/… $\endgroup$
    – Alexey Romanov
    Jan 13, 2014 at 18:39
  • $\begingroup$ User feedback, provided after the fact? $\endgroup$
    – Raphael ♦
    Jan 13, 2014 at 19:35
  • $\begingroup$ think I understand but it would be better if you clarify what you mean by "ground truth". you seem to want some kind of benchmark or quality measurement of the results. $\endgroup$
    – vzn
    Jan 14, 2014 at 3:32
  • $\begingroup$ @vzn I meant that I don't have any results to verify the quality or accuracy of results $\endgroup$
    – krammer
    Jan 14, 2014 at 16:47
  • $\begingroup$ this is still a bit vague. are you recommending businesses at particular locations? is it within a city? a country? worldwide? etc? it would help to give an example... also how does the recommendation system work roughly, can you sketch it out? how is the data organized? etc.. for example how would one know what a "popular trend" is, or the users interest(s)? etc $\endgroup$
    – vzn
    Jan 14, 2014 at 17:10

2 Answers 2

Reset to default
3
$\begingroup$

One of the best ways to benchmark such big systems is to use a service like Amazon Mechanical Turk (AMT). First you implement a user interface for everyday users where they could specify whether or not the recommendations are relevant. Then you submit this to Amazon Turk and let many people try it out.

You could further process the results of an Amazon Turk to really benchmark your system and obtain quantification of accuracy of it.

Team of Fei Fei Li used similar technique for benchmarking their Fine Grained segmentation algorithm. The paper is here: http://vision.stanford.edu/pdf/DengKrauseFei-Fei_CVPR2013.pdf

Naturally using AMT won't give you a realtime solution, where you get a benchmark everytime a recommendation occurs, however you can prepare a subset of your dataset and let AMT guys evaluate the system using GUI you provide. Again, a sample data collection system is described here: http://ai.stanford.edu/~jkrause/papers/fgvc13.pdf

I have no intention of advertising it but AMT is also an economical workaround.

Improve this answer
$\endgroup$
2
  • $\begingroup$ "Use the Turk" is not conceptual advice and borders on advertisement. Can you please stress the conceptual parts of your answer? In particular, how should such feedback be incorporated in the recommender? $\endgroup$
    – Raphael ♦
    Jan 13, 2014 at 19:36
  • $\begingroup$ Mechanical Turk is commonly used by researchers in CS for this sort of problem, so I would not view it as any sort of advertisement. One procedure would be to simply give the "Turkers" the pairs of (input, output) to the system and ask them to rate the output of each pair on a scale according to some metric. $\endgroup$
    – usul
    Jan 16, 2014 at 0:59
1
$\begingroup$

the basic strategy in [supervised] machine learning is to split the data into a "learning/training set" and a "test set". the algorithm when trained has no access to the test set which also acts something like a scientific control group. the algorithm attempts to learn a pattern. then you compare its performance on the data that it was "blind" to during training (also called "held out"), and if training/"generalization" occurred, the performance on the test data will be similar. briefly stated when there is a large drop in performance on training vs test data its called "overfitting" or "memorization" and the algorithm learned irrelevant details of the training data and did not "generalize". one "rule of thumb" balance on training vs test set is sometimes 80% vs 20% ie 4/5ths of the data is used for training and 1/5th is for test.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.