PageRank is heavily skewed in that a small proportion of webpages have a high PageRank score while most of the others have a very low PageRank score. Does this undermine the effectiveness of the TF-IDF function? For example, if PageRank scores were to be combined with TF-IDF scores, then for a few of the webpages, the TF-IDF would make very little difference, as they would have a high PR score, correct? And for the majority of the webpages with low PageRank, the TF-IDF score would tend to dominate, right?

  • $\begingroup$ Presumably it depends on how you combined PageRank and TF-IDF. Also, it's not clear what you mean by "undermines the effectiveness of TF-IDF" when you're hypothesizing a combination of PageRank + TF-IDF -- undermines it, compared to what? $\endgroup$ – D.W. Feb 20 '20 at 5:08
  • $\begingroup$ Google's full ranking algorithm is secret. I don't think we even know how it combines PageRank and TF-IDF, let alone variant of TF-IDF it uses, and what else it does. $\endgroup$ – Pseudonym Feb 20 '20 at 5:58
  • $\begingroup$ I would like to know if there is any theoretical difficulty because of the skew of PageRank. For example, if PageRank values were linear, would it make it easier to combine with TF-IDF? $\endgroup$ – Joebevo Feb 21 '20 at 1:43
  • 1
    $\begingroup$ Well PageRank gives a certain score to websites. Now you can sort all the scores and give a website a score based on the index in this list. The transformation from some score distribution to a linear score is trivial. So no the skew of PageRank doesn't really matter and can be corrected easily. $\endgroup$ – plshelp Jul 7 '20 at 21:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.