1
$\begingroup$

Disclaimer: First of all I've had a lot of trouble finding the right stack exchange site for this question and I don't know if it's correct here either.

The Problem: I'm working on a search algorithm for a django powered website which allows people to write articles on various subjects and search for articles on the webpage.

I've got a database in which I store articles for people to read on a website. I'm storing each article with a bit of information like:

Articles:

  • A title.
  • A predetermined category where the writer can choose from when adding an article (sports, economy, politics).
  • Tags which the writer of the article can input themselves.

  • Length of the article in characters

  • A short description
  • Dated added
  • Date updated

I've a few easy ways a user can search an article based on the title ( for example shortest first or newest first)

I'm now working on adding a way to search based on popularity but I have a hard time to determine what I define as popular.

I'm saving article views in a different database table which contains:

Views:

  • User ID
  • article ID
  • Timestamp

Each row inside this table is considered 1 view so I can see exactly when certain articles are doing well in terms of amount of views it gets. If I would search for an article with the most views in the database it would look something like this in django:

Article.objects.order_by('-views')[:30]

In this case views is stored in the table, However I store it in a different table so I can have the functionality described above.

The problem however is that if I have 1 article which is a few years old with a high number of views, for example lets say there has been a sports event (like the Olympic games of 2014) and a lot of people have read the article but the article is no longer very relevant as this same event has been done again (that would be the Olympic games of 2018 happening this year).

Now lets say someone searches for popular articles about the Olympic games. Even though the articles about the Olympic games of 2014 have gathered more views than those of 2018 (as they have been written roughly 4 years ago and therefor have had more time to gather views)

In this case the more relevant articles should be presented to the user before the more popular articles.

My Potential Solution:

I'm thinking of adding a column to the article table for memorizing the 'importance' of each article, then I can have Importance be determined by other factors

for example if I want to have the amount of views be a factor for each view importance gets a bit higher, but if the view has been within the last week it might count as 2 views. therefor favoring more actual articles slightly more over articles with more views but written a few years ago. this also allows me to for example measure the time each reader spends on an article and have that influence importance as well.

However I'm not sure if this would be the best way of doing things as there are a few problems like:

  1. If importance is stored in the database it has to be updated regularly and periodically which may cause performance problems whenever the user accesses an article while the database is updating it's articles importance.

  2. Calculating the importance in real time every time a user doe a search will also take a huge amount of processing power if the amount of users is very high, it will also probably cause a lot of redundancy whenever 2 users search for the same articles and the exact same importance factor is calculated twice.

How would I go about implementing this way of sorting the search results without needing lots of processing power? Is this the best way to do it? How do other search engines handle this problem?

$\endgroup$
  • $\begingroup$ I don't see how this is an answerable question about CS.SE; until you define what you mean by relevance, we can't tell you how to find an article with highest relevance or how to implement the search & ranking code. $\endgroup$ – D.W. Jan 3 '18 at 17:11
1
$\begingroup$

It seems to me like 'relevance' is a qualitative abstraction. Thus you need to define your own heuristic for your service which defines what relevance is.

I think it would be best to look for solutions in third party applications such as elasticsearch instead of implementing your own.

$\endgroup$
  • $\begingroup$ Yes, I've done this with another search indexing system in the past. @ale's formula below is pretty much exactly what I used (along with a roughly logarithmic scaling of "popularity"). $\endgroup$ – Will Crawford Jan 3 '18 at 18:09
0
$\begingroup$

One option would be to add a metric which weights the views by age. The older the view the less weight it has. weight = 1 - constant * ((date(today) - date(posted)) / date(today))

$\endgroup$
  • $\begingroup$ this is a good idea and it also seems expandable... as mentioned above the weight of each article could depend on multiple factors and I've not decided yet what exactly I'm going to use for measuring importance (or what you call weight). However the bigger problem is how I would implement such a system without the need for a lot of processing power when working with a lot of data (many articles may potentially be posted to this website) $\endgroup$ – BRHSM Jan 3 '18 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.