Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [information-retrieval]

The tag has no usage guidance.

0
votes
0answers
10 views

How to use HITS algorithm in a web search engine?

I implemented the classical non-weighted converged version of HITS algorithm. I create a root set and a base set from the root set and then I applied the HITS algorithm and got the authorities and ...
0
votes
0answers
19 views

How does single pass in-memory indexing work?

I don't understand how single pass in-memory indexing works in the field of information retrieval since doesn't the postings list in an inverted index have to be sorted? As far as I understand, the ...
0
votes
0answers
17 views

How do you determine k in a k-gram index?

In information retrieval theory, k-gram indexes can be used for wildcard queries. I often see 3-gram examples, but if I were to build a system how would I choose $k$?
1
vote
0answers
11 views

How to use HITS algorithm to rank pages

I am trying to implement a simple web search engine mechanism with using HITS algorithm. I understand how the algorithm work and produce hubs and authority values for each page. But how can I use this ...
0
votes
0answers
10 views

Semantic Question Answering system vs Cross Language Information Retrieval

I've been trying to understand Bilingual Question Answering (QA) systems and Cross-Language Information retrieval systems (CLIR). I understand that QA systems give a precise response instead of a set ...
1
vote
0answers
29 views

Single Pass in Memory Indexing

I was reading single pass in memory indexing and had few doubts. Why is the time complexity of SPIMI O(T) where T are the token. We know that before writing blocks to Disk, we have to sort ...
0
votes
2answers
99 views

How does a Trie work?

Specifically, many say that to speed up an HTTP routing code going from regex to a Trie is the solution. Sometimes also a compressed Trie or Radix Tree. The problem though is that after reading ...
3
votes
1answer
39 views

Calculating Frequency of Terms with Zipf's Law

I'm studying for an exam and came across this question. I feel like it's much simpler than I am making it out to be, but I'm not sure. Suppose we have a corpus in which Zipf’s Law holds. If the most ...
1
vote
1answer
36 views

Query Logs and evaluation of search engines

I'm taking a course about Information Retrieval and we had a quiz involving the evaluation of search engines. We had a problematic question regarding "Query Logs" (basically recording the queries ...
2
votes
1answer
29 views

Encoding a list of numbers using gamma coding

I'm trying to work out how to find the gamma $\gamma -code$ that encodes the postings list (12, 18, 21, 22). Is the postings list encoded just by a list of the individual $\gamma$-codes for each ...
0
votes
0answers
20 views

Can LSTMs help in text summarization?

I know LSTMs are used for text prediction; but can they be useful for text and document summarization without using document statistics such as tf-idf vectors?
1
vote
1answer
30 views

How to figure out whether two texts refer to the same object or event

Let's assume there is something happen in the world - Football world cup final. And team-1 beat team-2 with the score 3:2. So there is whole bunch of articles on every website about it, each contains ...
0
votes
0answers
16 views

Learning material on information visualization

If somebody with a computer science background wants to learn about information visualization, again from a computer science perspective, which will be the best learning material for him? A textbook ...
0
votes
0answers
41 views

Algorithm of finding relationships between two sets given a bijection relationship between their cosets to another set

Let $C$ be the set of all available classes, $S$ be the set of all available students, $P$ be the set of all available phone numbers. Each student many have zero to many phone numbers, but each phone ...
2
votes
1answer
283 views

How is average precision related to the precision-recall curve?

As background, in Information Retrieval, there are many metrics to assess how well search results are retrieving information. A simple one is precision, where we calculate the fraction of documents ...
0
votes
1answer
35 views

Does this classify as Information retrieval?

I’m trying to figure out if what I’m doing would classify as Information Retrieval. According to this Thread: Information retrieval is based on a query - you specify what information you need and ...
0
votes
0answers
48 views

Gensim LDA all topics zero

I'm trying to run Gensim's LdaModel on a corpus. I load the corpus myself from two XML files. I made a class to iterate over the text pieces in the XML files. It ...
1
vote
0answers
34 views

Evaluation of Information extraction a task

I have to evaluate my information extraction program. I have to evaluate it by the expectation-driven way. I don't know what that means. I would appreciate if you could help me. Thx.
7
votes
1answer
334 views

What is the most recent comparison of signature files and inverted indices?

Modern papers on search indices often contain a statement that inverted indexes (posting lists) are categorically superior to signature files (bloom filters). Here are some examples from papers ...
5
votes
0answers
346 views

What's known about web search query distribution?

Silverstein et al., Analysis of a Very Large AltaVista Query Log, SRC Technical Note '98 looked at 43 days of queries in 1998 and pulled out how many terms there are per query, what the top 25 queries ...
4
votes
1answer
5k views

What is the difference between R precision and precision at K?

I am trying to understand the difference between these two but it looks like as if they are calculated the same way. What is the difference? R precision: R precision is the precision at the Rth ...
4
votes
1answer
93 views

What is a difference between cross-lingual IR and multi-lingual IR?

In papers usually Cross-Lingual Information Retrieval (CLIR) and Multi-Lingual Information Retrieval (MLIR) use equivalently or distinctly. I want to know is there any difference between these two ...
3
votes
2answers
106 views

Measuring the information of a document?

I'd like to measure how much information a document $D$ contains. Clearly, the New York Times published yesterday contains more information than my diary wrote on the same day. But, I do not know ...
6
votes
1answer
514 views

What would a formal grammar for a binary file format look like?

Binary structures often feature length specifiers; the parser is supposed to read them and then consume the specified amount of symbols. Because of this, the grammar is context-sensitive. What would ...
-1
votes
1answer
60 views

How do I calculate values in K-means (clustering)?

I'm learning some Data Retrieval concepts, one of them being K-means clustering. I understand that it's meant to help understand data, but that's where I begin having issues. I'm working on this ...
1
vote
2answers
120 views

How do search engines efficiently retrieve documents in a sorted (e.g. page rank) order from an already sorted (by document id) inverted index?

The standard way a search engine retrieves documents is by using an inverse index from words in the query to document ids. Since the ids are sorted, a query like "word1 AND word2" would fetch the ...
0
votes
0answers
177 views

Skip-Gram Window?

I'm reading about Skip-Gram and how it represents words as vectors. If I'm correct it not just represents a single words as a vector, but the word and it's neighbours. So if I have the dataset "the ...
1
vote
0answers
182 views

retrieve file format from unknown binary file [closed]

Is it possible, somehow, maybe by using some kind of brute force algorithm, to try and figure out what kind of file one is dealing with when one only has the raw binary without file format or meta ...
0
votes
1answer
159 views

Why isn't there a universal separator character for data files? [closed]

I am not sure if this is the correct forum for this question, but... After so much time and usage of spreadsheets and data files in general, there is a need to cater for all types of files containing ...
1
vote
1answer
234 views

Metrics for measuring the tradeoff between efficiency and effectiveness in Information Retrieval

I'm comparing several methods for building inverted indexes (n-grams, hash boundaries, etc.). Given a set of queries, these indexes perform differently. On one hand, I get performances metrics (...
2
votes
2answers
144 views

Can you get O(n) with a word frequency algorithm?

By a word frequency algorithm: An algorithm gets a document as an input, and returns each unique word along with the number of times it has appeared in the document. For example: in:"Hello my name ...
1
vote
0answers
69 views

Does Google's search algorithm really respect order of relevance imposed by PageRank? [closed]

I'm in doubt about that because in Google's results for a search on "bill clinton" [1], the server "The 'Unofficial' Bill Clinton" (94.06%) appears first than the server "President Bill Clinton - The ...
2
votes
1answer
51 views

How do IR researchers evaluate the ranks of documents?

I am developing a new IR system in a specialized context. I understand that a traditional IR system (like a search engine) should rank documents in terms of their relevance for a query. The most ...
3
votes
2answers
108 views

Calculate Pagerank of three nodes

Soon I will have an exam where I will be asked to calculate pagerank. So far, I have been thought to use power iteration. However, this can be a very expensive operation to do during an exam, ...
1
vote
1answer
91 views

phrase searches and vector space model

Consider a system that indexes documents based on vector space model and a simple query, such as qwe asd. When searching we assign weights to both words ...
2
votes
1answer
472 views

How to Know Whether a Topic is Publishable in Computer Science

Forgive my ignorance, but I am very new to the publishing process in Computer Science. I have some research that I have done involving analyzing data formats used in web-based image processing ...
3
votes
1answer
113 views

Natural language query processing

I am trying to implement a natural language query preprocessing module which would, given a query formulated in natural language, extract the keywords from that query and submit it to an IR (...
1
vote
2answers
1k views

Application of cosine similarity to detect plagiarism

Can anyone tell me how using cosine similarity to see the correlation between two documents actually shows you if someone is plagiarising the other? I understand how cosine similarity works but don't ...
0
votes
0answers
23 views

How to decode multiple-digit gamma codes and get the gap sequence? [duplicate]

How to decode gamma code ($\gamma$ code): 1110001110101011111101101111011 and get the gap sequence? Detailed information about Gamma codes ($\gamma$ codes) ...
2
votes
2answers
3k views

How to decode multiple-digit gamma codes and get the gap sequence?

How to decode gamma code ($\gamma$ code): 1110001110101011111101101111011 and get the gap sequence? Detailed information about Gamma codes ($\gamma$ codes) ...
3
votes
2answers
2k views

How to filter a very, very large file

I have a very large unsorted file, 1000GB, of ID pairs ID:ABC123 ID:ABC124 ID:ABC123 ID:ABC124 ID:ABC123 ID:ABA122 ID:ABC124 ID:ABC123 ID:ABC124 ID:ABC126 I would like to filter the file for 1) ...
5
votes
2answers
642 views

Estimate entropy, based upon observed frequency counts

Suppose I have $n$ independent observations $x_1,\dots,x_n$ from some unknown distribution over a known alphabet $\Sigma$, and I want to estimate the entropy of the distribution. I can count the ...
2
votes
1answer
337 views

Document definition in information retrieval

Is there common formal definition of Document in information retrieval field? In many researches authors don't define the term Document(maybe because it is evident for them). Wikipedia says "text file,...