Questions tagged [information-retrieval]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
1 answer
17 views

Efficient search of a large set of documents to find documents that only contain a particular set of words

Say I have a set of documents $D = \{d_1, d_2, \dots, d_n\}$ in some natural language. Each document $d_i$ consists of a subset of words from a word pool $W = \{w_1, w_2, \dots, w_k\}$. For example, $\...
user avatar
0 votes
1 answer
37 views

How do I find the most similar phrase in "Extending Multi-Sense Word Embedding to Phrases and Sentences for Unsupervised Semantic Applications"?

This question is about de paper Extending Multi-Sense Word Embedding to Phrases and Sentences for Unsupervised Semantic Applications, depicted in the following picture: I am interested in more ...
user avatar
  • 139
0 votes
0 answers
20 views

Explain better how CompuBERT handles math tokens on input

My question is about CompuBERT (Three is Better than One Ensembling Math Information Retrieval Systems) It is written on page 21 (table 2): "we assigned a distinct mask for math tokens on input, ...
user avatar
  • 139
0 votes
0 answers
28 views

What is the best semantifier of mathematical notation?

I need an algorithm to read a formula like $E=mc^2$ and retrieves $E$ is energy. The best I could find are: Semantification of Identifiers in Mathematics for Better Math Information Retrieval. ...
user avatar
  • 139
0 votes
0 answers
17 views

Explain better the training mechanism of CompuBERT

This question is about CompuBERT (new implementation). If I have a question with an answer with no common words or mathematical notation, how CompuBERT will behave in this situation? Reasons why I am ...
user avatar
  • 139
0 votes
1 answer
44 views

Does CompuBERT suffer from the curse of dimensionality?

This question is about CompuBERT (new implementation). I have read textual data have high dimensionality so I would like to know the behaviour of CompuBERT which uses a dot product for question ...
user avatar
  • 139
0 votes
0 answers
56 views

Is Multiple Negatives Ranking Loss currently the best loss for question answering in CompuBERT?

This question is about CompuBERT (new implementation). (The multiple questions are closely related so don't close my question) Is Multiple Negatives Ranking Loss currently the best loss for question ...
user avatar
  • 139
0 votes
1 answer
38 views

A question about the paper "Ensembling Ten Math Information Retrieval Systems"

My question is about the paper Ensembling Ten Math Information Retrieval Systems. I am interested in the task of finding answers. Which of the ten system are able to answer questions using only dot ...
user avatar
  • 139
0 votes
0 answers
24 views

using TRIE for strings?

I saw the following question online: Init - Initlize data structure in O(1). Insert(s) - Add string s to your Data Structure in ...
user avatar
  • 1
1 vote
0 answers
33 views

Probability of a document is not the same as 1/number of documents, is it?

Currently, I am writing code in Python for a paper that I'm reading and has been implemented in R already by the author. For this paper we have a corpus with $n$ documents and $m$ words. In one of the ...
user avatar
  • 121
0 votes
2 answers
105 views

If we want to map abbreviations of full-English words (e.g. map "Jan" to "January"), how can we identify abbreviations which map to multiple words?

Short Version: How can we construct a trie which maps abbreviations of names-of-the-month to full-month (we map the abbreviation "mar" to "march")? The set of all abbreviations is ...
user avatar
2 votes
1 answer
85 views

How does an inverted index reduce storage requirements?

In p. 7 of the book "Introduction to Information Retrieval" (by Manning et al), the authors explain how, given a collection of text documents, an inverted index is built by tokenizing, then ...
user avatar
  • 21
1 vote
2 answers
49 views

Retrieving data from website - Parser vs AI

I'm currently working on a personal software project (as a hobby) and would like to know which kind of approach I should follow according to you. My application scans some websites (like Amazon or ...
user avatar
  • 111
0 votes
1 answer
40 views

Data extaction vs information extraction

How is information extraction related to data extraction? Is it a subfield or can it be used equivalently? Data is also information. What is the relation and what are the differences? I want to write ...
user avatar
  • 181
2 votes
1 answer
151 views

What is the point of the "check" array in a triple-array trie?

I am currently reading the definition of a triple-array trie as described here. More precisely, I am trying to understand the nature of the three arrays: base. Each element in base corresponds ...
user avatar
  • 123
0 votes
1 answer
28 views

F-measure with $\beta > 1$

Consider the F-measure $F_{\beta}$. I know that with $\beta < 1$ precision is given more importance, ending with $F_{0} = precision$. With $\beta > 1$ means recall gets the upper hand, with the ...
user avatar
0 votes
1 answer
18 views

A survey on ranking keyword search results

To rank the keyword search results, I'm trying to crack the way the Airbnb algorithm or similar ones work. I'm not asking which features they are using since those are different depending on the ...
user avatar
  • 227
1 vote
0 answers
115 views

Does the way that PageRank is heavily skewed lower the effectiveness of TF-IDF calculations?

PageRank is heavily skewed in that a small proportion of webpages have a high PageRank score while most of the others have a very low PageRank score. Does this undermine the effectiveness of the TF-...
user avatar
  • 41
2 votes
0 answers
28 views

Poset data structure to find least element, greater or equal to given

Let $A$ be a finite set, and $S \subset \mathcal{P}(A)$. Is there a data structure for $S$ that would allow to quickly retrieve an element $q \in S $, given a key $p \in \mathcal{P}(A)$, such that $q$ ...
user avatar
  • 145
1 vote
1 answer
26 views

What do "patterns, which separately match parts of a page" and "interactions between patterns" mean?

Database System Concepts by Abraham Silberschatz, Henry F. Korth and S. Sudarshan says Information extraction using simple patterns, which separately match parts of a page, is relatively error ...
user avatar
  • 1
3 votes
1 answer
55 views

Why does the GS1 DataMatrix encode diagonally instead of vertically or horizontally?

Perhaps this is the wrong exchange, so if there is a better place to post this please tell me! The DataMatrix encodes diagonally in a sort of zig zag pattern, as shown below. Why don't they encode ...
user avatar
-1 votes
1 answer
805 views

Find the Most Frequent Ordered Word Pair In a Document

This is a problem from Skiena's excellent book The Algorithm Design Manual: Give an algorithm for finding an ordered word pair(e.g."New York") occurring with the greatest frequency in a given ...
user avatar
1 vote
0 answers
59 views

How to use HITS algorithm to rank pages

I am trying to implement a simple web search engine mechanism with using HITS algorithm. I understand how the algorithm work and produce hubs and authority values for each page. But how can I use this ...
user avatar
1 vote
0 answers
141 views

Single Pass in Memory Indexing

I was reading single pass in memory indexing and had few doubts. Why is the time complexity of SPIMI O(T) where T are the token. We know that before writing blocks to Disk, we have to sort ...
user avatar
1 vote
2 answers
485 views

How does a Trie work?

Specifically, many say that to speed up an HTTP routing code going from regex to a Trie is the solution. Sometimes also a compressed Trie or Radix Tree. The problem though is that after reading ...
user avatar
3 votes
1 answer
203 views

Calculating Frequency of Terms with Zipf's Law

I'm studying for an exam and came across this question. I feel like it's much simpler than I am making it out to be, but I'm not sure. Suppose we have a corpus in which Zipf’s Law holds. If the most ...
user avatar
1 vote
2 answers
124 views

Query Logs and evaluation of search engines

I'm taking a course about Information Retrieval and we had a quiz involving the evaluation of search engines. We had a problematic question regarding "Query Logs" (basically recording the queries ...
user avatar
  • 135
2 votes
1 answer
480 views

Encoding a list of numbers using gamma coding

I'm trying to work out how to find the gamma $\gamma -code$ that encodes the postings list (12, 18, 21, 22). Is the postings list encoded just by a list of the individual $\gamma$-codes for each ...
user avatar
  • 31
1 vote
1 answer
36 views

How to figure out whether two texts refer to the same object or event

Let's assume there is something happen in the world - Football world cup final. And team-1 beat team-2 with the score 3:2. So there is whole bunch of articles on every website about it, each contains ...
user avatar
  • 141
0 votes
0 answers
16 views

Learning material on information visualization

If somebody with a computer science background wants to learn about information visualization, again from a computer science perspective, which will be the best learning material for him? A textbook ...
user avatar
  • 337
0 votes
0 answers
78 views

Algorithm of finding relationships between two sets given a bijection relationship between their cosets to another set

Let $C$ be the set of all available classes, $S$ be the set of all available students, $P$ be the set of all available phone numbers. Each student many have zero to many phone numbers, but each phone ...
user avatar
  • 101
2 votes
1 answer
439 views

How is average precision related to the precision-recall curve?

As background, in Information Retrieval, there are many metrics to assess how well search results are retrieving information. A simple one is precision, where we calculate the fraction of documents ...
user avatar
  • 413
0 votes
1 answer
41 views

Does this classify as Information retrieval?

I’m trying to figure out if what I’m doing would classify as Information Retrieval. According to this Thread: Information retrieval is based on a query - you specify what information you need and ...
user avatar
  • 103
0 votes
0 answers
118 views

Gensim LDA all topics zero

I'm trying to run Gensim's LdaModel on a corpus. I load the corpus myself from two XML files. I made a class to iterate over the text pieces in the XML files. It ...
user avatar
1 vote
0 answers
34 views

Evaluation of Information extraction a task

I have to evaluate my information extraction program. I have to evaluate it by the expectation-driven way. I don't know what that means. I would appreciate if you could help me. Thx.
user avatar
  • 41
7 votes
1 answer
401 views

What is the most recent comparison of signature files and inverted indices?

Modern papers on search indices often contain a statement that inverted indexes (posting lists) are categorically superior to signature files (bloom filters). Here are some examples from papers ...
user avatar
  • 529
5 votes
0 answers
439 views

What's known about web search query distribution?

Silverstein et al., Analysis of a Very Large AltaVista Query Log, SRC Technical Note '98 looked at 43 days of queries in 1998 and pulled out how many terms there are per query, what the top 25 queries ...
user avatar
  • 529
13 votes
2 answers
17k views

What is the difference between R precision and precision at K?

I am trying to understand the difference between these two but it looks like as if they are calculated the same way. What is the difference? R precision: R precision is the precision at the Rth ...
user avatar
4 votes
1 answer
790 views

What is a difference between cross-lingual IR and multi-lingual IR?

In papers usually Cross-Lingual Information Retrieval (CLIR) and Multi-Lingual Information Retrieval (MLIR) use equivalently or distinctly. I want to know is there any difference between these two ...
user avatar
  • 177
4 votes
2 answers
230 views

Measuring the information of a document?

I'd like to measure how much information a document $D$ contains. Clearly, the New York Times published yesterday contains more information than my diary wrote on the same day. But, I do not know ...
user avatar
6 votes
1 answer
882 views

What would a formal grammar for a binary file format look like?

Binary structures often feature length specifiers; the parser is supposed to read them and then consume the specified amount of symbols. Because of this, the grammar is context-sensitive. What would ...
user avatar
-1 votes
1 answer
111 views

How do I calculate values in K-means (clustering)?

I'm learning some Data Retrieval concepts, one of them being K-means clustering. I understand that it's meant to help understand data, but that's where I begin having issues. I'm working on this ...
user avatar
  • 101
1 vote
2 answers
181 views

How do search engines efficiently retrieve documents in a sorted (e.g. page rank) order from an already sorted (by document id) inverted index?

The standard way a search engine retrieves documents is by using an inverse index from words in the query to document ids. Since the ids are sorted, a query like "word1 AND word2" would fetch the ...
user avatar
0 votes
0 answers
215 views

Skip-Gram Window?

I'm reading about Skip-Gram and how it represents words as vectors. If I'm correct it not just represents a single words as a vector, but the word and it's neighbours. So if I have the dataset "the ...
user avatar
1 vote
0 answers
336 views

retrieve file format from unknown binary file [closed]

Is it possible, somehow, maybe by using some kind of brute force algorithm, to try and figure out what kind of file one is dealing with when one only has the raw binary without file format or meta ...
user avatar
0 votes
1 answer
500 views

Why isn't there a universal separator character for data files? [closed]

I am not sure if this is the correct forum for this question, but... After so much time and usage of spreadsheets and data files in general, there is a need to cater for all types of files containing ...
user avatar
  • 11
1 vote
1 answer
513 views

Metrics for measuring the tradeoff between efficiency and effectiveness in Information Retrieval

I'm comparing several methods for building inverted indexes (n-grams, hash boundaries, etc.). Given a set of queries, these indexes perform differently. On one hand, I get performances metrics (...
user avatar
2 votes
2 answers
385 views

Can you get O(n) with a word frequency algorithm?

By a word frequency algorithm: An algorithm gets a document as an input, and returns each unique word along with the number of times it has appeared in the document. For example: in:"Hello my name ...
user avatar
  • 21
1 vote
0 answers
74 views

Does Google's search algorithm really respect order of relevance imposed by PageRank? [closed]

I'm in doubt about that because in Google's results for a search on "bill clinton" [1], the server "The 'Unofficial' Bill Clinton" (94.06%) appears first than the server "President Bill Clinton - The ...
user avatar
  • 139
2 votes
1 answer
64 views

How do IR researchers evaluate the ranks of documents?

I am developing a new IR system in a specialized context. I understand that a traditional IR system (like a search engine) should rank documents in terms of their relevance for a query. The most ...
user avatar