# Questions tagged [big-data]

The tag has no usage guidance.

27 questions
Filter by
Sorted by
Tagged with
-3 votes
0 answers
25 views

### Replication and sharding in Big Data

Is there any database that uses both replication and sharding ? If yes, please explain the basic working of it .
• 1
3 votes
2 answers
90 views

### How do XFast and Y Fast Tries compare to B trees in performance?

I learned that Y fast tries support amortized loglog(u) time insertions , deletions. and loglog(u) time membership, successor and predecessor operations with O(n) space. So when n is closer to U in ...
• 123
0 votes
0 answers
19 views

### Need advice to run a unit test effectively on 14gb compressed data in dictionary form

I've got 14GB of compressed data for a class assignment. I need to run a unit test on modules I've created to make use of the uncompressed data, which will be an object of nested dictionaries with the ...
1 vote
1 answer
1k views

### What is the difference between FLOPS and OPS?

I have typically heard computer performance discussed in terms of FLOPS. However, I have recently seen multiple references instead using OPS i.e. operations per second, typically in the context of Big ...
• 111
1 vote
1 answer
79 views

### Solving number of distinct elements in $O(\frac{n\ell}{p})$ space complexity with $2p$ passes over data

Suppose there is an n-element stream with elements from $\{0,1\}^\ell$ which means each element is in set $\{0, \dots , 2^\ell-1\}$. Also may assume $2^\ell >n^2$. How can I with $2p$ passes over ...
• 47
3 votes
2 answers
64 views

### Semi-streaming algorithm for $s$-$t$ connectivity

Let $G=(V,E)$ be an undirected graph. Given a pair of vertices $s,t \in V$, how can we construct a semi-streaming algorithm which determines is $s$ and $t$ are connected? Is there any way to construct ...
• 181
0 votes
0 answers
45 views

### Hashfunction for unique character distributions

The original problem is given a large input file, with n input lines of random string, find the number of pairs-> meaning same number and type of characters, in the file. Constraint on type of ...
0 votes
0 answers
39 views

### Is it efficient to use large scale data algorithms on disk?

Is it efficient to use algorithms like locality sensitive hashing and bloom filters on disk instead of memory for very large datasets where even these structures cannot be saved in memory?
0 votes
1 answer
54 views

### How to check rapidly if an element is present in a large set of data

I am trying to harvest scientific publications data from different online sources like Core, PMC, arXiv etc. From these sources I keep the metadata of the articles (title, authors, abstract etc.) and ...
• 103
1 vote
2 answers
70 views

### Learning a perceptron from stream data

I want to train a Perceptron using stochastic gradient rulefrom the stream data. I have very limited amount of memory and i can store only $N$ examples. Suppose my population consist of point as ...
• 253
0 votes
1 answer
2k views

### Large-scale string clustering

I've more than 10 million strings of length 1-100 characters. This number will be even bigger in the future. I'm interested in clustering this data, but I'm not quite sure what would be effective at ...
1 vote
0 answers
29 views

### Clarification on MapReduce description in textbook

I am reading through chapter 2 of of the free textbook "Mining of Massive Datasets" (http://www.mmds.org/). On page 28 the following is stated: "It is reasonable to create one Map task for every ...
0 votes
0 answers
30 views

### Locality sensitive hashing with non-scalal values

Locality sensitive hashing works well when matching is between vectors of scalars, but I now need to extend LSH to compare matrices. Each matrix is formed of n readings from m sensors forming a n by m ...
• 101
0 votes
1 answer
104 views

### Where do you exactly use data structures?

I know what data structures is, How it works & importance of it. I just have some doubts on where do we actually use it. When database can do filtering, fetching, sorting efficiently, Why we ...
0 votes
1 answer
175 views

### Bias or not when finding patterns using data mining techniques?

I am currently following a course on Data Mining and i am very curious about the deeper underlying method. As far as i have learned so far data mining is about finding unknown patterns that can be ...
0 votes
1 answer
54 views

### Number of seeks in an I/O efficent algroithm

A seek is the task of positioning the read/write head of the disk to the required data block, and irrespective of the size of the data block, all contiguous data can be accessed using a single seek. ...
• 1,509
1 vote
1 answer
40 views

### Why splitted text files is bigger than a large one with the same content?

I have this large text file that when unzipped has about 2GB. I split this one into multiple(more than 5 million) files and now I have a folder of about 20GB, how is this possible?
1 vote
0 answers
23 views

### What are internal clustering index for binary data ? And if possible applicable to massive cluster ?

I was wondering what are the current existing internal clustering index for binary data. I know already the silhouette and Davis Bouldin for euclidian space, i suppose they work as well in binary ...
• 235
1 vote
1 answer
118 views

### Taking intersection in large search

As I understand, you can build the the word -> pages index in Google or large SQL database since indexed search has complexity O(1) -- lookup gives you a billion-page result at once ...
2 votes
1 answer
2k views

### (a,b)-tree vs B-tree

I would like to know what are the differences between (a,b)-tree and a B-tree. It has been a few days I am studying different papers and I am seeing different definitions that make me confused. For ...
• 1,509
1 vote
0 answers
535 views

### Examples of real world graphs that are too big for a single commodity-type machine

I've been reading on distributed systems for processing on large graphs. The most prominent examples include Pregel (developed by Google) and Apache Giraph. Most of these systems argue their existence ...
2 votes
2 answers
396 views

### Hashing by doing modulo $m$ for $m=p^2$ for a prime $p$ instead of using a prime $m$ - is it that bad?

I am doing an exercise from a Big Data course I'm taking on Coursera (this exercise is for experimenting with a big-data problem and is not for any credit or homework) , the assignment was described ...
• 267
3 votes
4 answers
944 views

### High Dimensional Data Structures

I have a 20-dimensional dataset, with a large amount of data points. I would like to have each dimension discretized into bins. Per bin, I would like to be able to access two neighbours per dimension (...
• 133
2 votes
0 answers
1k views

### Looking for dynamic network data sets [closed]

There are a number of collections of network (or graph) data sets freely available on the web, e.g. http://snap.stanford.edu/data/index.html http://www.cc.gatech.edu/dimacs10/downloads.shtml I am ...
• 21
3 votes
4 answers
2k views

### If I have a large random array of 0s and 1s that I want to sort what kind of an algorithm and data structures should I consider?

What are the types of things that need to be considered if I need to sort a large random array of 0s and 1s? You can assume large array is in the order of million or billions. I understand there ...
• 207
3 votes
2 answers
2k views

### Bloom Filter for 208 million URLs

I need to create a bloom filter of 208 million URLs. What would be a good choice of bit vector size and number of hash functions? I tried a bit vector of size 1 GB and 4 hash functions, but it ...
• 157
8 votes
4 answers
428 views

### Applying algorithms on large data

Is there any book or tutorial that teaches us how to efficiently apply the common algorithms (sorting, searching, etc.) on large data (i.e. data that cannot be fully loaded into main memory) and how ...
• 503