Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-mining]

Using the techniques of artificial intelligence and machine learning to extract patterns from large data sets and transforming those data into a useful, organized form for future processing.

0
votes
0answers
13 views

Difference between Graph clustering and Node clustering

I have confused with Node clustering and graph-clustering, what is the difference? Can anyone help me to distinguish these two properly with small examples?
1
vote
2answers
59 views

What kind of logarithm does $\log$ mean in Wikipedia articles?

Sorry if this seems like a dumb question, but what what type of logarithm is $\log$ in Wikipedia articles? Cheers.
2
votes
0answers
16 views

In topological data analysis, do bar codes that begin and end at the same index mean anything?

The typical workflow in topological data analysis is from point cloud data to filtration to a list of bar codes corresponding to each dimension. A filtration is a sequence of simplicial complexes, ...
0
votes
0answers
17 views

Conditional probability in Bayesian network

I have the following Bayesian network: I want to state that P(S|B) = 1 - P(!S|B) In the solution they use "P(S|B) = P(S|B,F) + P(S|B,!F)", which I understand but I don't understand why can't I use ...
2
votes
0answers
24 views

What is the definition of a “Clustering Feature” in BIRCH algorithm?

The paper for BIRCH (a clustering algorithm) contains definitions of a Clustering Feature (CF) where the notation is unclear (cf. PDF page 3 / section 4). A cluster contains N d-dimensional entries $ ...
0
votes
0answers
23 views

Lower Bound Space complexity one pass algorithm / Heavy-Hitters Problem

I am confronted with the following problem: Let S be the family of all m-subsets of $[n] = [2m]$ let $S_1, S_2 \in S$ be distinct sets and let the state of storage be $State_1$ after stream $S_1$ is ...
0
votes
0answers
20 views

What are Key benefits of Ontologies in Systematic Literature Review?

I am working on a Systematic Literature Review (SLR) and about to done with data synthesis. After SLR, I want to create an Ontology and include different details of the SLR in Ontology. I have almost ...
0
votes
1answer
54 views

Bias or not when finding patterns using data mining techniques?

I am currently following a course on Data Mining and i am very curious about the deeper underlying method. As far as i have learned so far data mining is about finding unknown patterns that can be ...
1
vote
0answers
38 views

Combining Computer Science and Humanities

I currently hold a bachelors in Computer science and a masters in Art History. I really want to combine the two and I know of Digital Humanities but I'm not completely aware of where Digital Humanists ...
0
votes
0answers
19 views

Can one extract if-then association rules from database using decision trees?

I'm learning the field of data-mining to extract meaningful patterns from a database, so far I saw Apriori and FP-Growth, but being familiar with decision trees I was sure I was going to find an ...
0
votes
1answer
31 views

How is density of a substance in grams per cubic centimeter a discrete attribute?

I'm going over the different types of attributes in data-mining (nominal, ordinal, discrete, etc) and there is one example in the book that states that the density of a substance in grams per cubic ...
0
votes
0answers
42 views

Naviers Stokes equation and machine learning

I am looking for a reference explaining how to solve Navier-Stokes numerically using Machine learning algorithms . Thank you in advance for your help .
1
vote
0answers
65 views

Understanding Time Series Data for Classification

I have collected data from numerous volunteers driving a simulator in 8 different scenarios (classes). A volunteer drives in a map for 4 minutes in one scenario (one experiment), then he drives in the ...
3
votes
1answer
60 views

Finding (and possibly extracting) source code in heterogenous text data set

I'm looking for a way to recognize and possibly extract source code from text files that may contain only source code, source code mixed with plain text or just plain text without any source code. ...
1
vote
1answer
30 views

How to figure out whether two texts refer to the same object or event

Let's assume there is something happen in the world - Football world cup final. And team-1 beat team-2 with the score 3:2. So there is whole bunch of articles on every website about it, each contains ...
1
vote
0answers
296 views

Algorithms for tabulating/counting/frequency counting?

It is common in data science to receive two equal length vectors (array of dimension 1), say Categories and Weights. We aim to find all unique values of Categories and sum up the corresponding ...
1
vote
1answer
67 views

Data Searching from a large data set without reading each element

I have just started learning algorithms and data structures and I came by an interesting problem. I need some help in solving the problem. There is a data set given to me. Within the data set are ...
1
vote
0answers
23 views

How may I look for 'regions' of text in a larger corpus of different texts

I have an extremely large (100GB+) corpus of many different texts. All of them are in English and 'well' formatted. They are not loaded into any kind of database, think of them as a huge collection of ...
2
votes
1answer
99 views

Specific Examples with Explanation of Similarities and Differences of how Distance Functions are used Across Different Fields [closed]

I took a tangent from a student project I had done a number of years ago and spent some time studying distance functions. (please note that the above link contains the full question with links as I ...
1
vote
0answers
146 views

What is the best stream data clustering algorithm that can handle non-static, uncertain data? [closed]

I have gone through many algorithms including streaming k-means, CluStream etc and they all have their pros and cons. What is the best performing algorithm in terms of Computational Complexity ...
1
vote
0answers
32 views

which paper to take that would best fit to becoming a data scientist? [closed]

My name is Webber from Auckland, New Zealand. I am currently in my second year studying the bachelor of computer and information science. I have chosen to do a double major in computational ...
0
votes
1answer
307 views

List count of occurrences pairs, triplets, etc. from sets

A receipt is an array of products. I have an array of receipts. I need to generate a report in where I can find the products often bought together. For instance, for a single receipt where the ...
1
vote
0answers
36 views

Outline detection from patterns in a list of textual articles

Are there NLP algorithms dealing with detecting the repeating patterns in a a list of texts from which a topic keywords and other associative keywords can be derived? I will show it as an example: ...
1
vote
2answers
181 views

How to use Neural Network classification if data not same size?

I have data like this. [0 1 0 1 0] [0 1 0 1 0 1 1] [0 1 0 1 ] [0 1 0 1 0 1 1 1 1 0] ... I want to classify with Neural Network but my data different size . I can ...
2
votes
0answers
21 views

Where to cut a category tree

Since I don't have CS background I will most probably ask this question the wrong way. I need to choose a node from a tree, where I include all beneath this node leafs in a validation. I have a data ...
10
votes
4answers
5k views

Data Science vs Operations Research

The general question, as the title suggests, is: What is the difference between DS and OR/optimization. On a conceptual level I understand that DS tries to extract knowledge from the available data ...
1
vote
0answers
62 views

Indoor path prediction with machine learning and pattern matching

Ok. You'll have to bear with me because I'm new to this. I have an idea for a research project. It involves trying to predict a path that someone might take in an indoor environment. My idea is to use ...
1
vote
1answer
182 views

Method for finding correlation between data sets

Let's say that I have $N$ data sets where I have data points at some fixed frequency, such as "daily". What would be a good method for finding correlation between any of the data sets, or choosing a ...
0
votes
0answers
53 views

LSH and bands for document similarity

I have 50 documents and im using LSH+Minhashing Jaccard and i get this results: for bands:25 -> false positives: 200 for bands:10 -> false positives: 80 for bands:2 -> false positives: 28 for bands:1 ...
0
votes
0answers
79 views

Minhash Jaccard to find similarity between documents

I wrote a program to check the similarity of 40 documents. Firstly i use exact Jaccard method and secondly Minhash Jaccard method. I consider that the 2 methods differ if their values are differ more ...
6
votes
0answers
108 views

Guided mining of common substructures in large set of graphs

Disclaimer: I'm not a CS so I basically have no idea what I'm talking about I have a large (>1000) set of directed acyclic graphs with a large (>1000) set of vertices each; the vertices are labeled. ...
3
votes
1answer
39 views

A decision algorithm to choose what partner to use

Background I have a setup where I need to make an API call to a partner to initiate a financial transaction. I have multiple partners, and I need to choose which one to use each time a transaction ...
1
vote
1answer
160 views

Does a distance matrix have to be Euclidean in order to be clustered by an average-linking algorithm (UPGMA)?

What are the exact assumptions behind the use of UPGMA? Can I use a non-Euclidean metric? This may result in a non-Euclidean distance matrix. What kind of bias may I encounter if I do so? References ...
7
votes
2answers
236 views

any hope for a universal automatic parser?

Say you are a program, and you are given some source code but you don't know in what language, it can be C++/Java/Python/Lisp/... all you know is that it is highly structured and LR(1) parse-able, and ...
0
votes
1answer
72 views

What is the metadata in microblogging platform?

I'm reading papers regarding microblog credibility perception of non-text cues. Tweeting is believing? Understanding microblog credibility perceptions by Morris et al., 2012 Microblog Credibility ...
3
votes
2answers
107 views

Measuring the information of a document?

I'd like to measure how much information a document $D$ contains. Clearly, the New York Times published yesterday contains more information than my diary wrote on the same day. But, I do not know ...
7
votes
1answer
2k views

Computer vision: object detection with labels that are single coordinates

Are there papers in the literature that address the following object detection task ? The task can be described as follows: Given a set of images, the labels are just coordinates (x,y) that ...
1
vote
0answers
19 views

What is the name for this form of analysis? [closed]

Sorry for the vague question, but I'm trying to figure out what to call something and a more specific question title would likely require too much space. Anyway... I'm getting into ROM hacking for ...
3
votes
1answer
3k views

Detecting events in time series data

I am collecting data from a sensor over time, and I'm trying to figure out how to detect "events" in the data - specifically, when a given event begins and ends. The frequency, duration, and amplitude ...
1
vote
1answer
59 views

Find string patterns preferably in regex for string streams

I am trying to classify the data in a database columns. DB has about 90 million entries.The main goal is to find different patterns in columns to leverage it for create look alike data. The data ...
2
votes
0answers
45 views

Clustering with probabilities / vector quantization with arbitrary distance measures

Suppose I'm given $n$ points $x_1,\dots,x_n$ in some space $\mathcal{S}$ (think: $\mathbb{R}^d$), and probabilities $p_1,\dots,p_n$ that form a probability distribution (so $p_1 + \dots + p_n=1$). ...
2
votes
2answers
67 views

Gramatical evolution doesn't result on creating good constants

I'm using Grammatical Evolution for symbolic regression tasks (i.e. searching the space of mathematical expressions to find the model that best fits a given dataset). I'm evolving solutions according ...
1
vote
0answers
40 views

detecting anomalies in time series data

I work on a web project that handles region based user submissions. We currently have about 50 regions, some receive a large number of submissions, some receive next to none. We also ingest case ...
2
votes
2answers
197 views

How to generate response variable during machine learning?

I am analyzing data of insurance companies and for some reasons the data that was provided doesn't have any response variable of whether an insurance claim is legit or suspicious. Are there any ways ...
0
votes
1answer
700 views

About the MNIST data-set

Is this known as a fact or from some analysis that the MNIST data-set is almost as if its sampled from some low (~10?) dimensional manifold? Is there a locally linear embedding to a low-dimensional ...
1
vote
0answers
100 views

Can file entropy be predictable?

I am working on some lossless compression topic and looking on any resources or prior studies on lossless file entropy predictability. Assuming we know type of the file (we can tell its belongs to ...
7
votes
1answer
9k views

Machine Learning: Identify Patterns in Time-Series Data

I work in renewable energy. My company gathers a lot of data from equipment. This typically includes process data (such as transformer temperature, line voltages, currents, etc.) and discrete alarms (...
2
votes
0answers
2k views

two ways of calculating the entropy in attribute selection (decision tree)

The definition of the entropy is $$H(Y) = -\sum p(y_j)\log_2 p(y_j)\,.$$ Now my text book says to compute the entropy for each attribute we consider the grouping of the data by that attribute now ...
3
votes
1answer
243 views

In Data Mining, what does it mean to be greedy?

I am looking at a number of algorithms in Data Mining and some are described as being greedy. My issue is that they seem to be using the term greedy in different ways, which seems contradictory. For ...
3
votes
1answer
97 views

Standardizing Data for Neural Networks

Let's say we have a data set with following features [age, sex, country, city, annual income] [35, male, USA, New York, 73000]. I came across the article which ...