Questions tagged [data-mining]

Using the techniques of artificial intelligence and machine learning to extract patterns from large data sets and transforming those data into a useful, organized form for future processing.

Filter by
Sorted by
Tagged with
7
votes
1answer
2k views

Computer vision: object detection with labels that are single coordinates

Are there papers in the literature that address the following object detection task ? The task can be described as follows: Given a set of images, the labels are just coordinates (x,y) that ...
1
vote
0answers
21 views

What is the name for this form of analysis? [closed]

Sorry for the vague question, but I'm trying to figure out what to call something and a more specific question title would likely require too much space. Anyway... I'm getting into ROM hacking for ...
4
votes
1answer
6k views

Detecting events in time series data

I am collecting data from a sensor over time, and I'm trying to figure out how to detect "events" in the data - specifically, when a given event begins and ends. The frequency, duration, and amplitude ...
1
vote
1answer
64 views

Find string patterns preferably in regex for string streams

I am trying to classify the data in a database columns. DB has about 90 million entries.The main goal is to find different patterns in columns to leverage it for create look alike data. The data ...
2
votes
0answers
52 views

Clustering with probabilities / vector quantization with arbitrary distance measures

Suppose I'm given $n$ points $x_1,\dots,x_n$ in some space $\mathcal{S}$ (think: $\mathbb{R}^d$), and probabilities $p_1,\dots,p_n$ that form a probability distribution (so $p_1 + \dots + p_n=1$). ...
3
votes
2answers
85 views

Gramatical evolution doesn't result on creating good constants

I'm using Grammatical Evolution for symbolic regression tasks (i.e. searching the space of mathematical expressions to find the model that best fits a given dataset). I'm evolving solutions according ...
1
vote
0answers
44 views

detecting anomalies in time series data

I work on a web project that handles region based user submissions. We currently have about 50 regions, some receive a large number of submissions, some receive next to none. We also ingest case ...
2
votes
2answers
279 views

How to generate response variable during machine learning?

I am analyzing data of insurance companies and for some reasons the data that was provided doesn't have any response variable of whether an insurance claim is legit or suspicious. Are there any ways ...
0
votes
1answer
1k views

About the MNIST data-set

Is this known as a fact or from some analysis that the MNIST data-set is almost as if its sampled from some low (~10?) dimensional manifold? Is there a locally linear embedding to a low-dimensional ...
1
vote
0answers
116 views

Can file entropy be predictable?

I am working on some lossless compression topic and looking on any resources or prior studies on lossless file entropy predictability. Assuming we know type of the file (we can tell its belongs to ...
8
votes
1answer
11k views

Machine Learning: Identify Patterns in Time-Series Data

I work in renewable energy. My company gathers a lot of data from equipment. This typically includes process data (such as transformer temperature, line voltages, currents, etc.) and discrete alarms (...
2
votes
0answers
2k views

two ways of calculating the entropy in attribute selection (decision tree)

The definition of the entropy is $$H(Y) = -\sum p(y_j)\log_2 p(y_j)\,.$$ Now my text book says to compute the entropy for each attribute we consider the grouping of the data by that attribute now ...
3
votes
1answer
412 views

In Data Mining, what does it mean to be greedy?

I am looking at a number of algorithms in Data Mining and some are described as being greedy. My issue is that they seem to be using the term greedy in different ways, which seems contradictory. For ...
2
votes
1answer
99 views

Standardizing Data for Neural Networks

Let's say we have a data set with following features [age, sex, country, city, annual income] [35, male, USA, New York, 73000]. I came across the article which ...
2
votes
1answer
57 views

Exhaustive list of ways to distribute n objects to k sets

I am working on a research paper and we are developing a brute force algorithm to examine another clustering technique. In this brute force algorithm we test every possible clustering example and see ...
3
votes
1answer
140 views

Algorithm to find pronounciation rules

Suppose that you have a large dictionary with spellings and pronounciations of foreign words, and you want to find a set of pronunciation rules. They should have the simplest form: a sequence of ...
0
votes
0answers
51 views

What are some data mining algoritms used in finance?

I want to test the applicability of homomorphic encryption in the financial domain, as suggested in Can homomorphic encryption be practical ?. Now, I want to know what algorithms are used for ...
0
votes
2answers
401 views

What to do when the information gain on decision trees is 0 for all possible splits?

I just started studying decisions trees and I am trying to construct a tree for a training set which uses Status as the class label. I am using the misclassification error as measure of impurity. ...
3
votes
1answer
70 views

Expected number of common edges for a given tree with any other tree

So I am working on a problem where I have a set of (labeled) nodes and I have a tree structure (rooted) over that set of nodes. The goal for me is to automatically generate that tree structure. To ...
1
vote
0answers
533 views

Text data comparison

Okay lets say i have two data structures . two phone data for example containing their Name and spec ( cpu , ram , display etc ) . I want to check if these two phones are the same or not . Their names ...
1
vote
1answer
952 views

How to identify labels in unsupervised learning?

Let's say I am working on handwritten digit recognition (0 to 9). I know for instance that if I use clustering then I need to look for 10 clusters. But once I have the 10 clusters,how do I identify ...
0
votes
2answers
128 views

Transforming training data for machine learning algorithms

If you want to make good predictions with machine learning (supervised learning in particular), you need a good training set. And relevant predictors in your feature set can be overshadowed by ...
0
votes
0answers
30 views

what is the best theory/model to use for prediction in multivariate data?

I use a software for pollutant propagation on rivers that takes as input a set of parameters (p1, p2, ...pn) and creates an output file which is basically a matrix where on each row there is ...
9
votes
2answers
158 views

What are some efficient ways to find the differences between two large corpuses of text that have similar, but differently ordered content?

I have two large files containing paragraphs of English text: The first text is about 200 pages long and has about 10 paragraphs per page (each paragraph is 5 sentences long). The second text ...
0
votes
1answer
48 views

Definition and properties of support

From Xiong, Hui, Shashi Shekhar, Pang-Ning Tan, and Vipin Kumar. “TAPER: A Two-Step Approach for All-Strong-Pairs Correlation Query in Large Databases.” Knowledge and Data Engineering, IEEE ...
2
votes
2answers
307 views

Can you get O(n) with a word frequency algorithm?

By a word frequency algorithm: An algorithm gets a document as an input, and returns each unique word along with the number of times it has appeared in the document. For example: in:"Hello my name ...
0
votes
0answers
35 views

Adjust FCM algorithm with size constrains

I want to cluster a list of property addressed based on distance and limit the size of each cluster. The properties are currently assigned to modules by discretion of managers. But I hope to develop ...
1
vote
1answer
210 views

Adding concept drift to data sets

I'm about to work with concept drift problem in data streams. I need to start with real data sets from UCI machine learning repository and add to them concept drift (in attributes domain). Do you ...
2
votes
1answer
78 views

Most frequently tools or programming language for implementation text processing and nlp algorithms in academic papers and journals [closed]

I want to prototype and try some idea (some algorithm) in the field of text processing and nlp and if the results was good I want to publish some paper or journal article about that. I am familiar ...
2
votes
1answer
76 views

Looking for a paper comparing NLP methods with Data Mining techniques

I've recently attended a conference, where one of the participants mentioned a recent paper published by a Google employee, which showed that using data mining techniques in application to NLP might ...
1
vote
0answers
185 views

Formula for number of parameters in an undirected graphical (probability) model

I have googled endlessly, and I cannot find it. Can anyone point me to a reference that gives a way to calculate the number of parameters in an undirected Graphical Model? Adapting from the similar ...
0
votes
0answers
22 views

Is this an accepted/valid clustering evaluation metric?

We have a clustering algorithm where the number of clusters isn't known to the algorithm - it iteratively creates clusters out of similar-looking data points. The evaluation metric we're currently ...
4
votes
1answer
97 views

Extracting maximum information from a set of exam answers and their scores

Imagine we have a multiple-choice exam with N questions. Suppose we have a set of K answer sheets to the exam and their total scores (1 for a correct answer on a question, 0 for incorrect). How much ...
2
votes
2answers
2k views

Application of cosine similarity to detect plagiarism

Can anyone tell me how using cosine similarity to see the correlation between two documents actually shows you if someone is plagiarising the other? I understand how cosine similarity works but don't ...
0
votes
0answers
77 views

Is k-means parallelizable (other than the data parallelism of the distance function computation)

The k-means algorithm is known to be NP-hard. While the distance function computation in the algorithm loop is data parallel, the algorithm is iterative and may become exponential in the number of ...
0
votes
1answer
716 views

Automatically generate meaningful queries for a data table

My field of research is not Database or AI. But I have some problems to solve, and would like to know which branch this kind of problems belong to, and what are the results. The main question is: ...
0
votes
0answers
172 views

Activity prediction in a kitchen

Here is the scenario: There are three chefs(A- main chef, B and C- assistant) working together to prepare a diner set. The sequence of the event is as following. Start: The three chefs enter the ...
8
votes
1answer
13k views

difference between multilayer perceptron and linear regression

What is the difference between multilayer perceptron and linear regression classifier. I am trying to learn a model with numerical attributes, and predict a numerical value. Thanks
2
votes
0answers
67 views

Baseline approaches for likes prediction

I have a small user-item matrix (25k x 1.8k) describing how users liked or disliked some of the items. Users don't have any attributes but items have several features. I would like to be able to ...
-1
votes
1answer
491 views

Training Error & Convergence to True Error

I Take some online class for Machine Learning. one of teacher say this sentence. if we have m data points, the training error converges to the true error as m → ∞. i thought, this sentence not ...
0
votes
1answer
4k views

VC Dimension Calculation for Intervals

As i See in ML Course a VC dimension calculation is very theoretical. What is the VC-dimension of intervals in R? The target function is specifieed by an interval, and labels any example positive ...
-1
votes
1answer
68 views

Policy function π in Reinforcement learning unclear

I have one question about policy function in Reinforcement learning. in fact this function indicates which action should be done in each state? Or this function indicate for get the ...
-1
votes
1answer
102 views

Selecting a M.S. in CS focus [closed]

I need to pick a focus in my Masters program. The running candidates are Networking Bioinformatics Data mining Computer Image Recognition In order to avoid this becoming a discussion, I'd be ...
13
votes
5answers
17k views

Word Frequency with Ordering in O(n) Complexity

During an interview for a Java developer position, I was asked the following: Write a function that takes two params: a String representing a text document and an integer providing the ...
0
votes
2answers
46 views

Determine Epsilon for identification [closed]

I have a project in which I need to compare different distances in a database with a distance in input in order to identify a person. For that I use this expression: DistanceDB - DistanceInput < ...
2
votes
2answers
2k views

Need an algorithm to find the input factors that are most affecting the output

I apologize if this question is already answered and appreciate any pointers to existing answers. I'm not familiar with statistical or data mining terms so my search was limited to basic words used in ...
2
votes
1answer
641 views

Best algorithm for correlation between time series?

I have some biological data (ECG), which are quite chaotic in nature, and and some other data; that are not chaotic but related in some way, like fatigue. I want to find out how the time series, ...
3
votes
0answers
77 views

What are some of the methods that NLP practitioners use to automatically learn linguistic features from text? [closed]

I am learning about NLP, with an eye to starting some practical NLP projects. I see that many of the algorithms for relation extraction and named entity recognition require you to identify linguistic ...
2
votes
1answer
42 views

Subspace clustering with random transformation

One approach for clustering a high dimensional dataset is to use linear transformation, and the most common approaches are PCA and random projection (where random projection arises from the Johnson-...
3
votes
1answer
923 views

Why linear transformation can improve classification accuracy when the dimensionality of data is high?

Let $X$ be an $m\times n$ ($m$: number of records, and $n$: number of attributes) dataset. When the number of attributes $n$ is large and the dataset $X$ is noisy, classification gets more ...