Questions tagged [data-mining]

Using the techniques of artificial intelligence and machine learning to extract patterns from large data sets and transforming those data into a useful, organized form for future processing.

46 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
10
votes
0answers
128 views

Applying the graph mining algorithm Leap Search in an unlabeled setting

I am reading Mining Significant Graph Patterns by Leap Search (Yan et al., 2008), and I am unclear on how their technique translates to the unlabeled setting, since $p$ and $q$ (the frequency ...
7
votes
0answers
112 views

Guided mining of common substructures in large set of graphs

Disclaimer: I'm not a CS so I basically have no idea what I'm talking about I have a large (>1000) set of directed acyclic graphs with a large (>1000) set of vertices each; the vertices are labeled. ...
3
votes
0answers
80 views

Document clustering for summarization

I am curious as to what steps one would reasonably need to take to perform an extraction-based text summarizer. I've taken a look at some papers I've found on Google such as this one, which explains ...
2
votes
0answers
21 views

In topological data analysis, do bar codes that begin and end at the same index mean anything?

The typical workflow in topological data analysis is from point cloud data to filtration to a list of bar codes corresponding to each dimension. A filtration is a sequence of simplicial complexes, ...
2
votes
0answers
69 views

What is the definition of a “Clustering Feature” in BIRCH algorithm?

The paper for BIRCH (a clustering algorithm) contains definitions of a Clustering Feature (CF) where the notation is unclear (cf. PDF page 3 / section 4). A cluster contains N d-dimensional entries $ ...
2
votes
0answers
23 views

Where to cut a category tree

Since I don't have CS background I will most probably ask this question the wrong way. I need to choose a node from a tree, where I include all beneath this node leafs in a validation. I have a data ...
2
votes
0answers
52 views

Clustering with probabilities / vector quantization with arbitrary distance measures

Suppose I'm given $n$ points $x_1,\dots,x_n$ in some space $\mathcal{S}$ (think: $\mathbb{R}^d$), and probabilities $p_1,\dots,p_n$ that form a probability distribution (so $p_1 + \dots + p_n=1$). ...
2
votes
0answers
2k views

two ways of calculating the entropy in attribute selection (decision tree)

The definition of the entropy is $$H(Y) = -\sum p(y_j)\log_2 p(y_j)\,.$$ Now my text book says to compute the entropy for each attribute we consider the grouping of the data by that attribute now ...
2
votes
0answers
67 views

Baseline approaches for likes prediction

I have a small user-item matrix (25k x 1.8k) describing how users liked or disliked some of the items. Users don't have any attributes but items have several features. I would like to be able to ...
2
votes
0answers
138 views

What limits the implementation of proximity operators for text indexing and searching?

I was asking on academia.se, if anyone knows scientific search engines offering a proximity operator like Google Web Search does, while Google Scholar Search does not. That's sad, because this ...
1
vote
0answers
10 views

About the paper Privacy-preserving in association rule mining using an improved discrete binary artificial bee colony

I don't understand two parts in this paper: The min notion on page 4 line 357 (equation 10d): I understand this as to find all the $M_{10}$, $M_{11}$, $M_{01}$ first and then try to minimize the ...
1
vote
0answers
42 views

Can anyone think of applications of a 3 way (k-way) dot product in computer science or data mining

I have developed a locality sensitive hashing algorithm for the 3-way or k-way dot product. When I say 3-way dot product I mean the following. Suppose we have $x,y,z \in [-1,1]^{S}$ for $S \in \...
1
vote
1answer
77 views

Why is it not always possible to compute the centroid of feature vectors?

Hi in the data mining and machine learning course that I'm taking there is a subject on feature spaces and there is this part about feature vector aggregation and metric spaces that I don't really ...
1
vote
0answers
16 views

While computing signature matrix in min hashing, can I take nth row of the permutation P in which document d has value 1?

I am learning about some techniques to find similarity between documents. One of the methods is Min Hashing. According to Min Hashing we can find a signature matrix given a random permutation, P. ...
1
vote
0answers
66 views

Combining Computer Science and Humanities

I currently hold a bachelors in Computer science and a masters in Art History. I really want to combine the two and I know of Digital Humanities but I'm not completely aware of where Digital Humanists ...
1
vote
0answers
73 views

Understanding Time Series Data for Classification

I have collected data from numerous volunteers driving a simulator in 8 different scenarios (classes). A volunteer drives in a map for 4 minutes in one scenario (one experiment), then he drives in the ...
1
vote
0answers
590 views

Algorithms for tabulating/counting/frequency counting?

It is common in data science to receive two equal length vectors (array of dimension 1), say Categories and Weights. We aim to find all unique values of Categories and sum up the corresponding ...
1
vote
0answers
24 views

How may I look for 'regions' of text in a larger corpus of different texts

I have an extremely large (100GB+) corpus of many different texts. All of them are in English and 'well' formatted. They are not loaded into any kind of database, think of them as a huge collection of ...
1
vote
0answers
37 views

Outline detection from patterns in a list of textual articles

Are there NLP algorithms dealing with detecting the repeating patterns in a a list of texts from which a topic keywords and other associative keywords can be derived? I will show it as an example: ...
1
vote
0answers
75 views

Indoor path prediction with machine learning and pattern matching

Ok. You'll have to bear with me because I'm new to this. I have an idea for a research project. It involves trying to predict a path that someone might take in an indoor environment. My idea is to use ...
1
vote
0answers
44 views

detecting anomalies in time series data

I work on a web project that handles region based user submissions. We currently have about 50 regions, some receive a large number of submissions, some receive next to none. We also ingest case ...
1
vote
0answers
116 views

Can file entropy be predictable?

I am working on some lossless compression topic and looking on any resources or prior studies on lossless file entropy predictability. Assuming we know type of the file (we can tell its belongs to ...
1
vote
0answers
533 views

Text data comparison

Okay lets say i have two data structures . two phone data for example containing their Name and spec ( cpu , ram , display etc ) . I want to check if these two phones are the same or not . Their names ...
1
vote
0answers
185 views

Formula for number of parameters in an undirected graphical (probability) model

I have googled endlessly, and I cannot find it. Can anyone point me to a reference that gives a way to calculate the number of parameters in an undirected Graphical Model? Adapting from the similar ...
1
vote
0answers
62 views

How to cluster nodes based on the number of dependencies

I have a problem where, there are a set of nodes and dependencies between them. I want to cluster them based on the maximum number of dependencies. Dependencies can be thought of as number of edges ...
1
vote
0answers
48 views

Is it possible to discern the quality of web data from networks derived from it?

One topic I've recently looked at is co-occurence networks formed from Twitter tweets. This is how I felt after looking at the tweets of random people: This leads me to the question: Question: Is ...
0
votes
0answers
14 views

When outputting every timestep of an LSTM, is outputs in timesteps with missing training data also affected by overfitting?

I'm referring to things like the "timedistributed" layer in Keras or how in Pytorch LSTM the output is included in every timestep. An example illustrating my question, say I'm trying to ...
0
votes
0answers
8 views

Which model to apply on such panel data with so may rows but for each unique id rows are 6-8 rows per unique id?

I am new to such panel data where I have multiple observation for same ID in different Quarter and I am not sure what kind of machine learning algorithm I can apply. I have data from Q1-18 till Q4-...
0
votes
0answers
9 views

building a unsupervised learning model to detect suspicious transactions using DBSCAN

I am working for the first time on building a unsupervised learning model to detect suspicious transactions using DBSCAN, Do I train the model on all data columns(Columns like account number, ...
0
votes
0answers
27 views

How to handle distribution of values with same attributes into different classes

I'm a student studying a data mining course and have come across a problem. I need to explain the problem with the help of an example scenario as I do not know how to explain the problem in any other ...
0
votes
0answers
36 views

Multi-Interval Discretization of Continuous-Valued Attributes for Classification Learning

I'm trying to figure out Fayyad and Irani (1993) MDLP discretization of continuous variables (here is link to the original paper). I understand how algorithm works, but I have some doubts about first ...
0
votes
1answer
54 views

PageRank vector

I computed the PageRank vector for the example given in https://en.wikipedia.org/wiki/PageRank (where the picture shows that node B ends up with a score of 38.4, node C with 34.3, node D with 3.9). I ...
0
votes
0answers
21 views

ML - What are the benefits of transforming variables with long tail?

What are the obvious benefits of transforming variables with long tail distributions? To extend it more, why the machine learning algorithms will perform better afer such a transformation?
0
votes
1answer
104 views

What is a range of an attribute in data mining?

In my data mining course, we are working with a data set. I have to identity the ranges of each attribute. I am given the set of data for all 150 points and i am given the visualization (charts) of ...
0
votes
1answer
41 views

Dimension Reduction - Which feature should remove to reduce the dimension of the matrix

Let's suppose that we have the following 2 tables: If we want to reduce the dimension by one(in every table) which feature we should remove and why ? I am confused about the way that i should work ...
0
votes
0answers
25 views

What are Key benefits of Ontologies in Systematic Literature Review?

I am working on a Systematic Literature Review (SLR) and about to done with data synthesis. After SLR, I want to create an Ontology and include different details of the SLR in Ontology. I have almost ...
0
votes
0answers
68 views

Naviers Stokes equation and machine learning

I am looking for a reference explaining how to solve Navier-Stokes numerically using Machine learning algorithms . Thank you in advance for your help .
0
votes
0answers
74 views

LSH and bands for document similarity

I have 50 documents and im using LSH+Minhashing Jaccard and i get this results: for bands:25 -> false positives: 200 for bands:10 -> false positives: 80 for bands:2 -> false positives: 28 for bands:1 ...
0
votes
0answers
96 views

Minhash Jaccard to find similarity between documents

I wrote a program to check the similarity of 40 documents. Firstly i use exact Jaccard method and secondly Minhash Jaccard method. I consider that the 2 methods differ if their values are differ more ...
0
votes
0answers
51 views

What are some data mining algoritms used in finance?

I want to test the applicability of homomorphic encryption in the financial domain, as suggested in Can homomorphic encryption be practical ?. Now, I want to know what algorithms are used for ...
0
votes
0answers
30 views

what is the best theory/model to use for prediction in multivariate data?

I use a software for pollutant propagation on rivers that takes as input a set of parameters (p1, p2, ...pn) and creates an output file which is basically a matrix where on each row there is ...
0
votes
0answers
35 views

Adjust FCM algorithm with size constrains

I want to cluster a list of property addressed based on distance and limit the size of each cluster. The properties are currently assigned to modules by discretion of managers. But I hope to develop ...
0
votes
0answers
22 views

Is this an accepted/valid clustering evaluation metric?

We have a clustering algorithm where the number of clusters isn't known to the algorithm - it iteratively creates clusters out of similar-looking data points. The evaluation metric we're currently ...
0
votes
0answers
75 views

Is k-means parallelizable (other than the data parallelism of the distance function computation)

The k-means algorithm is known to be NP-hard. While the distance function computation in the algorithm loop is data parallel, the algorithm is iterative and may become exponential in the number of ...
0
votes
0answers
172 views

Activity prediction in a kitchen

Here is the scenario: There are three chefs(A- main chef, B and C- assistant) working together to prepare a diner set. The sequence of the event is as following. Start: The three chefs enter the ...
-1
votes
1answer
491 views

Training Error & Convergence to True Error

I Take some online class for Machine Learning. one of teacher say this sentence. if we have m data points, the training error converges to the true error as m → ∞. i thought, this sentence not ...