The tag has no usage guidance.

learn more… | top users | synonyms

1
vote
1answer
13 views

Best practices for normalizing up training, validation, and test sets

I was reading up on how to normalize my training, validation, and test sets for a neural network, when I read this snippet: An important point to make about the preprocessing is that any ...
-3
votes
1answer
43 views

Is it okay to select the best performers of test cases for scientific publication in neural network machine learning [closed]

If I split my data properly into 75% train, 15% test, and 15% validation, and there are over 100,000 samples, is it appropriate for me to train 100s of neural networks then select only a couple based ...
2
votes
3answers
54 views

How to calculate IV, EV and optimal k for K-means?

Could someone explain how to calculate the following 3 evaluative properties: Intercluster Variability (IV) - How different are the data points within the same cluster Extracluster Variability (EV) -...
1
vote
1answer
26 views

What ML methods exist to categorize signal from noise? Red noise? Spatially correlated noise?

Let's say we are given measurements of some sort. In many cases, it is safe to assume that noise is white noise, serially uncorrelated, and zero mean with some finite variance. But in other cases, ...
0
votes
0answers
17 views

Most impactful factor in the given set of values

My question is somewhat similar to this question Need an algorithm to find the input factors that are most affecting the output but the answers does not solve my problem. My question is: I am ...
3
votes
2answers
75 views

Achieving better than the theoretical False Positive Rate for Bloom Filters

I implemented a standard Bloom Filter in C++, and tested it on different sizes, with varying values of the ratio ${c = n/m}$ where ${n}$ is the size of the filter, and ${m}$ is the number of elements ...
2
votes
0answers
19 views

error measure (of ML agos) that takes confidence into account

when calculating the error measure such as mean absolute error, we use the real values and the predicted values. Many machine learning algorithms can give a confidence measure of each value being ...
1
vote
1answer
37 views

Advice for statistics/ML problem

I've been studying a particular problem recently, and it seems like there might be techniques from statistics or ML that could be applied. Any advice or comments would be appreciated. We're given ...
1
vote
0answers
33 views

Known distributions that generate sparse vectors?

I have data that comes in the form of a vector. Each vector is sparse. Is there a commonly used distribution that will generate sparse vectors? I am working on a project where I am passing a bunch ...
1
vote
2answers
44 views

Learning parameters of noise and filter coefficients from data where data and noise both have Gaussian distributions

Assume $X$ and $N$ are two sets of vectors (observations) from a normal distribution, where $X$ represents clean data and $N$ represents noise; and $A$ a projection matrix of a filter. The scenario is ...
24
votes
12answers
7k views

Why is overfitting bad?

I've studied this lots, and they say overfitting the actions in machine learning is bad, yet our neurons do become very strong and find the best actions/senses that we go by or avoid, plus can be de-...
1
vote
0answers
33 views

Linear regression - iterative approach

I have a single output variable $y$ and a number of inputs $x_1$, $x_2$, etc. These are time series. Each $x_i$ explains the changes in $y$ in specific circumstances, and the goal is to have a linear ...
0
votes
1answer
21 views

Confused by extremely high autocorrelation

I have the following python code ...
3
votes
1answer
50 views

Use subset of training data as prediction data

At our company, we've started using Amazon Machine Learning to predict the likelihood of a certain segment of our customers cancelling their subscription. We only have 500 customers in that segment ...
2
votes
1answer
138 views

Best Fit Memory Allocation Algorithm Statistical Analysis

I've been reading book on Operating System, in which author writes : Somewhat surprisingly, it(Best fit) also results in more wasted memory than first fit or next fit because it tends to fill up ...
2
votes
0answers
45 views

Using the random forest algorithm to predict vectors [duplicate]

I know this might sound like a newbie question, but bear with me. I have read a paper where researchers use a random forest to predict species distribution, but in their study, they only predict a ...
3
votes
0answers
38 views

Infer probabilities, for concatenation of words

Fix an alphabet $\Sigma$, and a set of words, $W = \{w_1,\dots,w_n\} \subseteq \Sigma^*$. I have a randomized model that works like this: Alice generates a random sequence of words, using some ...
3
votes
1answer
202 views

Approximate Bayesian Computation VS Monte Carlo Simulation

I am a little confused about the differences between Approximation Bayesian Computation (ABC) and Monte Carlo Methods (MCM). Citing from wikipedia: Approximate Bayesian computation (ABC) ...
0
votes
1answer
44 views

Definition and properties of support

From Xiong, Hui, Shashi Shekhar, Pang-Ning Tan, and Vipin Kumar. “TAPER: A Two-Step Approach for All-Strong-Pairs Correlation Query in Large Databases.” Knowledge and Data Engineering, IEEE ...
0
votes
2answers
35 views

“Practical” bounding box?

For the sake of simplicity, lets say I have a bunch of 2d points, each have X and Y. The points are distributed somewhat randomly but not completely, they will be biased to be closer to the world ...
2
votes
2answers
277 views

Maintain statistics over a sliding window (robust & efficient)

I am looking for an algorithms to maintain several statistics over a sliding window. The setup is as follows: There is a datastream consisting of (real value,timestamp) tuples. The values for the last ...
2
votes
0answers
140 views

How to prevent overflow and underflow in the Euclidean distance and Mahalanobis distance

I was working in my project when I was struck by the question of whether it would be necessary, or at least cautious, prevent overflow and underflow in the calculation of these two distances. I ...
1
vote
1answer
45 views

What are the tradeoffs between implenting dataframes as a series of columns vs a series of tuples

I was reading the documentation for Julia's Dataframe package, and the package implements Dataframe by a series of a columns each of a single type. From reading the docs, it looks like that's how R ...
1
vote
0answers
71 views

Formula for number of parameters in an undirected graphical (probability) model

I have googled endlessly, and I cannot find it. Can anyone point me to a reference that gives a way to calculate the number of parameters in an undirected Graphical Model? Adapting from the similar ...
4
votes
0answers
68 views

Space complexity of statistic functions

When computing statistics on a list of data it occurred to me that most of the standard statistic functions, such as mean, min, max can be computed in O(N) time with O(1) space. They can also be ...
0
votes
1answer
28 views

Bayes nets - calculating probabilities

Given a Bayesian network, say a -> b -> c, all binary random variables (I won't show the CPTs, assume they are given). You are told b and c are true. How do you calculate the P(a=True)?
0
votes
0answers
66 views

Merging two disconnected graphs

Firstly, I'd like to apologize for any misused terms or ways I could have made the description much more succinct. It's been a while since I took machine learning during my bachelor's. I have two ...
3
votes
2answers
131 views

How do I measure the reliability of a confidence value in a predictive algorithm?

Supposing I have some algorithm that is able to provide me with a confidence value for some event occurring. Let's say on day 1 it tells me that there is a 80% chance it will rain, on day 2 it tells ...
1
vote
2answers
189 views

Need an algorithm to find the input factors that are most affecting the output

I apologize if this question is already answered and appreciate any pointers to existing answers. I'm not familiar with statistical or data mining terms so my search was limited to basic words used in ...
2
votes
0answers
31 views

Do online quantile estimation algorithms that support deletions exist?

There are several online quantile estimation algorithms, but I haven't seen any that supports deletions (i.e. unobserve elements observed in the past). Are there any such known algorithms? To ...
4
votes
1answer
179 views

Name for this algorithm?

I'm trying to figure out if there is a proper or commonly accepted name for this particular function (f). ...
2
votes
1answer
155 views

Best algorithm for correlation between time series?

I have some biological data (ECG), which are quite chaotic in nature, and and some other data; that are not chaotic but related in some way, like fatigue. I want to find out how the time series, ...
5
votes
1answer
226 views

What does it mean for a random number generator's sequence to be only 1-dimensionally equidistributed?

Whilst reading up on Xorshift I came across the following (emphases added): The following xorshift+ generator, instead, has 128 bits of state, a maximal period of 2^128 − 1 and passes BigCrush: ...
3
votes
2answers
407 views

How to determine if a black-box is polynomial or exponential

I have a problem which essentially reduces to this: You have a black-box function that accepts inputs of length $n$. You can measure the amount of time the function takes to return the answer, but ...
0
votes
0answers
54 views

TF-IDF query engine in context of terms weight

I'm looking for public algorithm which gives the engine these abilities: Query by ranked terms Limit outcome by date/time range Basically, i'd like to concentrate articles (generally ...
10
votes
4answers
293 views

Machine learning - importance of correlation vs. causation

It is a well-known fact that "Correlation doesn't equal causation", but machine learning seems to be almost entirely based on correlation. I'm working on a system to estimate the performance of ...
1
vote
0answers
64 views

Applications of algorithms to stock trading analysis

There is a new Quantitative Finance SE site. However, I am interested in asking the "CS crowd": What are some interesting key references or surveys on applying algorithms to stock trading ...
2
votes
2answers
223 views

What would be a decent threshold for classification problem?

I'm using machine-learning algorithms to solve binary classification problem (i.e. classification can be 'good' or 'bad'). I'm using SVM based algorithms, ...
2
votes
0answers
151 views

Efficiently estimating latency quantiles of a distributed system

I'm building a load test harness for a distributed system. Currently I'm using the "Cormode, Korn, Muthukrishnan, and Srivastava" method to estimate latency quantiles of system responses. I'm now ...
3
votes
1answer
269 views

Estimate entropy, based upon observed frequency counts

Suppose I have $n$ independent observations $x_1,\dots,x_n$ from some unknown distribution over a known alphabet $\Sigma$, and I want to estimate the entropy of the distribution. I can count the ...
2
votes
1answer
41 views

Finding Statistical Signifigance for a Classifier

I recently did work in the area of machine learning for one of my jobs and was able to build a classifier which evaluated to an F score of 85%. I also have access to correctly and incorrectly ...
2
votes
1answer
634 views

Understanding an example of coin toss expectation maximization [duplicate]

I've been trying to get my head around Expectation maximization algorithms, and I thought I'd start simple. I found this 3-coin example here: http://cs.dartmouth.edu/~cs104/CS104_11.04.22.pdf I ...
2
votes
2answers
652 views

distance between histograms

I have 2 histograms that represent the height of characters in 2 images. example: 1 ** 2 **** 3 **** . . . 100 ****** For these 2 histograms I compute the peaks. And To check if these 2 images ...
6
votes
1answer
162 views

Conditional Probabilities as Tensors?

Is it proper to view conditional probabilities, such as the forms: P(a|c) P(a|c,d) P(a, b|c, d) ...and so forth, as being tensors? If so, does anyone know of a decent introductory text (online ...
3
votes
1answer
522 views

VC dimension of linear separator in 3D

I am confused about the Vapnik-Chervonenkis dimension of a linear separator in 3 dimensions. In three dimensions, a linear separator would be a plane, and the classification model would be "...
3
votes
2answers
401 views

Predicting energy consumption of households

I have the dataset which you can find here, containing many different characteristics of different houses, including their types of heating, or the number of adults and children living in the house. ...
1
vote
1answer
251 views

Null Hypothesis in Analysis and Testing

I have my end of year exams next Thursday. I'm generally doing fine but I am having some major issues with this strand of my course, this has to be the biggest issue I have. So, here is the question ...
3
votes
1answer
2k views

Differences between Fuzzy C-Means and EM

When clustering a set of data points, what exactly are the differences between Fuzzy C-Means (aka Soft K-Means) and Expectation Maximization? In slide 30 and 32 of this lecture I found, it says that ...
14
votes
1answer
4k views

Applying Expectation Maximization to coin toss examples

I've been self-studying the Expectation Maximization lately, and grabbed myself some simple examples in the process: From here: There are three coins $c_0$, $c_1$ and $c_2$ with $p_0$, $p_1$ and $p_2$...
1
vote
1answer
88 views

Maximum variance and useful information of dataset

I am reading through PCA and it says that the maximum variance principal component has most of the information. Can we apply that to any data set? If a data set has n attributes and most of the ...