45

The best explanation I've heard is this: When you're doing machine learning, you assume you're trying to learn from data that follows some probabilistic distribution. This means that in any data set, because of randomness, there will be some noise: data will randomly vary. When you overfit, you end up learning from your noise, and including it in your ...


40

ELI5 Version This is basically how I explained it to my 6 year old. Once there was a girl named Mel ("Get it? ML?" "Dad, you're lame."). And every day Mel played with a different friend, and every day she played it was a sunny, wonderful day. Mel played with Jordan on Monday, Lily on Tuesday, Mimi on Wednesday, Olive on Thursday .. and then on Friday Mel ...


14

Overfitting implies that your learner won't generalize well. For example, consider a standard supervised learning scenario in which you try to divide points into two classes. Suppose that you are given $N$ training points. You can fit a polynomial of degree $N$ that outputs 1 on training points of the first class and -1 on training points of the second class....


12

(This answer uses the second link you gave.) $\newcommand{\Like}{\text{L}}\newcommand{\E}{\text{E}}$Recall the definition of likelihood: $$\Like[\theta | X] = \Pr[X| \theta] = \sum_Z \Pr[X, Z | \theta]$$ where in our case $\theta = (\theta_A, \theta_B)$ are the estimators for the probability that coins A and B respectively land heads, $X = (X_1, \dotsc, X_5)...


12

Theoretically speaking, this is impossible to accomplish, basically because "polynomial" and "exponential" are asymptotic concepts, and no prefix of the data guarantees anything about the behavior at infinity. Practically speaking, you can try to compute $t(n)^{1/n}$ and so if it approaches a constant bounded away from 1. If so, it is exponential. To test ...


11

Not all of AI works on correlation, Bayesian Belief Networks are built around the probability that A causes B. I'm working on a system to estimate the performance of students on questions based on their past performances. I don't think you need causation for this. A past performance does not cause a current performance. Answering on an early question ...


9

Roughly speaking, over-fitting typically occurs when the ratio is too high. Think of over-fitting as a situation where your model learn the training data by heart instead of learning the big pictures which prevent it from being able to generalized to the test data: this happens when the model is too complex with respect to the size of the training data, ...


8

You have a small (but crucial) misunderstanding of what the VC dimension of a class is. The VC dimension is the maximal number $d$ such that there exists a set of $d$ points that is shattered by the class. It doesn't mean that every set is shattered. In this case, indeed 4 co-planar points cannot be shattered, but if they are placed in a tetraeder then ...


8

This is sometimes called "0-1 normalization" or "feature scaling". http://en.wikipedia.org/wiki/Feature_scaling http://en.wikipedia.org/wiki/Normalization_(statistics) The entire function f() is a Piecewise function, which consists of 3 domains, defined separately for v <= min v is between min and max, and v >= max. The primary function is the ...


8

You can't. Randomness is a property of the source, not a property of the values you get from that source. In other words, randomness is a statement about the probability distribution, not about some specific values sampled from that distribution; from a finite sample, you can't give a definite answer to your question. Or, to quote Dilbert: What you can ...


7

The true error of a classifier $h$ is $$ \begin{align*} L_D(h) &= \sideset{\mathbb{E}}{}{}_{x,y \sim D} \Pr[h(x) \neq y] \\ &= \sideset{\mathbb{E}}{}{}_{x,y \sim D} \begin{cases} \Pr[y \neq 0|x] & \text{if } h(x) = 0, \\ \Pr[y \neq 1|x] & \text{if } h(x) = 1. \end{cases} \end{align*} $$ (All probabilities are with respect to $D$.) The ...


6

machine learning seems to be almost entirely based on correlation I don't think so, not in general at least. For example, the main assumption for ML algorithms in terms of PAC analysis and VC dimension analysis, is that training/testing data come from the same distribution that future data will. So in your system, you would have to assume, that each ...


5

The typical reason for smoothing in the first place is to handle cases where $\#\{X_i = x_i | Y = y\} = 0$. If this wasn't done, we would always get $P(Y=y|X=x) = 0$ whenever this was the case. This happens when, for example, classifying text documents you encounter a word that wasn't in your training data, or just didn't appear in some particular class. ...


5

This website provides the answer: A xorshift* generator with an n-bit state space is $n/64$-dimensionally equidistributed: every $n/64$-tuple of consecutive 64-bit values appears exactly once in the output, except for the zero tuple (and this is the best possible for 64-bit values). A xorshift+ generator is however only $(n/64 − 1)$-dimensionally ...


5

There is no such test. All automated tests for examining the randomness of a PRNG have limitations. If you read some papers in the literature that propose PRNGs, you might notice that they do more than just run an automated test against their scheme. Of course you can always write up anything at any time. For the research community to find it interesting,...


4

Yes, ABC is a specific application of Monte Carlo method. That application is approximating likelihood functions. Anything that happens in a computer is actually deterministic. However, ABC, like any other MCM, must have a good pseudo-random generator. There is no difference in the amount of randomness required by the methods. Examples of Monte Carlo ...


4

I think we should consider two situations: Finite training There is a finite amount of data we use to train our model. After that we want to use the model. In this case, if you overfit, you will not make a model of the phenomenon that yielded the data, but you will make a model of your data set. If your data set is not perfect - I have trouble imagining a ...


4

No. You must use the test set only once. Otherwise your result will not be representative of the true performance of your scheme. That's bad. Instead, use the validation set for checking the performance of your algorithm on different parameters, different initialization strategies, different architectures, etc. That's what the validation set is there ...


4

It's not equivalent, and I suspect there will be a loss of statistical randomization/mixing. The core step that offers mixing of the bits is multiplication by FNV_PRIME modulo $2^{32}$. The original version (operating on bytes) does this once for each byte of the input. The alternative version (operating on integers) does this once for each integer, i.e., ...


4

There are two parts to your question that need to be addressed. What transformation are you performing? Why does that help learning? The transformation of taking your input and turning them into "Z-scores" is really just centering and standardizing the variance for each variable. The reason I put "Z-score" in quotes is because there is no ...


4

Why does this help? Part of the reason is because of properties of the activation function. Typically, most activation functions have their most interesting behavior around 0. For instance, the ReLu activation function switches from $f(x)=0$ to $f(x)=x$ at $x=0$. The sigmoid activation function has most of its interesting behavior at $x=0$, and plateaus ...


4

Given two sorted arrays $A[1],\ldots,A[\ell]$ and $B[1],\ldots,B[m]$, the goal is to find the median. Let's assume for simplicity that $n := \ell+m$ is odd, and that all elements are distinct. Therefore we are looking for the element which has exactly $\frac{n-1}{2}$ below it in both arrays. Given an element in one of the arrays, it is easy to find whether ...


4

Maintain two data structures as you move along. A hash $H$ table of frequencies. An array of sets $A$ containing the integers in each frequency. So $A[1]$ is the set of all integers that occur once in the window, $A[2]$ all that occur twice, etc. When you move the window, you need to do two updates, one to increment a frequency of the new integer and one ...


4

Precision and recall are the two basic measures and most of the other measures can be written in terms of them. If a model has both better precision and better recall, then it can be seen as strictly better than another model. If two models are incomparable in the sense that another has better precision and another has better recall, then it depends on the ...


3

There is no universal answer. Instead, it depends on your application. What counts as useful for your application? That determines what should count as a useful or good-enough machine learning algorithm. What counts as useful will vary widely from application to application; some applications require 99.99% accuracy, others might be happy with 52% ...


3

This is definitely possible, although the tensor has of course certain additional structure (constraints). If you consider the following conditional defined for a categorical response $Y$ on categorical predictors $X_i$: $P(Y|X_1,\ldots,X_n)$ this correspond to a conditional probability tensor of size $d_0 \cdot d_1 \cdot \ldots \cdot d_n$. Here $Y$ (as a ...


3

The difference between the two schemes is the model of the clusters. Fuzzy C-means and K-means model their clusters as circles (spheres in n-dimensional space), EM-clustering models the clusters as probability density functions (PDFs). In Euclidean space, the latter can have elliptical shapes (using Gaussian PDFs), determined by their covariance matrices. ...


3

Like most things of this nature the best method is best found by empirical evaluation. One thing worth noting is that most smoothing schemes can be thought of as the incorporation of a prior into your likelihood estimate. For example if you are trying to estimate the parameter $\theta$ of a binary random variable $X$ and you have data $\mathcal{D} = \{x_1, \...


3

You can find the median approximately with this: Approximate Medians and other Quantiles in One Pass and With Limited Memory, to some degree of confidence. That algorithm works, but it's pretty slow. One of the authors of that paper, Gurmeet Manku, has some other publications that might be what you're looking for, also. There's a stackoverflow question ...


3

First, the definition is clear: support is exactly "the fraction of transactions that contain a particular subset of items." This is a data-mining term, not a statistics term. $supp(A)$ is the fraction of transactions that contain item $A$. $supp(B)$ is the fraction of transactions that contain item $B$. $supp(A,B)$ is the fraction of transactions that ...


Only top voted, non community-wiki answers of a minimum length are eligible