Questions tagged [statistics]
The statistics tag has no usage guidance.
0
votes
1answer
26 views
How can we get small test error reducing only train error?
My question is about mathematical part of machine learning algorithms, especially about using it in neural networks. We train network reducing train error and I was thinking about how then test error ...
3
votes
1answer
33 views
How good (or bad) is my makeshift PRNG?
Say I have designed a makeshift PRNG for my personal amusement, now I would like to see how good it is. How do I benchmark its "randomness"? Ideally, I want to know a statistics test, such that if I ...
1
vote
0answers
16 views
How to go about designing a quiz that changes difficulty based on user performance?
I thought about using a machine learning algorithm to learn the performance patterns of the quiz taker and model the next quiz based on how he/she performed. So each question will have a difficulty ...
2
votes
2answers
29 views
identify abrupt frame changes in a video
The frames of live streams or videos, in general, do not flow smoothly. A sports game would have multiple cameras recording the game and hence, there will be camera switches in the stream. I am ...
0
votes
2answers
76 views
probability of collision
my data's range is from 1 to 9 and I have two subsets of integers from this range.
the hash function takes each of this subsets and calculate product of these three integers and maps this set to the ...
0
votes
0answers
31 views
Optimize Algorithm For Shut in Box Game
This game is a simplified version of Shut the Box. There are 9 tiles (1,2,3,4,5...9). The initial tiles are unflipped. You have 2 dices.
Each round, the player rolls the 2 dices and their sum is S.
...
0
votes
1answer
47 views
Is there a fast algorithm for computing the rolling mode of an array of integers?
I was wondering if there exists an efficient algorithm for calculating the "rolling mode of an array of integers.
By rolling mode I mean that we have an array of integers of size $n$ and a sliding ...
1
vote
1answer
380 views
Median of two sorted arrays of different size in logarithmic time?
The solution given in Cormen is as follows:
Reading this solution, the first doubt that comes in my mind is what if the median lies in array B. According to this solution, we are applying binary ...
0
votes
0answers
21 views
Locality sensitive hashing with non-scalal values
Locality sensitive hashing works well when matching is between vectors of scalars, but I now need to extend LSH to compare matrices. Each matrix is formed of n readings from m sensors forming a n by m ...
0
votes
1answer
23 views
How to find a possibility of match in Naive Bayes Classifier?
My math level is very very poor so I can't get the statistics.
Can anyone explain in simple words?
I.e. if I have frequency data:
...
0
votes
0answers
9 views
What statistics can compare individual survey responses & others that were measured as a group (from which we need to extract individual responses)?
The organization I belong to studies indigenous populations through primarily survey research. One study for example aims to identify the relationship between culture and the evaluation of odors. We ...
0
votes
1answer
22 views
Testing two distributions, both accept null hypothesis
I have a sample that is collected to verify the accuracy of a new random number generator.
Applying the goodness of fit test to check if this sample comes from the Standard Normal Distribution and ...
1
vote
0answers
69 views
Examples of bad statistics in Computer Science
I am preparing class materials for the applied statistics class for Ph.D. students and want to present them with examples of bad statistics in different fields of computer science. I am especially ...
1
vote
0answers
62 views
What is the reason behind getting same result by maximum likelihood estimate and smoothing?
I know that Good-Turing smoothing helps us to trim a bit of probability from some more frequent events and give it to the events we've never seen. Thus it keeps our model from assigning zero ...
0
votes
0answers
46 views
SimRank++ on a weighted graph (why the formula reflects the influncee of the weight)
In the paper "Simrank++：Query Rewriting through Link Analysis of the Click Graph"(http://www.vldb.org/pvldb/1/1453903.pdf), the formula to compute the similarity between $q$ and $q'$ is as follows:
\...
1
vote
1answer
42 views
Inequality over the entropy of an integer-valued random variable
I stumbled upon this inequality over a course on Information Theory :
If $Z\in\mathbb{N}$ with finite mean, then $H(Z) \leq E(Z)\times h(\frac 1 {E(Z)})$ where $h$ is the binary entropy function
...
6
votes
2answers
109 views
Applying graph “adjustment” algorithms to Elo rating system
I'm trying to address what I perceive to be a potential shortcoming in the Elo Rating System (predominantly used by the international Chess community to rate + rank players). I have a two-player game ...
1
vote
0answers
220 views
Understanding Sliding Window Algorithm
I am trying to implement sliding window algorithm for rainfall forecasting. The algorithm is follows:
This algorithm is from An enhancement of sliding window algorithm for
rainfall forecasting in ...
0
votes
1answer
201 views
maximizing inner product of vectors in an ellipsoid and a given vector
I have been wrestling with this for quite a long time but couldn't convince myself that the following is true:
What I do understand: $\theta_a$ denotes the set of points that are within the ellipsoid....
1
vote
0answers
37 views
Update model parameter with new data, discarding old data
I have this dataset, and I am using y = (a * x^n) / (b + x^n) Hill function as the model, where a is the limit of the Hill curve,...
3
votes
1answer
26 views
HMM tagger - Baum Welch training
I am trying to implement a trigram HMM tagger for a language that has over 1000 tags. In my training data I have 459 tags. Now if we consider that states of the HMM are all possible bigrams of tags, ...
1
vote
0answers
29 views
Unsupervised learning: necessity of labels and dependency between features and labels?
I have logs of activities without labels, which describe whether an activity is normal or not. Assuming that normal behaviors will follow a Gaussian distribution, I fit Gaussian distributions on ...
0
votes
2answers
161 views
Trying to Classify MNIST where pixels are shuffled with a seed.Why is it not working?
I transformed the MNIST dataset as follows:(X (70000 x 784) is the training matrix)
...
0
votes
2answers
73 views
Estimating run time in the absence of knowledge
I’ve always wondered how to make a best guess in the absence of any knowledge.
Let’s say you have a computation running, and you don’t know when it will complete (just that it will). Can you make any ...
3
votes
0answers
106 views
What would be the probabilty of a randomly generated tree to be a Red-Black Tree
The question is not related to the homework
I was working on a homework, and the specification was to generate a random tree with n elements(n being in the thousands for the assignment) and asked me ...
1
vote
1answer
351 views
how to interpret the coefficients of binary logistic regression?
I'm working on a kNN classifier to classify whether a text is written by a man or a woman on the basis of the most frequent words. However, a kNN doesn't show which features were the most important in ...
1
vote
2answers
35 views
If we can $\epsilon-$cover one set with $\frac{K}{2}$ elements and another with 2 elements why isn't their set product $2^{\frac{K}{2}}$ cover?
I was reading the following paper on Robustness and Generalization (and covering numbers)and I was reading their example 1:
which didn't quite make sense to me. The reason it didn't make sense is ...
0
votes
0answers
76 views
“Sensitivity” of algorithms
Let us assume that we have two algorithms, $A$ and $B$. Based on 3 different values of a parameter, $P$, $A$ takes $a_1, a_2, a_3$ seconds and $B$ takes $b_1, b_2, b_3$ seconds. What would be a proper ...
1
vote
1answer
29 views
What is the theory behind testing fixes for intermittent bugs?
I ask this question because I deal with alot of intermittent bugs and want to be able to answer the question "Can I be at least 95% sure that my changes to the code fix the bug?". I've got some ideas ...
0
votes
1answer
43 views
How to find a good saw function for PRNG?
As part of an assignment I have to write a PRNG using a sin (or any other trigonometric function) and a saw function which I'm struggling a bit with. So how would you find a good saw function with ...
-1
votes
2answers
78 views
finding normality from a set of samples of MEAN
I have set of 1000 samples. each sample represents MEAN of X amount transactions response time.
Now I have a running transaction , I know it's current response time but I want to know if this ...
0
votes
0answers
40 views
What is the probability distribution of arrival times in a distributed system?
I'm going to simulate an asynchronous distributed system. In a distributed system, when a node sends a message, after $\Delta t$ seconds the message reaches to its destination.
My Question: What is ...
4
votes
1answer
2k views
Showing that Bayes classifier is optimal
Consider domain $X$, label set $ Y=\{0,1\}$ and the zero-one loss.
Given any probability distribution D over $ X\times \{0,1\} $, we've defined the Bayes classifier $ f_D $ to be-
$$ f_{D}(x)=
\...
1
vote
0answers
53 views
Approximate conditional entropy
Given a set of random variables $X = \{x_1, x_2, \dots, x_n\}$. If the conditional entropy for all $Y \subset X - \{X_i\}$ where $|Y| \leq 5$. How to approximate conditional entropy when $|Y| = 10$ ...
1
vote
0answers
175 views
EM-algorithm for categorical hidden variables
I have the following model: Let's say two indepentent weighted six-sided dice $X$ and $Y$ with unknown probabilities (i.e. probability of 1 is unkown etc) and they have not necessarily the same ...
6
votes
2answers
142 views
How best to statistically verify random numbers?
Lets say I have 1000 bytes that are supposedly random. I want to verify to a certain certainty that they are indeed random and evenly distributed across all byte values. Aside from calculating the ...
3
votes
2answers
78 views
Redistributing a set of uniformly distributed numbers to an arbitrarily defined shape
Lets say I have a random number generator that spits out uniform numbers from 0 to 1
Next, I have a shape defined by a series of vertices, like { [0, 0.4], [0.5, 0.2], [1, 0.4] }
In those vertices, ...
0
votes
0answers
341 views
What is role of parameter learning rate, lr, and momentum constant, mc in Neural Networks?
can anyone describes the more simplified mathematical formulation of learning rate, lr, and momentum constant, mc in Neural Networks while training the data?
4
votes
2answers
1k views
Z-score relations to perceptrons
I am learning about preceptrons and my professor noted that z-scores are a commmon pre-processing step to normalizing input variables. Following this, I am having trouble understanding why z-scores ...
3
votes
1answer
167 views
Least squares fit of a 1D lattice of points to a 2D dataset
Given a set of data points (shown in red), it is possible to fit a line $y = mx + c$ through the points using linear least squares regression.
I would like to modify this to fit a 1D lattice (grid) ...
3
votes
1answer
28 views
Constructing a list of functions/formulas which describes a set of grid points in a 3D matrix
Given a 3D matrix of size $N \times N \times N$, let $\mathcal{S}$ be a set of points in the Matrix and $\mathcal{S}'$ be the complement of $\mathcal{S}$. Can we find a set of equations of the form:
$...
2
votes
1answer
527 views
Hash multiple integers directly using FNV-1a
An alternative version of FNV-1a hash spread on the internet, which operates directly on integers instead of bytes. The offset basis and prime are the same used in the original version, which operates ...
1
vote
1answer
69 views
Best practices for normalizing up training, validation, and test sets
I was reading up on how to normalize my training, validation, and test sets for a neural network, when I read this snippet:
An important point to make about the preprocessing is that any
...
-2
votes
1answer
60 views
Is it okay to select the best performers of test cases for scientific publication in neural network machine learning [closed]
If I split my data properly into 75% train, 15% test, and 15% validation, and there are over 100,000 samples, is it appropriate for me to train 100s of neural networks then select only a couple based ...
2
votes
3answers
221 views
How to calculate IV, EV and optimal k for K-means?
Could someone explain how to calculate the following 3 evaluative properties:
Intercluster Variability (IV) - How different are the data points within the same cluster
Extracluster Variability (EV) - ...
1
vote
1answer
42 views
What ML methods exist to categorize signal from noise? Red noise? Spatially correlated noise?
Let's say we are given measurements of some sort.
In many cases, it is safe to assume that noise is white noise, serially uncorrelated, and zero mean with some finite variance.
But in other cases, ...
3
votes
2answers
344 views
Achieving better than the theoretical False Positive Rate for Bloom Filters
I implemented a standard Bloom Filter in C++, and tested it on different sizes, with varying values of the ratio ${c = n/m}$ where ${n}$ is the size of the filter, and ${m}$ is the number of elements ...
2
votes
0answers
30 views
error measure (of ML agos) that takes confidence into account
when calculating the error measure such as mean absolute error, we use the real values and the predicted values. Many machine learning algorithms can give a confidence measure of each value being ...
1
vote
1answer
44 views
Advice for statistics/ML problem
I've been studying a particular problem recently, and it seems like there might be techniques from statistics or ML that could be applied. Any advice or comments would be appreciated.
We're given ...
1
vote
0answers
46 views
Known distributions that generate sparse vectors?
I have data that comes in the form of a vector. Each vector is sparse. Is there a commonly used distribution that will generate sparse vectors?
I am working on a project where I am passing a bunch ...
