I am new to machine learning. I have read several papers where they have employed deep learning for various applications and have used the term "prior" in most of the model design cases, say prior in human body pose estimation. Can someone explain what does it actually means. I could only find the mathematical formulation of prior and posterior in the tutorials.
$\begingroup$
$\endgroup$
3
-
5$\begingroup$ It's a mathematical concept so, yes, it's formulated mathematically. However, the Wikipedia page seems to give plenty of intuition. Did you check it? If so, could you say more about what you didn't understand and what you're looking for in an answer? $\endgroup$– David RicherbyJun 11, 2017 at 13:08
-
$\begingroup$ @David Richerby. Thank you for your response. Yes I had checked that wikipedia page and I could gather a vague idea that its something about knowledge or information about a variable. I had been reading papers on body pose estimation where there were mentions of body pose priors, body kinematic prior, modeling of priors over 3D human pose, learning priors, prior to estimate 3D human pose. I could not clearly figure out what the term "prior" actually means in this context. $\endgroup$– AmyJun 11, 2017 at 14:58
-
$\begingroup$ jonathanweisberg.org/pdf/VarietiesvF.pdf $\endgroup$– AndrewAug 18, 2019 at 14:14
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
7
Put simply, and without any mathematical symbols, prior means initial beliefs about an event in terms of probability distribution. You then set up an experiment and get some data, and then "update" your belief (and hence the probability distribution) according to the outcome of the experiment, (the posteriori probability distribution).
Example: Assume we are given two coins. But we don't know which coin is fake. Coin 1 is unbiased (HEADS and TAILS have 50% probability), and Coin 2 is biased, say, we know it gives HEADS with probability 60%. Mathematically:
Given we have HEADS, the probability that it is Coin 1 is 0.4 $$p(H | Coin_1) = 0.4$$ and probability it is Coin 2 is 0.6 $$p(H| Coin_2) = 0.6$$
So, that is all what we know before we set up an experiment.
Now we are going to pick a coin toss it, and based on the information what we have (H or T) we are going to guess what coin we have chosen (Coin 1 or Coin 2).
Initially we assume $p(Coin_1) = p(Coin_2) = 0.5$ both coins have equal chances, because we have no information yet. This is our prior. It is a uniform distribution.
Now we take randomly one coin, toss it, and have a HEADS. At this moment everything happens. We compute posterior probability/distribution using Bayesian formula: $$p(Coin_1 | H) = \frac{p(H | Coin_1)p(Coin_1)}{p(H | Coin_1)p(Coin_1) + p(H | Coin_2)p(Coin_2)} = \frac{0.4\times0.5}{0.4\times0.5 + 0.6\times0.5} = 0.4$$
$$p(Coin_2 | H) = \frac{p(H | Coin_2)p(Coin_2)}{p(H | Coin_1)p(Coin_1) + p(H | Coin_2)p(Coin_2)} = \frac{0.6\times0.5}{0.4\times0.5 + 0.6\times0.5} = 0.6$$
So, initially we had $0.5$ probability for each coin, but now after the experiment our beliefs has changed, now we believe that the coin is Coin 1 with probability 0.4 and it is Coin 2 with the probability 0.6. This is our posterior distribution, Bernoulli distribution.
This is the basic principle of Bayesian inference and statistics used in Machine learning.
answered Jun 11, 2017 at 20:51
-
6$\begingroup$ You need to fix the example above. That calculation shows that both coins are biased (First one with prob of Heads 40% and the second one with probability of heads 60%) In case the first one is biased It is still a Bernoulli distribution but with probabilities P(Coin1|H) =5/11 and P(Coin2|H)=6/11 $\endgroup$ Jan 23, 2018 at 16:33
-
1$\begingroup$ Should "Given we have HEADS, the probability that it is Coin 1 is 0.4" be rewritten as "Given that we have Coin 1, the probability that it is HEADS is 0.4"? $\endgroup$ Dec 9, 2019 at 18:36
-
1$\begingroup$ The explanation doesn't explain in terms of machine learning. $\endgroup$ Apr 5, 2020 at 22:42
-
1$\begingroup$ I would also like to reiterate, the explanation doesn't explain in terms of machine learning. $\endgroup$ Jun 23, 2020 at 21:41
-
$\begingroup$ Does it have to be a probability distribution? Can I say "CNNs are good architectural priors for image processing tasks"? $\endgroup$– YashasAug 5, 2021 at 6:02
$\begingroup$
$\endgroup$
0
It's roughly any pre-training choices you encode into your model
In machine learning a prior is, according to the book "Deep Learning" by Goodfellow, Bengio, and Courville, a probability distribution over the parameters of your machine learning model. So as I understand it according to this definition your priors determine how you randomly select your initial parameters before you start training. An alternative definition according to this website is more general, basically any configuration of parameters or model choices that you make prior to starting training.
So it looks like the term prior in machine learning is sometimes applied less rigorously than in statistics, though it is either based on, or at least inspired by the corresponding term in the field of statistics.
$\begingroup$
$\endgroup$
In Bayesian statistics, a "prior" represents the beliefs we have before observing some data. Then, after we observe some data, we update our beliefs; those updated beliefs are called the "posterior".
Statistical machine learning can be viewed through this lens. The "priors" are the beliefs we hold or we try to enforce/encourage a model to hold even before it is trained with any training data. The observed data is the training data, and the machine learning model is trained on this data. After it finishes training, the final model is the posterior. Here are some examples:
In natural language processing, one "prior" belief we might have is that if two words tend to be used in the same context, there is a decent chance they are related. For instance, even if you didn't know the meaning of "good" or "delicious", if I told you that "lip-smackingly" was often followed by either the word "good" or the word "delicious", you might suspect that "good" and "delicious" might mean something similar (or are somehow related). That's a prior belief -- it's something we might believe is true, even before we've seen any data. Starting from that prior, we might design a word embedding like GloVe, use it to encode sentences before feeding them to a machine learning model, and train the model. In this way, we have build a model that respects our "priors" (our beliefs before we saw the training data).
In image classification, one "prior" belief we might have is that if it's useful to look for a particular pattern -- e.g., straight lines -- in the upper-right of an image, it's also useful to look for them everywhere in the image. That's a "prior" because it's an assumption or belief or hypothesis we hold, even before training on any training data. We might then build a neural network architecture, such as a convolutional neural network, that behaves in a way that is motivated by or is consistent with this "prior". Then, we can train such a network on training data, to obtain a specific model. This illustrates how an architecture can embed a prior (in this case, translation-equivarance), either deliberately or inadvertently. Convolutional neural networks are biased towards performing well on types of images that are consistent with this "prior", and performing poorly on types of images that are inconsistent with this "prior".
In some applications of statistical modeling, one prior is that there are many features but only a few are relevant to the quantity we wish to predict. Lasso regression was invented to deal with this situation: it is linear regression, but starting from a prior that the coefficients for most features should be zero, and probably only a few features should have a non-zero coefficient. In contrast, linear regression doesn't start with any assumption or information about what the coefficients are likely to look like. Lasso regression will perform better than linear regression on tasks where the assumption is appropriate.
In particular, having an informative "prior" can be a good thing. It can help the model perform better, given a limited amount of training data, because the model starts with some knowledge -- it doesn't start from nothing.
