Skip to main content
Followed my own suggestion and traced through the proof.
Source Link
Louis
  • 2.9k
  • 16
  • 25

If you have some information about the first two moments of the summandsYes, youwe can get something usefula bound like this. For exampleTo see why, we will need to look a fairlylittle more closely at how Chernoff bounds are proved. A relatively standard form of the Chernoffthis kind of tail bound would say something like ifassume that $$ X = X_1 + \cdots + X_n $$ with all $X_i$ independent, discrete, supported in $[-1,1]$, centeredwith mean (i.e.$\mu_i = 0$, variance $\mathbb{E}[X_i]=0$), and independent, then$\sigma_i^2$. The resulting tail bound ends up being that: $$\operatorname{Pr}[X > \lambda\sigma]\le e^{-\lambda^2/4}$$ where $\sigma^2 = \sum_{i\in [n]} \operatorname{Var}[X_i]$$\sigma^2 = \sum_{i\in [n]} \sigma^2_i$ is the variance of $X$ and $\lambda\in[0,2\sigma]$. Here (Here is a proof by Van Vu.)

If, insteadI'm more or less going to first reproduce the proof, so you can get an idea of why decreasing the expectation turns out to be ok.

All Chernoff bounds are willingbased on applying Markov's inequality to assume$e^{tX}$ to get that $\operatorname{E}[X_i]\le 0$$\operatorname{Pr}[X > \lambda\sigma]\le \mathbb{E}[e^{tX}]e^{-t\lambda\sigma}$. So the general method is to work out $\mathbb{E}[e^{tX}]$ and replacethen optimize $\sigma^2$ with$t$. We can do the sumMGF computation a little differently from the link, namely as $$\mathbb{E}[e^{tX}] = \prod_{i\in [n]}\mathbb{E}[e^{tX_i}] \le \prod_{i\in [n]}\mathbb{E}[(1 + tX_i + t^2X_i^2)]$$ where we first used independence of the $\operatorname{E}[X_i^2]$$X_i$ and then $e^x\le 1 + x + x^2$ for $x\in [0,1]$. Under the hypotheses we started with and linearity of expectation, you'll $$\mathbb{E}[(1 + tX_i + t^2X_i^2)] = 1 + t^2\sigma_i^2\le e^{t^2\sigma_i^2}$$ so we get a reasonably nice looking bound by just tracing through the argumentthat $$ \operatorname{Pr}[X > \lambda\sigma] \le e^{t^2\sigma^2 - t\lambda\sigma}$$ With $t = \lambda/2\sigma$ this is what we wanted (and since $t\in[0,1]$, all the inequalities we used are valid).

Also, this will explain why you can't just infer a general answerNow let's assume that $\mu_i\le 0$ (instead of $= 0$ as before) and otherwise the same set of hypotheses. The natural units Returning to the MGF computation, we get $$ \mathbb{E}[(1 + tX_i + t^2X_i^2)] = 1 + t\mu_i + t^2\mathbb{E}[X_i^2] = 1 + t\mu_i + t^2\sigma_i^2 + t^2\mu_i^2 $$ Using that $x^2 + x < 0$ for tail bounds are standard deviations from$x\in (-1,0)$, this implies that $$ \mathbb{E}[(1 + tX_i + t^2X_i^2)] \le 1 + t^2\sigma_i^2 $$ and so we get the meansame tail bound as before. In this case

Finally, ifnote you move the mean away from zeroalways have $\sigma^2_i\le 4$, thenso $\operatorname{E}[X_i^2]$ isn't quite the variance any more$\sigma^2\le 4n$.

If you have some information about the first two moments of the summands, you can get something useful. For example, a fairly standard form of the Chernoff bound would say something like if $$ X = X_1 + \cdots + X_n $$ with all $X_i$ discrete, supported in $[-1,1]$, centered (i.e., $\mathbb{E}[X_i]=0$), and independent, then $$\operatorname{Pr}[X > \lambda\sigma]\le e^{-\lambda^2/4}$$ where $\sigma^2 = \sum_{i\in [n]} \operatorname{Var}[X_i]$ is the variance of $X$. Here is a proof by Van Vu.

If, instead, you are willing to assume that $\operatorname{E}[X_i]\le 0$ and replace $\sigma^2$ with the sum of $\operatorname{E}[X_i^2]$, you'll get a reasonably nice looking bound by just tracing through the argument.

Also, this will explain why you can't just infer a general answer. The natural units for tail bounds are standard deviations from the mean. In this case, if you move the mean away from zero, then $\operatorname{E}[X_i^2]$ isn't quite the variance any more.

Yes, we can get a bound like this. To see why, we will need to look a little more closely at how Chernoff bounds are proved. A relatively standard form of this kind of tail bound would assume that $$ X = X_1 + \cdots + X_n $$ with all $X_i$ independent, discrete, supported in $[-1,1]$, with mean $\mu_i = 0$, variance $\sigma_i^2$. The resulting tail bound ends up being that: $$\operatorname{Pr}[X > \lambda\sigma]\le e^{-\lambda^2/4}$$ where $\sigma^2 = \sum_{i\in [n]} \sigma^2_i$ is the variance of $X$ and $\lambda\in[0,2\sigma]$. (Here is a proof by Van Vu.)

I'm more or less going to first reproduce the proof, so you can get an idea of why decreasing the expectation turns out to be ok.

All Chernoff bounds are based on applying Markov's inequality to $e^{tX}$ to get that $\operatorname{Pr}[X > \lambda\sigma]\le \mathbb{E}[e^{tX}]e^{-t\lambda\sigma}$. So the general method is to work out $\mathbb{E}[e^{tX}]$ and then optimize $t$. We can do the MGF computation a little differently from the link, namely as $$\mathbb{E}[e^{tX}] = \prod_{i\in [n]}\mathbb{E}[e^{tX_i}] \le \prod_{i\in [n]}\mathbb{E}[(1 + tX_i + t^2X_i^2)]$$ where we first used independence of the $X_i$ and then $e^x\le 1 + x + x^2$ for $x\in [0,1]$. Under the hypotheses we started with and linearity of expectation, $$\mathbb{E}[(1 + tX_i + t^2X_i^2)] = 1 + t^2\sigma_i^2\le e^{t^2\sigma_i^2}$$ so we get that $$ \operatorname{Pr}[X > \lambda\sigma] \le e^{t^2\sigma^2 - t\lambda\sigma}$$ With $t = \lambda/2\sigma$ this is what we wanted (and since $t\in[0,1]$, all the inequalities we used are valid).

Now let's assume that $\mu_i\le 0$ (instead of $= 0$ as before) and otherwise the same set of hypotheses. Returning to the MGF computation, we get $$ \mathbb{E}[(1 + tX_i + t^2X_i^2)] = 1 + t\mu_i + t^2\mathbb{E}[X_i^2] = 1 + t\mu_i + t^2\sigma_i^2 + t^2\mu_i^2 $$ Using that $x^2 + x < 0$ for $x\in (-1,0)$, this implies that $$ \mathbb{E}[(1 + tX_i + t^2X_i^2)] \le 1 + t^2\sigma_i^2 $$ and so we get the same tail bound as before.

Finally, note you always have $\sigma^2_i\le 4$, so $\sigma^2\le 4n$.

Define "centered"
Source Link
D.W.
  • 165.6k
  • 21
  • 230
  • 490

If you have some information about the first two moments of the summands, you can get something useful. For example, a fairly standard form of the Chernoff bound would say something like if $$ X = X_1 + \cdots + X_n $$ with all $X_i$ discrete, supported in $[-1,1]$, centered (i.e., $\mathbb{E}[X_i]=0$), and independent, then $$\operatorname{Pr}[X > \lambda\sigma]\le e^{-\lambda^2/4}$$ where $\sigma^2 = \sum_{i\in [n]} \operatorname{Var}[X_i]$ is the variance of $X$. Here is a proof by Van Vu.

If, instead, you are willing to assume that $\operatorname{E}[X_i]\le 0$ and replace $\sigma^2$ with the sum of $\operatorname{E}[X_i^2]$, you'll get a reasonably nice looking bound by just tracing through the argument.

Also, this will explain why you can't just infer a general answer. The natural units for tail bounds are standard deviations from the mean. In this case, if you move the mean away from zero, then $\operatorname{E}[X_i^2]$ isn't quite the variance any more.

If you have some information about the first two moments of the summands, you can get something useful. For example, a fairly standard form of the Chernoff bound would say something like if $$ X = X_1 + \cdots + X_n $$ with all $X_i$ discrete, supported in $[-1,1]$, centered, and independent, then $$\operatorname{Pr}[X > \lambda\sigma]\le e^{-\lambda^2/4}$$ where $\sigma^2 = \sum_{i\in [n]} \operatorname{Var}[X_i]$ is the variance of $X$. Here is a proof by Van Vu.

If, instead, you are willing to assume that $\operatorname{E}[X_i]\le 0$ and replace $\sigma^2$ with the sum of $\operatorname{E}[X_i^2]$, you'll get a reasonably nice looking bound by just tracing through the argument.

Also, this will explain why you can't just infer a general answer. The natural units for tail bounds are standard deviations from the mean. In this case, if you move the mean away from zero, then $\operatorname{E}[X_i^2]$ isn't quite the variance any more.

If you have some information about the first two moments of the summands, you can get something useful. For example, a fairly standard form of the Chernoff bound would say something like if $$ X = X_1 + \cdots + X_n $$ with all $X_i$ discrete, supported in $[-1,1]$, centered (i.e., $\mathbb{E}[X_i]=0$), and independent, then $$\operatorname{Pr}[X > \lambda\sigma]\le e^{-\lambda^2/4}$$ where $\sigma^2 = \sum_{i\in [n]} \operatorname{Var}[X_i]$ is the variance of $X$. Here is a proof by Van Vu.

If, instead, you are willing to assume that $\operatorname{E}[X_i]\le 0$ and replace $\sigma^2$ with the sum of $\operatorname{E}[X_i^2]$, you'll get a reasonably nice looking bound by just tracing through the argument.

Also, this will explain why you can't just infer a general answer. The natural units for tail bounds are standard deviations from the mean. In this case, if you move the mean away from zero, then $\operatorname{E}[X_i^2]$ isn't quite the variance any more.

Fix wording.
Source Link
D.W.
  • 165.6k
  • 21
  • 230
  • 490

If you have some information about the first two moments of the summands, you can get something useful. For example, a fairly standard form of the Chernoff bound would say something like if $$ X = X_1 + \cdots X_n $$$$ X = X_1 + \cdots + X_n $$ with all $X_i$ discrete, supported in $[-1,1]$, centered, and independent, then $$\operatorname{Pr}[X > \lambda\sigma]\le e^{-\lambda^2/4}$$ Wherewhere $\sigma^2 = \sum_{i\in [n]} \operatorname{Var}[X_i]$ is the variance of $X$. Here is a proof by Vana proof by Van Vu.

If, instead, you are willing to assume that $\operatorname{E}[X_i]\le 0$ and replace $\sigma^2$ with the sum of $\operatorname{E}[X_i^2]$, you'll get a reasonably nice looking bound by just tracing through the argument.

Also, this will explain why you can't just infer a general answer. The natural units for tail bounds are standard deviations from the mean. In this case, if you move the mean away from zero, then $\operatorname{E}[X_i^2]$ isn't quite the variance any more.

If you have some information about the first two moments of the summands, you can get something useful. For example, a fairly standard form of the Chernoff bound would say something like if $$ X = X_1 + \cdots X_n $$ with all $X_i$ discrete, supported in $[-1,1]$, centered, and independent, then $$\operatorname{Pr}[X > \lambda\sigma]\le e^{-\lambda^2/4}$$ Where $\sigma^2 = \sum_{i\in [n]} \operatorname{Var}[X_i]$ is the variance of $X$. Here is a proof by Van Vu.

If, instead, you are willing to assume that $\operatorname{E}[X_i]\le 0$ and replace $\sigma^2$ with the sum of $\operatorname{E}[X_i^2]$, you'll get a reasonably nice looking bound by just tracing through the argument.

Also, this will explain why you can't just infer a general answer. The natural units for tail bounds are standard deviations from the mean. In this case, if you move the mean away from zero, then $\operatorname{E}[X_i^2]$ isn't quite the variance any more.

If you have some information about the first two moments of the summands, you can get something useful. For example, a fairly standard form of the Chernoff bound would say something like if $$ X = X_1 + \cdots + X_n $$ with all $X_i$ discrete, supported in $[-1,1]$, centered, and independent, then $$\operatorname{Pr}[X > \lambda\sigma]\le e^{-\lambda^2/4}$$ where $\sigma^2 = \sum_{i\in [n]} \operatorname{Var}[X_i]$ is the variance of $X$. Here is a proof by Van Vu.

If, instead, you are willing to assume that $\operatorname{E}[X_i]\le 0$ and replace $\sigma^2$ with the sum of $\operatorname{E}[X_i^2]$, you'll get a reasonably nice looking bound by just tracing through the argument.

Also, this will explain why you can't just infer a general answer. The natural units for tail bounds are standard deviations from the mean. In this case, if you move the mean away from zero, then $\operatorname{E}[X_i^2]$ isn't quite the variance any more.

Added independence assumption, discussion of why knowing the expectation helps.
Source Link
Louis
  • 2.9k
  • 16
  • 25
Loading
Source Link
Louis
  • 2.9k
  • 16
  • 25
Loading