In a setting where I have n clients each owns a subset of the dataset, each computes the mean and variance on his data and send them to the server. The server would average all means and variances. For mean, I simply do the averaging by: $\mu = \frac{\mu_1+\mu_2+.....+\mu_n}{n}$

However, for the variance, I can't figure out to average n variances. For only two I found this answer here

$\sigma^2 = \frac{1}{2}(\sigma^2_1 + \sigma^2_2) + (\frac{\mu_1-\mu_2}{2})^2$

But how about n variances? thank you


Let's assume you have $n$ datasets, $x_k$ is the $k$th dataset. It's length we denote by $l_k$, its elements are $$x_k =\big\{x_k(1), x_k(2), \dots, x_k(l_k)\big\},$$

and its mean and variance are respectively \begin{align} \mu_k &= \frac{1}{l_k}\sum_{i=1}^{l_k} x_k(i),\\ \sigma_k^2 &= \frac{1}{l_k}\sum_{i=1}^{l_k} x_k(i)^2 - \mu_k^2. \end{align}

Then, noting that $l_k\mu_k = \sum_{i=1}^{l_k} x_k(i)$:

\begin{align} \mu &= \frac{1}{\sum_{k=1}^nl_k}\sum_{k=1}^n\sum_{i=1}^{l_k} x_k(i)\\ \mu &= \frac{1}{\sum_{k=1}^nl_k}\sum_{k=1}^nl_k\mu_k \end{align}

and in similar fashion

\begin{align} \sigma^2 &= \frac{1}{\sum_{k=1}^nl_k}\sum_{k=1}^n\sum_{i=1}^{l_k} x_k(i)^2 - \mu^2\\ \sigma^2 &= \frac{1}{\sum_{k=1}^nl_k}\sum_{k=1}^nl_k\left(\sigma^2_k + \mu_k^2\right) - \mu^2 \end{align}

When $l_k = N$ for all $k$ the above simplifies:

\begin{align} \mu &= \frac{1}{n}\sum_{k=1}^n\mu_k\\ \sigma^2 &= \frac{1}{n}\sum_{k=1}^n\left(\sigma^2_k + \mu_k^2\right) - \mu^2 \end{align}

however when the datasets have different sizes the above does not simplify, and you need to take that into account.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.