随机算法 (Fall 2016)/Concentration of measure

Balls into Bins

Consider throwing $m$ balls into $n$ bins uniformly and independently at random. This is equivalent to a random mapping $f:[m]\to [n]$. Needless to say, random mapping is an important random model and may have many applications in Computer Science, e.g. hashing.

We are concerned with the following three questions regarding the balls into bins model:

• birthday problem: the probability that every bin contains at most one ball (the mapping is 1-1);
• coupon collector problem: the probability that every bin contains at least one ball (the mapping is on-to);
• occupancy problem: the maximum load of bins.

Birthday Problem

There are $m$ students in the class. Assume that for each student, his/her birthday is uniformly and independently distributed over the 365 days in a years. We wonder what the probability that no two students share a birthday.

Due to the pigeonhole principle, it is obvious that for $m>365$, there must be two students with the same birthday. Surprisingly, for any $m>57$ this event occurs with more than 99% probability. This is called the birthday paradox. Despite the name, the birthday paradox is not a real paradox.

We can model this problem as a balls-into-bins problem. $m$ different balls (students) are uniformly and independently thrown into 365 bins (days). More generally, let $n$ be the number of bins. We ask for the probability of the following event $\mathcal{E}$

• $\mathcal{E}$: there is no bin with more than one balls (i.e. no two students share birthday).

We first analyze this by counting. There are totally $n^m$ ways of assigning $m$ balls to $n$ bins. The number of assignments that no two balls share a bin is $(\genfrac{}{}{0ex}{}{n}{m})m!$.

Thus the probability is given by:

$\begin{array}{r}Pr[\mathcal{E}]=\frac{(\genfrac{}{}{0ex}{}{n}{m})m!}{n^m}.\end{array}$

Recall that $(\genfrac{}{}{0ex}{}{n}{m})=\frac{n!}{(n-m)!m!}$. Then

$\begin{array}{r}Pr[\mathcal{E}]=\frac{(\genfrac{}{}{0ex}{}{n}{m})m!}{n^m}=\frac{n!}{n^m(n-m)!}=\frac{n}{n}\cdot \frac{n-1}{n}\cdot \frac{n-2}{n}\cdots \frac{n-(m-1)}{n}=\prod _{k=1}^{m-1}(1-\frac{k}{n}).\end{array}$

There is also a more "probabilistic" argument for the above equation. Consider again that $m$ students are mapped to $n$ possible birthdays uniformly at random.

The first student has a birthday for sure. The probability that the second student has a different birthday from the first student is $(1-\frac{1}{n})$. Given that the first two students have different birthdays, the probability that the third student has a different birthday from the first two students is $(1-\frac{2}{n})$. Continuing this on, assuming that the first $k-1$ students all have different birthdays, the probability that the $k$th student has a different birthday than the first $k-1$, is given by $(1-\frac{k-1}{n})$. By the chain rule, the probability that all $m$ students have different birthdays is:

$\begin{array}{rl}Pr[\mathcal{E}]=(1-\frac{1}{n})\cdot (1-\frac{2}{n})\cdots (1-\frac{m-1}{n})& =\prod _{k=1}^{m-1}(1-\frac{k}{n}),\end{array}$

which is the same as what we got by the counting argument.

There are several ways of analyzing this formular. Here is a convenient one: Due to Taylor's expansion, $e^{-k/n}\approx 1-k/n$. Then

$\begin{array}{rl}\prod _{k=1}^{m-1}(1-\frac{k}{n})& \approx \prod _{k=1}^{m-1}{e}^{-\frac{k}{n}}\\ & =\exp (-\sum _{k=1}^{m-1}\frac{k}{n})\\ & =e^{-m(m-1)/2n}\\ & \approx e^{-m^2/2n}.\end{array}$

The quality of this approximation is shown in the Figure.

Therefore, for $m=\sqrt{2n\ln \frac{1}{ϵ}}$, the probability that $Pr[\mathcal{E}]\approx ϵ$.

Coupon Collector

Suppose that a chocolate company releases $n$ different types of coupons. Each box of chocolates contains one coupon with a uniformly random type. Once you have collected all $n$ types of coupons, you will get a prize. So how many boxes of chocolates you are expected to buy to win the prize?

The coupon collector problem can be described in the balls-into-bins model as follows. We keep throwing balls one-by-one into $n$ bins (coupons), such that each ball is thrown into a bin uniformly and independently at random. Each ball corresponds to a box of chocolate, and each bin corresponds to a type of coupon. Thus, the number of boxes bought to collect $n$ coupons is just the number of balls thrown until none of the $n$ bins is empty.

Theorem

Let $X$ be the number of balls thrown uniformly and independently to $n$ bins until no bin is empty. Then $\mathbf{E}[X]=nH(n)$, where $H(n)$ is the $n$th harmonic number.

Proof. Let $X_i$ be the number of balls thrown while there are exactly $i-1$ nonempty bins, then clearly $X=\sum _{i=1}^n{X}_i$. When there are exactly $i-1$ nonempty bins, throwing a ball, the probability that the number of nonempty bins increases (i.e. the ball is thrown to an empty bin) is $p_i=1-\frac{i-1}{n}.$ $X_i$ is the number of balls thrown to make the number of nonempty bins increases from $i-1$ to $i$, i.e. the number of balls thrown until a ball is thrown to a current empty bin. Thus, $X_i$ follows the geometric distribution, such that $Pr[X_i=k]=(1-p_i{)}^{k-1}{p}_i$ For a geometric random variable, $\mathbf{E}[X_i]=\frac{1}{p_i}=\frac{n}{n-i+1}$. Applying the linearity of expectations, $\begin{array}{rl}\mathbf{E}[X]& =\mathbf{E}\left[\sum _{i=1}^n{X}_i\right]\\ & =\sum _{i=1}^n\mathbf{E}\left[X_i\right]\\ & =\sum _{i=1}^n\frac{n}{n-i+1}\\ & =n\sum _{i=1}^n\frac{1}{i}\\ & =nH(n),\end{array}$ where $H(n)$ is the $n$th Harmonic number, and $H(n)=\ln n+O(1)$. Thus, for the coupon collectors problem, the expected number of coupons required to obtain all $n$ types of coupons is $n\ln n+O(n)$. $◻$

---

Only knowing the expectation is not good enough. We would like to know how fast the probability decrease as a random variable deviates from its mean value.

Theorem

Let $X$ be the number of balls thrown uniformly and independently to $n$ bins until no bin is empty. Then $Pr[X\ge n\ln n+cn]<e^{-c}$ for any $c>0$.

Proof. For any particular bin $i$, the probability that bin $i$ is empty after throwing $n\ln n+cn$ balls is ${(1-\frac{1}{n})}^{n\ln n+cn}<e^{-(\ln n+c)}=\frac{1}{n{e}^c}.$ By the union bound, the probability that there exists an empty bin after throwing $n\ln n+cn$ balls is $Pr[X\ge n\ln n+cn]<n\cdot \frac{1}{n{e}^c}=e^{-c}.$ $◻$

Occupancy Problem

Now we ask about the loads of bins. Assuming that $m$ balls are uniformly and independently assigned to $n$ bins, for $1\le i\le n$, let $X_i$ be the load of the $i$th bin, i.e. the number of balls in the $i$th bin.

An easy analysis shows that for every bin $i$, the expected load $\mathbf{E}[X_i]$ is equal to the average load $m/n$.

Because there are totally $m$ balls, it is always true that $\sum _{i=1}^n{X}_i=m$.

Therefore, due to the linearity of expectations,

$\begin{array}{rl}\sum _{i=1}^n\mathbf{E}[X_i]& =\mathbf{E}\left[\sum _{i=1}^n{X}_i\right]=\mathbf{E}\left[m\right]=m.\end{array}$

Because for each ball, the bin to which the ball is assigned is uniformly and independently chosen, the distributions of the loads of bins are identical. Thus $\mathbf{E}[X_i]$ is the same for each $i$. Combining with the above equation, it holds that for every $1\le i\le m$, $\mathbf{E}[X_i]=\frac{m}{n}$. So the average is indeed the average!

---

Next we analyze the distribution of the maximum load. We show that when $m=n$, i.e. $n$ balls are uniformly and independently thrown into $n$ bins, the maximum load is $O\left(\frac{\log n}{\log \log n}\right)$ with high probability.

Theorem

Suppose that $n$ balls are thrown independently and uniformly at random into $n$ bins. For $1\le i\le n$, let $X_i$ be the random variable denoting the number of balls in the $i$th bin. Then $Pr[\underset{1\le i\le n}{max}{X}_i\ge \frac{3\ln n}{\ln \ln n}]<\frac{1}{n}.$

Proof. Let $M$ be an integer. Take bin 1. For any particular $M$ balls, these $M$ balls are all thrown to bin 1 with probability $(1/n{)}^M$, and there are totally $(\genfrac{}{}{0ex}{}{n}{M})$ distinct sets of $M$ balls. Therefore, applying the union bound, $\begin{array}{rl}Pr[X_1\ge M]& \le (\genfrac{}{}{0ex}{}{n}{M}){\left(\frac{1}{n}\right)}^M\\ & =\frac{n!}{M!(n-M)!n^M}\\ & =\frac{1}{M!}\cdot \frac{n(n-1)(n-2)\cdots (n-M+1)}{n^M}\\ & =\frac{1}{M!}\cdot \prod _{i=0}^{M-1}(1-\frac{i}{n})\\ & \le \frac{1}{M!}.\end{array}$ According to Stirling's approximation, $M!\approx \sqrt{2\pi M}{\left(\frac{M}{e}\right)}^M$, thus $\frac{1}{M!}\le {\left(\frac{e}{M}\right)}^M.$ Figure 1 Due to the symmetry. All $X_i$ have the same distribution. Apply the union bound again, $\begin{array}{rl}Pr[\underset{1\le i\le n}{max}{X}_i\ge M]& =Pr[(X_1\ge M)\vee (X_2\ge M)\vee \cdots \vee (X_n\ge M)]\\ & \le nPr[X_1\ge M]\\ & \le n{\left(\frac{e}{M}\right)}^M.\end{array}$ When $M=3\ln n/\ln \ln n$, $\begin{array}{rl}{\left(\frac{e}{M}\right)}^M& ={\left(\frac{e\ln \ln n}{3\ln n}\right)}^{3\ln n/\ln \ln n}\\ & <{\left(\frac{\ln \ln n}{\ln n}\right)}^{3\ln n/\ln \ln n}\\ & =e^{3(\ln \ln \ln n-\ln \ln n)\ln n/\ln \ln n}\\ & =e^{-3\ln n+3\ln \ln \ln n\ln n/\ln \ln n}\\ & \le e^{-2\ln n}\\ & =\frac{1}{n^2}.\end{array}$ Therefore, $\begin{array}{rlr}Pr[\underset{1\le i\le n}{max}{X}_i\ge \frac{3\ln n}{\ln \ln n}]& \le n{\left(\frac{e}{M}\right)}^M& <\frac{1}{n}.\end{array}$ $◻$

When $m>n$, Figure 1 illustrates the results of several random experiments, which show that the distribution of the loads of bins becomes more even as the number of balls grows larger than the number of bins.

Formally, it can be proved that for $m=\mathrm{\Omega }(n\log n)$, with high probability, the maximum load is within $O\left(\frac{m}{n}\right)$, which is asymptotically equal to the average load.

The Chernoff Bound

Suppose that we have a fair coin. If we toss it once, then the outcome is completely unpredictable. But if we toss it, say for 1000 times, then the number of HEADs is very likely to be around 500. This striking phenomenon, illustrated in the right figure, is called the concentration. The Chernoff bound captures the concentration of independent trials.

The Chernoff bound is also a tail bound for the sum of independent random variables which may give us exponentially sharp bounds.

Before proving the Chernoff bound, we should talk about the moment generating functions.

Moment generating functions

The more we know about the moments of a random variable $X$, the more information we would have about $X$. There is a so-called moment generating function, which "packs" all the information about the moments of $X$ into one function.

Definition

The moment generating function of a random variable $X$ is defined as $\mathbf{E}\left[{\mathrm{e}}^{\lambda X}\right]$ where $\lambda$ is the parameter of the function.

By Taylor's expansion and the linearity of expectations,

$\begin{array}{rl}\mathbf{E}\left[{\mathrm{e}}^{\lambda X}\right]& =\mathbf{E}\left[\sum _{k=0}^{\mathrm{\infty }}\frac{{\lambda }^k}{k!}{X}^k\right]\\ & =\sum _{k=0}^{\mathrm{\infty }}\frac{{\lambda }^k}{k!}\mathbf{E}\left[X^k\right]\end{array}$

The moment generating function $\mathbf{E}\left[{\mathrm{e}}^{\lambda X}\right]$ is a function of $\lambda$.

The Chernoff bound

The Chernoff bounds are exponentially sharp tail inequalities for the sum of independent trials. The bounds are obtained by applying Markov's inequality to the moment generating function of the sum of independent trials, with some appropriate choice of the parameter $\lambda$.

Chernoff bound (the upper tail)

Let $X=\sum _{i=1}^n{X}_i$, where $X_1,X_2,\dots ,X_n$ are independent Poisson trials. Let $\mu =\mathbf{E}[X]$. Then for any $\delta >0$, $Pr[X\ge (1+\delta )\mu ]\le {\left(\frac{e^{\delta }}{(1+\delta {)}^{(1+\delta )}}\right)}^{\mu }.$

Proof. For any $\lambda >0$, $X\ge (1+\delta )\mu$ is equivalent to that $e^{\lambda X}\ge e^{\lambda (1+\delta )\mu }$, thus $\begin{array}{rl}Pr[X\ge (1+\delta )\mu ]& =Pr[e^{\lambda X}\ge e^{\lambda (1+\delta )\mu }]\\ & \le \frac{\mathbf{E}\left[e^{\lambda X}\right]}{e^{\lambda (1+\delta )\mu }},\end{array}$ where the last step follows by Markov's inequality. Computing the moment generating function $\mathbf{E}[e^{\lambda X}]$: $\begin{array}{rl}\mathbf{E}\left[e^{\lambda X}\right]& =\mathbf{E}\left[e^{\lambda \sum _{i=1}^n{X}_i}\right]\\ & =\mathbf{E}\left[\prod _{i=1}^n{e}^{\lambda X_i}\right]\\ & =\prod _{i=1}^n\mathbf{E}\left[e^{\lambda X_i}\right].& (\text{for independent random variables})\end{array}$ Let $p_i=Pr[X_i=1]$ for $i=1,2,\dots ,n$. Then, $\mu =\mathbf{E}[X]=\mathbf{E}\left[\sum _{i=1}^n{X}_i\right]=\sum _{i=1}^n\mathbf{E}[X_i]=\sum _{i=1}^n{p}_i$. We bound the moment generating function for each individual $X_i$ as follows. $\begin{array}{rl}\mathbf{E}\left[e^{\lambda X_i}\right]& =p_i\cdot e^{\lambda \cdot 1}+(1-p_i)\cdot e^{\lambda \cdot 0}\\ & =1+p_i(e^{\lambda }-1)\\ & \le e^{p_i(e^{\lambda }-1)},\end{array}$ where in the last step we apply the Taylor's expansion so that $e^y\ge 1+y$ where $y=p_i(e^{\lambda }-1)\ge 0$. (By doing this, we can transform the product to the sum of $p_i$, which is $\mu$.) Therefore, $\begin{array}{rl}\mathbf{E}\left[e^{\lambda X}\right]& =\prod _{i=1}^n\mathbf{E}\left[e^{\lambda X_i}\right]\\ & \le \prod _{i=1}^n{e}^{p_i(e^{\lambda }-1)}\\ & =\exp (\sum _{i=1}^n{p}_i(e^{\lambda }-1))\\ & =e^{(e^{\lambda }-1)\mu }.\end{array}$ Thus, we have shown that for any $\lambda >0$, $\begin{array}{rl}Pr[X\ge (1+\delta )\mu ]& \le \frac{\mathbf{E}\left[e^{\lambda X}\right]}{e^{\lambda (1+\delta )\mu }}\\ & \le \frac{e^{(e^{\lambda }-1)\mu }}{e^{\lambda (1+\delta )\mu }}\\ & ={\left(\frac{e^{(e^{\lambda }-1)}}{e^{\lambda (1+\delta )}}\right)}^{\mu }\end{array}$. For any $\delta >0$, we can let $\lambda =\ln (1+\delta )>0$ to get $Pr[X\ge (1+\delta )\mu ]\le {\left(\frac{e^{\delta }}{(1+\delta {)}^{(1+\delta )}}\right)}^{\mu }.$ $◻$

The idea of the proof is actually quite clear: we apply Markov's inequality to $e^{\lambda X}$ and for the rest, we just estimate the moment generating function $\mathbf{E}[e^{\lambda X}]$. To make the bound as tight as possible, we minimized the $\frac{e^{(e^{\lambda }-1)}}{e^{\lambda (1+\delta )}}$ by setting $\lambda =\ln (1+\delta )$, which can be justified by taking derivatives of $\frac{e^{(e^{\lambda }-1)}}{e^{\lambda (1+\delta )}}$.

---

We then proceed to the lower tail, the probability that the random variable deviates below the mean value:

Chernoff bound (the lower tail)

Let $X=\sum _{i=1}^n{X}_i$, where $X_1,X_2,\dots ,X_n$ are independent Poisson trials. Let $\mu =\mathbf{E}[X]$. Then for any $0<\delta <1$, $Pr[X\le (1-\delta )\mu ]\le {\left(\frac{e^{-\delta }}{(1-\delta {)}^{(1-\delta )}}\right)}^{\mu }.$

Proof. For any $\lambda <0$, by the same analysis as in the upper tail version, $\begin{array}{rl}Pr[X\le (1-\delta )\mu ]& =Pr[e^{\lambda X}\ge e^{\lambda (1-\delta )\mu }]\\ & \le \frac{\mathbf{E}\left[e^{\lambda X}\right]}{e^{\lambda (1-\delta )\mu }}\\ & \le {\left(\frac{e^{(e^{\lambda }-1)}}{e^{\lambda (1-\delta )}}\right)}^{\mu }.\end{array}$ For any $0<\delta <1$, we can let $\lambda =\ln (1-\delta )<0$ to get $Pr[X\ge (1-\delta )\mu ]\le {\left(\frac{e^{-\delta }}{(1-\delta {)}^{(1-\delta )}}\right)}^{\mu }.$ $◻$

Useful forms of the Chernoff bounds

Some useful special forms of the bounds can be derived directly from the above general forms of the bounds. We now know better why we say that the bounds are exponentially sharp.

Useful forms of the Chernoff bound

Let $X=\sum _{i=1}^n{X}_i$, where $X_1,X_2,\dots ,X_n$ are independent Poisson trials. Let $\mu =\mathbf{E}[X]$. Then 1. for $0<\delta \le 1$, $Pr[X\ge (1+\delta )\mu ]<\exp (-\frac{\mu {\delta }^2}{3});$ $Pr[X\le (1-\delta )\mu ]<\exp (-\frac{\mu {\delta }^2}{2});$ 2. for $t\ge 2e\mu$, $Pr[X\ge t]\le 2^{-t}.$

Proof. To obtain the bounds in (1), we need to show that for $0<\delta <1$, $\frac{e^{\delta }}{(1+\delta {)}^{(1+\delta )}}\le e^{-{\delta }^2/3}$ and $\frac{e^{-\delta }}{(1-\delta {)}^{(1-\delta )}}\le e^{-{\delta }^2/2}$. We can verify both inequalities by standard analysis techniques. To obtain the bound in (2), let $t=(1+\delta )\mu$. Then $\delta =t/\mu -1\ge 2e-1$. Hence, $\begin{array}{rl}Pr[X\ge (1+\delta )\mu ]& \le {\left(\frac{e^{\delta }}{(1+\delta {)}^{(1+\delta )}}\right)}^{\mu }\\ & \le {\left(\frac{e}{1+\delta }\right)}^{(1+\delta )\mu }\\ & \le {\left(\frac{e}{2e}\right)}^t\\ & \le 2^{-t}\end{array}$ $◻$

Applications to balls-into-bins

Throwing $m$ balls uniformly and independently to $n$ bins, what is the maximum load of all bins with high probability? In the last class, we gave an analysis of this problem by using a counting argument.

Now we give a more "advanced" analysis by using Chernoff bounds.

For any $i\in [n]$ and $j\in [m]$, let $X_{ij}$ be the indicator variable for the event that ball $j$ is thrown to bin $i$. Obviously

$\mathbf{E}[X_{ij}]=Pr[\text{ball }j\text{ is thrown to bin }i]=\frac{1}{n}$

Let $Y_i=\sum _{j\in [m]}{X}_{ij}$ be the load of bin $i$.

Then the expected load of bin $i$ is

$(\ast )\mu =\mathbf{E}[Y_i]=\mathbf{E}\left[\sum _{j\in [m]}{X}_{ij}\right]=\sum _{j\in [m]}\mathbf{E}[X_{ij}]=m/n.$

For the case $m=n$, it holds that $\mu =1$

Note that $Y_i$ is a sum of $m$ mutually independent indicator variable. Applying Chernoff bound, for any particular bin $i\in [n]$,

$Pr[Y_i>(1+\delta )\mu ]\le {\left(\frac{e^{\delta }}{(1+\delta {)}^{1+\delta }}\right)}^{\mu }.$

The $m=n$ case

When $m=n$, $\mu =1$. Write $c=1+\delta$. The above bound can be written as

$Pr[Y_i>c]\le \frac{e^{c-1}}{c^c}.$

Let $c=\frac{e\ln n}{\ln \ln n}$, we evaluate $\frac{e^{c-1}}{c^c}$ by taking logarithm to its reciprocal.

$\begin{array}{rl}\ln \left(\frac{c^c}{e^{c-1}}\right)& =c\ln c-c+1\\ & =c(\ln c-1)+1\\ & =\frac{e\ln n}{\ln \ln n}(\ln \ln n-\ln \ln \ln n)+1\\ & \ge \frac{e\ln n}{\ln \ln n}\cdot \frac{2}{e}\ln \ln n+1\\ & \ge 2\ln n.\end{array}$

Thus,

$Pr[Y_i>\frac{e\ln n}{\ln \ln n}]\le \frac{1}{n^2}.$

Applying the union bound, the probability that there exists a bin with load $>12\ln n$ is

$n\cdot Pr[Y_1>\frac{e\ln n}{\ln \ln n}]\le \frac{1}{n}$.

Therefore, for $m=n$, with high probability, the maximum load is $O\left(\frac{e\ln n}{\ln \ln n}\right)$.

The $m>\ln n$ case

When $m\ge n\ln n$, then according to $(\ast )$, $\mu =\frac{m}{n}\ge \ln n$

We can apply an easier form of the Chernoff bounds,

$Pr[Y_i\ge 2e\mu ]\le 2^{-2e\mu }\le 2^{-2e\ln n}<\frac{1}{n^2}.$

By the union bound, the probability that there exists a bin with load $\ge 2e\frac{m}{n}$ is,

$n\cdot Pr[Y_1>2e\frac{m}{n}]=n\cdot Pr[Y_1>2e\mu ]\le \frac{1}{n}$.

Therefore, for $m\ge n\ln n$, with high probability, the maximum load is $O\left(\frac{m}{n}\right)$.