# 随机算法 (Fall 2015)/Balls and Bins

# Probability Space

The axiom foundation of probability theory is laid by Kolmogorov, one of the greatest mathematician of the 20th century, who advanced various very different fields of mathematics.

**Definition (Probability Space)**A

**probability space**is a triple .- is a set, called the
**sample space**. - is the set of all
**events**, satisfying:- (K1). and . (The
*certain*event and the*impossible*event.) - (K2). If , then . (Intersection, union, and diference of two events are events).

- (K1). and . (The
- A
**probability measure**is a function that maps each event to a nonnegative real number, satisfying- (K3). .
- (K4). If (such events are call
*disjoint*events), then . - (K5*). For a decreasing sequence of events of events with , it holds that .

- is a set, called the

- Remark

- In general, the set may be continuous, but we only consider
**discrete**probability in this lecture, thus we assume that is either finite or countably infinite. - Sometimes it is convenient to assume , i.e. the events enumerates all subsets of . But in general, a probability space is well-defined by any satisfying (K1) and (K2). Such is called a -algebra defined on .
- The last axiom (K5*) is redundant if is finite, thus it is only essential when there are infinitely many events. The role of axiom (K5*) in probability theory is like Zorn's Lemma (or equivalently the Axiom of Choice) in axiomatic set theory.

Useful laws for probability can be deduced from the *axioms* (K1)-(K5).

**Proposition**- Let . It holds that .
- If then .

**Proof.**- The events and are disjoint and . Due to Axiom (K4) and (K3), .
- The events and are disjoint and since . Due to Axiom (K4), , thus .

- Notation

An event can be represented as with a predicate .

The predicate notation of probability is

- .

We will mostly use the predicate notation instead of subset notation.

## Conditional Probability

In probability theory, the word "condition" is a verb. "Conditioning on the event ..." means that it is assumed that the event occurs.

**Definition (conditional probability)**- The
**conditional probability**that event occurs given that event occurs is

- The

The conditional probability is well-defined only if .

For independent events and , it holds that

It supports our intuition that for two independent events, whether one of them occurs will not affect the chance of the other.

### Law of total probability

The following fact is known as the law of total probability. It computes the probability by averaging over all possible cases.

**Theorem (law of total probability)**- Let be
*mutually disjoint*events, and is the sample space. - Then for any event ,

- Let be

**Proof.**Since are mutually disjoint and , events are also mutually disjoint, and . Then the additivity of disjoint events, we have

The law of total probability provides us a standard tool for breaking a probability into sub-cases. Sometimes this will help the analysis.

### The chain rule

By the definition of conditional probability, . Thus, . This hints us that we can compute the probability of the AND of events by conditional probabilities. Formally, we have the following theorem:

**Theorem**- Let be any events. Then

- Let be any events. Then

**Proof.**It holds that . Thus, let and , then Recursively applying this equation to until there is only left, the theorem is proved.

## Random Variable

**Definition (random variable)**- A random variable on a sample space is a real-valued function . A random variable X is called a
**discrete**random variable if its range is finite or countably infinite.

- A random variable on a sample space is a real-valued function . A random variable X is called a

For a random variable and a real value , we write "" for the event , and denote the probability of the event by

- .

The independence can also be defined for variables:

**Definition (Independent variables)**- Two random variables and are
**independent**if and only if - for all values and . Random variables are
**mutually independent**if and only if, for any subset and any values , where ,

- Two random variables and are

Note that in probability theory, the "mutual independence" is not equivalent with "pair-wise independence", which we will learn in the future.

## Linearity of Expectation

Let be a discrete **random variable**. The expectation of is defined as follows.

**Definition (Expectation)**- The
**expectation**of a discrete random variable , denoted by , is given by - where the summation is over all values in the range of .

- The

Perhaps the most useful property of expectation is its **linearity**.

**Theorem (Linearity of Expectations)**- For any discrete random variables , and any real constants ,

- For any discrete random variables , and any real constants ,

**Proof.**By the definition of the expectations, it is easy to verify that (try to prove by yourself): for any discrete random variables and , and any real constant ,

- ;
- .

The theorem follows by induction.

The linearity of expectation gives an easy way to compute the expectation of a random variable if the variable can be written as a sum.

- Example
- Supposed that we have a biased coin that the probability of HEADs is . Flipping the coin for n times, what is the expectation of number of HEADs?
- It looks straightforward that it must be np, but how can we prove it? Surely we can apply the definition of expectation to compute the expectation with brute force. A more convenient way is by the linearity of expectations: Let indicate whether the -th flip is HEADs. Then , and the total number of HEADs after n flips is . Applying the linearity of expectation, the expected number of HEADs is:
- .

The real power of the linearity of expectations is that it does not require the random variables to be independent, thus can be applied to any set of random variables. For example:

However, do not exaggerate this power!

- For an arbitrary function (not necessarily linear), the equation does not hold generally.
- For variances, the equation does not hold without further assumption of the independence of and .

## Conditional Expectation

Conditional expectation can be accordingly defined:

**Definition (conditional expectation)**- For random variables and ,
- where the summation is taken over the range of .

- For random variables and ,

There is also a **law of total expectation**.

**Theorem (law of total expectation)**- Let and be two random variables. Then

- Let and be two random variables. Then

# Distributions of Coin Flips

We introduce several important probability distributions induced by independent coin flips (independent trials), including: Bernoulli trial, geometric distribution, binomial distribution.

## Bernoulli trial (Bernoulli distribution)

Bernoulli trial describes the probability distribution of a single (biased) coin flip. Suppose that we flip a (biased) coin where the probability of HEADS is . Let be the 0-1 random variable which indicates whether the result is HEADS. We say that follows the Bernoulli distribution with parameter . Formally,

- .

## Geometric distribution

Suppose we flip the same coin repeatedly until HEADS appears, where each coin flip is independent and follows the Bernoulli distribution with parameter . Let be the random variable denoting the total number of coin flips. Then has the geometric distribution with parameter . Formally, .

For geometric , . This can be verified by directly computing by the definition of expectations. There is also a smarter way of computing , by using indicators and the linearity of expectations. For , let be the 0-1 random variable such that if and only if none of the first coin flipings are HEADS, thus . A key observation is that . Thus, due to the linearity of expectations,

## Binomial distribution

Suppose we flip the same (biased) coin for times, where each coin flip is independent and follows the Bernoulli distribution with parameter . Let be the number of HEADS. Then has the binomial distribution with parameters and . Formally, .

A binomial random variable with parameters and is usually denoted by .

As we saw above, by applying the linearity of expectations, it is easy to show that for an .

# Balls into Bins

Consider throwing balls into bins uniformly and independently at random. This is equivalent to a random mapping . Needless to say, random mapping is an important random model and may have many applications in Computer Science, e.g. hashing.

We are concerned with the following three questions regarding the balls into bins model:

- birthday problem: the probability that every bin contains at most one ball (the mapping is 1-1);
- coupon collector problem: the probability that every bin contains at least one ball (the mapping is on-to);
- occupancy problem: the maximum load of bins.

## Birthday Problem

There are students in the class. Assume that for each student, his/her birthday is uniformly and independently distributed over the 365 days in a years. We wonder what the probability that no two students share a birthday.

Due to the pigeonhole principle, it is obvious that for , there must be two students with the same birthday. Surprisingly, for any this event occurs with more than 99% probability. This is called the **birthday paradox**. Despite the name, the birthday paradox is not a real paradox.

We can model this problem as a balls-into-bins problem. different balls (students) are uniformly and independently thrown into 365 bins (days). More generally, let be the number of bins. We ask for the probability of the following event

- : there is no bin with more than one balls (i.e. no two students share birthday).

We first analyze this by counting. There are totally ways of assigning balls to bins. The number of assignments that no two balls share a bin is .

Thus the probability is given by:

Recall that . Then

There is also a more "probabilistic" argument for the above equation. Consider again that students are mapped to possible birthdays uniformly at random.

The first student has a birthday for sure. The probability that the second student has a different birthday from the first student is . Given that the first two students have different birthdays, the probability that the third student has a different birthday from the first two students is . Continuing this on, assuming that the first students all have different birthdays, the probability that the th student has a different birthday than the first , is given by . By the chain rule, the probability that all students have different birthdays is:

which is the same as what we got by the counting argument.

There are several ways of analyzing this formular. Here is a convenient one: Due to Taylor's expansion, . Then

The quality of this approximation is shown in the Figure.

Therefore, for , the probability that .

## Coupon Collector

Suppose that a chocolate company releases different types of coupons. Each box of chocolates contains one coupon with a uniformly random type. Once you have collected all types of coupons, you will get a prize. So how many boxes of chocolates you are expected to buy to win the prize?

The coupon collector problem can be described in the balls-into-bins model as follows. We keep throwing balls one-by-one into bins (coupons), such that each ball is thrown into a bin uniformly and independently at random. Each ball corresponds to a box of chocolate, and each bin corresponds to a type of coupon. Thus, the number of boxes bought to collect coupons is just the number of balls thrown until none of the bins is empty.

**Theorem**- Let be the number of balls thrown uniformly and independently to bins until no bin is empty. Then , where is the th harmonic number.

**Proof.**Let be the number of balls thrown while there are *exactly*nonempty bins, then clearly .When there are exactly nonempty bins, throwing a ball, the probability that the number of nonempty bins increases (i.e. the ball is thrown to an empty bin) is

is the number of balls thrown to make the number of nonempty bins increases from to , i.e. the number of balls thrown until a ball is thrown to a current empty bin. Thus, follows the geometric distribution, such that

For a geometric random variable, .

Applying the linearity of expectations,

where is the th Harmonic number, and . Thus, for the coupon collectors problem, the expected number of coupons required to obtain all types of coupons is .

Only knowing the expectation is not good enough. We would like to know how fast the probability decrease as a random variable deviates from its mean value.

**Theorem**- Let be the number of balls thrown uniformly and independently to bins until no bin is empty. Then for any .

**Proof.**For any particular bin , the probability that bin is empty after throwing balls is By the union bound, the probability that there exists an empty bin after throwing balls is

### Stable Marriage

We now consider the famous **stable marriage problem** or **stable matching problem** (SMP). This problem captures two aspects: allocations (matchings) and stability, two central topics in economics.

An instance of stable marriage consists of:

- men and women;
- each person associated with a strictly ordered
*preference list*containing all the members of the opposite sex.

Formally, let be the set of men and be the set of women. Each man is associated with a permutation of elemets in and each woman is associated with a permutation of elements in .

A *matching* is a one-one correspondence . We said a man and a woman are *partners* in if .

**Definition (stable matching)**- A pair of a man and woman is a
**blocking pair**in a matching if and are not partners in but- prefers to , and
- prefers to .

- A matching is
**stable**if there is no blocking pair in it.

- A pair of a man and woman is a

It is unclear from the definition itself whether stable matchings always exist, and how to efficiently find a stable matching. Both questions are answered by the following proposal algorithm due to Gale and Shapley.

**The proposal algorithm (Gale-Shapley 1962)**- Initially, all person are not married;
- in each step (called a
**proposal**):- an arbitrary unmarried man proposes to the woman who is ranked highest in his preference list among all the women who has not yet rejected ;
- if is still single then accepts the proposal and is married to ;
- if is married to another man who is ranked lower than in her preference list then divorces (thus becomes single again and considers himself as rejected by ) and is married to ;
- if otherwise rejects ;

The algorithm terminates when the last single woman receives a proposal. Since for every pair of man and woman, proposes to at most once. The algorithm terminates in at most proposals in the worst case.

It is obvious to see that the algorithm retruns a macthing, and this matching must be stable. To see this, by contradiction suppose that the algorithm resturns a macthing , such that two men are macthed to two women in respectively, but and prefers each other to their partners and respectively. By definition of the algorithm, would have proposed to before proposing to , by which time must either be single or be matched to a man ranked lower than in her list (because her final partner is ranked lower than ), which means must have accepted 's proposal, a contradiction.

We are interested in the average-case performance of this algorithm, that is, the expected number of proposals if everyone's preference list is a uniformly and independently random permutation.

The following **principle of deferred decisions** is quite useful in analysing performance of algorithm with random input.

**Principle of deferred decisions**- The decision of random choice in the random input can be deferred to the running time of the algorithm.

Apply the principle of deferred decisions, the deterministic proposal algorithm with random permutations as input is equivalent to the following random process:

- At each step, a man choose a woman uniformly and independently at random to propose, among all the women who have not rejected him yet. (
**sample without replacement**)

We then compare the above process with the following modified process:

- The man repeatedly samples a uniform and independent woman to propose among all women, until he successfully samples a woman who has not rejected him and propose to her. (
**sample with replacement**)

It is easy to see that the modified process (sample with replacement) is no more efficient than the original process (sample without replacement) because it simulates the original process if at each step we only count the last proposal to the woman who has not rejected the man. Such comparison of two random processes by forcing them to be related in some way is called coupling.

Note that in the modified process (sample with replacement), each proposal, no matter from which man, is going to a uniformly and independently random women. And we know that the algorithm terminated once the last single woman receives a proposal, i.e. once all women have received at least one proposal. This is the coupon collector problem with proposals as balls (cookie boxes) and women as bins (coupons). Due to our analysis of the coupon collector problem, the expected number of proposals is bounded by .

## Occupancy Problem

Now we ask about the loads of bins. Assuming that balls are uniformly and independently assigned to bins, for , let be the **load** of the th bin, i.e. the number of balls in the th bin.

An easy analysis shows that for every bin , the expected load is equal to the average load .

Because there are totally balls, it is always true that .

Therefore, due to the linearity of expectations,

Because for each ball, the bin to which the ball is assigned is uniformly and independently chosen, the distributions of the loads of bins are identical. Thus is the same for each . Combining with the above equation, it holds that for every , . So the average is indeed the average!

Next we analyze the distribution of the maximum load. We show that when , i.e. balls are uniformly and independently thrown into bins, the maximum load is with high probability.

**Theorem**- Suppose that balls are thrown independently and uniformly at random into bins. For , let be the random variable denoting the number of balls in the th bin. Then

- Suppose that balls are thrown independently and uniformly at random into bins. For , let be the random variable denoting the number of balls in the th bin. Then

**Proof.**Let be an integer. Take bin 1. For any particular balls, these balls are all thrown to bin 1 with probability , and there are totally distinct sets of balls. Therefore, applying the union bound, According to Stirling's approximation, , thus

Due to the symmetry. All have the same distribution. Apply the union bound again,

When ,

Therefore,

When , Figure 1 illustrates the results of several random experiments, which show that the distribution of the loads of bins becomes more even as the number of balls grows larger than the number of bins.

Formally, it can be proved that for , with high probability, the maximum load is within , which is asymptotically equal to the average load.