随机算法 (Spring 2013)/Random Variables and Expectations
Random Variable
Definition (random variable) - A random variable on a sample space is a real-valued function . A random variable X is called a discrete random variable if its range is finite or countably infinite.
For a random variable and a real value , we write "" for the event , and denote the probability of the event by
- .
The independence can also be defined for variables:
Definition (Independent variables) - Two random variables and are independent if and only if
- for all values and . Random variables are mutually independent if and only if, for any subset and any values , where ,
- Two random variables and are independent if and only if
Note that in probability theory, the "mutual independence" is not equivalent with "pair-wise independence", which we will learn in the future.
Expectation
Let be a discrete random variable. The expectation of is defined as follows.
Definition (Expectation) - The expectation of a discrete random variable , denoted by , is given by
- where the summation is over all values in the range of .
- The expectation of a discrete random variable , denoted by , is given by
Linearity of Expectation
Perhaps the most useful property of expectation is its linearity.
Theorem (Linearity of Expectations) - For any discrete random variables , and any real constants ,
- For any discrete random variables , and any real constants ,
Proof. By the definition of the expectations, it is easy to verify that (try to prove by yourself): for any discrete random variables and , and any real constant ,
- ;
- .
The theorem follows by induction.
The linearity of expectation gives an easy way to compute the expectation of a random variable if the variable can be written as a sum.
- Example
- Supposed that we have a biased coin that the probability of HEADs is . Flipping the coin for n times, what is the expectation of number of HEADs?
- It looks straightforward that it must be np, but how can we prove it? Surely we can apply the definition of expectation to compute the expectation with brute force. A more convenient way is by the linearity of expectations: Let indicate whether the -th flip is HEADs. Then , and the total number of HEADs after n flips is . Applying the linearity of expectation, the expected number of HEADs is:
- .
The real power of the linearity of expectations is that it does not require the random variables to be independent, thus can be applied to any set of random variables. For example:
However, do not exaggerate this power!
- For an arbitrary function (not necessarily linear), the equation does not hold generally.
- For variances, the equation does not hold without further assumption of the independence of and .
Conditional Expectation
Conditional expectation can be accordingly defined:
Definition (conditional expectation) - For random variables and ,
- where the summation is taken over the range of .
- For random variables and ,
There is also a law of total expectation.
Theorem (law of total expectation) - Let and be two random variables. Then
- Let and be two random variables. Then
Random Quicksort
Given as input a set of numbers, we want to sort the numbers in in increasing order. One of the most famous algorithm for this problem is the Quicksort algorithm.
- if do:
- pick an as the pivot;
- partition into , , and , where all numbers in are smaller than and all numbers in are larger than ;
- recursively sort and ;
The time complexity of this sorting algorithm is measured by the number of comparisons.
For the deterministic quicksort algorithm, the pivot is picked from a fixed position (e.g. the first number in the array). The worst-case time complexity in terms of number of comparisons is .
We consider the following randomized version of the quicksort.
- if do:
- uniformly pick a random as the pivot;
- partition into , , and , where all numbers in are smaller than and all numbers in are larger than ;
- recursively sort and ;
Analysis of Random Quicksort
Our goal is to analyze the expected number of comparisons during an execution of RandQSort with an arbitrary input . We achieve this by measuring the chance that each pair of elements are compared, and summing all of them up due to Linearity of Expectation.
Let denote the th smallest element in . Let be the random variable which indicates whether and are compared during the execution of RandQSort. That is:
Elements and are compared only if one of them is chosen as pivot. After comparison they are separated (thus are never compared again). So we have the following observations:
Observation 1: Every pair of and are compared at most once.
Therefore the sum of for all pair gives the total number of comparisons. The expected number of comparisons is . Due to Linearity of Expectation, . Our next step is to analyze for each .
By the definition of expectation and ,
We are going to bound this probability.
Observation 2: and are compared if and only if one of them is chosen as pivot when they are still in the same subset.
This is easy to verify: just check the algorithm. The next one is a bit complicated.
Observation 3: If and are still in the same subset then all are in the same subset.
We can verify this by induction. Initially, itself has the property described above; and partitioning any with the property into and will preserve the property for both and . Therefore Claim 3 holds.
Combining Observation 2 and 3, we have:
Observation 4: and are compared only if one of is chosen from .
And,
Observation 5: Every one of is chosen equal-probably.
This is because the Random Quicksort chooses the pivot uniformly at random.
Observation 4 and 5 together imply:
Remark: Perhaps you feel confused about the above argument. You may ask: "The algorithm chooses pivots for many times during the execution. Why in the above argument, it looks like the pivot is chosen only once?" Good question! Let's see what really happens by looking closely.
For any pair and , initially are all in the same set (obviously!). During the execution of the algorithm, the set which containing are shrinking (due to the pivoting), until one of is chosen, and the set is partitioned into different subsets. We ask for the probability that the chosen one is among . So we really care about "the last" pivoting before is split. Formally, let be the random variable denoting the pivot element. We know that for each , with the same probability, and with an unknown probability (remember that there might be other elements in the same subset with ). The probability we are looking for is actually , which is always , provided that is uniform over . The conditional probability rules out the irrelevant events in a probabilistic argument. |
Summing all up:
is the th Harmonic number. It holds that
- .
Therefore, for an arbitrary input of numbers, the expected number of comparisons taken by RandQSort to sort is .
Distributions of Coin Flips
We introduce several important distributions induced by independent coin flips (independent probabilistic experiments), including: Bernoulli trial, geometric distribution, binomial distribution.
Bernoulli trial (Bernoulli distribution)
Bernoulli trial describes the probability distribution of a single (biased) coin flip. Suppose that we flip a (biased) coin where the probability of HEADS is . Let be the 0-1 random variable which indicates whether the result is HEADS. We say that follows the Bernoulli distribution with parameter . Formally,
- .
Geometric distribution
Suppose we flip the same coin repeatedly until HEADS appears, where each coin flip is independent and follows the Bernoulli distribution with parameter . Let be the random variable denoting the total number of coin flips. Then has the geometric distribution with parameter . Formally, .
For geometric , . This can be verified by directly computing by the definition of expectations. There is also a smarter way of computing , by using indicators and the linearity of expectations. For , let be the 0-1 random variable such that if and only if none of the first coin flipings are HEADS, thus . A key observation is that . Thus, due to the linearity of expectations,
Binomial distribution
Suppose we flip the same (biased) coin for times, where each coin flip is independent and follows the Bernoulli distribution with parameter . Let be the number of HEADS. Then has the binomial distribution with parameters and . Formally, .
A binomial random variable with parameters and is usually denoted by .
As we saw above, by applying the linearity of expectations, it is easy to show that for an .
Balls into Bins
Birthday Problem
There are students in the class. Assume that for each student, his/her birthday is uniformly and independently distributed over the 365 days in a years. We wonder what the probability that no two students share a birthday.
Due to the pigeonhole principle, it is obvious that for , there must be two students with the same birthday. Surprisingly, for any this event occurs with more than 99% probability. This is called the birthday paradox. Despite the name, the birthday paradox is not a real paradox.
We can model this problem as a balls-into-bins problem. different balls (students) are uniformly and independently thrown into 365 bins (days). More generally, let be the number of bins. We ask for the probability of the following event
- : there is no bin with more than one balls (i.e. no two students share birthday).
We first analyze this by counting. There are totally ways of assigning balls to bins. The number of assignments that no two balls share a bin is .
Thus the probability is given by:
Recall that . Then
There is also a more "probabilistic" argument for the above equation. To be rigorous, we need the following theorem, which holds generally and is very useful for computing the AND of many events.
By the definition of conditional probability, . Thus, . This hints us that we can compute the probability of the AND of events by conditional probabilities. Formally, we have the following theorem: Theorem:
- Let be any events. Then
Proof: It holds that . Thus, let and , then
Recursively applying this equation to until there is only left, the theorem is proved.
- Let be any events. Then
Now we are back to the probabilistic analysis of the birthday problem, with a general setting of students and possible birthdays (imagine that we live in a planet where a year has days).
The first student has a birthday (of course!). The probability that the second student has a different birthday is . Given that the first two students have different birthdays, the probability that the third student has a different birthday from the first two is . Continuing this on, assuming that the first students all have different birthdays, the probability that the th student has a different birthday than the first , is given by . So the probability that all students have different birthdays is the product of all these conditional probabilities:
which is the same as what we got by the counting argument.
There are several ways of analyzing this formular. Here is a convenient one: Due to Taylor's expansion, . Then
The quality of this approximation is shown in the Figure.
Therefore, for , the probability that .
Coupon Collector
Suppose that a chocolate company releases different types of coupons. Each box of chocolates contains one coupon with a uniformly random type. Once you have collected all types of coupons, you will get a prize. So how many boxes of chocolates you are expected to buy to win the prize?
The coupon collector problem can be described in the balls-into-bins model as follows. We keep throwing balls one-by-one into bins (coupons), such that each ball is thrown into a bin uniformly and independently at random. Each ball corresponds to a box of chocolate, and each bin corresponds to a type of coupon. Thus, the number of boxes bought to collect coupons is just the number of balls thrown until none of the bins is empty.
Theorem - Let be the number of balls thrown uniformly and independently to bins until no bin is empty. Then , where is the th harmonic number.
Proof. Let be the number of balls thrown while there are exactly nonempty bins, then clearly . When there are exactly nonempty bins, throwing a ball, the probability that the number of nonempty bins increases (i.e. the ball is thrown to an empty bin) is
is the number of balls thrown to make the number of nonempty bins increases from to , i.e. the number of balls thrown until a ball is thrown to a current empty bin. Thus, follows the geometric distribution, such that
For a geometric random variable, .
Applying the linearity of expectations,
where is the th Harmonic number, and . Thus, for the coupon collectors problem, the expected number of coupons required to obtain all types of coupons is .
Only knowing the expectation is not good enough. We would like to know how fast the probability decrease as a random variable deviates from its mean value.
Theorem - Let be the number of balls thrown uniformly and independently to bins until no bin is empty. Then for any .
Proof. For any particular bin , the probability that bin is empty after throwing balls is By the union bound, the probability that there exists an empty bin after throwing balls is
Occupancy Problem
Now we ask about the loads of bins. Assuming that balls are uniformly and independently assigned to bins, for , let be the load of the th bin, i.e. the number of balls in the th bin.
An easy analysis shows that for every bin , the expected load is equal to the average load .
Because there are totally balls, it is always true that .
Therefore, due to the linearity of expectations,
Because for each ball, the bin to which the ball is assigned is uniformly and independently chosen, the distributions of the loads of bins are identical. Thus is the same for each . Combining with the above equation, it holds that for every , . So the average is indeed the average!
Next we analyze the distribution of the maximum load. We show that when , i.e. balls are uniformly and independently thrown into bins, the maximum load is with high probability.
Theorem - Suppose that balls are thrown independently and uniformly at random into bins. For , let be the random variable denoting the number of balls in the th bin. Then
- Suppose that balls are thrown independently and uniformly at random into bins. For , let be the random variable denoting the number of balls in the th bin. Then
Proof. Let be an integer. Take bin 1. For any particular balls, these balls are all thrown to bin 1 with probability , and there are totally distinct sets of balls. Therefore, applying the union bound, According to Stirling's approximation, , thus
Due to the symmetry. All have the same distribution. Apply the union bound again,
When ,
Therefore,
When , Figure 1 illustrates the results of several random experiments, which show that the distribution of the loads of bins becomes more even as the number of balls grows larger than the number of bins.
Formally, it can be proved that for , with high probability, the maximum load is within , which is asymptotically equal to the average load.