随机算法 (Fall 2011)/Limited independence

From EtoneWiki
Jump to: navigation, search

k-wise independence

Recall the definition of independence between events:

Definition (Independent events)
Events are mutually independent if, for any subset ,

Similarly, we can define independence between random variables:

Definition (Independent variables)
Random variables are mutually independent if, for any subset and any values , where ,

Mutual independence is an ideal condition of independence. The limited notion of independence is usually defined by the k-wise independence.

Definition (k-wise Independenc)
1. Events are k-wise independent if, for any subset with
2. Random variables are k-wise independent if, for any subset with and any values , where ,

A very common case is pairwise independence, i.e. the 2-wise independence.

Definition (pairwise Independent random variables)
Random variables are pairwise independent if, for any where and any values

Note that the definition of k-wise independence is hereditary:

  • If are k-wise independent, then they are also -wise independent for any .
  • If are NOT k-wise independent, then they cannot be -wise independent for any .

Construction via XOR

Suppose we have mutually independent and uniform random bits . We are going to extract pairwise independent bits from these mutually independent bits.

Enumerate all the nonempty subsets of in some order. Let be the th subset. Let

where is the exclusive-or, whose truth table is as follows.

0 0 0
0 1 1
1 0 1
1 1 0

There are such , because there are nonempty subsets of . An equivalent definition of is

.

Sometimes, is called the parity of the bits in .

We claim that are pairwise independent and uniform.

Theorem
For any and any ,
For any that and any ,

The proof is left for your exercise.

Therefore, we extract exponentially many pairwise independent uniform random bits from a sequence of mutually independent uniform random bits.

Note that are not 3-wise independent. For example, consider the subsets and the corresponding random bits . Any two of would decide the value of the third one.

Construction via modulo a prime

We now consider constructing pairwise independent random variables ranging over for some prime . Unlike the above construction, now we only need two independent random sources , which are uniformly and independently distributed over .

Let be defined as:

Theorem
The random variables are pairwise independent uniform random variables over .
Proof.
We first show that are uniform. That is, we will show that for any ,

Due to the law of total probability,

For prime , for any , there is exact one value in of satisfying . Thus, and the above probability is .

We then show that are pairwise independent, i.e. we will show that for any that and any ,

The event is equivalent to that

Due to the Chinese remainder theorem, there exists a unique solution of and in to the above linear congruential system. Thus the probability of the event is .

Tools for limited independence

Let be random variables. The variance of their sum is

If are pairwise independent, then for any since the covariance of a pair of independent random variables is 0. This gives us the following theorem of linearity of variance for pairwise independent random variables.

Theorem
For pairwise independent random variables ,

The theorem relies on that the covariances of pairwise independent random variables are 0, which in turn is actually a consequence of a more general theorem.

Theorem (-wise independence fools -degree polynomials)
Let be mutually independent random variables and be -wise independent random variables, with that the marginal distribution of is identical to the marginal distribution of , , that is, for any , .
Let be a multivariate polynomial of degree at most . Then

This phenomenon is sometimes called that the -degree polynomials are fooled by -wise independence. In other words, a -degree polynomial behaves the same on the -wise independent random variables as on the mutual independent random variables.

This theorem is implied by the following lemma.

Lemma
Let be mutually independent random variables. Then

The lemma can be proved by directly compute the expectation. We omit the detailed proof.

By the linearity of expectation, the expectation of a polynomial is reduced to the sum of the expectations of terms. For a k-degree polynomial, each term has at most variables. Due to the above lemma, with k-wise independence, the expectation of each term behaves exactly the same as mutual independence.

Since the th moment is the expectation of a k-degree polynomial of random variables, the tools based on the th moment can be safely used for the k-wise independence. In particular, Chebyshev's inequality for pairwise independent random variables:

Chebyshev's inequality
Let , where are pairwise independent Poisson trials. Let .
Then

Application: Derandomizing MAX-CUT

Let be an undirected graph, and be a vertex set. The cut defined by is .

Given as input an undirected graph , find the whose cut value is maximized. This problem is called the maximum cut (MAX-CUT) problem, which is NP-hard. The decision version of one of the weighted version of the problem is one of the Karp's 21 NP-complete problems. The problem has a -approximation algorithm by rounding a semidefinite programming. Assuming that the unique game conjecture (UGC), there does not exist a poly-time algorithm with better approximation ratio unless .

Here we give a very simple -approximation algorithm. The "algorithm" has a one-line description:

  • Put each into independently with probability 1/2.

We then analyze the approximation ratio of this algorithm.

For each , let indicate whether , that is

For each edge , let indicate whether contribute to the cut , i.e. whether or , that is

Then . Due to the linearity of expectation,

.

The maximum cut of a graph is at most . Thus, the algorithm returns in expectation a cut with size at least half of the maximum cut.

We then show how to dereandomize this algorithm by pairwise independent bits.

Suppose that and enumerate the vertices by in an arbitrary order. Let . Sample bits uniformly and independently at random. Enumerate all nonempty subsets of by . For each vertex , let . The MAX-CUT algorithm uses these bits to construct the solution :

  • For , put into if .

We have shown that , , are uniform and pairwise independent. Thus we still have that . The above analysis still holds, so that the algorithm returns in expectation a cut with size at least .

Finally, we notice that there are only total random bits in the new algorithm. We can enumerate all possible strings of bits, run the above algorithm with the bit strings as the "random sources", and output the maximum cut returned. There must exist a bit string on which the algorithm returns a cut of size (why?). This gives us a deterministic polynomial time (actually time) -approximation algorithm.

Application: Two-point sampling

Consider a Monte Carlo randomized algorithm with one-sided error for a decision problem . We formulate the algorithm as a deterministic algorithm that takes as input and a uniform random number where is a prime, such that for any input :

  • If , then , where the probability is taken over the random choice of .
  • If , then for any .

We call the random source for the algorithm.

For the that , we call the that makes a witness for . For a positive , at least half of are witnesses. The random source has polynomial number of bits, which means that is exponentially large, thus it is infeasible to find the witness for an input by exhaustive search. Deterministic overcomes this by having sophisticated deterministic rules for efficiently searching for a witness. Randomization, on the other hard, reduce this to a bit of luck, by randomly choosing an and winning with a probability of 1/2.

We can boost the accuracy (equivalently, reduce the error) of any Monte Carlo randomized algorithm with one-sided error by running the algorithm for a number of times.

Suppose that we sample values uniformly and independently from , and run the following scheme:

return ;

That is, return 1 if any instance of . For any that , due to the independence of , the probability that returns an incorrect result is at most . On the other hand, never makes mistakes for the that since has no false positives. Thus, the error of the Monte Carlo algorithm is reduced to .

Sampling mutually independent random numbers from can be quite expensive since it requires random bits. Suppose that we can only afford random bits. In particular, we sample two independent uniform random number and from . If we use and directly bu running two independent instances and , we only get an error upper bound of 1/4.

The following scheme reduces the error significantly with the same number of random bits:

Algorithm

Choose two independent uniform random number and from . Construct random number by:

Run .

Due to the discussion in the last section, we know that for , are pairwise independent and uniform over . Let and . Due to the uniformity of and our definition of , for any that , it holds that

By the linearity of expectations,

Since is Bernoulli trial with a probability of success at least . We can estimate the variance of each as follows.

Applying Chebyshev's inequality, we have that for any that ,

The error is reduced to with only two random numbers. This scheme works as long as .