View source for 高级算法 (Fall 2016)/''Lovász'' Local Lemma

 [[File:Under_construction.png‎|60px]] <font color=red size =5>Under construction.</font> 

=Lovász Local Lemma=
Assume that <math>A_1,A_2,\ldots,A_n</math> are "bad" events. We are looking at the "rare event" that none of these bad events occurs, formally, the event <math>\bigwedge_{i=1}^n\overline{A_i}</math>. How can we guarantee this rare event occurs with positive probability? The ''Lovász Local Lemma'' provides an answer to this fundamental question by using the information about the dependencies between the bad events.

==The dependency graph ==
The notion of mutual independence between an event and a set of events is formally defined as follows.
{{Theorem|Definition (mutual independence)|
:An event <math>A</math> is said to be '''mutually independent''' of events <math>B_1,B_2,\ldots, B_k</math>, if for any disjoint <math>I^+,I^-\subseteq\{1,2,\ldots,k\}</math>, it holds that
::<math>\Pr\left[A \mid \left(\bigwedge_{i\in I^+}B_i\right) \wedge \left(\bigwedge_{i\in I^-}\overline{B_i}\right)\right]=\Pr[A]</math>.
}}

Given a sequence of events <math>A_1,A_2,\ldots,A_n</math>, we use the '''dependency graph''' to describe the dependencies between these events.

{{Theorem
|Definition (dependency graph)|
:Let <math>A_1,A_2,\ldots,A_n</math> be a sequence of events. A graph <math>D=(V,E)</math> on the set of vertices <math>V=\{1,2,\ldots,n\}</math> is called a '''dependency graph''' for the events <math>A_1,\ldots,A_n</math> if for each <math>i</math>, <math>1\le i\le n</math>, the event <math>A_i</math> is mutually independent of all the events <math>\{A_j\mid (i,j)\not\in E\}</math>.
:Furthermore, for each event <math>A_i</math>:
:* define <math>\Gamma(A_i)=\{A_j\mid (i,j)\in E\}</math> as the '''neighborhood''' of event <math>A_i</math> in the dependency graph; 
:* define <math>\Gamma^+(A_i)=\Gamma(A_i)\cup\{A_i\}</math> as the '''inclusive neighborhood''' of <math>A_i</math>, i.e. the set of events adjacent to <math>A_i</math> in the dependency graph, including <math>A_i</math> itself.
}}

;Example
:Let <math>X_1,X_2,\ldots,X_m</math> be a set of ''mutually independent'' random variables. Each event <math>A_i</math> is a predicate defined on a number of variables among <math>X_1,X_2,\ldots,X_m</math>. Let <math>v(A_i)</math> be the unique smallest set of variables which determine <math>A_i</math>. The dependency graph <math>D=(V,E)</math> is defined by 
:::<math>(i,j)\in E</math> iff <math>v(A_i)\cap v(A_j)\neq \emptyset</math>.

== The local lemma ==
The following lemma, known as the Lovász local lemma, first proved by Erdős and Lovász in 1975, is an extremely powerful tool, as it supplies a way for dealing with rare events.

{{Theorem
|Lovász Local Lemma (symmetric case)|
:Let <math>A_1,A_2,\ldots,A_n</math> be a set of events, and assume that there is a <math>p\in[0,1)</math> such that the followings are satisfied:
:#for all <math>1\le i\le n</math>, <math>\Pr[A_i]\le p</math>;
:#the maximum degree of the dependency graph for the events <math>A_1,A_2,\ldots,A_n</math> is <math>d</math>, and 
:::<math>\mathrm{e}p\cdot (d+1)\le 1</math>.
:Then
::<math>\Pr\left[\bigwedge_{i=1}^n\overline{A_i}\right]>0</math>.
}}

The following is a general asymmetric version of the local lemma. This generalization is due to Spencer.
{{Theorem
|Lovász Local Lemma (general case)|
:Let <math>A_1,A_2,\ldots,A_n</math> be a sequence of events. Suppose there exist real numbers <math>x_1,x_2,\ldots, x_n</math> such that <math>0\le x_i<1</math> and for all <math>1\le i\le n</math>,
::<math>\Pr[A_i]\le x_i\prod_{A_j\in \Gamma(A_i)}(1-x_j)</math>.
:Then 
::<math>\Pr\left[\bigwedge_{i=1}^n\overline{A_i}\right]\ge\prod_{i=1}^n(1-x_i)</math>.
}}

To see that the general LLL implies symmetric LLL, we set <math>x_i=\frac{1}{d+1}</math> for all <math>i=1,2,\ldots,n</math>. Then we have <math>\left(1-\frac{1}{d+1}\right)^d>\frac{1}{\mathrm{e}}</math>.

Assume the condition in the symmetric LLL:
:#for all <math>1\le i\le n</math>, <math>\Pr[A_i]\le p</math>;
:#<math>\mathrm{e}p\cdot(d+1)\le 1</math>;
then it is easy to verify that for all <math>1\le i\le n</math>,
:<math>\Pr[A_i]\le p\le\frac{1}{\mathrm{e}(d+1)}<\frac{1}{d+1}\left(1-\frac{1}{d+1}\right)^d\le x_i\prod_{A_j\in\Gamma(A_i)}(1-x_j)</math>.
Due to the general LLL, we have
:<math>\Pr\left[\bigwedge_{i=1}^n\overline{A_i}\right]\ge\prod_{i=1}^n(1-x_i)=\left(1-\frac{1}{d+1}\right)^n>0</math>.
This proves the symmetric LLL.

Alternatively, by setting <math>x_i=\frac{1}{d}</math> and assuming without loss of generality that the maximum degree of the dependency graph has <math>d\ge 2</math>, we can have another symmetric version of the local lemma.
{{Theorem
|Lovász Local Lemma (symmetric case, alternative form)|
:Let <math>A_1,A_2,\ldots,A_n</math> be a set of events, and assume that there is a <math>p\in[0,1)</math> such that the followings are satisfied:
:#for all <math>1\le i\le n</math>, <math>\Pr[A_i]\le p</math>;
:#the maximum degree of the dependency graph for the events <math>A_1,A_2,\ldots,A_n</math> is <math>d</math>, and 
:::<math>4p d\le 1</math>.
:Then
::<math>\Pr\left[\bigwedge_{i=1}^n\overline{A_i}\right]>0</math>.
}}

The original proof of the Lovász Local Lemma is by induction. See [[高级算法 (Fall 2016)/Nonconstructive Proof of Lovász Local Lemma| this note]] for the original non-constructive proof of Lovász Local Lemma.

=Random Search for <math>k</math>-SAT=
We start by giving the definition of <math>k</math>-CNF and <math>k</math>-SAT.
{{Theorem|Definition (exact-<math>k</math>-CNF)|
:A logic expression <math>\phi</math> defined on <math>n</math> Boolean variables <math>x_1,x_2,\ldots,x_n\in\{\mathrm{true},\mathrm{false}\}</math> is said to be a '''conjunctive normal form (CNF)''' if:
:* <math>\phi</math> can be written as a conjunction(AND) of '''clauses''' as <math>\phi=C_1\wedge C_2\wedge\cdots\wedge C_m</math>;
:* each clause <math>C_i=\ell_{i_1}\vee \ell_{i_2}\vee\cdots\vee\ell_{i_k}</math> is a disjunction(OR) of '''literals''';
:* each literal <math>\ell_j</math> is either a variable <math>x_i</math> or the negation <math>\neg x_i</math> of a variable.
:We call a CNF formula  '''<math>k</math>-CNF''', or more precisely an '''exact-<math>k</math>-CNF''', if every clause consists of ''exact'' <math>k</math> ''distinct'' literals. 
}}
For example:
:<math>
(x_1\vee \neg x_2 \vee \neg x_3)\wedge (\neg x_1\vee \neg x_3\vee x_4)\wedge (x_1\vee x_2\vee x_4)\wedge (x_2\vee x_3\vee \neg x_4)
</math>
is a <math>3</math>-CNF formula by above definition.

;Remark
:The notion of <math>k</math>-CNF defined here is slightly more restrictive than the standard definition of <math>k</math>-CNF, where each clause consists of ''at most'' <math>k</math> variables. See [https://en.wikipedia.org/wiki/Boolean_satisfiability_problem#3-satisfiability here] for a discussion of the subtle differences between these two definitions.

A logic expression <math>\phi</math> is said to be '''satisfiable''' if there is an assignment of values of true or false to the variables <math>\boldsymbol{x}=(x_1,x_2,\ldots,x_n)</math> so that <math>\phi(\boldsymbol{x})</math> is true. For a CNF <math>\phi</math>, this mean that there is an assignment that satisfies all clauses in <math>\phi</math> simultaneously.

The '''<math>k</math>-satisfiability (<math>k</math>-SAT)''' problem is that given as input a <math>k</math>-CNF formula <math>\phi</math> decide whether <math>\phi</math> is satisfiable. 
{{Theorem|<math>k</math>-SAT|
:'''Input:''' a <math>k</math>-CNF formula <math>\phi</math>.
:Determines whether <math>\phi</math> is satisfiable.
}}
It is well known that <math>k</math>-SAT is '''NP-complete''' for any <math>k\ge 3</math>.

== Satisfiability of <math>k</math>-CNF==
As in the Lovasz local lemma, we consider the ''dependencies'' between clauses in a CNF formula. 

We say that a CNF formula <math>\phi</math> has '''maximum degree''' at most <math>d</math> if every clause in <math>\phi</math> shares variables with at most <math>d</math> other clauses in <math>\phi</math>.

By the Lovasz local lemma, we almost immediately have the following theorem for the satisfiability of <math>k</math>-CNF with bounded degree.
{{Theorem|Theorem|
:Let <math>\phi</math> be a <math>k</math>-CNF formula with maximum degree at most <math>d</math>. If <math>d\le 2^{k-2}</math> then <math>\phi</math> is always satisfiable.
}}
{{Proof|
Let <math>X_1,X_2,\ldots,X_n</math> be Boolean random variables sampled uniformly and independently from <math>\{\text{true},\text{false}\}</math>. We are going to show that <math>\phi</math> is satisfied by this random assignment with positive probability. Due to the probabilistic method, this will prove the existence of a satisfying assignment for <math>\phi</math>.

Suppose there are <math>m</math> clauses <math>C_1,C_2,\ldots,C_m</math> in <math>\phi</math>. Let <math>A_i</math> denote the bad event that <math>C_i</math> is not satisfied by the random assignment <math>X_1,X_2,\ldots,X_n</math>. Clearly, each <math>A_i</math> is dependent with at most <math>d</math> other <math>A_j</math>'s, which means the maximum degree of the dependency graph for <math>A_1, A_2, \ldots, A_m</math> is at most <math>d</math>.

Recall that in a <math>k</math>-CNF <math>\phi</math>, every clause <math>C_i</math> consists of precisely <math>k</math> variable, and <math>C_i</math> is violated by only one assignment among all <math>2^k</math> assignments of the <math>k</math> variables in <math>C_i</math>. Therefore, the probability of <math>C_i</math> being violated is <math>p=\Pr[A_i]=2^{-k}</math>.

If <math>d\le 2^{k-2}</math>, that is, <math>4pd\le 1</math>, then due to Lovasz local lemma (symmetric case, alternative form), it holds that
:<math>\Pr\left[\bigwedge_{i=1}^m\overline{A_i}\right]>0</math>.
The existence of satisfying assignment follows by the probabilistic method.
}}

== Moser's recursive fix algorithm ==
The above theorem basically says that for a CNF if every individual clause is easy to satisfy and is dependent with few other clauses then the CNF should be always satisfiable. However, the theorem only states the existence of a satisfying solution, but does not gives a way to find such solution. 

In 2009, Moser gave a very simple randomized algorithm which efficiently finds a satisfying assignment with high probability under the condition <math>d\le 2^{k-5}</math>.

We need the following notations. Given as input a CNF formula <math>\phi</math>: 
* Let <math>\mathcal{X}=\{x_1,x_2,\ldots,x_n\}</math> be the set of Boolean variables on which <math>\phi</math> is defined.
* For each clause <math>C</math> in <math>\phi</math>, we denote by <math>\mathsf{vbl}(C)\subseteq\mathcal{X}</math> the set of variables on which <math>C</math> is defined.
* We also abuse the notation and denote by <math>\Gamma(C)=\{\text{clause }D\text{ in }\phi\mid D\neq C, \mathsf{vbl}(C)\cap\mathsf{vbl}(D)\neq\phi\}</math> the '''neighborhood''' of <math>C</math>, i.e. the set of ''other'' clauses in <math>\phi</math> that shares variables with <math>C</math>, and <math>\Gamma^+(C)=\Gamma(C)\cup\{C\}</math> the '''inclusive neighborhood''' of <math>C</math>, i.e. the set of all clauses, including <math>C</math> itself, that share variables with <math>C</math>.

The algorithm consists of two functions: the main function ''Solve''() and a recursive sub-routine ''Fix''().
{{Theorem
|Solve(CNF <math>\phi</math>)|
:Pick values of <math>x_1,x_2\ldots,x_n</math> uniformly and independently at random;
:while there is an ''unsatisfied'' clause <math>C</math> in <math>\phi</math>
:: '''Fix'''(<math>C</math>);
}}

The sub-routine '''Fix()''' is a recursive procedure:
{{Theorem
|Fix(Clause <math>C</math>)|
:Replace the values of variables in <math>\mathsf{vbl}(C)</math> with uniform and independent random values;
:while there is ''unsatisfied'' clause <math>D\in\Gamma^+(C)</math>
:: '''Fix'''(<math>D</math>);
}}

It is quite amazing to see that this simple algorithm works very well.

{{Theorem|Theorem|
:Let <math>\phi</math> be a <math>k</math>-CNF formula with maximum degree at most <math>d</math>. 
:There is a universal constant <math>c>0</math>, such that if <math>d< 2^{k-c}</math> then the algorithm ''Solve''(<math>\phi</math>) finds a satisfying assignment for <math>\phi</math> in time <math>O(n+km\log m)</math> with high probability.
}}

== Analysis of Moser's algorithm by entropy compression==

= Constructive Proof of General ''LLL'' =
* <math>\mathcal{X}</math> is a set of mutually independent random variables.
* <math>\mathcal{A}</math> is a set of events defined on variables in <math>\mathcal{X}</math>, where each <math>A\in\mathcal{A}</math> is a predicate defined on a subset of random variables in <math>\mathcal{X}</math>.
* For each <math>A\in\mathcal{A}</math>, denote by <math>\mathsf{vbl}(A)</math> the set of variables on which <math>A</math> is defined.
* For each <math>A\in\mathcal{A}</math>, the '''neighborhood''' of <math>A</math>, denoted by <math>\Gamma(A)</math>, is defined as
::<math>\Gamma(A)=\{B\in\mathcal{A}\mid B\neq A, \mathsf{vbl}(A)\cap\mathsf{vbl}(B)\neq\emptyset\}</math>;
*:and let <math>\Gamma^+(A)=\Gamma(A)\cup \{A\}</math> be the '''inclusive neighborhood''' of <math>A</math>.

We still interpret the events in <math>\mathcal{A}</math> as a series of bad events, and we want to make sure it is possible to avoid the occurrences of all bad events in <math>\mathcal{A}</math> simultaneously. Furthermore, we want to give an algorithm which can efficiently find an evaluation of the random variables in <math>\mathcal{X}</math> to make none of the bad events in <math>\mathcal{A}</math> occur.

== The Moser-Tardos random solver ==
We make the following assumptions:
* It is efficient to draw independent samples for every random variable <math>X\in\mathcal{X}</math> according to its distribution.
* It is efficient to evaluate whether <math>A</math> occurs on an evaluation of random variables in <math>\mathsf{vbl}(A)</math>.

{{Theorem
|Moser-Tardos Algorithm|
:Sample all <math>X\in\mathcal{X}</math> independently;
:while there is a bad event <math>A\in\mathcal{A}</math> that occurs
::resample all <math>X\in\mathsf{vbl}(A)</math>;
}}

{{Theorem|Theorem (Moser-Tardos 2010)|
:If there is an <math>\alpha:\mathcal{A}\to[0,1)</math> such that for every <math>A\in\mathcal{A}</math>
::<math>\Pr[A]\le \alpha(A)\prod_{B\in\Gamma(A)}(1-\alpha(B))</math>,
:then the Moser-Tardos algorithm returns an evaluation of random variables in <math>\mathcal{X}</math> satisfying <math>\bigwedge_{A\in\mathcal{A}}\overline{A}</math> using at most <math>\sum_{A\in\mathcal{A}}\frac{\alpha(A)}{1-\alpha(A)}</math> resamples in expectation.
}}

== Analysis of the  Moser-Tardos algorithm by the witness tree==