高级算法 (Fall 2016)/''Lovász'' Local Lemma

 Under construction. 

Lovász Local Lemma

Assume that [math]\displaystyle{ A_1,A_2,\ldots,A_n }[/math] are "bad" events. We are looking at the "rare event" that none of these bad events occurs, formally, the event [math]\displaystyle{ \bigwedge_{i=1}^n\overline{A_i} }[/math]. How can we guarantee this rare event occurs with positive probability? The Lovász Local Lemma provides an answer to this fundamental question by using the information about the dependencies between the bad events.

The dependency graph

The notion of mutual independence between an event and a set of events is formally defined as follows.

Definition (mutual independence)

An event [math]\displaystyle{ A }[/math] is said to be mutually independent of events [math]\displaystyle{ B_1,B_2,\ldots, B_k }[/math], if for any disjoint [math]\displaystyle{ I^+,I^-\subseteq\{1,2,\ldots,k\} }[/math], it holds that [math]\displaystyle{ \Pr\left[A \mid \left(\bigwedge_{i\in I^+}B_i\right) \wedge \left(\bigwedge_{i\in I^-}\overline{B_i}\right)\right]=\Pr[A] }[/math].

Given a sequence of events [math]\displaystyle{ A_1,A_2,\ldots,A_n }[/math], we use the dependency graph to describe the dependencies between these events.

Definition (dependency graph)

Let [math]\displaystyle{ A_1,A_2,\ldots,A_n }[/math] be a sequence of events. A graph [math]\displaystyle{ D=(V,E) }[/math] on the set of vertices [math]\displaystyle{ V=\{1,2,\ldots,n\} }[/math] is called a dependency graph for the events [math]\displaystyle{ A_1,\ldots,A_n }[/math] if for each [math]\displaystyle{ i }[/math], [math]\displaystyle{ 1\le i\le n }[/math], the event [math]\displaystyle{ A_i }[/math] is mutually independent of all the events [math]\displaystyle{ \{A_j\mid (i,j)\not\in E\} }[/math]. Furthermore, for each event [math]\displaystyle{ A_i }[/math]: define [math]\displaystyle{ \Gamma(A_i)=\{A_j\mid (i,j)\in E\} }[/math] as the neighborhood of event [math]\displaystyle{ A_i }[/math] in the dependency graph; define [math]\displaystyle{ \Gamma^+(A_i)=\Gamma(A_i)\cup\{A_i\} }[/math] as the inclusive neighborhood of [math]\displaystyle{ A_i }[/math], i.e. the set of events adjacent to [math]\displaystyle{ A_i }[/math] in the dependency graph, including [math]\displaystyle{ A_i }[/math] itself.

Example

Let [math]\displaystyle{ X_1,X_2,\ldots,X_m }[/math] be a set of mutually independent random variables. Each event [math]\displaystyle{ A_i }[/math] is a predicate defined on a number of variables among [math]\displaystyle{ X_1,X_2,\ldots,X_m }[/math]. Let [math]\displaystyle{ v(A_i) }[/math] be the unique smallest set of variables which determine [math]\displaystyle{ A_i }[/math]. The dependency graph [math]\displaystyle{ D=(V,E) }[/math] is defined by

[math]\displaystyle{ (i,j)\in E }[/math] iff [math]\displaystyle{ v(A_i)\cap v(A_j)\neq \emptyset }[/math].

The local lemma

The following lemma, known as the Lovász local lemma, first proved by Erdős and Lovász in 1975, is an extremely powerful tool, as it supplies a way for dealing with rare events.

Lovász Local Lemma (symmetric case)

Let [math]\displaystyle{ A_1,A_2,\ldots,A_n }[/math] be a set of events, and assume that there is a [math]\displaystyle{ p\in[0,1) }[/math] such that the followings are satisfied: for all [math]\displaystyle{ 1\le i\le n }[/math], [math]\displaystyle{ \Pr[A_i]\le p }[/math]; the maximum degree of the dependency graph for the events [math]\displaystyle{ A_1,A_2,\ldots,A_n }[/math] is [math]\displaystyle{ d }[/math], and [math]\displaystyle{ \mathrm{e}p\cdot (d+1)\le 1 }[/math]. Then [math]\displaystyle{ \Pr\left[\bigwedge_{i=1}^n\overline{A_i}\right]\gt 0 }[/math].

The following is a general asymmetric version of the local lemma. This generalization is due to Spencer.

Lovász Local Lemma (general case)

Let [math]\displaystyle{ A_1,A_2,\ldots,A_n }[/math] be a sequence of events. Suppose there exist real numbers [math]\displaystyle{ x_1,x_2,\ldots, x_n }[/math] such that [math]\displaystyle{ 0\le x_i\lt 1 }[/math] and for all [math]\displaystyle{ 1\le i\le n }[/math], [math]\displaystyle{ \Pr[A_i]\le x_i\prod_{A_j\in \Gamma(A_i)}(1-x_j) }[/math]. Then [math]\displaystyle{ \Pr\left[\bigwedge_{i=1}^n\overline{A_i}\right]\ge\prod_{i=1}^n(1-x_i) }[/math].

To see that the general LLL implies symmetric LLL, we set [math]\displaystyle{ x_i=\frac{1}{d+1} }[/math] for all [math]\displaystyle{ i=1,2,\ldots,n }[/math]. Then we have [math]\displaystyle{ \left(1-\frac{1}{d+1}\right)^d\gt \frac{1}{\mathrm{e}} }[/math].

Assume the condition in the symmetric LLL:

1. for all [math]\displaystyle{ 1\le i\le n }[/math], [math]\displaystyle{ \Pr[A_i]\le p }[/math];
2. [math]\displaystyle{ \mathrm{e}p\cdot(d+1)\le 1 }[/math];

then it is easy to verify that for all [math]\displaystyle{ 1\le i\le n }[/math],

[math]\displaystyle{ \Pr[A_i]\le p\le\frac{1}{\mathrm{e}(d+1)}\lt \frac{1}{d+1}\left(1-\frac{1}{d+1}\right)^d\le x_i\prod_{A_j\in\Gamma(A_i)}(1-x_j) }[/math].

Due to the general LLL, we have

[math]\displaystyle{ \Pr\left[\bigwedge_{i=1}^n\overline{A_i}\right]\ge\prod_{i=1}^n(1-x_i)=\left(1-\frac{1}{d+1}\right)^n\gt 0 }[/math].

This proves the symmetric LLL.

Alternatively, by setting [math]\displaystyle{ x_i=\frac{1}{d} }[/math] and assuming without loss of generality that the maximum degree of the dependency graph has [math]\displaystyle{ d\ge 2 }[/math], we can have another symmetric version of the local lemma.

Lovász Local Lemma (symmetric case, alternative form)

Let [math]\displaystyle{ A_1,A_2,\ldots,A_n }[/math] be a set of events, and assume that there is a [math]\displaystyle{ p\in[0,1) }[/math] such that the followings are satisfied: for all [math]\displaystyle{ 1\le i\le n }[/math], [math]\displaystyle{ \Pr[A_i]\le p }[/math]; the maximum degree of the dependency graph for the events [math]\displaystyle{ A_1,A_2,\ldots,A_n }[/math] is [math]\displaystyle{ d }[/math], and [math]\displaystyle{ 4p d\le 1 }[/math]. Then [math]\displaystyle{ \Pr\left[\bigwedge_{i=1}^n\overline{A_i}\right]\gt 0 }[/math].

The original proof of the Lovász Local Lemma is by induction. See this note for the original non-constructive proof of Lovász Local Lemma.

Random Search for [math]\displaystyle{ k }[/math]-SAT

We start by giving the definition of [math]\displaystyle{ k }[/math]-CNF and [math]\displaystyle{ k }[/math]-SAT.

Definition (exact-[math]\displaystyle{ k }[/math]-CNF)

A logic expression [math]\displaystyle{ \phi }[/math] defined on [math]\displaystyle{ n }[/math] Boolean variables [math]\displaystyle{ x_1,x_2,\ldots,x_n\in\{\mathrm{true},\mathrm{false}\} }[/math] is said to be a conjunctive normal form (CNF) if: [math]\displaystyle{ \phi }[/math] can be written as a conjunction(AND) of clauses as [math]\displaystyle{ \phi=C_1\wedge C_2\wedge\cdots\wedge C_m }[/math]; each clause [math]\displaystyle{ C_i=\ell_{i_1}\vee \ell_{i_2}\vee\cdots\vee\ell_{i_k} }[/math] is a disjunction(OR) of literals; each literal [math]\displaystyle{ \ell_j }[/math] is either a variable [math]\displaystyle{ x_i }[/math] or the negation [math]\displaystyle{ \neg x_i }[/math] of a variable. We call a CNF formula [math]\displaystyle{ k }[/math]-CNF, or more precisely an exact-[math]\displaystyle{ k }[/math]-CNF, if every clause consists of exact [math]\displaystyle{ k }[/math] distinct literals.

For example:

[math]\displaystyle{ (x_1\vee \neg x_2 \vee \neg x_3)\wedge (\neg x_1\vee \neg x_3\vee x_4)\wedge (x_1\vee x_2\vee x_4)\wedge (x_2\vee x_3\vee \neg x_4) }[/math]

is a [math]\displaystyle{ 3 }[/math]-CNF formula by above definition.

Remark

The notion of [math]\displaystyle{ k }[/math]-CNF defined here is slightly more restrictive than the standard definition of [math]\displaystyle{ k }[/math]-CNF, where each clause consists of at most [math]\displaystyle{ k }[/math] variables. See here for a discussion of the subtle differences between these two definitions.

A logic expression [math]\displaystyle{ \phi }[/math] is said to be satisfiable if there is an assignment of values of true or false to the variables [math]\displaystyle{ \boldsymbol{x}=(x_1,x_2,\ldots,x_n) }[/math] so that [math]\displaystyle{ \phi(\boldsymbol{x}) }[/math] is true. For a CNF [math]\displaystyle{ \phi }[/math], this mean that there is an assignment that satisfies all clauses in [math]\displaystyle{ \phi }[/math] simultaneously.

The [math]\displaystyle{ k }[/math]-satisfiability ([math]\displaystyle{ k }[/math]-SAT) problem is that given as input a [math]\displaystyle{ k }[/math]-CNF formula [math]\displaystyle{ \phi }[/math] decide whether [math]\displaystyle{ \phi }[/math] is satisfiable.

[math]\displaystyle{ k }[/math]-SAT

Input: a [math]\displaystyle{ k }[/math]-CNF formula [math]\displaystyle{ \phi }[/math]. Determines whether [math]\displaystyle{ \phi }[/math] is satisfiable.

It is well known that [math]\displaystyle{ k }[/math]-SAT is NP-complete for any [math]\displaystyle{ k\ge 3 }[/math].

Satisfiability of [math]\displaystyle{ k }[/math]-CNF

As in the Lovasz local lemma, we consider the dependencies between clauses in a CNF formula.

We say that a CNF formula [math]\displaystyle{ \phi }[/math] has maximum degree at most [math]\displaystyle{ d }[/math] if every clause in [math]\displaystyle{ \phi }[/math] shares variables with at most [math]\displaystyle{ d }[/math] other clauses in [math]\displaystyle{ \phi }[/math].

By the Lovasz local lemma, we almost immediately have the following theorem for the satisfiability of [math]\displaystyle{ k }[/math]-CNF with bounded degree.

Theorem

Let [math]\displaystyle{ \phi }[/math] be a [math]\displaystyle{ k }[/math]-CNF formula with maximum degree at most [math]\displaystyle{ d }[/math]. If [math]\displaystyle{ d\le 2^{k-2} }[/math] then [math]\displaystyle{ \phi }[/math] is always satisfiable.

Proof. Let [math]\displaystyle{ X_1,X_2,\ldots,X_n }[/math] be Boolean random variables sampled uniformly and independently from [math]\displaystyle{ \{\text{true},\text{false}\} }[/math]. We are going to show that [math]\displaystyle{ \phi }[/math] is satisfied by this random assignment with positive probability. Due to the probabilistic method, this will prove the existence of a satisfying assignment for [math]\displaystyle{ \phi }[/math]. Suppose there are [math]\displaystyle{ m }[/math] clauses [math]\displaystyle{ C_1,C_2,\ldots,C_m }[/math] in [math]\displaystyle{ \phi }[/math]. Let [math]\displaystyle{ A_i }[/math] denote the bad event that [math]\displaystyle{ C_i }[/math] is not satisfied by the random assignment [math]\displaystyle{ X_1,X_2,\ldots,X_n }[/math]. Clearly, each [math]\displaystyle{ A_i }[/math] is dependent with at most [math]\displaystyle{ d }[/math] other [math]\displaystyle{ A_j }[/math]'s, which means the maximum degree of the dependency graph for [math]\displaystyle{ A_1, A_2, \ldots, A_m }[/math] is at most [math]\displaystyle{ d }[/math]. Recall that in a [math]\displaystyle{ k }[/math]-CNF [math]\displaystyle{ \phi }[/math], every clause [math]\displaystyle{ C_i }[/math] consists of precisely [math]\displaystyle{ k }[/math] variable, and [math]\displaystyle{ C_i }[/math] is violated by only one assignment among all [math]\displaystyle{ 2^k }[/math] assignments of the [math]\displaystyle{ k }[/math] variables in [math]\displaystyle{ C_i }[/math]. Therefore, the probability of [math]\displaystyle{ C_i }[/math] being violated is [math]\displaystyle{ p=\Pr[A_i]=2^{-k} }[/math]. If [math]\displaystyle{ d\le 2^{k-2} }[/math], that is, [math]\displaystyle{ 4pd\le 1 }[/math], then due to Lovasz local lemma (symmetric case, alternative form), it holds that [math]\displaystyle{ \Pr\left[\bigwedge_{i=1}^m\overline{A_i}\right]\gt 0 }[/math]. The existence of satisfying assignment follows by the probabilistic method. [math]\displaystyle{ \square }[/math]

Moser's recursive fix algorithm

The above theorem basically says that for a CNF if every individual clause is easy to satisfy and is dependent with few other clauses then the CNF should be always satisfiable. However, the theorem only states the existence of a satisfying solution, but does not gives a way to find such solution.

In 2009, Moser gave a very simple randomized algorithm which efficiently finds a satisfying assignment with high probability under the condition [math]\displaystyle{ d\le 2^{k-5} }[/math].

We need the following notations. Given as input a CNF formula [math]\displaystyle{ \phi }[/math]:

• let [math]\displaystyle{ \mathcal{X}=\{x_1,x_2,\ldots,x_n\} }[/math] be the Boolean variables on which [math]\displaystyle{ \phi }[/math] is defined, and [math]\displaystyle{ \mathcal{C} }[/math] the set of clauses in [math]\displaystyle{ \phi }[/math];
• for each clause [math]\displaystyle{ C\in \mathcal{C} }[/math], we denote by [math]\displaystyle{ \mathsf{vbl}(C)\subseteq\mathcal{X} }[/math] the set of variables on which [math]\displaystyle{ C }[/math] is defined;
• we also abuse the notation and denote by [math]\displaystyle{ \Gamma(C)=\{D\in\mathcal{C}\mid D\neq C, \mathsf{vbl}(C)\cap\mathsf{vol}(D)\neq\phi\} }[/math] the neighborhood of [math]\displaystyle{ C }[/math], i.e. the set of other clauses in [math]\displaystyle{ \phi }[/math] that shares variables with [math]\displaystyle{ C }[/math], and [math]\displaystyle{ \Gamma^+(C)=\Gamma(C)\cup\{C\} }[/math] the inclusive neighborhood of [math]\displaystyle{ C }[/math], i.e. the set of all clauses, including [math]\displaystyle{ C }[/math] itself, that share variables with [math]\displaystyle{ C }[/math].

The algorithm consists of two components: the main function Solve() and a sub-routine Fix().

Solve(CNF [math]\displaystyle{ \phi }[/math])

Pick values of [math]\displaystyle{ x_1,x_2\ldots,x_n }[/math] uniformly and independently at random; While there is an unsatisfied clause [math]\displaystyle{ C }[/math] in [math]\displaystyle{ \phi }[/math] Fix([math]\displaystyle{ C }[/math]);

The sub-routine Fix() is a recursive procedure:

Fix(Clause [math]\displaystyle{ C }[/math])

Replace the values of variables in [math]\displaystyle{ \mathsf{vbl}(C) }[/math] with new uniform and independent random values; While there is unsatisfied clause [math]\displaystyle{ D\in\Gamma^+(C) }[/math] Fix([math]\displaystyle{ D }[/math]);

It is quite amazing to see that this simple algorithm works very well.

Theorem

Let [math]\displaystyle{ \phi }[/math] be a [math]\displaystyle{ k }[/math]-CNF formula with maximum degree at most [math]\displaystyle{ d }[/math]. There is a universal constant [math]\displaystyle{ c\gt 0 }[/math], such that if [math]\displaystyle{ d\lt 2^{k-c} }[/math] then the algorithm Solve([math]\displaystyle{ \phi }[/math]) finds a satisfying assignment for [math]\displaystyle{ \phi }[/math] in time [math]\displaystyle{ O(n+km\log m) }[/math] with high probability.

The analysis is based on a technique called entropy compression. This is a very clever idea and may be very different from what you might have seen so far about algorithm analysis. We first give a high-level description of this idea:

• We use [math]\displaystyle{ \mathsf{Alg}(r, \phi) }[/math] to abstractly denote an algorithm [math]\displaystyle{ \mathsf{Alg} }[/math] running on an input [math]\displaystyle{ \phi }[/math] with random bits [math]\displaystyle{ r\in\{0,1\}^* }[/math]. For an algorithm with no access to the random bits [math]\displaystyle{ r }[/math], once the input [math]\displaystyle{ \phi }[/math] is fixed, the behavior of the algorithm as well as its output is deterministic. But for randomized algorithms, the behavior of [math]\displaystyle{ \mathsf{Alg}(r, \phi) }[/math] is a random variable even when the input [math]\displaystyle{ \phi }[/math] is fixed.
• Fix an arbitrary (worst-case) input [math]\displaystyle{ \phi }[/math]. We try to construct a succinct representation [math]\displaystyle{ c }[/math] of the behavior of [math]\displaystyle{ \mathsf{Alg}(r, \phi) }[/math] in such a manner that the random bits [math]\displaystyle{ r }[/math] can always be fully recovered from this succinct representation [math]\displaystyle{ c }[/math]. In other words, [math]\displaystyle{ \mathsf{Alg}(r, \phi) }[/math] gives an encoding (a 1-1 mapping) of the random bits [math]\displaystyle{ r }[/math] to a succinct representation [math]\displaystyle{ c }[/math].
• It is a fundamental law that random bits cannot be compressed significantly by any encoding. Therefore if a longer running time of [math]\displaystyle{ \mathsf{Alg}(r, \phi) }[/math] would imply that the random bits [math]\displaystyle{ r }[/math] can be encoded to a succinct representation [math]\displaystyle{ c }[/math] which is much shorter than [math]\displaystyle{ r }[/math], then we prove the running time of the algorithm [math]\displaystyle{ \mathsf{Alg}(r, \phi) }[/math] cannot be too long.

• A natural way to reach this last contradiction is to have the following situation: As the running time of [math]\displaystyle{ \mathsf{Alg}(r, \phi) }[/math] grows, naturally both lengths of random bits [math]\displaystyle{ r }[/math] and the succinct representation [math]\displaystyle{ c }[/math] of the behavior of [math]\displaystyle{ \mathsf{Alg}(r, \phi) }[/math] grow. So if the former grows much faster than the latter as the running time grows, then a large running time may cause the length of [math]\displaystyle{ r }[/math] significantly greater than the length of [math]\displaystyle{ c }[/math].

---

We now proceed to the analysis of [math]\displaystyle{ \text{Solve}(\phi) }[/math] and prove the above theorem.

From now on we assume that the input instance [math]\displaystyle{ \phi }[/math] is an arbitrarily fixed exact-[math]\displaystyle{ k }[/math]-CNF formula with maximum degree [math]\displaystyle{ d }[/math]:

• Let [math]\displaystyle{ T }[/math] denote the total number of time the function [math]\displaystyle{ \text{Fix}() }[/math] is being called (including both the calls in [math]\displaystyle{ \text{Solve}(\phi) }[/math] and the recursive calls).
• Let [math]\displaystyle{ t=\min\{T,2^m\} }[/math].

Note that [math]\displaystyle{ T }[/math] and [math]\displaystyle{ t }[/math] are random variables which depend solely on the random bits used by the algorithm (after the input [math]\displaystyle{ \phi }[/math] being fixed). We are going to show that [math]\displaystyle{ t=O(m\log m) }[/math] with high probability, which means [math]\displaystyle{ T=O(m\log m) }[/math] with high probability.

The reason we consider [math]\displaystyle{ t=\min\{T,2^m\} }[/math] is that we are not sure whether [math]\displaystyle{ T }[/math] is finite or infinite, so we "truncate" [math]\displaystyle{ T }[/math] if it becomes too large. This truncation threshold is quite arbitrary as long as it is significantly greater than [math]\displaystyle{ m\log m }[/math].

We start by making the following simple observations:

• A clause [math]\displaystyle{ C }[/math] in [math]\displaystyle{ \phi }[/math] will be satisfied after Fix([math]\displaystyle{ C }[/math]) returned and will remain as being satisfied afterwards.
• At the moment a Fix([math]\displaystyle{ C }[/math]) being called, the clause [math]\displaystyle{ C }[/math] must be unsatisfied.

Proposition

There are at most [math]\displaystyle{ m }[/math] top-level callings to Fix(), where [math]\displaystyle{ m }[/math] is the total number of clauses in [math]\displaystyle{ \phi }[/math]. If a Fix([math]\displaystyle{ C }[/math]) is being called, the values of variables in [math]\displaystyle{ C }[/math] are uniquely determined.