View source for 高级算法 (Fall 2023)/Problem Set 1

*'''作业目前正在更新中，不是最终版'''

*每道题目的解答都要有完整的解题过程，中英文不限。

*我们推荐大家使用LaTeX, markdown等对作业进行排版。

== Problem 1 (Min-cut/Max-cut) ==

* ['''Counting <math>\alpha</math>-approximate min-cut'''] For any '''<math>\alpha \ge 1</math>''', a cut is called an '''<math>\alpha</math>'''-approximate min-cut in a multigraph '''<math>G</math>''' if the number of edges in it is at most '''<math>\alpha</math>''' times that of the min-cut.  Prove that the number of '''<math>\alpha</math>'''-approximate min-cuts in a multigraph '''<math>G</math>''' is at most '''<math>n^{2\alpha} / 2</math>'''.  Hint: Run Karger's algorithm until it has '''<math>\lceil 2\alpha \rceil</math>''' supernodes. What is the chance that a particular '''<math>\alpha</math>'''-approximate min-cut is still available? How many possible cuts does this collapsed graph have?
* ['''Weighted min-cut problem'''] Modify the Karger's Contraction algorithm so that it works for the ''weighted min-cut problem''. Prove that the modified algorithm returns a weighted minimum cut with probability at least <math>{2}/{n(n-1)}</math>. The weighted min-cut problem is defined as follows.
** '''Input''': an undirected weighted graph <math>G(V, E)</math>, where every edge <math>e \in E</math> is associated with a positive real weight <math>w_e</math>
** '''Output''': a cut <math>C</math> in <math>G</math> such that <math>\sum_{e \in C} w_e</math> is minimized.

* ['''Max directed-cut'''] In the ''maximum directed cut'' (MAX-DICUT) problem, we are given as input a directed graph <math>G(V,E)</math>. Each directed arc <math>(i, j) \in E</math> has nonnegative weight <math>w_{ij} \ge 0</math>.  The goal is to partition <math>V</math> into disjoint sets <math>U</math> and <math>W=V\setminus U</math>  so as to maximize the total weight of the arcs going from <math>U</math> to <math>W</math>. Give a randomized <math>1/4</math>-approximation algorithm for this problem.

== Problem 2 (Fingerprinting) ==

*['''Polynomial Identity Testing'''] Consider the function <math>f:\mathbb{R}^n\to\mathbb{R}</math> defined as

:<math>f(\vec x)=f(x_1,x_2,\dots,x_n)=\prod_{i=1}^{n}(a_ix_i-b_i)</math>,
:where <math>\{a_i\}_{1\le i\le n}</math> and <math>\{b_i\}_{1\le i\le n}</math> are '''unknown''' coefficients satisfy that <math>a_i, b_i\in \mathbb{Z}</math> and <math>0\le a_i, b_i \le n</math> for all <math>1\le i\le n</math>.

:Let <math>p>n</math> be the smallest prime strictly greater than <math>n</math>. The function <math>g:\mathbb{Z}_p^n\to\mathbb{Z}_p</math> is defined as

:<math>g(\vec x)=g(x_1,x_2,\dots,x_n)=\prod_{i=1}^{n}(a_ix_i-b_i)</math>,

:where <math>+</math> and <math>\cdot</math> are defined over the finite field <math>\mathbb{Z}_p</math>. By the properties of finite field, for any value <math>\vec r\in\mathbb{Z}_p^n</math>, it holds that <math>g(\vec r)=f(\vec r)\bmod p</math>. Since the coefficients <math>\{a_i\}_{1\le i\le n}</math> and <math>\{b_i\}_{1\le i\le n}</math> are unknown, you can't calculate <math>f(\vec x)</math> directly. However, there exists an oracle <math>O</math>, each time <math>O</math> gets an input <math>\vec x</math>, it immediately outputs the value of <math>g(\vec x)</math>.

# Prove that <math>f\not\equiv 0 \Rightarrow g\not\equiv 0</math>.
# Use the oracle <math>O</math> to design an algorithm to determine whether <math>f\equiv 0</math>, with error probability at most <math>\epsilon</math>, where <math>\epsilon\in (0,1)</math> is a constant.

*['''Test isomorphism of rooted tree'''] Two rooted trees <math>T_1</math> and <math>T_2</math> are said to be isomorphic if there exists a one to one mapping <math>f</math> from the nodes of <math>T_1</math> to those of <math>T_2</math> satisfying the following condition: <math>v</math> is a child of <math>w</math> in <math>T_1</math> if and only if <math>f(v)</math> is a child of <math>f(w)</math> in <math>T_2</math>. Observe that no ordering is assumed on the children of any vertex. Devise an efficient randomized algorithm for testing the isomorphism of rooted trees and analyze its performance. '''''Hint:''''' Recursively associate a polynomial <math>P_v</math> with each vertex <math>v</math> in a tree <math>T</math>. 

*['''2D pattern matching'''] Consider the problem of image matching. You are given an <math>n\times n</math>-bit matrix as "text" and an <math>m\times m</math>-bit “pattern” you want to find in the text.  Devise an efficient (expected time <math>O(n^2)</math> ) algorithm for the problem. '''''Hint:''''' The key is rapidly updating a fingerprint as you shift the “window” over the matrix. You may first want to consider the case of an <math>n \times m</math>-bit “text.” 

== Problem 3 (Hashing) ==

* ['''Set differences'''] Fix a universe <math>U</math> and two subset <math>A,B \subseteq U</math>, both with size <math>n</math>. we create both Bloom filters <math>F_A</math>(<math>F_B</math>) for <math>A</math> (<math>B</math>), using the same number of bits <math> m</math> and the same <math>k</math> hash functions.
# Let <math>F_C = F_A \land F_B</math> be the Bloom filter formed by computing the bitwise AND of <math>F_A</math> and <math>F_B</math>. Argue that <math>F_C</math> may not always be the same as the Bloom filter that are created for <math>A\cap B </math>.
# Bloom filters can be used to estimate set differences. Express the expected number of bits where <math>F_A</math> and <math>F_B</math> differ as a function of <math>m, n, k</math> and <math>|A\cap B|</math>.

* ['''Count Distinct Element'''] In class, we saw how to estimate the number of distinct elements in a data stream using the Flajolet-Martin algorithm. Consider the following alternative formulation of the distinct elements problem: given an <math>N</math> dimensional vector <math>x</math>, we want to process a stream of arbitrary increments to entries in <math>x</math>. In other words, if we see a number <math>i\in 1,\dots,N</math> in the stream, we update entry <math>x_i\gets x_i + 1</math>. Our goal is to estimate <math>\left \|x\right \|_0</math>, which measures the number of non-zero entries in <math>x</math>. With <math>x</math> viewed as a histogram that maintains counts for <math>N</math> potential elements, <math>\left \|x\right \|_0</math> is exactly the number of distinct elements processed. In this problem we will develop an alternative algorithm for estimating <math>\left \|x\right \|_0</math> that can also handle '''decrements''' to entries in <math>x</math>. Specifically, instead of the stream containing just indices <math>i</math>, it contains pairs <math>(i, +)</math> and <math>(i, −)</math>. On receiving <math>(i, +)</math>, <math>x</math> should update so that <math>x_i\gets x_i + 1</math> and on receiving <math>(i, −)</math>, <math>x</math> should update so that <math>x_i\gets x_i - 1</math>. For this problem we will assume that, at the end of our stream, each <math>x_i \ge 0</math> (i.e. for a specific index we can’t receive more decrements than increments).
# Consider a simpler problem. For a given value <math>T</math>, let’s design an algorithm that succeeds with probability <math>(1 − \delta)</math>, outputing '''LOW''' if <math>T < \frac{1}{2}\left \|x\right \|_0</math> and '''HIGH''' if <math>T > 2\left \|x\right \|_0</math>:
#* Assume we have access to a completely random hash function <math>h(\cdot)</math> that maps each <math>i</math> to a random point in <math>[0, 1]</math>. We maintain the estimator <math>s=\sum_{i:h(i)<\frac{1}{2T}}x_i</math> as we receive increment and decrement updates. Show that, at the end of our stream,    (i)  If <math>T < \frac{1}{2}\left \|x\right \|_0</math>, <math>\Pr_h[s=0]<1/e\approx 0.37</math> and (ii)  If <math>T > 2\left \|x\right \|_0</math>, <math>\Pr_h[s=0]>0.5</math>.
#* Using this fact, show how to use <math>k=O(\log 1/\delta)</math> independent random hash functions, and corresponding individual estimators <math>s_1, s_2, . . . , s_k</math>, to output '''LOW''' if <math>T < \frac{1}{2}\left \|x\right \|_0</math> and '''HIGH''' if <math>T > 2\left \|x\right \|_0</math>. If neither event occurs you can output either '''LOW''' or '''HIGH'''. Your algorithm should succeed with probability <math>(1 − \delta)</math>.
# Using <math>O(\log N)</math> repetitions of your algorithm for the above decision problem (with <math>\delta</math> set appropriately), show how to obtain an estimate <math>F</math> for <math>\left \|x\right \|_0</math> such that <math>\frac{1}{4}\left \|x\right \|_0\le F\le 4\left \|x\right \|_0</math> w.h.p.(with probability <math>1-O(1/N)</math>).

*['''Rethinking bloom filter'''] In class, we argued that with the right setting of parameters, the probability of a false positive for an <math>m </math>-bit bloom filter on  <math>n </math> items was roughly  <math>(.6185)^{m/n} </math>.  This problem will consider another approach. Divide the  <math>m</math> bits into  <math>r </math> "blocks" of <math>m/r </math> bits.  Associate a (random) <math>2 </math>-universal hash function with each block. For each item, hash it to one bit in each block using the block’s hash function, and set those bits to  <math>1</math>.  Now consider an item  <math>x </math> not in the inserted set, and suppose we hash it into the blocks in the same way.  We wish to bound the probability we get a false positive in this data structure (i.e.  all bits  <math>1 </math>).
# Bound the probability that in a particular block,  <math>x </math> hashes to a set bit, and from that bound derive the probability that  <math>x </math> is a false positive.
# Determine the best choice of <math>r</math> to minimize the above probability as a function of  <math>m </math> and  <math>n </math>.

== Problem 4 (Concentration of measure) ==

* ['''<math>k</math>-th moment bound'''] Let <math>X</math> be a random variable with expectation <math>0</math> such that moment generating function <math>\mathbf{E}[\exp(t|X|)]</math> is finite for some <math> t > 0 </math>. We can use the following two kinds of tail inequalities for <math> X </math>.

:'''''Chernoff Bound'''''
:<math>
\begin{align}
\mathbf{Pr}[|X| \geq \delta] \leq \min_{t \geq 0} \frac{\mathbf{E}[e^{t|X|}]}{e^{t\delta}}
\end{align}
</math>

:'''''<math>k</math>th-Moment Bound'''''
:<math>
\begin{align}
\mathbf{Pr}[|X| \geq \delta] \leq \frac{\mathbf{E}[|X|^k]}{\delta^k}
\end{align}
</math>

# Show that for each <math>\delta</math>, there exists a choice of <math>k</math> such that the <math>k</math>th-moment bound is stronger than the Chernoff bound. '''''Hint''''': Consider the Taylor expansion of the moment generating function and apply the probabilistic method.
# Why would we still prefer the Chernoff bound to the (seemingly) stronger <math>k</math>-th moment bound?
* ['''The median trick'''] Suppose we want to estimate the value of  <math>Z</math>. Let  <math>\mathcal{A}</math> be an algorithm that outputs <math>\widehat{Z}</math> satisfying
:<math>\Pr[ (1-\epsilon)Z \leq \widehat{Z} \leq (1+\epsilon )Z] \geq \frac{3}{4} .</math>

:We run  <math>\mathcal{A}</math> independently for <math>s</math> times, and obtain the outputs <math>\widehat{Z}_1,\widehat{Z}_2,\ldots,\widehat{Z}_s</math>. Let <math>X</math> be the '''median''' (中位数) of <math>\widehat{Z}_1,\widehat{Z}_2,\ldots,\widehat{Z}_s</math>. Find the number <math>s</math> such that 
:<math>\Pr[ (1-\epsilon)Z \leq X \leq (1+\epsilon )Z] \geq 1 - \delta .</math>

:Use Chernoff bound to express <math>s</math> as a function of <math>\delta</math>. Try to make <math>s</math> as small as possible.
* ['''Cut size in random graph'''] Show that with probability at least <math>2/3</math>, the size of the max-cut in [[wikipedia:Erdős–Rényi_model#Definition|Erdős–Rényi random graph]] <math>G(n,1/2)</math> is at most <math>n^2/8 + O(n^{1.5})</math>. In the <math>G(n,1/2)</math> model, each edge is included in the graph with probability <math>1/2</math>, independently of every other edge.
* ['''Code rate of boolean code'''] A '''boolean code''' is a mapping <math>C:\{0,1\}^k\rightarrow\{0,1\}^n</math>. Each <math>x\in\{0,1\}^k</math> is called a '''message''' and <math>y=C(x)</math> is called a '''codeword'''. The '''code rate''' <math>r</math> of a code <math>C</math> is <math>r=\frac{k}{n}</math>. A boolean code <math>C:\{0,1\}^k\rightarrow\{0,1\}^n</math> is a '''linear code''' if it is a linear transformation, i.e. there is a matrix <math>A\in\{0,1\}^{n\times k}</math> such that <math>C(x)=Ax</math> for any <math>x\in\{0,1\}^k</math>, where the additions and multiplications are defined over the finite field of order two, <math>(\{0,1\},+_{\bmod 2},\times_{\bmod 2})</math>. The '''distance''' between two codeword <math>y_1</math> and <math>y_2</math>, denoted by <math>d(y_1,y_2)</math>, is defined as the Hamming distance between them. Formally, <math>d(y_1,y_2)=\|y_1-y_2\|_1=\sum_{i=1}^n|y_1(i)-y_2(i)|</math>. The distance of a code <math>C</math> is the minimum distance between any two codewords. Formally, <math>d=\min_{x_1,x_2\in \{0,1\}^k\atop x_1\neq x_2}d(C(x_1),C(x_2))</math>. Usually we want to make both the code rate <math>r</math> and the code distance <math>d</math> as large as possible, because a larger rate means that the amount of actual message per transmitted bit is high, and a larger distance allows for more error correction and detection. Use the probabilistic method to prove that there exists a boolean code <math>C:\{0,1\}^k\rightarrow\{0,1\}^n</math> of code rate <math>r</math> and distance <math>\left(\frac{1}{2}-\Theta\left(\sqrt{r}\right)\right)n</math>. Try to optimize the constant in <math>\Theta(\cdot)</math>.
* ['''Balls into bins with the "power of two choices"'''] In Balls-and-Bins model, we throw <math>m</math> balls independently and uniformly at random into <math>n</math> bins. We know that the maximum load is <math>\Theta\left(\frac{\log n}{\log\log n}\right)</math> with high probability when <math>m=\Theta(n)</math>. The two-choice paradigm is another way to throw <math>m</math> balls into <math>n</math> bins: each ball is thrown into the least loaded of two bins chosen independently and uniformly at random(it could be the case that the two chosen bins are exactly the same, and then the ball will be thrown into that bin), and breaks the tie arbitrarily. When <math>m=\Theta(n)</math>, the maximum load of two-choice paradigm is known to be <math>\Theta(\log\log n)</math> with high probability, which is exponentially less than the maxim load when there is only one random choice. This phenomenon is called '''''the power of two choices'''''. Here are the questions:

#Consider the following paradigm: we throw <math>n</math> balls into <math>n</math> bins. The first <math>\frac{n}{2}</math> balls are thrown into bins independently and uniformly at random. The remaining <math>\frac{n}{2}</math> balls are thrown into bins using the two-choice paradigm. What is the maximum load with high probability? You need to give an asymptotically tight bound (in the form of <math>\Theta(\cdot)</math>).
#Replace the above paradigm to the following: the first <math>\frac{n}{2}</math> balls are thrown into bins using the  two-choice paradigm while the remaining <math>\frac{n}{2}</math> balls are thrown into bins independently and uniformly at random.  What is the maximum load with high probability in this case? You need to give an asymptotically tight bound.
#Replace the above paradigm to the following: assume all <math>n</math> balls are thrown in a sequence. For every <math>1\le i\le n</math>, if <math>i</math> is odd, we throw <math>i</math>-th ball into bins independently and uniformly at random, otherwise, we throw it into bins using the two-choice paradigm. What is the maximum load with high probability in this case? You need to give an asymptotically tight bound.

== Problem 5 (Dimension reduction) ==

* ['''Inner product'''] Fix parameters <math>d>0, \delta,\epsilon\in(0,1)</math>. Let <math>A\in \mathbb{R}^{k\times d}</math> be a random matrix with <math>k = O(\log(1/\delta)/\epsilon^2)</math> rows, and entries of <math>A</math> are chosen i.i.d. from Gaussian distribution with mean <math>0</math> and variance <math>1/k</math>. Prove that for any <math>x,y\in \mathbb{R}^d</math>: <math>|x^\top y - (Ax)^\top(Ay)|\leq \epsilon(\|x\|_2^2 + \|y\|_2^2)</math> with probability <math>\geq 1-\delta</math>.
* ['''Linear separability'''] In machine learning, the goal of many classification methods is to seperate data into classes using a hyperplane. A hyperplane in <math>\mathbb{R}^d</math> is characterized by a unit vector <math>a\in \mathbb{R}^d (\|a\|_2 = 1)</math> and <math>c\in \mathbb{R}</math>. It contains all <math>z\in \mathbb{R}^d</math> such that <math>a^\top z = c</math>. Suppose our dataset consists of <math>n</math> '''unit''' vectors in <math>\mathbb{R}^d</math>. These points can be separated into two linearly separable sets <math>X,Y</math> where <math>|X|+|Y| = n</math>. That is, for all <math>x\in X</math>, <math>a^\top x>c</math> and for all <math>y\in T</math>, <math>a^\top y<c</math> (or vice versa). Furthermore, suppose that the <math>\ell_2</math> distance of each point in <math>X</math> and <math>Y</math> to this separating hyperplane is at least <math>\epsilon</math>. When this is the case, the hyperplane is said to have margin <math>\epsilon</math>.
# Show that <math>X,Y</math> can be separated by a hyperplane with margin <math>\epsilon</math> is equivalently to the following condition: for all <math>x\in X</math>, <math>a^\top x >c+\epsilon</math> and for all <math>y\in Y</math>, <math>a^\top y < c-\epsilon</math> (or vice versa).
# Show that if we use a Johnson-Lindenstrauss map <math>A\in \mathbb{R}^{k\times d}</math> (the scaled Gaussian matrix given in the lecture) to reduce our data points to <math>O(\log n/\epsilon^2)</math> dimensions, then the dimension reduced data can still be separated by a hyperplane with margin <math>\epsilon/4</math> with probability at least <math>9/10</math>. ('''''Hint''''': use the fact that JLT preserves inner product.)
* ['''Sparse vector'''] A <math>k</math>-sparse vector is any vector with <math>k</math> nonzero entries. Let <math>S_k</math> be the set of all <math>k</math>-sparse vectors in <math>\mathbb{R}^d</math> (<math>d>k</math>). Show that there exists a random matrix <math>A\in \mathbb{R}^{s\times d}</math> with <math>s = O(k\log d/\epsilon^2)</math> rows such that with probability at least <math>9/10</math>, for '''all''' <math>x\in S_k</math>:
:<math>(1-\epsilon)\|x\|_2^2 \leq \|Ax\|^2 \leq (1+\epsilon)\|x\|^2_2.</math>

:In this question, you can directly use the following fact without needing to provide a proof: fix any <math>\delta\in (0,1)</math> and integer <math>m>0</math>, let <math>S = \{x\in \mathbb{R}^m: \|x\|_2 = 1\}</math> be the unit sphere in <math>\mathbb{R}^m</math>, there exists a discrete set <math>P = \{p_1, \cdots p_\ell\}</math> of size <math>\ell = (3/\delta)^k</math> and <math>P\subset S</math> such that for all <math>x\in S</math>, <math>\min_{i\in[\ell]} \|p_i - x\|_2 \leq \delta</math>. Such <math>P</math> is called a "''<math>\delta</math>-net''". Note that any point in the sphere is at most <math>\delta</math> away from a point in <math>P</math>.

== Problem 6 (Lovász Local Lemma) ==

* ['''Colorable hypergrap'''] A <math>k</math>-uniform hypergraph <math>H = (V,E)</math> is an ordered pair <math>G = (V,E)</math>, where <math>V</math> denotes the set of vertices and <math>E</math> denotes the set of edges. Moreover, each edge in <math>E</math> contains <math>k</math> distinct vertices (note that a <math>2</math>-uniform hypergraph is a normal graph). A hypergraph is called <math>2</math>-colorable if we can assign two colors to the vertices in such a way that every edge includes vertices of both colors. That is, no edge is monochromatic.
# Use the union bound to show that if <math>H</math> is a <math>k</math>-uniform hypergraph having at most <math>2^{k-1} - 1</math> edges, then <math>H</math> is <math>2</math>-colorable.
# Use the Lovász Local Lemma to prove another statement that if <math>H</math> is a <math>k</math>-uniform hypergraph having maximum degree at most <math>\frac{1}{k}(2^{k-3}-1)</math>. The degree of a vertex in <math>H</math> is the number of edges of which it is a member.
# A hypergraph is <math>k</math> regular if all vertices have degree <math>k</math>. Show that for sufficiently large <math>k</math>, the vertices of a <math>k</math>-uniform, <math>k</math>-regular hypergraph can be <math>2</math>-colored. What is the smallest value of <math>k</math> you can achieve?
* ['''Directed cycle'''] Let <math>D=(V,E)</math> be a simple directed graph with '''minimum outdegree''' <math>\delta</math> and '''maximum indegree''' <math>\Delta</math>. Prove that for any positive integer <math> k</math>, if <math>\mathrm{e}(\Delta \delta+1)(1-\frac{1}{k})^{\delta}<1</math>, then  <math>D</math> contains a (directed,simple) cycle of length <math>0\pmod k.</math>
:'''Hint''': Let <math>f:V\rightarrow \{0,1,\dots,k-1\}</math> be a random coloring of <math> V </math> obtained by choosing uniformly and independently at random for each <math> v\in V</math>. Try to use Lovász Local Lemma to show something interesting!
* ['''Algorithmic LLL'''] Let  <math>G </math> be a  <math>d </math>-regular graph (a normal graph, not a hypergraph) for sufficiently large  <math>d </math>. 
# Prove that the vertices of  <math>G </math> can be colored by  <math>k = \frac{d}{3\ln d} </math> colors in such a way that every vertex has at least one neighbor of each color.
# Propose a random algorithm for achieving such a coloring (described above). The expected run-time of your algorithm should be a polynomial in  <math> n </math>.