imported>Etone |
imported>Etone |
| Line 1: |
Line 1: |
| = <math>k</math>-wise independence = | | = Parameter Estimation = |
| Recall the definition of independence between events:
| | Consider the following abstract problem of parameter estimation. |
| {{Theorem
| |
| |Definition (Independent events)|
| |
| :Events <math>\mathcal{E}_1, \mathcal{E}_2, \ldots, \mathcal{E}_n</math> are '''mutually independent''' if, for any subset <math>I\subseteq\{1,2,\ldots,n\}</math>,
| |
| ::<math>\begin{align}
| |
| \Pr\left[\bigwedge_{i\in I}\mathcal{E}_i\right]
| |
| &=
| |
| \prod_{i\in I}\Pr[\mathcal{E}_i].
| |
| \end{align}</math>
| |
| }}
| |
| Similarly, we can define independence between random variables:
| |
| {{Theorem
| |
| |Definition (Independent variables)|
| |
| :Random variables <math>X_1, X_2, \ldots, X_n</math> are '''mutually independent''' if, for any subset <math>I\subseteq\{1,2,\ldots,n\}</math> and any values <math>x_i</math>, where <math>i\in I</math>,
| |
| ::<math>\begin{align}
| |
| \Pr\left[\bigwedge_{i\in I}(X_i=x_i)\right]
| |
| &=
| |
| \prod_{i\in I}\Pr[X_i=x_i].
| |
| \end{align}</math>
| |
| }}
| |
| | |
| Mutual independence is an ideal condition of independence. The limited notion of independence is usually defined by the '''k-wise independence'''.
| |
| {{Theorem
| |
| |Definition (k-wise Independenc)|
| |
| :1. Events <math>\mathcal{E}_1, \mathcal{E}_2, \ldots, \mathcal{E}_n</math> are '''k-wise independent''' if, for any subset <math>I\subseteq\{1,2,\ldots,n\}</math> with <math>|I|\le k</math>
| |
| :::<math>\begin{align}
| |
| \Pr\left[\bigwedge_{i\in I}\mathcal{E}_i\right]
| |
| &=
| |
| \prod_{i\in I}\Pr[\mathcal{E}_i].
| |
| \end{align}</math>
| |
| :2. Random variables <math>X_1, X_2, \ldots, X_n</math> are '''k-wise independent''' if, for any subset <math>I\subseteq\{1,2,\ldots,n\}</math> with <math>|I|\le k</math> and any values <math>x_i</math>, where <math>i\in I</math>,
| |
| :::<math>\begin{align}
| |
| \Pr\left[\bigwedge_{i\in I}(X_i=x_i)\right]
| |
| &=
| |
| \prod_{i\in I}\Pr[X_i=x_i].
| |
| \end{align}</math>
| |
| }}
| |
| | |
| A very common case is pairwise independence, i.e. the 2-wise independence.
| |
| {{Theorem
| |
| |Definition (pairwise Independent random variables)|
| |
| :Random variables <math>X_1, X_2, \ldots, X_n</math> are '''pairwise independent''' if, for any <math>X_i,X_j</math> where <math>i\neq j</math> and any values <math>a,b</math>
| |
| :::<math>\begin{align}
| |
| \Pr\left[X_i=a\wedge X_j=b\right]
| |
| &=
| |
| \Pr[X_i=a]\cdot\Pr[X_j=b].
| |
| \end{align}</math>
| |
| }}
| |
| | |
| Note that the definition of k-wise independence is hereditary:
| |
| * If <math>X_1, X_2, \ldots, X_n</math> are k-wise independent, then they are also <math>\ell</math>-wise independent for any <math>\ell<k</math>.
| |
| * If <math>X_1, X_2, \ldots, X_n</math> are NOT k-wise independent, then they cannot be <math>\ell</math>-wise independent for any <math>\ell>k</math>.
| |
| | |
| == Pairwise Independent Bits ==
| |
| Suppose we have <math>m</math> mutually independent and uniform random bits <math>X_1,\ldots, X_m</math>. We are going to extract <math>n=2^m-1</math> pairwise independent bits from these <math>m</math> mutually independent bits.
| |
| | |
| Enumerate all the nonempty subsets of <math>\{1,2,\ldots,m\}</math> in some order. Let <math>S_j</math> be the <math>j</math>th subset. Let
| |
| :<math>
| |
| Y_j=\bigoplus_{i\in S_j} X_i,
| |
| </math>
| |
| where <math>\oplus</math> is the exclusive-or, whose truth table is as follows.
| |
| :{|cellpadding="4" border="1"
| |
| |-
| |
| |<math>a</math>
| |
| |<math>b</math>
| |
| |<math>a</math><math>\oplus</math><math>b</math>
| |
| |-
| |
| | 0 || 0 ||align="center"| 0
| |
| |-
| |
| | 0 || 1 ||align="center"| 1
| |
| |-
| |
| | 1 || 0 ||align="center"| 1
| |
| |-
| |
| | 1 || 1 ||align="center"| 0
| |
| |}
| |
| | |
| There are <math>n=2^m-1</math> such <math>Y_j</math>, because there are <math>2^m-1</math> nonempty subsets of <math>\{1,2,\ldots,m\}</math>. An equivalent definition of <math>Y_j</math> is
| |
| :<math>Y_j=\left(\sum_{i\in S_j}X_i\right)\bmod 2</math>.
| |
| Sometimes, <math>Y_j</math> is called the '''parity''' of the bits in <math>S_j</math>.
| |
| | |
| We claim that <math>Y_j</math> are pairwise independent and uniform.
| |
| | |
| {{Theorem
| |
| |Theorem|
| |
| :For any <math>Y_j</math> and any <math>b\in\{0,1\}</math>,
| |
| ::<math>\begin{align}
| |
| \Pr\left[Y_j=b\right]
| |
| &=
| |
| \frac{1}{2}.
| |
| \end{align}</math>
| |
| :For any <math>Y_j,Y_\ell</math> that <math>j\neq\ell</math> and any <math>a,b\in\{0,1\}</math>,
| |
| ::<math>\begin{align}
| |
| \Pr\left[Y_j=a\wedge Y_\ell=b\right]
| |
| &=
| |
| \frac{1}{4}.
| |
| \end{align}</math>
| |
| }}
| |
| | |
| The proof is left for your exercise.
| |
| | |
| Therefore, we extract exponentially many pairwise independent uniform random bits from a sequence of mutually independent uniform random bits.
| |
| | |
| Note that <math>Y_j</math> are not 3-wise independent. For example, consider the subsets <math>S_1=\{1\},S_2=\{2\},S_3=\{1,2\}</math> and the corresponding random bits <math>Y_1,Y_2,Y_3</math>. Any two of <math>Y_1,Y_2,Y_3</math> would decide the value of the third one.
| |
| | |
| == Pairwise Independent Variables ==
| |
| We now consider constructing pairwise independent random variables ranging over <math>[p]=\{0,1,2,\ldots,p-1\}</math> for some prime <math>p</math>. Unlike the above construction, now we only need two independent random sources <math>X_0,X_1</math>, which are uniformly and independently distributed over <math>[p]</math>.
| |
| | |
| Let <math>Y_0,Y_1,\ldots, Y_{p-1}</math> be defined as:
| |
| :<math>
| |
| \begin{align}
| |
| Y_i=(X_0+i\cdot X_1)\bmod p &\quad \mbox{for }i\in[p].
| |
| \end{align}
| |
| </math>
| |
| | |
| {{Theorem
| |
| |Theorem|
| |
| : The random variables <math>Y_0,Y_1,\ldots, Y_{p-1}</math> are pairwise independent uniform random variables over <math>[p]</math>.
| |
| }}
| |
| {{Proof| We first show that <math>Y_i</math> are uniform. That is, we will show that for any <math>i,a\in[p]</math>,
| |
| :<math>\begin{align}
| |
| \Pr\left[(X_0+i\cdot X_1)\bmod p=a\right]
| |
| &=
| |
| \frac{1}{p}.
| |
| \end{align}</math>
| |
| Due to the law of total probability,
| |
| :<math>\begin{align}
| |
| \Pr\left[(X_0+i\cdot X_1)\bmod p=a\right]
| |
| &=
| |
| \sum_{j\in[p]}\Pr[X_1=j]\cdot\Pr\left[(X_0+ij)\bmod p=a\right]\\
| |
| &=\frac{1}{p}\sum_{j\in[p]}\Pr\left[X_0\equiv(a-ij)\pmod{p}\right].
| |
| \end{align}</math>
| |
| For prime <math>p</math>, for any <math>i,j,a\in[p]</math>, there is exact one value in <math>[p]</math> of <math>X_0</math> satisfying <math>X_0\equiv(a-ij)\pmod{p}</math>. Thus, <math>\Pr\left[X_0\equiv(a-ij)\pmod{p}\right]=1/p</math> and the above probability is <math>\frac{1}{p}</math>.
| |
| | |
| We then show that <math>Y_i</math> are pairwise independent, i.e. we will show that for any <math>Y_i,Y_j</math> that <math>i\neq j</math> and any <math>a,b\in[p]</math>,
| |
| :<math>\begin{align}
| |
| \Pr\left[Y_i=a\wedge Y_j=b\right]
| |
| &=
| |
| \frac{1}{p^2}.
| |
| \end{align}</math>
| |
| | |
| The event <math>Y_i=a\wedge Y_j=b</math> is equivalent to that
| |
| :<math>
| |
| \begin{cases}
| |
| (X_0+iX_1)\equiv a\pmod{p}\\
| |
| (X_0+jX_1)\equiv b\pmod{p}
| |
| \end{cases}
| |
| </math>
| |
| Due to the [http://en.wikipedia.org/wiki/Chinese_remainder_theorem Chinese remainder theorem], there exists a unique solution of <math>X_0</math> and <math>X_1</math> in <math>[p]</math> to the above linear congruential system. Thus the probability of the event is <math>\frac{1}{p^2}</math>.
| |
| }}
| |
| | |
| ==Application: Derandomizing MAX-CUT==
| |
| Let <math>G(V,E)</math> be an undirected graph, and <math>S\subset V</math> be a vertex set. The '''cut''' defined by <math>S</math> is <math>C(S,\bar{S})=|\{uv\in E\mid u\in S, v\not\in S\}|</math>.
| |
| | |
| Given as input an undirected graph <math>G(V,E)</math>, find the <math>S\subset V</math> whose cut value <math>C(S,\bar{S})</math> is maximized. This problem is called the [http://en.wikipedia.org/wiki/Maximum_cut maximum cut (MAX-CUT) problem], which is NP-hard. The decision version of one of the weighted version of the problem is one of the [http://en.wikipedia.org/wiki/Karp%27s_21_NP-complete_problems Karp's 21 NP-complete problems]. The problem has a <math>0.878</math>-approximation algorithm by rounding a [http://en.wikipedia.org/wiki/Semidefinite_programming semidefinite programming]. Assuming that the [http://en.wikipedia.org/wiki/Unique_games_conjecture unique game conjecture (UGC)], there does not exist a poly-time algorithm with better approximation ratio unless <math>P=NP</math>.
| |
| | |
| Here we give a very simple <math>0.5</math>-approximation algorithm. The "algorithm" has a one-line description:
| |
| *Put each <math>v\in V</math> into <math>S</math> independently with probability 1/2.
| |
| We then analyze the approximation ratio of this algorithm.
| |
| | |
| For each <math>v\in V</math>, let <math>Y_v</math> indicate whether <math>v\in S</math>, that is
| |
| :<math>Y_v=\begin{cases}1& v\in S,\\
| |
| 0& v\not\in S.\end{cases}</math>
| |
| For each edge <math>uv\in E</math>, let <math>Y_{uv}</math> indicate whether <math>uv</math> contribute to the cut <math>C(S,\bar{S})</math>, i.e. whether <math>u\in S, v\not\in S</math> or <math>u\not\in S, v\in S</math>, that is
| |
| :<math>Y_{uv}=\begin{cases}1&Y_u\neq Y_v,\\0&\text{otherwise}.\end{cases}</math>
| |
| Then <math>C(S,\bar{S})=\sum_{uv\in E}Y_{uv}</math>. Due to the linearity of expectation,
| |
| :<math>\mathbf{E}\left[C(S,\bar{S})\right]=\sum_{uv\in E}\mathbf{E}[Y_{uv}]=\sum_{uv\in E}\Pr[Y_u\neq Y_v]=\frac{|E|}{2}</math>.
| |
| The maximum cut of a graph is at most <math>|E|</math>. Thus, the algorithm returns in expectation a cut with size at least half of the maximum cut.
| |
| | |
| We then show how to dereandomize this algorithm by pairwise independent bits.
| |
| | |
| Suppose that <math>|V|=n</math> and enumerate the <math>n</math> vertices by <math>v_1,v_2,\ldots, v_n</math> in an arbitrary order. Let <math>m=\lceil\log_2 (n+1)\rceil</math>. Sample <math>m</math> bits <math>X_1,\ldots, X_m\in\{0,1\}</math> uniformly and independently at random. Enumerate all nonempty subsets of <math>\{1,2,\ldots,m\}</math> by <math>S_1,S_2,\ldots,S_{2^m-1}</math>. For each vertex <math>v_j</math>, let <math>Y_{v_j}=\bigoplus_{i\in S_j}X_i</math>. The MAX-CUT algorithm uses these bits to construct the solution <math>S</math>:
| |
| * For <math>j=1,2,\ldots,n</math>, put <math>v_j</math> into <math>S</math> if <math>Y_{v_j}=1</math>.
| |
| | |
| We have shown that <math>Y_{v_j}</math>, <math>j=1,2,\ldots,n</math>, are uniform and pairwise independent. Thus we still have that <math>\Pr[Y_{u}\neq Y_{v}]=\frac{1}{2}</math>. The above analysis still holds, so that the algorithm returns in expectation a cut with size at least <math>\frac{|E|}{2}</math>.
| |
| | |
| Finally, we notice that there are only <math>m=\lceil\log_2 (n+1)\rceil</math> total random bits in the new algorithm. We can enumerate all <math>2^m\le 2(n+1)</math> possible strings of <math>m</math> bits, run the above algorithm with the bit strings as the "random sources", and output the maximum cut returned. There must exist a bit string <math>X_1,\ldots, X_m\in\{0,1\}</math> on which the algorithm returns a cut of size <math>\ge \frac{|E|}{2}</math> (why?). This gives us a deterministic polynomial time (actually <math>O(n^2)</math> time) <math>1/2</math>-approximation algorithm.
| |
| | |
| == Application: Two-point sampling ==
| |
| Consider a Monte Carlo randomized algorithm with one-sided error for a decision problem <math>f</math>. We formulate the algorithm as a deterministic algorithm <math>A</math> that takes as input <math>x</math> and a uniform random number <math>r\in[p]</math> where <math>p</math> is a prime, such that for any input <math>x</math>: | |
| * If <math>f(x)=1</math>, then <math>\Pr[A(x,r)=1]\ge\frac{1}{2}</math>, where the probability is taken over the random choice of <math>r</math>.
| |
| * If <math>f(x)=0</math>, then <math>A(x,r)=0</math> for any <math>r</math>.
| |
| We call <math>r</math> the '''random source''' for the algorithm.
| |
| | |
| For the <math>x</math> that <math>f(x)=1</math>, we call the <math>r</math> that makes <math>A(x,r)=1</math> a '''witness''' for <math>x</math>. For a positive <math>x</math>, at least half of <math>[p]</math> are witnesses. The random source <math>r</math> has polynomial number of bits, which means that <math>p</math> is exponentially large, thus it is infeasible to find the witness for an input <math>x</math> by exhaustive search. Deterministic overcomes this by having sophisticated deterministic rules for efficiently searching for a witness. Randomization, on the other hard, reduce this to a bit of luck, by randomly choosing an <math>r</math> and winning with a probability of 1/2.
| |
| | |
| We can boost the accuracy (equivalently, reduce the error) of any Monte Carlo randomized algorithm with one-sided error by running the algorithm for a number of times.
| |
| | |
| Suppose that we sample <math>t</math> values <math>r_1,r_2,\ldots,r_t</math> uniformly and independently from <math>[p]</math>, and run the following scheme:
| |
| :{|border="1"
| |
| |<math>B(x,r_1,r_2,\ldots,r_t):</math>
| |
| :return <math>\bigvee_{i=1}^t A(x,r_i)</math>;
| |
| |}
| |
| | |
| That is, return 1 if any instance of <math>A(x,r_i)=1</math>. For any <math>x</math> that <math>f(x)=1</math>, due to the independence of <math>r_1,r_2,\ldots,r_t</math>, the probability that <math>B(x,r_1,r_2,\ldots,r_t)</math> returns an incorrect result is at most <math>2^{-t}</math>. On the other hand, <math>B</math> never makes mistakes for the <math>x</math> that <math>f(x)=0</math> since <math>A</math> has no false positives. Thus, the error of the Monte Carlo algorithm is reduced to <math>2^{-t}</math>.
| |
| | |
| Sampling <math>t</math> mutually independent random numbers from <math>[p]</math> can be quite expensive since it requires <math>\Omega(t\log p)</math> random bits. Suppose that we can only afford <math>O(\log p)</math> random bits. In particular, we sample two independent uniform random number <math>a</math> and <math>b</math> from <math>[p]</math>. If we use <math>a</math> and <math>b</math> directly bu running two independent instances <math>A(x,a)</math> and <math>A(x,b)</math>, we only get an error upper bound of 1/4.
| |
| | |
| The following scheme reduces the error significantly with the same number of random bits:
| |
| :{{Theorem
| |
| |Algorithm|
| |
| Choose two independent uniform random number <math>a</math> and <math>b</math> from <math>[p]</math>.
| |
| Construct <math>t</math> random number <math>r_1,r_2,\ldots,r_t</math> by:
| |
| :<math>
| |
| \begin{align}
| |
| \forall 1\le i\le t, &\quad \mbox{let }r_i = (a\cdot i+b)\bmod p.
| |
| \end{align}
| |
| </math>
| |
| Run <math>B(x,r_1,r_2,\ldots,r_t):</math>.
| |
| }}
| |
| | |
| Due to the discussion in the last section, we know that for <math>t\le p</math>, <math>r_1,r_2,\ldots,r_t</math> are pairwise independent and uniform over <math>[p]</math>. Let <math>X_i=A(x,r_i)</math> and <math>X=\sum_{i=1}^tX_i</math>. Due to the uniformity of <math>r_i</math> and our definition of <math>A</math>, for any <math>x</math> that <math>f(x)=1</math>, it holds that
| |
| :<math>
| |
| \Pr[X_i=1]=\Pr[A(x,r_i)=1]\ge\frac{1}{2}.
| |
| </math>
| |
| By the linearity of expectations,
| |
| :<math>
| |
| \mathbf{E}[X]=\sum_{i=1}^t\mathbf{E}[X_i]=\sum_{i=1}^t\Pr[X_i=1]\ge\frac{t}{2}.
| |
| </math>
| |
| Since <math>X_i</math> is Bernoulli trial with a probability of success at least <math>p=1/2</math>.
| |
| We can estimate the variance of each <math>X_i</math> as follows.
| |
| :<math>
| |
| \mathbf{Var}[X_i]=p(1-p)\le\frac{1}{4}.
| |
| </math>
| |
| | |
| Applying Chebyshev's inequality, we have that for any <math>x</math> that <math>f(x)=1</math>,
| |
| :<math>\begin{align}
| |
| \Pr\left[\bigvee_{i=1}^t A(x,r_i)=0\right]
| |
| &=
| |
| \Pr[X=0]\\
| |
| &\le
| |
| \Pr[|X-\mathbf{E}[X]|\ge \mathbf{E}[X]]\\
| |
| &\le
| |
| \Pr\left[|X-\mathbf{E}[X]|\ge \frac{t}{2}\right]\\
| |
| &\le
| |
| \frac{4}{t^2}\sum_{i=1}^t\mathbf{Var}[X_i]\\
| |
| &\le \frac{1}{t}.
| |
| \end{align}
| |
| </math>
| |
| The error is reduced to <math>1/t</math> with only two random numbers. This scheme works as long as <math>t\le p</math>.
| |
| | |
| | |
| =Universal Hashing =
| |
| Hashing is one of the oldest tools in Computer Science. Knuth's memorandum in 1963 on analysis of hash tables is now considered to be the birth of the area of analysis of algorithms.
| |
| * Knuth. Notes on "open" addressing, July 22 1963. Unpublished memorandum.
| |
| | |
| The idea of hashing is simple: an unknown set <math>S</math> of <math>n</math> data '''items''' (or keys) are drawn from a large '''universe''' <math>U=[N]</math> where <math>N\gg n</math>; in order to store <math>S</math> in a table of <math>M</math> entries (slots), we assume a consistent mapping (called a '''hash function''') from the universe <math>U</math> to a small range <math>[M]</math>.
| |
| | |
| This idea seems clever: we use a consistent mapping to deal with an arbitrary unknown data set. However, there is a fundamental flaw for hashing.
| |
| * For sufficiently large universe (<math>N> M(n-1)</math>), for any function, there exists a bad data set <math>S</math>, such that all items in <math>S</math> are mapped to the same entry in the table.
| |
| | |
| A simple use of pigeonhole principle can prove the above statement.
| |
| | |
| To overcome this situation, randomization is introduced into hashing. We assume that the hash function is a random mapping from <math>[N]</math> to <math>[M]</math>. In order to ease the analysis, the following ideal assumption is used:
| |
| | |
| '''Simple Uniform Hash Assumption''' ('''SUHA''' or '''UHA''', a.k.a. the random oracle model):
| |
| :A ''uniform'' random function <math>h:[N]\rightarrow[M]</math> is available and the computation of <math>h</math> is efficient.
| |
| | |
| == Families of universal hash functions ==
| |
| The assumption of completely random function simplifies the analysis. However, in practice, truly uniform random hash function is extremely expensive to compute and store. Thus, this simple assumption can hardly represent the reality.
| |
| | |
| There are two approaches for implementing practical hash functions. One is to use ''ad hoc'' implementations and wish they may work. The other approach is to construct class of hash functions which are efficient to compute and store but with weaker randomness guarantees, and then analyze the applications of hash functions based on this weaker assumption of randomness.
| |
| | |
| This route was took by Carter and Wegman in 1977 while they introduced universal families of hash functions.
| |
| | |
| {{Theorem
| |
| |Definition (universal hash families)|
| |
| :Let <math>[N]</math> be a universe with <math>N\ge M</math>. A family of hash functions <math>\mathcal{H}</math> from <math>[N]</math> to <math>[M]</math> is said to be '''<math>k</math>-universal''' if, for any items <math>x_1,x_2,\ldots,x_k\in [N]</math> and for a hash function <math>h</math> chosen uniformly at random from <math>\mathcal{H}</math>, we have
| |
| ::<math>
| |
| \Pr[h(x_1)=h(x_2)=\cdots=h(x_k)]\le\frac{1}{M^{k-1}}.
| |
| </math>
| |
| | |
| :A family of hash functions <math>\mathcal{H}</math> from <math>[N]</math> to <math>[M]</math> is said to be '''strongly <math>k</math>-universal''' if, for any items <math>x_1,x_2,\ldots,x_k\in [N]</math>, any values <math>y_1,y_2,\ldots,y_k\in[M]</math>, and for a hash function <math>h</math> chosen uniformly at random from <math>\mathcal{H}</math>, we have
| |
| ::<math>
| |
| \Pr[h(x_1)=y_1\wedge h(x_2)=y_2 \wedge \cdots \wedge h(x_k)=y_k]=\frac{1}{M^{k}}.
| |
| </math>
| |
| }}
| |
| In particular, for a 2-universal family <math>\mathcal{H}</math>, for any elements <math>x_1,x_2\in[N]</math>, a uniform random <math>h\in\mathcal{H}</math> has
| |
| :<math>
| |
| \Pr[h(x_1)=h(x_2)]\le\frac{1}{M}.
| |
| </math>
| |
| For a strongly 2-universal family <math>\mathcal{H}</math>, for any elements <math>x_1,x_2\in[N]</math> and any values <math>y_1,y_2\in[M]</math>, a uniform random <math>h\in\mathcal{H}</math> has
| |
| :<math>
| |
| \Pr[h(x_1)=y_1\wedge h(x_2)=y_2]=\frac{1}{M^2}.
| |
| </math>
| |
| This behavior is exactly the same as uniform random hash functions on any pair of inputs. For this reason, a strongly 2-universal hash family are also called pairwise independent hash functions.
| |
| | |
| == 2-universal hash families ==
| |
| | |
| The construction of pairwise independent random variables via modulo a prime introduced in Section 1 already provides a way of constructing a strongly 2-universal hash family.
| |
| | |
| Let <math>p</math> be a prime. The function <math>h_{a,b}:[p]\rightarrow [p]</math> is defined by
| |
| :<math>
| |
| h_{a,b}(x)=(ax+b)\bmod p,
| |
| </math>
| |
| and the family is
| |
| :<math>
| |
| \mathcal{H}=\{h_{a,b}\mid a,b\in[p]\}.
| |
| </math>
| |
| | |
| {{Theorem
| |
| |Lemma|
| |
| :<math>\mathcal{H}</math> is strongly 2-universal.
| |
| }}
| |
| {{Proof| In Section 1, we have proved the pairwise independence of the sequence of <math>(a i+b)\bmod p</math>, for <math>i=0,1,\ldots, p-1</math>, which directly implies that <math>\mathcal{H}</math> is strongly 2-universal.
| |
| }}
| |
| | |
| ;The original construction of Carter-Wegman
| |
| What if we want to have hash functions from <math>[N]</math> to <math>[M]</math> for non-prime <math>N</math> and <math>M</math>? Carter and Wegman developed the following method.
| |
| | |
| Suppose that the universe is <math>[N]</math>, and the functions map <math>[N]</math> to <math>[M]</math>, where <math>N\ge M</math>. For some prime <math>p\ge N</math>, let
| |
| :<math>
| |
| h_{a,b}(x)=((ax+b)\bmod p)\bmod M,
| |
| </math>
| |
| and the family
| |
| :<math>
| |
| \mathcal{H}=\{h_{a,b}\mid 1\le a\le p-1, b\in[p]\}.
| |
| </math>
| |
| Note that unlike the first construction, now <math>a\neq 0</math>.
| |
| {{Theorem
| |
| |Lemma (Carter-Wegman)|
| |
| :<math>\mathcal{H}</math> is 2-universal.
| |
| }}
| |
| {{Proof| Due to the definition of <math>\mathcal{H}</math>, there are <math>p(p-1)</math> many different hash functions in <math>\mathcal{H}</math>, because each hash function in <math>\mathcal{H}</math> corresponds to a pair of <math>1\le a\le p-1</math> and <math>b\in[p]</math>. We only need to count for any particular pair of <math>x_1,x_2\in[N]</math> that <math>x_1\neq x_2</math>, the number of hash functions that <math>h(x_1)=h(x_2)</math>.
| |
| | |
| We first note that for any <math>x_1\neq x_2</math>, <math>a x_1+b\not\equiv a x_2+b \pmod p</math>. This is because <math>a x_1+b\equiv a x_2+b \pmod p</math> would imply that <math>a(x_1-x_2)\equiv 0\pmod p</math>, which can never happen since <math>1\le a\le p-1</math> and <math>x_1\neq x_2</math> (note that <math>x_1,x_2\in[N]</math> for an <math>N\le p</math>). Therefore, we can assume that <math>(a x_1+b)\bmod p=u</math> and <math>(a x_2+b)\bmod p=v</math> for <math>u\neq v</math>.
| |
| | |
| Due to the Chinese remainder theorem, for any <math>x_1,x_2\in[N]</math> that <math>x_1\neq x_2</math>, for any <math>u,v\in[p]</math> that <math>u\neq v</math>, there is exact one solution to <math>(a,b)</math> satisfying:
| |
| :<math>
| |
| \begin{cases}
| |
| a x_1+b \equiv u \pmod p\\
| |
| a x_2+b \equiv v \pmod p.
| |
| \end{cases}
| |
| </math>
| |
| After modulo <math>M</math>, every <math>u\in[p]</math> has at most <math>\lceil p/M\rceil -1</math> many <math>v\in[p]</math> that <math>v\neq u</math> but <math>v\equiv u\pmod M</math>. Therefore, for every pair of <math>x_1,x_2\in[N]</math> that <math>x_1\neq x_2</math>, there exist at most <math>p(\lceil p/M\rceil -1)\le p(p-1)/M</math> pairs of <math>1\le a\le p-1</math> and <math>b\in[p]</math> such that <math>((ax_1+b)\bmod p)\bmod M=((ax_2+b)\bmod p)\bmod M</math>, which means there are at most <math> p(p-1)/M</math> many hash functions <math>h\in\mathcal{H}</math> having <math>h(x_1)=h(x_2)</math> for <math>x_1\neq x_2</math>. For <math>h</math> uniformly chosen from <math>\mathcal{H}</math>, for any <math>x_1\neq x_2</math>,
| |
| :<math>
| |
| \Pr[h(x_1)=h(x_2)]\le \frac{p(p-1)/M}{p(p-1)}=\frac{1}{M}.
| |
| </math>
| |
| We prove that <math>\mathcal{H}</math> is 2-universal.
| |
| }}
| |
| | |
| ;A construction used in practice
| |
| The main issue of Carter-Wegman construction is the efficiency. The mod operation is very slow, and has been so for more than 30 years.
| |
| | |
| The following construction is due to Dietzfelbinger ''et al''. It was published in 1997 and has been practically used in various applications of universal hashing.
| |
| | |
| The family of hash functions is from <math>[2^u]</math> to <math>[2^v]</math>. With a binary representation, the functions map binary strings of length <math>u</math> to binary strings of length <math>v</math>.
| |
| Let
| |
| :<math>
| |
| h_{a}(x)=\left\lfloor\frac{a\cdot x\bmod 2^u}{2^{u-v}}\right\rfloor,
| |
| </math>
| |
| and the family
| |
| :<math>
| |
| \mathcal{H}=\{h_{a}\mid a\in[2^v]\mbox{ and }a\mbox{ is odd}\}.
| |
| </math>
| |
| | |
| This family of hash functions does not exactly meet the requirement of 2-universal family. However, Dietzfelbinger ''et al'' proved that <math>\mathcal{H}</math> is close to a 2-universal family. Specifically, for any input values <math>x_1,x_2\in[2^u]</math>, for a uniformly random <math>h\in\mathcal{H}</math>,
| |
| :<math>
| |
| \Pr[h(x_1)=h(x_2)]\le\frac{1}{2^{v-1}}.
| |
| </math>
| |
| So <math>\mathcal{H}</math> is within an approximation ratio of 2 to being 2-universal. The proof uses the fact that odd numbers are relative prime to a power of 2.
| |
|
| |
|
| The function is extremely simple to compute in c language.
| | Let <math>U</math> be a finite set of known size, and let <math>G\subseteq U</math>. We want to estimate the ''parameter'' <math>|G|</math>, i.e. the size of <math>G</math>. |
| We exploit that C-multiplication (*) of unsigned u-bit numbers is done <math>\bmod 2^u</math>, and have a one-line C-code for computing the hash function:
| |
| h_a(x) = (a*x)>>(u-v)
| |
| The bit-wise shifting is a lot faster than modular. It explains the popularity of this scheme in practice than the original Carter-Wegman construction.
| |
|
| |
|
| == Collision number ==
| | We assume two devices: |
| Consider a 2-universal family <math>\mathcal{H}</math> of hash functions from <math>[N]</math> to <math>[M]</math>. Let <math>h</math> be a hash function chosen uniformly from <math>\mathcal{H}</math>. For a fixed set <math>S</math> of <math>n</math> distinct elements from <math>[N]</math>, say <math>S=\{x_1,x_2,\ldots,x_n\}</math>, the elements are mapped to the hash values <math>h(x_1), h(x_2), \ldots, h(x_n)</math>. This can be seen as throwing <math>n</math> balls to <math>M</math> bins, with pairwise independent choices of bins.
| | * A '''uniform sampler''' <math>\mathcal{U}</math>, which uniformly and independently samples a member of <math>U</math> upon each calling. |
| | * A '''membership oracle''' of <math>G</math>, denoted <math>\mathcal{O}</math>. Given as the input an <math>x\in U</math>, <math>\mathcal{O}(x)</math> indicates whether or not <math>x</math> is a member of <math>G</math>. |
|
| |
|
| As in the balls-into-bins with full independence, we are curious about the questions such as the birthday problem or the maximum load. These questions are interesting not only because they are natural to ask in a balls-into-bins setting, but in the context of hashing, they are closely related to the performance of hash functions.
| | Equipped by <math>\mathcal{U}</math> and <math>\mathcal{O}</math>, we can have the following Monte Carlo algorithm: |
| | *Choose <math>N</math> independent samples from <math>U</math> by the uniform sampler <math>\mathcal{U}</math>, represented by the random variables <math>X_1,X_2,\ldots, X_N</math>. |
| | * Let <math>Y_i</math> be the indicator random variable defined as <math>Y_i=\mathcal{O}(X_i)</math>, namely, <math>Y_i</math> indicates whether <math>X_i\in G</math>. |
| | * Define the estimator random variable |
| | ::<math>Z=\frac{|U|}{N}\sum_{i=1}^N Y_i.</math> |
|
| |
|
| The old techniques for analyzing balls-into-bins rely too much on the independence of the choice of the bin for each ball, therefore can hardly be extended to the setting of 2-universal hash families. However, it turns out several balls-into-bins questions can somehow be answered by analyzing a very natural quantity: the number of '''collision pairs'''.
| | It is easy to see that <math>\mathbf{E}[Z]=|G|</math> and we might hope that with high probability the value of <math>Z</math> is close to <math>|G|</math>. Formally, <math>Z</math> is called an '''<math>\epsilon</math>-approximation''' of <math>|G|</math> if |
| | |
| A collision pair for hashing is a pair of elements <math>x_1,x_2\in S</math> which are mapped to the same hash value, i.e. <math>h(x_1)=h(x_2)</math>. Formally, for a fixed set of elements <math>S=\{x_1,x_2,\ldots,x_n\}</math>, for any <math>1\le i,j\le n</math>, let the random variable
| |
| :<math> | | :<math> |
| X_{ij}
| | (1-\epsilon)|G|\le Z\le (1+\epsilon)|G|. |
| =
| |
| \begin{cases}
| |
| 1 & \text{if }h(x_i)=h(x_j),\\ | |
| 0 & \text{otherwise.}
| |
| \end{cases}
| |
| </math> | | </math> |
| The total number of collision pairs among the <math>n</math> items <math>x_1,x_2,\ldots,x_n</math> is
| |
| :<math>X=\sum_{i<j} X_{ij}.\,</math>
| |
|
| |
| Since <math>\mathcal{H}</math> is 2-universal, for any <math>i\neq j</math>,
| |
| :<math>
| |
| \Pr[X_{ij}=1]=\Pr[h(x_i)=h(x_j)]\le\frac{1}{M}.
| |
| </math>
| |
|
| |
| The expected number of collision pairs is
| |
| :<math>\mathbf{E}[X]=\mathbf{E}\left[\sum_{i<j}X_{ij}\right]=\sum_{i<j}\mathbf{E}[X_{ij}]=\sum_{i<j}\Pr[X_{ij}=1]\le{n\choose 2}\frac{1}{M}<\frac{n^2}{2M}.
| |
| </math>
| |
|
| |
| In particular, for <math>n=M</math>, i.e. <math>n</math> items are mapped to <math>n</math> hash values by a pairwise independent hash function, the expected collision number is <math>\mathbf{E}[X]<\frac{n^2}{2M}=\frac{n}{2}</math>.
| |
|
| |
| === Birthday problem ===
| |
| In the context of hash functions, the birthday problem ask for the probability that there is no collision at all. Since collision is something that we want to avoid in the applications of hash functions, we would like to lower bound the probability of zero-collision, i.e. to upper bound the probability that there exists a collision pair.
| |
|
| |
|
| The above analysis gives us an estimation on the expected number of collision pairs, such that <math>\mathbf{E}[X]<\frac{n^2}{2M}</math>. Apply the Markov's inequality, for <math>0<\epsilon<1</math>, we have | | The following theorem states that the probabilistic accuracy of the estimation depends on the number of samples and the ratio between <math>|G|</math> and <math>|U|</math> |
| :<math>
| |
| \Pr\left[X\ge \frac{n^2}{2\epsilon M}\right]\le\Pr\left[X\ge \frac{1}{\epsilon}\mathbf{E}[X]\right]\le\epsilon.
| |
| </math> | |
|
| |
|
| When <math>n\le\sqrt{2\epsilon M}</math>, the number of collision pairs is <math>X\ge1</math> with probability at most <math>\epsilon</math>, therefore with probability at least <math>1-\epsilon</math>, there is no collision at all. Therefore, we have the following theorem.
| |
| {{Theorem | | {{Theorem |
| |Theorem| | | |Theorem (estimator theorem)| |
| :If <math>h</math> is chosen uniformly from a 2-universal family of hash functions mapping the universe <math>[N]</math> to <math>[M]</math> where <math>N\ge M</math>, then for any set <math>S\subset [N]</math> of <math>n</math> items, where <math>n\le\sqrt{2\epsilon M}</math>, the probability that there exits a collision pair is | | :Let <math>\alpha=\frac{|G|}{|U|}</math>. Then the Monte Carlo method yields an <math>\epsilon</math>-approximation to <math>|G|</math> with probability at least <math>1-\delta</math> provided |
| ::<math>
| | ::<math>N\ge\frac{4}{\epsilon^2 \alpha}\ln\frac{2}{\delta}</math>. |
| \Pr[\mbox{collision occurs}]\le\epsilon. | |
| </math> | |
| }} | | }} |
| | {{Proof| |
| | Recall that <math>Y_i</math> indicates whether the <math>i</math>-th sample is in <math>G</math>. Let <math>Y=\sum_{i=1}^NY_i</math>. Then we have <math>Z=\frac{|U|}{N}Y</math>, and hence the event <math>(1-\epsilon)|G|\le Z\le (1+\epsilon)|G|</math> is equivalent to that <math>(1-\epsilon)\frac{|G|}{|U|}N\le Y\le (1+\epsilon)\frac{|G|}{|U|}N</math>. Note that each <math>Y_i</math> is a Bernoulli trial that succeeds with probability <math>\frac{|G|}{|U|}</math>, thus <math>\mathbb{E}[Y]=\frac{|G|}{|U|}N</math>. Then the rest is due to Chernoff bound.}} |
|
| |
|
| Recall that for mutually independent choices of bins, for some <math>n=\sqrt{2M\ln(1/\epsilon)}</math>, the probability that a collision occurs is about <math>\epsilon</math>. For constant <math>\epsilon</math>, this gives an essentially same bound as the pairwise independent setting. Therefore,
| | A counting algorithm for the set <math>G</math> has to deal with the following three issues: |
| the behavior of pairwise independent hash function is essentially the same as the uniform random hash function for the birthday problem. This is easy to understand, because birthday problem is about the behavior of collisions, and the definition of 2-universal hash function can be interpreted as "functions that the probability of collision is as low as a uniform random function".
| | # Implement the membership oracle <math>\mathcal{O}</math>. This is usually straightforward, or assumed by the model. |
| | # Implement the uniform sampler <math>\mathcal{U}</math>. This can be straightforward or highly nontrivial, depending on the problem. |
| | # Deal with exponentially small <math>\alpha=\frac{|G|}{|U|}</math>. This requires us to cleverly choose the universe <math>U</math>. Sometimes this needs some nontrivial ideas. |
|
| |
|
| = Perfect Hashing = | | = Counting DNFs = |
| Perfect hashing is a data structure for storing a ''static dictionary''. In a static dictionary, a set <math>S</math> of <math>n</math> items from the universe <math>[N]</math> are preprocessed and stored in a table. Once the table is constructed, it will nit be changed any more, but will only be used for search operations: a search for an item gives the location of the item in the table or returns that the item is not in the table. You may think of an application that we store an encyclopedia in a DVD, so that searches are very efficient but there will be no updates to the data.
| | A disjunctive normal form (DNF) formular is a disjunction (OR) of clauses, where each clause is a conjunction (AND) of literals. For example: |
| | :<math>(x_1\wedge \overline{x_2}\wedge x_3)\vee(x_2\wedge x_4)\vee(\overline{x_1}\wedge x_3\wedge x_4)</math>. |
| | Note the difference from the conjunctive normal forms (CNF). |
|
| |
|
| This problem can be solved by binary search on a sorted table or balanced search trees in <math>O(\log n)</math> time for a set <math>S</math> of <math>n</math> elements. We show how to solve this problem with <math>O(1)</math> time by perfect hashing.
| | Given a DNF formular <math>\phi</math> as the input, the problem is to count the number of satisfying assignments of <math>\phi</math>. This problem is [http://en.wikipedia.org/wiki/Sharp-P-complete '''#P-complete''']. |
|
| |
|
| == Perfect hashing using quadratic space==
| | Naively applying the Monte Carlo method will not give a good answer. Suppose that there are <math>n</math> variables. Let <math>U=\{\mathrm{true},\mathrm{false}\}^n</math> be the set of all truth assignments of the <math>n</math> variables. Let <math>G=\{x\in U\mid \phi(x)=\mathrm{true}\}</math> be the set of satisfying assignments for <math>\phi</math>. The straightforward use of Monte Carlo method samples <math>N</math> assignments from <math>U</math> and check how many of them satisfy <math>\phi</math>. This algorithm fails when <math>|G|/|U|</math> is exponentially small, namely, when exponentially small fraction of the assignments satisfy the input DNF formula. |
| The idea of perfect hashing is that we use a hash function <math>h</math> to map the <math>n</math> items to distinct entries of the table; store every item <math>x\in S</math> in the entry <math>h(x)</math>; and also store the hash function <math>h</math> in a fixed location in the table (usually the beginning of the table). The algorithm for searching for an item is as follows:
| |
|
| |
|
| :search for <math>x</math> in table <math>T</math>:
| |
| # retrieve <math>h</math> from a fixed location in the table;
| |
| # if <math>x=T[h(x)]</math> return <math>h(x)</math>; else return NOT_FOUND;
| |
|
| |
| This scheme works as long as that the hash function satisfies the following two conditions:
| |
| * The description of <math>h</math> is sufficiently short, so that <math>h</math> can be stored in one entry (or in constant many entries) of the table.
| |
| * <math>h</math> has no collisions on <math>S</math>, i.e. there is no pair of items <math>x_1,x_2\in S</math> that are mapped to the same value by <math>h</math>.
| |
|
| |
| The first condition is easy to guarantee for 2-universal hash families. As shown by Carter-Wegman construction, a 2-universal hash function can be uniquely represented by two integers <math>a</math> and <math>b</math>, which can be stored in two entries (or just one, if the word length is sufficiently large) of the table.
| |
|
| |
| Our discussion is now focused on the second condition. We find that it relies on the ''perfectness'' of the hash function for a data set <math>S</math>.
| |
|
| |
| A hash function <math>h</math> is '''perfect''' for a set <math>S</math> of items if <math>h</math> maps all items in <math>S</math> to different values, i.e. there is no collision.
| |
|
| |
| We have shown by the birthday problem for 2-universal hashing that when <math>n</math> items are mapped to <math>n^2</math> values, for an <math>h</math> chosen uniformly from a 2-universal family of hash functions, the probability that a collision occurs is at most 1/2. Thus
| |
| :<math>
| |
| \Pr[h\mbox{ is perfect for }S]\ge\frac{1}{2}
| |
| </math>
| |
| for a table of <math>n^2</math> entries.
| |
|
| |
|
| The construction of perfect hashing is straightforward then: | | ;The union of sets problem |
| :For a set <math>S</math> of <math>n</math> elements:
| | We reformulate the DNF counting problem in a more abstract framework, called the '''union of sets''' problem. |
| # uniformly choose an <math>h</math> from a 2-universal family <math>\mathcal{H}</math>; (for Carter-Wegman's construction, it means uniformly choose two integer <math>1\le a\le p-1</math> and <math>b\in[p]</math> for a sufficiently large prime <math>p</math>.)
| |
| # check whether <math>h</math> is perfect for <math>S</math>;
| |
| # if <math>h</math> is NOT perfect for <math>S</math>, start over again; otherwise, construct the table;
| |
|
| |
|
| This is a Las Vegas randomized algorithm, which construct a perfect hashing for a fixed set <math>S</math> with expectedly at most two trials (due to geometric distribution). The resulting data structure is a <math>O(n^2)</math>-size static dictionary of <math>n</math> elements which answers every search in deterministic <math>O(1)</math> time.
| | Let <math>V</math> be a finite universe. We are given <math>m</math> subsets <math>H_1,H_2,\ldots,H_m\subseteq V</math>. The following assumptions hold: |
| | *For all <math>i</math>, <math>|H_i|</math> is computable in poly-time. |
| | *It is possible to sample uniformly from each individual <math>H_i</math>. |
| | *For any <math>x\in V</math>, it can be determined in poly-time whether <math>x\in H_i</math>. |
|
| |
|
| == FKS perfect hashing ==
| | The goal is to compute the size of <math>H=\bigcup_{i=1}^m H_i</math>. |
| In the last section we see how to use <math>O(n^2)</math> space and constant time for answering search in a set. Now we see how to do it with linear space and constant time. This solves the problem of searching asymptotically optimal for both time and space.
| |
|
| |
|
| This was once seemingly impossible, until Yao's seminal paper:
| | DNF counting can be interpreted in this general framework as follows. Suppose that the DNF formula <math>\phi</math> is defined on <math>n</math> variables, and <math>\phi</math> contains <math>m</math> clauses <math>C_1,C_2,\ldots,C_m</math>, where clause <math>C_i</math> has <math>k_i</math> literals. Without loss of generality, we assume that in each clause, each variable appears at most once. |
| *Yao. Should tables be sorted? ''Journal of the ACM (JACM)'', 1981. | | * <math>V</math> is the set of all assignments. |
| | *Each <math>H_i</math> is the set of satisfying assignments for the <math>i</math>-th clause <math>C_i</math> of the DNF formular <math>\phi</math>. Then the union of sets <math>H=\bigcup_i H_i</math> gives the set of satisfying assignments for <math>\phi</math>. |
| | * Each clause <math>C_i</math> is a conjunction (AND) of literals. It is not hard to see that <math>|H_i|=2^{n-k_i}</math>, which is efficiently computable. |
| | * Sampling from an <math>H_i</math> is simple: we just fix the assignments of the <math>k_i</math> literals of that clause, and sample uniformly and independently the rest <math>(n-k_i)</math> variable assignments. |
| | * For each assignment <math>x</math>, it is easy to check whether it satisfies a clause <math>C_i</math>, thus it is easy to determine whether <math>x\in H_i</math>. |
|
| |
|
| Yao's paper shows a possibility of achieving linear space and constant time at the same time by exploiting the power of hashing, but assumes an unrealistically large universe.
| | ==The coverage algorithm== |
| | We now introduce the coverage algorithm for the union of sets problem. |
|
| |
|
| Inspired by Yao's work, Fredman, Komlós, and Szemerédi discover the first linear-space and constant-time static dictionary in a realistic setting:
| | Consider the multiset <math>U</math> defined by |
| * Fredman, Komlós, and Szemerédi. Storing a sparse table with O(1) worst case access time. ''Journal of the ACM (JACM)'', 1984.
| | :<math>U=H_1\uplus H_2\uplus\cdots \uplus H_m</math>, |
| | where <math>\uplus</math> denotes the multiset union. It is more convenient to define <math>U</math> as the set |
| | :<math>U=\{(x,i)\mid x\in H_i\}</math>. |
| | For each <math>x\in H</math>, there may be more than one instances of <math>(x,i)\in U</math>. We can choose a unique representative among the multiple instances <math>(x,i)\in U</math> for the same <math>x\in H</math>, by choosing the <math>(x,i)</math> with the minimum <math>i</math>, and form a set <math>G</math>. |
|
| |
|
| The idea of FKS hashing is to arrange hash table in two levels:
| | Formally, <math>G=\{(x,i)\in U\mid \forall (x,j)\in U, j\le i\}</math>. Every <math>x\in H</math> corresponds to a unique <math>(x,i)\in G</math> where <math>i</math> is the smallest among <math>x\in H_i</math>. |
| * In the first level, <math>n</math> items are hashed to <math>n</math> ''buckets'' by a 2-universal hash function <math>h</math>.
| |
| : Let <math>B_i</math> be the set of items hashed to the <math>i</math>th bucket.
| |
| * In the second level, construct a <math>|B_i|^2</math>-size perfect hashing for each bucket <math>B_i</math>.
| |
|
| |
|
| The data structure can be stored in a table. The first few entries are reserved to store the primary hash function <math>h</math>. To help the searching algorithm locate a bucket, we use the next <math>n</math> entries of the table as the "pointers" to the bucket: each entry stores the address of the first entry of the space to store a bucket. In the rest of table, the <math>n</math> buckets are stored in order, each using a <math>|B_i|^2</math> space as required by perfect hashing.
| | It is obvious that <math>G\subseteq U</math> and |
| | :<math>|G|=|H|</math>. |
|
| |
|
| ::[[File:FKS.png|600px]]
| | Therefore, estimation of <math>|H|</math> is reduced to estimation of <math>|G|</math> with <math>G\subseteq U</math>. Then <math>|G|</math> can have an <math>\epsilon</math>-approximation with probability <math>(1-\delta)</math> in poly-time, if we can uniformly sample from <math>U</math> and <math>|G|/|U|</math> is suitably small. |
|
| |
|
| It is easy to see that the search time is constant. To search for an item <math>x</math>, the algorithm does the followings:
| | An uniform sample from <math>U</math> can be implemented as follows: |
| * Retrieve <math>h</math>. | | * generate an <math>i\in\{1,2,\ldots,m\}</math> with probability <math>\frac{|H_i|}{\sum_{i=1}^m|H_i|}</math>; |
| * Retrieve the address for bucket <math>h(x)</math>. | | * uniformly sample an <math>x\in H_i</math>, and return <math>(x,i)</math>. |
| * Search by perfect hashing within bucket <math>h(x)</math>.
| |
| Each line takes constant time. So the worst-case search time is constant.
| |
|
| |
|
| We then need to guarantee that the space is linear to <math>n</math>. At the first glance, this seems impossible because each instance of perfect hashing for a bucket costs a square-size of space. We will prove that although the individual buckets use square-sized spaces, the sum of the them is still linear.
| | It is easy to see that this gives a uniform member of <math>U</math>. The above sampling procedure is poly-time because each <math>|H_i|</math> can be computed in poly-time, and sampling uniformly from each <math>H_i</math> is poly-time. |
|
| |
|
| For a fixed set <math>S</math> of <math>n</math> items, for a hash function <math>h</math> chosen uniformly from a 2-universe family which maps the items to <math>[n]</math>, called <math>n</math> ''buckets'', let <math>Y_i=|B_i|</math> be the number of items in <math>S</math> mapped to the <math>i</math>th bucket.
| | We now only need to lower bound the ratio |
| We are going to bound the following quantity: | | :<math>\alpha=\frac{|G|}{|U|}</math>. |
| :<math> | |
| Y=\sum_{i=1}^n Y_i^2.
| |
| </math> | |
| Since each bucket <math>B_i</math> use a space of <math>Y_i^2</math> for perfect hashing. <math>Y</math> gives the size of the space for storing the buckets.
| |
|
| |
|
| We will show that <math>Y</math> is related to the total number of collision pairs. (Indeed, the number of collision pairs can be computed by a degree-2 polynomial, just like <math>Y</math>.) | | We claim that |
| | :<math>\alpha\ge\frac{1}{m}</math>. |
| | It is easy to see this, because each <math>x\in H</math> has at most <math>m</math> instances of <math>(x,i)</math> in <math>U</math>, and we already know that <math>|G|=|H|</math>. |
|
| |
|
| Note that a bucket of <math>Y_i</math> items contributes <math>{Y_i\choose 2}</math> collision pairs. Let <math>X</math> be the total number of collision pairs.
| | Due to the estimator theorem, this needs <math>\frac{4m}{\epsilon^2}\ln\frac{2}{\delta}</math> uniform random samples from <math>U</math>. |
| <math>X</math> can be computed by summing over the collision pairs in every bucket:
| |
| :<math>
| |
| X=\sum_{i=1}^n{Y_i\choose 2}=\sum_{i=1}^n\frac{Y_i(Y_i-1)}{2}=\frac{1}{2}\left(\sum_{i=1}^nY_i^2-\sum_{i=1}^nY_i\right)=\frac{1}{2}\left(\sum_{i=1}^nY_i^2-n\right).
| |
| </math> | |
|
| |
|
| Therefore, the sum of squares of the sizes of buckets is related to collision number by:
| | This gives the coverage algorithm for the abstract problem of the union of sets. The DNF counting is a special case of it. |
| :<math>
| |
| \sum_{i=1}^nY_i^2=2X+n.
| |
| </math>
| |
| By our analysis of the collision number, we know that for <math>n</math> items mapped to <math>n</math> buckets, the expected number of collision pairs is: <math>\mathbf{E}[X]\le \frac{n}{2}</math>.
| |
| Thus,
| |
| :<math>
| |
| \mathbf{E}\left[\sum_{i=1}^nY_i^2\right]=\mathbf{E}[2X+n]\le 2n.
| |
| </math>
| |
| Due to Markov's inequality, <math>\sum_{i=1}^nY_i^2=O(n)</math> with a constant probability. For any set <math>S</math>, we can find a suitable <math>h</math> after expected constant number of trials, and FKS can be constructed with guaranteed (instead of expected) linear-size which answers each search in constant time.
| |