imported>Etone |
imported>Etone |
| Line 1: |
Line 1: |
| =Distinct Elements= | | {{Infobox |
| Consider the following problem of '''counting distinct elements''': Suppose that <math>\Omega</math> is a sufficiently large universe.
| | |name = Infobox |
| *'''Input:''' a sequence of (not necessarily distinct) elements <math>x_1,x_2,\ldots,x_n\in\Omega</math>;
| | |bodystyle = |
| *'''Output:''' an estimation of the total number of distinct elements <math>z=|\{x_1,x_2,\ldots,x_n\}|</math>.
| | |title = <font size=3>高级算法 |
| | <br>Advanced Algorithms</font> |
| | |titlestyle = |
|
| |
|
| A straightforward way of solving this problem is to maintain a dictionary data structure, which costs at least linear (<math>O(n)</math>) space. For ''big data'', where <math>n</math> is very large, this is still too expensive. However, due to an information-theoretical argument, linear space is necessary if you want to compute the ''exact'' value of <math>z</math>.
| | |image = |
| | |imagestyle = |
| | |caption = |
| | |captionstyle = |
| | |headerstyle = background:#ccf; |
| | |labelstyle = background:#ddf; |
| | |datastyle = |
|
| |
|
| Our goal is to relax the problem a little bit to significantly reduce the space cost by tolerating ''approximate'' answers. The form of approximation we consider is '''<math>(\epsilon,\delta)</math>-estimator'''.
| | |header1 =Instructor |
| {{Theorem|<math>(\epsilon,\delta)</math>-estimator|
| | |label1 = |
| : A random variable <math>\widehat{Z}</math> is an '''<math>(\epsilon,\delta)</math>-estimator''' of a quantity <math>z</math> if
| | |data1 = |
| ::<math>\Pr[\,(1-\epsilon)z\le \widehat{Z}\le (1+\epsilon)z\,]\ge 1-\delta</math>.
| | |header2 = |
| : <math>\widehat{Z}</math> is said to be an '''unbiased estimator''' of <math>z</math> if <math>\mathbb{E}[\widehat{Z}]=z</math>. | | |label2 = |
| | |data2 = 尹一通 |
| | |header3 = |
| | |label3 = Email |
| | |data3 = yinyt@nju.edu.cn chaodong@nju.edu.cn |
| | |header4 = |
| | |label4= office |
| | |data4= 计算机系 804 |
| | |header5 = Class |
| | |label5 = |
| | |data5 = |
| | |header6 = |
| | |label6 = Class meetings |
| | |data6 = Wednesday, 10am-12pm <br> 仙I-108 |
| | |header7 = |
| | |label7 = Place |
| | |data7 = |
| | |header8 = |
| | |label8 = Office hours |
| | |data8 = Wednesday, 4pm-6pm <br>804 |
| | |header9 = Textbooks |
| | |label9 = |
| | |data9 = |
| | |header10 = |
| | |label10 = |
| | |data10 = [[File:MR-randomized-algorithms.png|border|100px]] |
| | |header11 = |
| | |label11 = |
| | |data11 = Motwani and Raghavan. <br>''Randomized Algorithms''.<br> Cambridge Univ Press, 1995. |
| | |header12 = |
| | |label12 = |
| | |data12 = [[File:Approximation_Algorithms.jpg|border|100px]] |
| | |header13 = |
| | |label13 = |
| | |data13 = Vazirani. <br>''Approximation Algorithms''. <br> Springer-Verlag, 2001. |
| | |belowstyle = background:#ddf; |
| | |below = |
| }} | | }} |
| Usually <math>\epsilon</math> is called '''approximation error''' and <math>\delta</math> is called '''confidence error'''.
| |
|
| |
|
| We now present an elegant algorithm introduced by [https://en.wikipedia.org/wiki/Flajolet–Martin_algorithm Flajolet and Martin] in 1984. The algorithm can be implemented in [https://en.wikipedia.org/wiki/Streaming_algorithm '''data stream model''']: The input elements <math>x_1,x_2,\ldots,x_n</math> is presented to the algorithm one at a time, where the size of data <math>n</math> is unknown to the algorithm. The algorithm maintains a value <math>\widehat{Z}</math> which is an <math>(\epsilon,\delta)</math>-estimator of the total number of distinct elements <math>z=|\{x_1,x_2,\ldots,x_n\}|</math>, using only a small amount of memory space to memorize (with loss) the data set <math>\{x_1,x_2,\ldots,x_n\}</math>.
| | This is the webpage for the ''Advanced Algorithms'' class of fall 2018. Students who take this class should check this page periodically for content updates and new announcements. |
|
| |
|
| A famous quotation of Flajolet describes the performance of this algorithm as:
| | = Announcement = |
| | TBA |
|
| |
|
| "Using only memory equivalent to 5 lines of printed text, you can estimate with a typical accuracy of 5% and in a single pass the total vocabulary of Shakespeare."
| | = Course info = |
| | * '''Instructor ''': 尹一通 |
| | :*'''email''': yinyt@nju.edu.cn |
| | * '''Class meeting''': Wednesday 10am-12pm, 仙I-108. |
| | * '''Office hour''': Wednesday 4pm-6pm, 计算机系 804. |
|
| |
|
| == An estimator by hashing == | | = Syllabus = |
| Suppose that we can access to an idealized random hash function <math>h:\Omega\to[0,1]</math> which is uniformly distributed over all mappings from the universe <math>\Omega</math> to unit interval <math>[0,1]</math>.
| |
|
| |
|
| Recall that the input sequence <math>x_1,x_2,\ldots,x_n\in\Omega</math> consists of <math>z=|\{x_1,x_2,\ldots,x_n\}|</math> distinct elements. These elements are mapped by the random function <math>h</math> to <math>z</math> hash values uniformly and independently distributed in <math>[0,1]</math>. We could maintain these hash values instead of the original elements, but this would still be too expensive because in the worst case we still have up to <math>n</math> distinct values to maintain. However, due to the idealized random hash function, the unit interval <math>[0,1]</math> will be partitioned into <math>z+1</math> subintervals by these <math>z</math> uniform and independent hash values. The typical length of the subinterval gives an estimation of the number <math>z</math>.
| | === 先修课程 Prerequisites === |
| | * 必须：离散数学，概率论，线性代数。 |
| | * 推荐：算法设计与分析。 |
|
| |
|
| {{Theorem|Proposition|
| | === Course materials === |
| :<math>\mathbb{E}\left[\min_{1\le i\le n}h(x_i)\right]=\frac{1}{z+1}</math>.
| | * [[高级算法 (Fall 2019) / Course materials|<font size=3>教材和参考书</font>]] |
| }}
| |
| {{Proof|
| |
| The input sequence <math>x_1,x_2,\ldots,x_n\in\Omega</math> consisting of <math>z</math> distinct elements are mapped to <math>z</math> random hash values uniformly and independently distributed in <math>[0,1]</math>. These <math>z</math> hash values partition the unit interval <math>[0,1]</math> into <math>z+1</math> subintervals <math>[0,v_1],[v_1,v_2],[v_2,v_3]\ldots,[v_{z-1},v_z],[v_z,1]</math>, where <math>v_i</math> denotes the <math>i</math>-th smallest value among all hash values <math>\{h(x_1),h(x_2),\ldots,h(x_n)\}</math>. Clearly we have
| |
| :<math>v_1=\min_{1\le i\le n}h(x_i)</math>.
| |
| Meanwhile, since all hash values are uniformly and independently distributed in <math>[0,1]</math>, the lengths of all subintervals <math>v_1, v_2-v_1, v_3-v_2,\ldots, v_z-v_{z-1}, 1-v_z</math> are identically distributed. By symmetry, they have the same expectation, therefore
| |
| :<math>
| |
| (z+1)\mathbb{E}[v_1]=
| |
| \mathbb{E}[v_1]+\sum_{i=1}^{z-1}\mathbb{E}[v_{i+1}-v_i]+\mathbb{E}[1-v_z]
| |
| =\mathbb{E}\left[v_1+(v_2-v_1)+(v_3-v_2)+\cdots+(v_{z}-v_{z-1})+1-v_z\right]
| |
| =1,
| |
| </math>
| |
| which implies that
| |
| :<math>\mathbb{E}\left[\min_{1\le i\le n}h(x_i)\right]=\mathbb{E}[v_1]=\frac{1}{z+1}</math>.
| |
| }}
| |
| | |
| The quantity <math>\min_{1\le i\le n}h(x_i)</math> can be computed with small space cost (for storing the current smallest hash value) by scan the input sequence in a single pass. Because as we proved its expectation is <math>\frac{1}{z+1}</math>, the smallest hash value <math>Y=\min_{1\le i\le n}h(x_i)</math> gives an unbiased estimator for <math>\frac{1}{z+1}</math>. However, <math>\frac{1}{Y}-1</math> is not necessarily a good estimator for <math>z</math>. Actually, it is a rather poor estimator. Consider for example when <math>z=1</math>, all input elements are the same. In this case, there is only one hash value and <math>Y=\min_{1\le i\le n}h(x_i)</math> is distributed uniformly over <math>[0,1]</math>, thus <math>\frac{1}{Y}-1</math> fails to be close enough to the correct answer 1 with high probability.
| |
| | |
| ==Flajolet-Martin algorithm==
| |
| The reason that the above estimator of a single hash function performs poorly is that the unbiased estimator <math>\min_{1\le i\le n}h(x_i)</math> has large variance. So a natural way to reduce this variance is to have multiple independent hash functions and take the average. This is precisely what [https://en.wikipedia.org/wiki/Flajolet–Martin_algorithm '''''Flajolet-Martin algorithm'''''] does.
| |
| | |
| Suppose that we can access to <math>k</math> independent random hash functions <math>h_1,h_2,\ldots,h_k</math>, where each <math>h_j:\Omega\to[0,1]</math> is uniformly and independently distributed over all functions mapping <math>\Omega</math> to <math>[0,1]</math>. Here <math>k</math> is a parameter to be fixed by the desired approximation error <math>\epsilon</math> and confidence error <math>\delta</math>. The ''Flajolet-Martin algorithm'' is given by the following pseudocode.
| |
| | |
| {{Theorem|''Flajolet-Martin algorithm'' (Flajolet and Martin 1984)|
| |
| :Suppose that <math>h_1,h_2,\ldots,h_k:\Omega\to[0,1]</math> are <math>k</math> uniform and independent random hash functions, where <math>k</math> is a parameter to be fixed later.
| |
| -----
| |
| :Scan the input sequence <math>x_1,x_2,\ldots,x_n\in\Omega</math> in a single pass to compute:
| |
| ::* <math>Y_j=\min_{1\le i\le n}h_j(x_i)</math> for every <math>j=1,2,\ldots,k</math>;
| |
| ::* average value <math>\overline{Y}=\frac{1}{k}\sum_{j=1}^kY_j</math>;
| |
| :return <math>\widehat{Z}=\frac{1}{\overline{Y}}-1</math> as the estimator.
| |
| }}
| |
| | |
| The algorithm is easy to implement in data stream model, with a space cost of storing <math>k</math> hash values. The following theorem guarantees that the algorithm returns an <math>(\epsilon,\delta)</math>-estimator of the total number of distinct elements for a suitable <math>k=O\left(\frac{1}{\epsilon^2\delta}\right)</math>.
| |
| {{Theorem|Theorem|
| |
| :For any <math>\epsilon,\delta<1/2</math>, if <math>k\ge\left\lceil\frac{4}{\epsilon^2\delta}\right\rceil</math> then the output <math>\widehat{Z}</math> always gives an <math>(\epsilon,\delta)</math>-estimator of the correct answer <math>z</math>.
| |
| }}
| |
| | |
| In the following we prove this main theorem for Flajolet-Martin algorithm.
| |
| | |
| An obstacle to analyze the estimator <math>\widehat{Z}=\frac{1}{\overline{Y}}-1</math> is that it is a nonlinear function of <math>\overline{Y}</math> who is easier to analyze. Nevertheless, we observe that <math>\widehat{Z}</math> is an <math>(\epsilon,\delta)</math>-estimator of <math>z</math> as long as <math>\overline{Y}</math> is an <math>(\epsilon/2,\delta)</math>-estimator of <math>\frac{1}{z+1}</math>. This can be deduced by just verifying the following:
| |
| :<math>\frac{1-\epsilon/2}{z+1}\le \overline{Y}\le \frac{1+\epsilon/2}{z+1} \implies (1-\epsilon)z\le\frac{1}{\overline{Y}}-1\le (1+\epsilon)z</math>,
| |
| for <math>\epsilon<\frac{1}{2}</math>. Therefore,
| |
| :<math>\Pr\left[\,(1-\epsilon)z\le \widehat{Z} \le (1+\epsilon)z\,\right]\ge \Pr\left[\,\frac{1-\epsilon/2}{z+1}\le \overline{Y}\le \frac{1+\epsilon/2}{z+1}\,\right]
| |
| =\Pr\left[\,\left|\overline{Y}-\frac{1}{z+1}\right|\le \frac{\epsilon/2}{z+1}\,\right]</math>.
| |
| It is then sufficient to show that <math>\Pr\left[\,\left|\overline{Y}-\frac{1}{z+1}\right|\le \frac{\epsilon/2}{z+1}\,\right]\ge 1-\delta</math> for proving the main theorem above. We will see that this is equivalent to show the concentration inequality
| |
| :<math>\Pr\left[\,\left|\overline{Y}-\mathbb{E}\left[\overline{Y}\right]\right|\le \frac{\epsilon/2}{z+1}\,\right]\ge 1-\delta\quad\qquad({\color{red}*})</math>.
| |
| | |
| {{Theorem|Lemma|
| |
| :The followings hold for each <math>Y_j</math>, <math>j=1,2\ldots,k</math>, and <math>\overline{Y}=\frac{1}{k}\sum_{j=1}^kY_j</math>:
| |
| :*<math>\mathbb{E}\left[\overline{Y}\right]=\mathbb{E}\left[Y_j\right]=\frac{1}{z+1}</math>;
| |
| :*<math>\mathbf{Var}\left[Y_j\right]\le\frac{1}{(z+1)^2}</math>, and consequently <math>\mathbf{Var}\left[\overline{Y}\right]\le\frac{1}{k(z+1)^2}</math>.
| |
| }}
| |
| {{Proof|
| |
| As in the case of single hash function, by symmetry it holds that <math>\mathbb{E}[Y_j]=\frac{1}{z+1}</math> for every <math>j=1,2,\ldots,k</math>. Therefore,
| |
| :<math>\mathbb{E}\left[\overline{Y}\right]=\frac{1}{k}\sum_{j=1}^k\mathbb{E}[Y_j]=\frac{1}{z+1}</math>.
| |
| Recall that each <math>Y_j</math> is the minimum of <math>z</math> random hash values uniformly and independently distributed over <math>[0,1]</math>. By geometry probability, it holds that for any <math>y\in[0,1]</math>,
| |
| :<math>\Pr[Y_j>y]=(1-y)^z</math>,
| |
| which means <math>\Pr[Y_j\le y]=1-(1-y)^z</math>. Taking the derivative with respect to <math>y</math>, we obtain the probability density function of random variable <math>Y_j</math>, which is <math>z(1-y)^{z-1}</math>.
| |
| | |
| We then compute the second moment.
| |
| :<math>\mathbb{E}[Y_j^2]=\int^{1}_0y^2z(1-y)^{z-1}\,\mathrm{d}y=\frac{2}{(z+1)(z+2)}</math>.
| |
| The variance is bounded as
| |
| :<math>\mathbf{Var}\left[Y_j\right]=\mathbb{E}\left[Y_j^2\right]-\mathbb{E}\left[Y_j\right]^2=\frac{2}{(z+1)(z+2)}-\frac{1}{(z+1)^2}\le\frac{1}{(z+1)^2}</math>.
| |
| Due to the (pairwise) independence between <math>Y_j</math>'s,
| |
| ::<math>\mathbf{Var}\left[\overline{Y}\right]=\mathbf{Var}\left[\frac{1}{k}\sum_{j=1}^kY_j\right]=\frac{1}{k^2}\sum_{j=1}^k\mathbf{Var}\left[Y_j\right]\le \frac{1}{k(z+1)^2}</math>.
| |
| }}
| |
| | |
| We resume to prove the inequality <math>({\color{red}*})</math>. By [[高级算法_(Fall_2018)/Basic_tail_inequalities#Chebyshev.27s_inequality|Chebyshev's inequality]], it holds that
| |
| :<math>\Pr\left[\,\left|\overline{Y}-\mathbb{E}\left[\overline{Y}\right]\right|> \frac{\epsilon/2}{z+1}\,\right]
| |
| \le\frac{4}{\epsilon^2}(z+1)^2\mathbf{Var}\left[\overline{Y}\right]
| |
| \le\frac{4}{\epsilon^2k}</math>.
| |
| When <math>k\ge\left\lceil\frac{4}{\epsilon^2\delta}\right\rceil</math>, this probability is at most <math>\delta</math>. The inequality <math>({\color{red}*})</math> is proved. As we discussed above, this proves the above main theorem for Flajolet-Martin algorithm.
| |
| | |
| ==Uniform Hash Assumption (UHA)==
| |
| In above we assume we can access to idealized random hash functions <math>h:\Omega\to[0,1]</math> with real values. With a more careful calculation, one can show the same performance guarantee for hash functions with discrete values as <math>h:\Omega\to[M]</math> where <math>M=\mathrm{poly}(n)</math>, that is, the hash values are strings of <math>O(\log n)</math> bits.
| |
| | |
| Even with such improved analysis, a uniform random discrete function in form of <math>h:[N]\to[M]</math> is not really efficient to store or to compute. By an information-theretical argument, it takes at least <math>\Omega(N\log M)</math> bits to represent such a random hash function because this is the entropy of such uniform random function.
| |
| | |
| For the convenience of analysis, it is common to assume the following '''Uniform Hash Assumption (UHA)''' also known as '''Simple Uniform Hash Assumption (SUHA)'''.
| |
| {{Theorem|Uniform Hash Assumption (UHA)|
| |
| :A ''uniform'' random function <math>h:[N]\rightarrow[M]</math> is available and the computation of <math>h</math> is efficient.
| |
| }}
| |
| | |
| = Set Membership=
| |
| A basic question in Computer Science is:
| |
| :"<math>\mbox{Is }x\in S?</math>"
| |
| for a set <math>S</math> and an element <math>x</math>. This is the '''set membership''' problem.
| |
| | |
| Formally, given an arbitrary set <math>S</math> of <math>n</math> elements from a universe <math>\Omega</math>, we want to use a succinct '''data structure''' to represent this set <math>S</math>, so that upon each '''query''' of any element <math>x</math> from the universe <math>[N]</math>, the question of whether <math>x\in S</math> is efficiently answered. The complexity of such data structure is measured in two-fold:
| |
| * '''space cost''': size of the data structure to represent a set <math>S</math> of size <math>n</math>;
| |
| * '''time cost''': time complexity of answering each query by accessing to the data structure.
| |
| | |
| Suppose that the universe <math>\Omega</math> is of size <math>N</math>. Clearly, the membership problem can be solved by a '''dictionary data structure''', e.g.:
| |
| * '''sorted table / balanced search tree''': with space cost <math>O(n\log N)</math> bits and time cost <math>O(\log n)</math>;
| |
| * '''perfect hashing''' of ''Fredman, Komlós & Szemerédi'': with space cost <math>O(n\log N)</math> bits and time cost <math>O(1)</math>.
| |
| | |
| Note that <math>\log{N\choose n}=\Theta\left(n\log \frac{N}{n}\right)</math> is the entropy of sets <math>S</math> of <math>n</math> elements from a universe <math>\Omega</math> of size <math>N</math>. Therefore it is necessary to use so many bits to represent a set without losing any information. Nevertheless, we can do better than this if we use a loss representation of the input set <math>S</math> and tolerate a bounded error in answering queries. Such lossy representation of data is sometimes called a '''''sketch'''''.
| |
| | |
| == Bloom filter ==
| |
| The Bloom filter is a space-efficient hash table that solves the '''approximate membership''' problem with one-sided error (''false positive'').
| |
|
| |
|
| Given a set <math>S</math> of <math>n</math> elements from a universe <math>\Omega</math>, a Bloom filter consists of an array <math>A</math> of <math>cn</math> bits, and <math>k</math> hash functions <math>h_1,h_2,\ldots,h_k</math> map <math>\Omega</math> to <math>[cn]</math>, where both <math>c</math> and <math>k</math> are parameters that we can try to optimize later.
| | === 成绩 Grades === |
| | * 课程成绩：本课程将会有若干次作业和一次期末考试。最终成绩将由平时作业成绩和期末考试成绩综合得出。 |
| | * 迟交：如果有特殊的理由，无法按时完成作业，请提前联系授课老师，给出正当理由。否则迟交的作业将不被接受。 |
|
| |
|
| As before, we assume the '''Uniform Hash Assumption (UHA)''': <math>h_1,h_2,\ldots,h_k</math> are mutually independent hash function where each <math>h_i</math> is a uniform random hash function <math>h_i:\Omega\to[cn]</math>.
| | === <font color=red> 学术诚信 Academic Integrity </font>=== |
| | 学术诚信是所有从事学术活动的学生和学者最基本的职业道德底线，本课程将不遗余力的维护学术诚信规范，违反这一底线的行为将不会被容忍。 |
|
| |
|
| The Bloom filter works as follows:
| | 作业完成的原则：署你名字的工作必须由你完成。允许讨论，但作业必须独立完成，并在作业中列出所有参与讨论的人。不允许其他任何形式的合作——尤其是与已经完成作业的同学“讨论”。 |
| {{Theorem|''Bloom filter'' (Bloom 1970)|
| |
| :Suppose <math>h_1,h_2,\ldots,h_k:\Omega\to[cn]</math> are uniform and independent random hash functions.
| |
| -----
| |
| :'''Data structure construction:''' Given a set <math>S\subset\Omega</math> of size <math>n=|S|</math>, the data structure is a Boolean array <math>A</math> of <math>cn</math> bits constructed as
| |
| :* initialize all <math>cn</math> bits of the Boolean array <math>A</math> to 0;
| |
| :* for each <math>x\in S</math>, let <math>A[h_i(x)]=1</math> for all <math>1\le i\le k</math>.
| |
| ----
| |
| :'''Query resolution:''' Upon each query of an arbitrary <math>x\in\Omega</math>,
| |
| :* answer "yes" if <math>A[h_i(x)]=1</math> for all <math>1\le i\le k</math> and "no" if otherwise.
| |
| }}
| |
| The Boolean array is our data structure, whose size is <math>cn</math> bits. With Uniform Hash Assumption (UHA), the time cost of the data structure for answering each query is <math>O(k)</math>.
| |
|
| |
|
| When the answer returned by the algorithm is "no", it holds that <math>A[h_i(x)]=0</math> for some <math>1\le i\le k</math>, in which case the query <math>x</math> must not belong to the set <math>S</math>. Thus, the Bloom filter has no false negatives.
| | 本课程将对剽窃行为采取零容忍的态度。在完成作业过程中，对他人工作（出版物、互联网资料、其他人的作业等）直接的文本抄袭和对关键思想、关键元素的抄袭，按照 [http://www.acm.org/publications/policies/plagiarism_policy ACM Policy on Plagiarism]的解释，都将视为剽窃。剽窃者成绩将被取消。如果发现互相抄袭行为，<font color=red> 抄袭和被抄袭双方的成绩都将被取消</font>。因此请主动防止自己的作业被他人抄袭。 |
|
| |
|
| On the other hand, when the answer returned by the algorithm is "yes", <math>A[h_i(x)]=1</math> for all <math>1\le i\le k</math>. It is still possible for some <math>x\not\in S</math> that all bits <math>A[h_i(x)]</math> are set by elements in <math>S</math>. We want to bound such false positive, that is, the following probability for an <math>x\not\in S</math>:
| | 学术诚信影响学生个人的品行，也关乎整个教育系统的正常运转。为了一点分数而做出学术不端的行为，不仅使自己沦为一个欺骗者，也使他人的诚实努力失去意义。让我们一起努力维护一个诚信的环境。 |
| :<math>\Pr[\,\forall 1\le i\le k, A[h_i(x)]=1\,]</math>,
| |
| which by independence between different hash functions and by symmetry is equal to:
| |
| :<math>\Pr[\, A[h_1(x)]=1\,]^k=(1-\Pr[\, A[h_1(x)]=0\,])^k</math>.
| |
| For an element <math>x\not\in S</math>, its hash value <math>h_1(x)</math> is independent of all hash values <math>h_i(y)</math> for all <math>1\le i\le k</math> and all <math>y\in S</math>. This is due to the Uniform Hash Assumption. The hash value <math>h_1(x)</math> of <math>x\not\in S</math> is then independent of the content of the array <math>A</math>. Therefore, the probability of this position <math>A[h_1(x)]</math> missed by all <math>kn</math> updates to the Boolean array <math>A</math> caused by all <math>n</math> elements in <math>S</math> is:
| |
| :<math>
| |
| \Pr[\, A[h_1(x)]=0\,]=\left(1-\frac{1}{cn}\right)^{kn}\approx e^{-k/c}.
| |
| </math>
| |
|
| |
|
| Putting everything together, for any <math>x\not\in S</math>, the false positive is bounded as:
| | = Assignments = |
| :<math>
| | TBA |
| \begin{align}
| |
| \Pr[\,\text{wrongly answer ''yes''}\,]
| |
| &=\Pr[\,\forall 1\le i\le k, A[h_i(x)]=1\,]\\
| |
| &=\Pr[\, A[h_1(x)]=1\,]^k=(1-\Pr[\, A[h_1(x)]=0\,])^k\\
| |
| &=\left(1-\left(1-\frac{1}{cn}\right)^{kn}\right)^k\\
| |
| &\approx \left(1- e^{-k/c}\right)^k
| |
| \end{align}
| |
| </math>
| |
| which is <math>(0.6185)^c</math> when <math>k=c\ln 2</math>.
| |
|
| |
|
| Bloom filter solves the membership query with a small constant error of false positives using a data structure of <math>O(n)</math> bits which answers each query with <math>O(1)</math> time cost.
| | = Lecture Notes = |
| | # [[高级算法 (Fall 2019)/Min-Cut and Max-Cut|Min-Cut and Max-Cut]] |
| | #: [[高级算法 (Fall 2019)/Probability Basics|Probability basics]] |
|
| |
|
| = Frequency Estimation=
| |
| Suppose that <math>\Omega</math> is the data universe. The '''frequency estimation''' problem is defined as follows.
| |
| *'''Data:''' a sequence of (not necessarily distinct) elements <math>x_1,x_2,\ldots,x_n\in\Omega</math>;
| |
| *'''Query:''' an element <math>x\in\Omega</math>;
| |
| *'''Output:''' an estimation <math>\hat{f}_x</math> of the frequency <math>f_x\triangleq|\{i\mid x_i=x\}|</math> of <math>x</math> in input data.
| |
|
| |
|
| We still want to give an algorithm in the data stream model: the algorithm scan the input sequence <math>x_1,x_2,\ldots,x_n</math> to construct a succinct data structure, such that upon each query of <math>x\in\Omega</math>, the algorithm returns an estimation of the frequency <math>f_x</math>.
| | = Related Online Courses= |
| | | * Ankur Moitra's [http://people.csail.mit.edu/moitra/854.html Advanced Algorithms] at MIT. |
| Clearly this problem can always be solved by storing all appeared distinct elements along with their frequencies. However, the space cost of this straightforward solution is rather high. Instead, we want to use a lossy representation (a ''sketch'') of input data which uses significantly less space but can still answer queries with tolarable accuracy.
| |
| | |
| Formally, upon each query of <math>x\in\Omega</math>, the algorithm should return an answer <math>\hat{f}_x</math> satisfying:
| |
| :<math>\Pr\left[\,\left|\hat{f}_x-f_x\right|\le \epsilon n\,\right]\ge 1-\delta</math>.
| |
| Note that this notion of approximation is with bounded ''additive'' error which is weaker than the notion of <math>(\epsilon,\delta)</math>-estimator, whose error bound is ''multiplicative''.
| |
| | |
| With such weak accuracy guarantee, its is possible to give a succinct data structure whose size is determined only by the error bounds <math>\epsilon</math> and <math>\delta</math> but independent of <math>n</math>, because only the frequencies of those '''heavy hitters''' (elements <math>x</math> with high frequencies <math>f_x>\epsilon n</math>) need to be memorized, and there are at most <math>1/\epsilon</math> many such heavy hitters.
| |
| | |
| == Count-min sketch==
| |
| {{Theorem|''Count-min sketch'' (Cormode and Muthukrishnan 2003)|
| |
| :Suppose <math>h_1,h_2,\ldots,h_k:\Omega\to[m]</math> are uniform and independent random hash functions.
| |
| -----
| |
| :'''Data structure construction:''' Given a sequence <math>x_1,x_2,\ldots,x_n\in\Omega</math>, the data structure is a two-dimensional <math>k\times m</math> integer array <math>CMS[k][m]</math> constructed as | |
| :*initialize all entries of <math>CMS[k][m]</math> to 0.
| |
| :*for <math>i=1,2,\ldots,n</math>, upon receiving <math>x_i</math>:
| |
| ::: for every <math>1\le j\le k</math>, evaluate <math>h_j(x_i)</math> and <math>CMS[j][h_j(x_i)]++</math>.
| |
| ----
| |
| :'''Query resolution:''' Upon each query of an arbitrary <math>x\in\Omega</math>,
| |
| :* return <math>\hat{f}=\min_{1\le j\le k}CMS[j][h_j(x)]</math>.
| |
| }}
| |
| | |
| It is easy to see that the space cost of count-min sketch is <math>O(km)</math> integers, which is <math>O(km\log n)</math> bits. Each query is answered within time cost <math>O(k)</math>. We then analyze the error bounds.
| |
| | |
| First, it is easy to observe that for any query <math>x\in\Omega</math> and every hash function <math>1\le j\le k</math>, it always holds for the corresponding entry in the count-min sketch
| |
| :<math>CMS[j][h_j(x)]\ge f_x</math>,
| |
| because the appearances of element <math>x</math> in the input sequence contribute at least <math>f_x</math> to the value of <math>CMS[j][h_j(x)]</math>.
| |
| | |
| Therefore, for any query <math>x\in\Omega</math> it always holds for the answer <math>\hat{f}=\min_{1\le j\le k}CMS[j][h_j(x)]\ge f_x</math>, which means
| |
| :<math>\Pr\left[\,\left|\hat{f}- f_x\right|\le\epsilon n\,\right]=\Pr\left[\,\hat{f}- f_x\le\epsilon n\,\right]</math>.
| |
| | |
| Moreover, we have the following proposition for the expected value of <math>CMS[j][h_j(x)]</math>.
| |
| {{Theorem|Proposition|
| |
| :For any <math>x\in\Omega</math> and every <math>1\le j\le k</math>, it holds that <math>\mathbb{E}\left[CMS[j][h_j(x)]\right]\le f_x+\frac{n}{m}</math>.
| |
| }}
| |
| {{Proof|
| |
| The value of <math>CMS[j][h_j(x)]</math> is constituted by the frequency <math>f_x</math> of <math>x</math> and the frequencies <math>f_y</math> of all other elements <math>y\neq x</math> among <math>x_1,x_2,\ldots,x_n</math>, thus
| |
| :<math>
| |
| \begin{align}
| |
| CMS[j][h_j(x)]
| |
| &=f_x+\sum_{\scriptstyle y\in\{x_1,x_2,\ldots,x_n\}\setminus\{x\}\atop\scriptstyle h_j(y)=h_j(x)} f_y\\
| |
| &=f_x+\sum_{y\in\{x_1,x_2,\ldots,x_n\}\setminus\{x\}} f_y \cdot I[h_j(y)=h_j(x)]
| |
| \end{align}
| |
| </math>
| |
| where <math>I[h_j(y)=h_j(x)]</math> denotes the Boolean random variable that indicates the occurrence of event <math>h_j(y)=h_j(x)</math>.
| |
| | |
| By linearity of expectation,
| |
| :<math>\mathbb{E}[CMS[j][h_j(x)]]=f_x+\sum_{y\in\{x_1,x_2,\ldots,x_n\}\setminus\{x\}} f_y \cdot \Pr[h_j(y)=h_j(x)]</math>.
| |
| Due to Uniform Hash Assumption (UHA), <math>h_j:\Omega\to[m]</math> is a uniform random function. For any <math>y\neq x</math>, the probability of hash collision is
| |
| :<math>\Pr[h_j(y)=h_j(x)]=\frac{1}{m}</math>.
| |
| Therefore,
| |
| :<math>
| |
| \begin{align}
| |
| \mathbb{E}[CMS[j][h_j(x)]]
| |
| &=f_x+\frac{1}{m}\sum_{y\in\{x_1,x_2,\ldots,x_n\}\setminus\{x\}} f_y \\
| |
| &\le f_x+\frac{1}{m}\sum_{y\in\{x_1,x_2,\ldots,x_n\}} f_y\\
| |
| &=f_x+\frac{n}{m},
| |
| \end{align}
| |
| </math>
| |
| where the last equation is due to the obvious identity <math>\sum_{y\in\{x_1,x_2,\ldots,x_n\}}f_y=n</math>.
| |
| }}
| |
