TCS Wiki - User contributions [en]

Combinatorics (Fall 2010)/Duality, Matroid

2011-01-03T10:12:45Z

172.21.9.33: /* LP duality */

== Duality ==

Consider the following LP:
:<math>
\begin{align}
\text{minimize} && 7x_1+x_2+5x_3\\
\text{subject to} &&
x_1-x_2+3x_3 &\ge 10\\
&&
5x_1-2x_2-x_3 &\ge 6\\
&& x_1,x_2,x_3 &\ge 0
\end{align}
</math>

Let <math>OPT</math> be the value of the optimal solution. We want to estimate the upper and lower bound of <math>OPT</math>.

Since <math>OPT</math> is the minimum over the feasible set, every feasible solution forms an upper bound for <math>OPT</math>. For example <math>\boldsymbol{x}=(2,1,3)</math> is a feasible solution, thus <math>OPT\le 7\cdot 2+1+5\cdot 3=30</math>.

For the lower bound, all feasible solution <math>\boldsymbol{x}</math> satisfies the two constraints:
:<math>
\begin{align}
x_1-x_2+3x_3 &\ge 10,\\
5x_1-2x_2-x_3 &\ge 6.\\
\end{align}
</math>
Since the <math>\boldsymbol{x}</math> is restricted to be nonnegative, term-by-term comparison of coefficients shows that
:<math>7x_1+x_2+5x_3\ge(x_1-x_2+3x_3)+(5x_1-2x_2-x_3)\ge 16.</math>
The idea behind this lower bound process is that we are finding suitable nonnegative multipliers (in the above case the multipliers are all 1s) for the constraints so that when we take their sum, the coefficient of each <math>x_i</math> in the sum is dominated by the coefficient in the objective function. It is important to ensure that the multipliers are nonnegative, so they do not reverse the direction of the constraint inequality.

To find the best lower bound, we need to choose the multipliers in such a way that the sum is as large as possible. Interestingly, the problem of finding the best lower bound can be formulated as another LP:

::<math>
\begin{align}
\text{maximize} && 10y_1+6y_2\\
\text{subject to} &&
y_1+5y_2 &\le 7\\
&&
-y_1+2y_2 &\le 1\\
&&3y_1-y_2 &\le 5\\
&& y_1,y_2&\ge 0
\end{align}
</math>

Here <math>y_1</math> and <math>y_2</math> were chosen to be nonnegative multipliers for the first and the second constraint, respectively. We call the first LP the '''primal program''' and the second LP the '''dual program'''. By definition, every feasible solution to the dual program gives a lower bound for the primal program.

=== LP duality ===
Given an LP in canonical form, called the '''primal''' LP:
:<math>
\begin{align}
\text{minimize} && \boldsymbol{c}^T\boldsymbol{x}\\
\text{subject to} &&
A\boldsymbol{x} &\ge\boldsymbol{b}\\
&& \boldsymbol{x} &\ge \boldsymbol{0}
\end{align}
</math>

the '''dual''' LP is defined as follows:
:<math>
\begin{align}
\text{maximum} && \boldsymbol{b}^T\boldsymbol{y}\\
\text{subject to} &&
A^T\boldsymbol{y} &\ge\boldsymbol{c}\\
&& \boldsymbol{y} &\ge \boldsymbol{0}
\end{align}
</math>

;Surviving problem (diet problem)
Let us consider the surviving problem. Suppose we have <math>n</math> types of natural food, each containing up to <math>m</math> types of vitamins. The <math>j</math>th food has <math>a_{ij}</math> amount of vitamin <math>i</math>, and the price of the <math>j</math>th food is <math>c_j</math>. We need to consume <math>b_i</math> amount of vitamin <math>i</math> for each <math>1\le i\le m</math> to keep a good health. We want to minimize the total costs of food while keeping healthy. The problem can be formalized as the following LP:
:<math>
\begin{align}
\text{minimize} \quad& c_1x_1+c_2x_2+\cdots+c_nx_n\\
\begin{align}
\text{subject to} \\
\\
\end{align}
\quad &
\begin{align} a_{i1}x_{1}+a_{i2}x_{2}+\cdots+a_{in}x_{n} &\le b_{i} &\quad& \forall 1\le i\le m\\
x_{j}&\ge 0 &\quad& \forall 1\le j\le n
\end{align}
\end{align}
</math>
The dual LP is
:<math>
\begin{align}
\text{maximize} \quad& b_1y_1+b_2y_2+\cdots+b_ny_m\\
\begin{align}
\text{subject to} \\
\\
\end{align}
\quad &
\begin{align} a_{1j}y_{1}+a_{2j}y_{2}+\cdots+a_{mj}y_{m} &\le c_{j} &\quad& \forall 1\le j\le n\\
y_{i}&\ge 0 &\quad& \forall 1\le i\le m
\end{align}
\end{align}
</math>
The problem can be interpreted as follows: A food company produces <math>m</math> types of vitamin pills. The company wants to design a pricing system such that
* The vitamin <math>i</math> has a nonnegative price <math>y_i</math>.
* The price system should be competitive to any natural food. A costumer cannot replace the vitamins by any natural food and get a cheaper price, that is, <math>\sum_{i=1}^my_ja_{ij}\le c_j</math> for any <math>1\le j\le n</math>.
* The company wants to find the maximal profit, assuming that the customer only buy exactly the necessary amount of vitamins (<math>b_i</math> for vitamin <math>i</math>).

;Maximum flow problem
In the last lecture, we defined the maximum flow problem, whose LP is
:<math>
\begin{align}
\text{maximize} \quad& \sum_{v:(s,v)\in E}f_{sv}\\
\begin{align}
\text{subject to} \\
\\
\\
\\
\end{align}
\quad &
\begin{align} f_{uv}&\le c_{uv} &\quad& \forall (u,v)\in E\\
\sum_{u:(u,v)\in E}f_{uv}-\sum_{w:(v,w)\in E}f_{vw} &=0 &\quad& \forall v\in V\setminus\{s,t\}\\
f_{uv}&\ge 0 &\quad& \forall (u,v)\in E
\end{align}
\end{align}
</math>
where directed graph <math>G(V,E)</math> is the flow network, <math>s\in V</math> is the source, <math>t\in V</math> is the sink, and <math>c_{uv}</math> is the capacity of directed edge <math>(u,v)\in E</math>.

We add a new edge from <math>t</math> to <math>s</math> to <math>E</math>, and let the capacity be <math>c_{ts}=\infty</math>. Let <math>E'</math> be the new edge set. The LP for the max-flow problem can be rewritten as:
:<math>
\begin{align}
\text{maximize} \quad& f_{ts}\\
\begin{align}
\text{subject to} \\
\\
\\
\\
\end{align}
\quad &
\begin{align} f_{uv}&\le c_{uv} &\quad& \forall (u,v)\in E\\
\sum_{u:(u,v)\in E'}f_{uv}-\sum_{w:(v,w)\in E'}f_{vw} &\le0 &\quad& \forall v\in V\\
f_{uv}&\ge 0 &\quad& \forall (u,v)\in E'
\end{align}
\end{align}
</math>
The second set of inequalities seem weaker than the original conservation constraint of flows, however, if this inequality holds at every node, then in fact it must be satisfied with equality at every node, thereby implying the flow conservation.

To obtain the dual program we introduce variables <math>d_{uv}</math> and <math>p_v</math> corresponding to the two types of inequalities in the primal. The dual LP is:
:<math>
\begin{align}
\text{minimize} \quad& \sum_{(u,v)\in E}c_{uv}d_{uv}\\
\begin{align}
\text{subject to} \\
\\
\\
\\
\end{align}
\quad &
\begin{align} d_{uv}-p_u+p_v &\ge 0 &\quad& \forall (u,v)\in E\\
p_s-p_t &\ge1 \\
d_{uv} &\ge 0 &\quad& \forall (u,v)\in E\\
p_v&\ge 0 &\quad& \forall v\in V
\end{align}
\end{align}
</math>
It is more helpful to consider its integer version:
:<math>
\begin{align}
\text{minimize} \quad& \sum_{(u,v)\in E}c_{uv}d_{uv}\\
\begin{align}
\text{subject to} \\
\\
\\
\\
\end{align}
\quad &
\begin{align} d_{uv}-p_u+p_v &\ge 0 &\quad& \forall (u,v)\in E\\
p_s-p_t &\ge1 \\
d_{uv} &\in\{0,1\} &\quad& \forall (u,v)\in E\\
p_v&\in\{0,1\} &\quad& \forall v\in V
\end{align}
\end{align}
</math>
In the last lecture, we know that the LP for max-flow is totally unimordular, so is this dual LP, therefore the optimal solutions to the integer program are the optimal solutions to the LP.

The variables <math>p_v</math> defines a bipartition of vertex set <math>V</math>. Let <math>S=\{v\in V\mid p_v=1\}</math>. The complement <math>\bar{S}=\{v\in V\mid p_v=1\}</math>.

For 0/1-valued variables, the only way to satisfy <math>p_s-p_t\ge1</math> is to have <math>p_s=1</math> and <math>p_t=0</math>. Therefore, <math>(S,\bar{S})</math> is an <math>s</math>-<math>t</math> cut.

In an optimal solution, <math>d_{uv}=1</math> if and only if <math>u\in S,v\in\bar{S}</math> and <math>(u,v)\in E</math>. Therefore, the objective function of an optimal solution <math>\sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math> is the capacity of the minimum <math>s</math>-<math>t</math> cut <math>(S,\bar{S})</math>.

=== Duality theorems ===

== Matroid ==

=== Kruskal's greedy algorithm for MST ===

=== Matroids ===
Let <math>X</math> be a finite set and <math>\mathcal{F}\subseteq 2^X</math> be a family of subsets of <math>X</math>. A member set <math>S\in\mathcal{F}</math> is called '''maximal''' if <math>S\cup\{x\}\not\in\mathcal{F}</math> for any <math>x\in X\setminus S</math>.

For <math>Y\subseteq X</math>, denote <math>\mathcal{F}_Y=\{S\in\mathcal{F}\mid S\subseteq Y\}</math>. Clearly <math>\mathcal{F}_Y</math> is the restriction of <math>\mathcal{F}</math> over <math>2^Y\,</math>.

{{Theorem|Definition|
:A set system <math>\mathcal{F}\subseteq 2^X</math> is a '''matroid''' if it satisfies:
:*(hereditary) if <math>T\subseteq S\in\mathcal{F}</math> then <math>T\in\mathcal{F}</math>;
:*(matroid property) for every <math>Y\subseteq X</math>, all maximal <math>S\in\mathcal{F}_Y</math> have the same <math>|S|</math>.
}}

Suppose <math>\mathcal{F}</math> is a matroid. Some matroid terminologies:
* Each member set <math>S\in\mathcal{F}</math> is called an '''independent set'''.
* A maximal independent subset of a set <math>Y\subset X</math>, i.e., a maximal <math>S\in\mathcal{F}_Y</math>, is called a '''basis''' of <math>Y</math>.
* The size of the maximal <math>S\in\mathcal{F}_Y</math> is called the '''rank''' of <math>Y</math>, denoted <math>r(Y)</math>.

==== Graph matroids ====

==== Linear matroids ====

=== Greedy algorithms on weighted matroids ===

=== Matroid intersections ===

Combinatorics (Fall 2010)/Duality, Matroid

2011-01-03T09:34:42Z

172.21.9.33: /* LP duality */

Combinatorics (Fall 2010)/Duality, Matroid

2011-01-03T09:33:25Z

172.21.9.33: /* LP duality */

Combinatorics (Fall 2010)/Duality, Matroid

2011-01-03T08:51:23Z

172.21.9.33: /* LP duality */

Combinatorics (Fall 2010)/Duality, Matroid

2011-01-03T08:30:56Z

172.21.9.33: /* Duality */

Combinatorics (Fall 2010)/Duality, Matroid

2011-01-03T08:25:56Z

172.21.9.33: /* Duality */

Combinatorics (Fall 2010)/Duality, Matroid

2011-01-03T07:48:01Z

172.21.9.33:

== Duality ==

Consider the following LP:
:<math>
\begin{align}
\text{maximize} && 7x_1+x_2+5x_3\\
\text{subject to} &&
x_1-x_2+3x_3 &\ge 10\\
&&
5x_1-2x_2-x_3 &\ge 6\\
&& x_1,x_2,x_3 &\ge 0
\end{align}
</math>

Let <math>OPT</math> be the value of the optimal solution.

=== LP duality ===

=== Duality theorems ===

== Matroid ==

=== Kruskal's greedy algorithm for MST ===

=== Matroids ===
Let <math>X</math> be a finite set and <math>\mathcal{F}\subseteq 2^X</math> be a family of subsets of <math>X</math>. A member set <math>S\in\mathcal{F}</math> is called '''maximal''' if <math>S\cup\{x\}\not\in\mathcal{F}</math> for any <math>x\in X\setminus S</math>.

For <math>Y\subseteq X</math>, denote <math>\mathcal{F}_Y=\{S\in\mathcal{F}\mid S\subseteq Y\}</math>. Clearly <math>\mathcal{F}_Y</math> is the restriction of <math>\mathcal{F}</math> over <math>2^Y\,</math>.

{{Theorem|Definition|
:A set system <math>\mathcal{F}\subseteq 2^X</math> is a '''matroid''' if it satisfies:
:*(hereditary) if <math>T\subseteq S\in\mathcal{F}</math> then <math>T\in\mathcal{F}</math>;
:*(matroid property) for every <math>Y\subseteq X</math>, all maximal <math>S\in\mathcal{F}_Y</math> have the same <math>|S|</math>.
}}

Suppose <math>\mathcal{F}</math> is a matroid. Some matroid terminologies:
* Each member set <math>S\in\mathcal{F}</math> is called an '''independent set'''.
* A maximal independent subset of a set <math>Y\subset X</math>, i.e., a maximal <math>S\in\mathcal{F}_Y</math>, is called a '''basis''' of <math>Y</math>.
* The size of the maximal <math>S\in\mathcal{F}_Y</math> is called the '''rank''' of <math>Y</math>, denoted <math>r(Y)</math>.

==== Graph matroids ====

==== Linear matroids ====

=== Greedy algorithms on weighted matroids ===

=== Matroid intersections ===

Combinatorics (Fall 2010)/Duality, Matroid

2011-01-03T07:47:47Z

172.21.9.33: /* Duality */

Consider the following LP:
:<math>
\begin{align}
\text{maximize} && 7x_1+x_2+5x_3\\
\text{subject to} &&
x_1-x_2+3x_3 &\ge 10\\
&&
5x_1-2x_2-x_3 &\ge 6\\
&& x_1,x_2,x_3 &\ge 0
\end{align}
</math>

Let <math>OPT</math> be the value of the optimal solution.

== Duality ==

=== LP duality ===

=== Duality theorems ===

== Matroid ==

=== Kruskal's greedy algorithm for MST ===

=== Matroids ===
Let <math>X</math> be a finite set and <math>\mathcal{F}\subseteq 2^X</math> be a family of subsets of <math>X</math>. A member set <math>S\in\mathcal{F}</math> is called '''maximal''' if <math>S\cup\{x\}\not\in\mathcal{F}</math> for any <math>x\in X\setminus S</math>.

For <math>Y\subseteq X</math>, denote <math>\mathcal{F}_Y=\{S\in\mathcal{F}\mid S\subseteq Y\}</math>. Clearly <math>\mathcal{F}_Y</math> is the restriction of <math>\mathcal{F}</math> over <math>2^Y\,</math>.

{{Theorem|Definition|
:A set system <math>\mathcal{F}\subseteq 2^X</math> is a '''matroid''' if it satisfies:
:*(hereditary) if <math>T\subseteq S\in\mathcal{F}</math> then <math>T\in\mathcal{F}</math>;
:*(matroid property) for every <math>Y\subseteq X</math>, all maximal <math>S\in\mathcal{F}_Y</math> have the same <math>|S|</math>.
}}

Suppose <math>\mathcal{F}</math> is a matroid. Some matroid terminologies:
* Each member set <math>S\in\mathcal{F}</math> is called an '''independent set'''.
* A maximal independent subset of a set <math>Y\subset X</math>, i.e., a maximal <math>S\in\mathcal{F}_Y</math>, is called a '''basis''' of <math>Y</math>.
* The size of the maximal <math>S\in\mathcal{F}_Y</math> is called the '''rank''' of <math>Y</math>, denoted <math>r(Y)</math>.

==== Graph matroids ====

==== Linear matroids ====

=== Greedy algorithms on weighted matroids ===

=== Matroid intersections ===

Combinatorics (Fall 2010)/Flow and matching

2010-12-26T07:28:29Z

172.21.5.190: /* Integrality of polytopes */

== Flow and Cut==

=== Flows ===
An instance of the maximum flow problem consists of:
* a directed graph <math>G(V,E)</math>;
* two distinguished vertices <math>s</math> (the '''source''') and <math>t</math> (the '''sink'''), where the in-degree of <math>s</math> and the out-degree of <math>t</math> are both 0;
* the '''capacity function''' <math>c:E\rightarrow\mathbb{R}^+</math> which associates each directed edge <math>(u,v)\in E</math> a nonnegative real number <math>c_{uv}</math> called the '''capacity''' of the edge.

The quadruple <math>(G,c,s,t)</math> is called a '''flow network'''.

A function <math>f:E\rightarrow\mathbb{R}^+</math> is called a '''flow''' (to be specific an '''<math>s</math>-<math>t</math> flow''') in the network <math>G(V,E)</math> if it satisfies:
* '''Capacity constraint:''' <math>f_{uv}\le c_{uv}</math> for all <math>(u,v)\in E</math>.
* '''Conservation constraint:''' <math>\sum_{u:(u,v)\in E}f_{uv}=\sum_{w:(v,w)\in E}f_{vw}</math> for all <math>v\in V\setminus\{s,t\}</math>.

The '''value''' of the flow <math>f</math> is <math>\sum_{v:(s,v)\in E}f_{sv}</math>.

Given a flow network, the maximum flow problem asks to find the flow of the maximum value.

The maximum flow problem can be described as the following linear program.
:<math>
\begin{align}
\text{maximize} \quad& \sum_{v:(s,v)\in E}f_{sv}\\
\begin{align}
\text{subject to} \\
\\
\\
\\
\end{align}
\quad &
\begin{align} f_{uv}&\le c_{uv} &\quad& \forall (u,v)\in E\\
\sum_{u:(u,v)\in E}f_{uv}-\sum_{w:(v,w)\in E}f_{vw} &=0 &\quad& \forall v\in V\setminus\{s,t\}\\
f_{uv}&\ge 0 &\quad& \forall (u,v)\in E
\end{align}
\end{align}
</math>

=== Cuts ===
{{Theorem|Definition|
:Let <math>(G(V,E),c,s,t)</math> be a flow network. Let <math>S\subset V</math>. We call <math>(S,\bar{S})</math> an '''<math>s</math>-<math>t</math> cut''' if <math>s\in S</math> and <math>t\not\in S</math>.
:The '''value''' of the cut (also called the '''capacity''' of the cut) is defined as <math>\sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>.
}}

A fundamental fact in flow theory is that cuts always upper bound flows.

{{Theorem|Lemma|
:Let <math>(G(V,E),c,s,t)</math> be a flow network. Let <math>f</math> be an arbitrary flow in <math>G</math>, and let <math>(S,\bar{S})</math> be an arbitrary <math>s</math>-<math>t</math> cut. Then
::<math>\sum_{v:(s,v)}f_{sv}\le \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>,
:that is, the value of any flow is no greater than the value of any cut.
}}
{{Proof|By the definition of <math>s</math>-<math>t</math> cut, <math>s\in S</math> and <math>t\not\in S</math>.

Due to the conservation of flow,
:<math>\sum_{u\in S}\left(\sum_{v:(u,v)\in E}f_{uv}-\sum_{v:(v,u)\in E}f_{vu}\right)=\sum_{v:(s,v)\in E}f_{sv}+\sum_{u\in S\setminus\{s\}}\left(\sum_{v:(u,v)\in E}f_{uv}-\sum_{v:(v,u)\in E}f_{vu}\right)=\sum_{v:(s,v)\in E}f_{sv}\,.</math>
On the other hand, summing flow over edges,
:<math>\sum_{v\in S}\left(\sum_{u:(u,v)\in E}f_{uv}-\sum_{u:(v,u)\in E}f_{vu}\right)=\sum_{u\in S,v\in S\atop (u,v)\in E}\left(f_{uv}-f_{uv}\right)+\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}=\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}\,.</math>
Therefore,
:<math>\sum_{v:(s,v)\in E}f_{sv}=\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}\le\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}\le \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}\,,</math>
}}

=== Augmenting paths ===
{{Theorem|Definition (Augmenting path)|
:Let <math>f</math> be a flow in <math>G</math>. An '''augmenting path to <math>u_k</math>''' is a sequence of distinct vertices <math>P=(u_0,u_1,\cdots, u_k)</math>, such that
:* <math>u_0=s\,</math>;
:and each pair of consecutive vertices <math>u_{i}u_{i+1}\,</math> in <math>P</math> corresponds to either a '''forward edge''' <math>(u_{i},u_{i+1})\in E</math> or a '''reverse edge''' <math>(u_{i+1},u_{i})\in E</math>, and
:* <math>f(u_i,u_{i+1})<c(u_i,u_{i+1})\,</math> when <math>u_{i}u_{i+1}\,</math> corresponds to a forward edge <math>(u_{i},u_{i+1})\in E</math>, and
:* <math>f(u_{i+1},u_i)>0\,</math> when <math>u_{i}u_{i+1}\,</math> corresponds to a reverse edge <math>(u_{i+1},u_{i})\in E</math>.
:If <math>u_k=t\,</math>, we simply call <math>P</math> an '''augmenting path'''.
}}

Let <math>f</math> be a flow in <math>G</math>. Suppose there is an augmenting path <math>P=u_0u_1\cdots u_k</math>, where <math>u_0=s</math> and <math>u_k=t</math>. Let <math>\epsilon>0</math> be a positive constant satisfying
*<math>\epsilon \le c(u_{i},u_{i+1})-f(u_i,u_{i+1})</math> for all forward edges <math>(u_{i},u_{i+1})\in E</math> in <math>P</math>;
*<math>\epsilon \le f(u_{i+1},u_i)</math> for all reverse edges <math>(u_{i+1},u_i)\in E</math> in <math>P</math>.
Due to the definition of augmenting path, we can always find such a positive <math>\epsilon</math>.

Increase <math>f(u_i,u_{i+1})</math> by <math>\epsilon</math> for all forward edges <math>(u_{i},u_{i+1})\in E</math> in <math>P</math> and decrease <math>f(u_{i+1},u_i)</math> by <math>\epsilon</math> for all reverse edges <math>(u_{i+1},u_i)\in E</math> in <math>P</math>. Denote the modified flow by <math>f'</math>. It can be verified that <math>f'</math> satisfies the capacity constraint and conservation constraint thus is still a valid flow. On the other hand, the value of the new flow <math>f'</math>
:<math>\sum_{v:(s,v)\in E}f_{sv}'=\epsilon+\sum_{v:(s,v)\in E}f_{sv}>\sum_{v:(s,v)\in E}f_{sv}</math>.
Therefore, the value of the flow can be "augmented" by adjusting the flow on the augmenting path. This immediately implies that if a flow is maximum, then there is no augmenting path. Surprisingly, the converse is also true, thus maximum flows are "characterized" by augmenting paths.

{{Theorem|Lemma|
:A flow <math>f</math> is maximum if and only if there are no augmenting paths.
}}
{{Proof|We have already proved the "only if" direction above. Now we prove the "if" direction.

Let <math>S=\{u\in V\mid \exists\text{an augmenting path to }u\}</math>. Clearly <math>s\in S</math>, and since there is no augmenting path <math>t\not\in S</math>. Therefore, <math>(S,\bar{S})</math> defines an <math>s</math>-<math>t</math> cut.

We claim that
:<math>\sum_{v:(s,v)}f_{sv}= \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>,
that is, the value of flow <math>f</math> approach the value of the cut <math>(S,\bar{S})</math> defined above. By the above lemma, this will imply that the current flow <math>f</math> is maximum.

To prove this claim, we first observe that
:<math>\sum_{v:(s,v)}f_{sv}= \sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}</math>.
This identity is implied by the flow conservation constraint, and holds for any <math>s</math>-<math>t</math> cut <math>(S,\bar{S})</math>.

We then claim that
*<math>f_{uv}=c_{uv}</math> for all <math>u\in S,v\not\in S, (u,v)\in E</math>; and
*<math>f_{vu}=0</math> for all <math>u\in S,v\not\in S, (v,u)\in E</math>.
If otherwise, then the augmenting path to <math>u</math> apending <math>uv</math> becomes a new augmenting path to <math>v</math>, which contradicts that <math>S</math> includes all vertices to which there exist augmenting paths.

Therefore,
:<math>\sum_{v:(s,v)}f_{sv}= \sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu} = \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>.
As discussed above, this proves the theorem.
}}

=== The max-flow min-cut theorem ===
{{Theorem|Max-Flow Min-Cut Theorem|
:In a flow network, the maximum value of any <math>s</math>-<math>t</math> flow equals the minimum value of any <math>s</math>-<math>t</math> cut.
}}

{{Proof|
Let <math>f</math> be a flow with maximum value, so there is no augmenting path.

Again, let <math>S=\{u\in V\mid \exists\text{an augmenting path to }u\}</math>. As proved above, <math>(S,\bar{S})</math> forms an <math>s</math>-<math>t</math> cut, and
:<math>\sum_{v:(s,v)}f_{sv}= \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>,
that is, the value of flow <math>f</math> equals the value of cut <math>(S,\bar{S})</math>.

Since we know that all <math>s</math>-<math>t</math> flows are not greater than any <math>s</math>-<math>t</math> cut, the value of flow <math>f</math> equals the minimum value of any <math>s</math>-<math>t</math> cut.
}}

{{Theorem|Flow Integrality Theorem|
:Let <math>(G,c,s,t)</math> be a flow network with integral capacity <math>c</math>. There exists an integral flow which is maximum.
}}
{{Proof|
Let <math>f</math> be an integral flow of maximum value. If there is an augmenting path, since both <math>c</math> and <math>f</math> are integral, a new flow can be constructed of value 1+the value of <math>f</math>, contradicting that <math>f</math> is maximum over all integral flows. Therefore, there is no augmenting path, which means that <math>f</math> is maximum over all flows, integral or not.
}}

== Unimodularity ==

=== Integer Programming===

=== Integrality of polytopes ===
A point in an <math>n</math>-dimensional space is integral if all its coordinates are integers.

A polyhedron is said to be '''integral''' if all its vertices are integral.

There always exists an optimal solution which is a vertex in <math>P</math>. For integral <math>P</math>, all vertices are integral.

{{Theorem|Theorem (Hoffman 1974)|
: If a polyhedron <math>P</math> is integral then for all integer vectors <math>\boldsymbol{c}</math> there is an optimal solution to <math>\max\{\boldsymbol{c}^T\boldsymbol{x}\mid \boldsymbol{x}\in P\}</math> which is integral.
}}

=== Unimodularity and total unimodularity ===
{{Theorem|Definition (Unimodularity)|
:An <math>n\times n</math> integer matrix <math>A</math> is called '''unimodular''' if <math>\det(A)=\pm1</math>.
:An <math>m\times n</math> integer matrix <math>A</math> is called '''total unimodular''' if every square submatrix <math>B</math> of <math>A</math> has <math>\det(B)\in\{1,-1,0\}</math>, that is, every square, nonsingular submatrix of <math>A</math> is unimodular.
}}

{{Theorem|Theorem|
:Let <math>A</math> be an <math>m\times n</math> integer matrix.
:If <math>A</math> is totally unimodualr, then for any integer vector <math>\boldsymbol{b}\in\mathbb{Z}^n</math> the polyhedron <math>\{\boldsymbol{x}\in\mathbb{R}^n\mid A\boldsymbol{x}=\boldsymbol{b}, \boldsymbol{x}\ge \boldsymbol{0}\}</math> is integral.
}}
{{Proof|
Let <math>B</math> be a basis of <math>A</math>, and let <math>\boldsymbol{b}'</math> be the corresponding coordinates in <math>\boldsymbol{b}</math>. A basic solution is formed by <math>B^{-1}\boldsymbol{b}'</math> and zeros. Since <math>A</math> is totally unimodular and <math>B</math> is a basis thus nonsingular, <math>\det(B)\in\{1,-1,0\}</math>. By [http://en.wikipedia.org/wiki/Cramer's_rule Cramer's rule], <math>B^{-1}</math> has integer entries, thus <math>B^{-1}\boldsymbol{b}'</math> is integral. Therefore, any basic solution of <math>A\boldsymbol{x}=\boldsymbol{b}, \boldsymbol{x}\ge \boldsymbol{0}</math> is integral, which means the polyhedron <math>\{\boldsymbol{x}\in\mathbb{R}^n\mid A\boldsymbol{x}=\boldsymbol{b}, \boldsymbol{x}\ge \boldsymbol{0}\}</math> is integral.
}}

{{Theorem|Theorem (Hoffman-Kruskal 1956)|
:Let <math>A</math> be an <math>m\times n</math> integer matrix.
:If <math>A</math> is totally unimodualr, then for any integer vector <math>\boldsymbol{b}\in\mathbb{Z}^n</math> the polyhedron <math>\{\boldsymbol{x}\in\mathbb{R}^n\mid A\boldsymbol{x}\ge\boldsymbol{b}, \boldsymbol{x}\ge \boldsymbol{0}\}</math> is integral.
}}
{{Proof|
Let <math>A'=\begin{bmatrix}A & -I\end{bmatrix}</math>. We claim that <math>A'</math> is also totally unimodular. Any square submatrix <math>B</math> of <math>A</math> can be written in the following form after permutation:
:<math>B=\begin{bmatrix}
C & 0\\
D & I
\end{bmatrix}</math>
where <math>C</math> is a square submatrix of <math>A</math> and <math>I</math> is identity matrix. Therefore,
:<math>\det(B)=\det(C)\in\{1,-1,0\}</math>,
thus <math>A'</math> is totally unimodular.

Add slack variables to transform the constraints to the standard form <math>A'\boldsymbol{z}=\boldsymbol{b},\boldsymbol{z}\ge\boldsymbol{0}</math>. The polyhedron <math>\{\boldsymbol{x}\mid A\boldsymbol{x}\ge\boldsymbol{b}, \boldsymbol{x}\ge \boldsymbol{0}\}</math> is integral if the polyhedron <math>\{\boldsymbol{z}\mid A'\boldsymbol{z}=\boldsymbol{b}, \boldsymbol{z}\ge \boldsymbol{0}\}</math> is integral, which is implied by the total unimodularity of <math>A'\,</math>.
}}

Combinatorics (Fall 2010)/Flow and matching

2010-12-26T07:10:55Z

172.21.5.190: /* Cuts */

== Flow and Cut==

=== Flows ===
An instance of the maximum flow problem consists of:
* a directed graph <math>G(V,E)</math>;
* two distinguished vertices <math>s</math> (the '''source''') and <math>t</math> (the '''sink'''), where the in-degree of <math>s</math> and the out-degree of <math>t</math> are both 0;
* the '''capacity function''' <math>c:E\rightarrow\mathbb{R}^+</math> which associates each directed edge <math>(u,v)\in E</math> a nonnegative real number <math>c_{uv}</math> called the '''capacity''' of the edge.

The quadruple <math>(G,c,s,t)</math> is called a '''flow network'''.

A function <math>f:E\rightarrow\mathbb{R}^+</math> is called a '''flow''' (to be specific an '''<math>s</math>-<math>t</math> flow''') in the network <math>G(V,E)</math> if it satisfies:
* '''Capacity constraint:''' <math>f_{uv}\le c_{uv}</math> for all <math>(u,v)\in E</math>.
* '''Conservation constraint:''' <math>\sum_{u:(u,v)\in E}f_{uv}=\sum_{w:(v,w)\in E}f_{vw}</math> for all <math>v\in V\setminus\{s,t\}</math>.

The '''value''' of the flow <math>f</math> is <math>\sum_{v:(s,v)\in E}f_{sv}</math>.

Given a flow network, the maximum flow problem asks to find the flow of the maximum value.

The maximum flow problem can be described as the following linear program.
:<math>
\begin{align}
\text{maximize} \quad& \sum_{v:(s,v)\in E}f_{sv}\\
\begin{align}
\text{subject to} \\
\\
\\
\\
\end{align}
\quad &
\begin{align} f_{uv}&\le c_{uv} &\quad& \forall (u,v)\in E\\
\sum_{u:(u,v)\in E}f_{uv}-\sum_{w:(v,w)\in E}f_{vw} &=0 &\quad& \forall v\in V\setminus\{s,t\}\\
f_{uv}&\ge 0 &\quad& \forall (u,v)\in E
\end{align}
\end{align}
</math>

=== Cuts ===
{{Theorem|Definition|
:Let <math>(G(V,E),c,s,t)</math> be a flow network. Let <math>S\subset V</math>. We call <math>(S,\bar{S})</math> an '''<math>s</math>-<math>t</math> cut''' if <math>s\in S</math> and <math>t\not\in S</math>.
:The '''value''' of the cut (also called the '''capacity''' of the cut) is defined as <math>\sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>.
}}

A fundamental fact in flow theory is that cuts always upper bound flows.

{{Theorem|Lemma|
:Let <math>(G(V,E),c,s,t)</math> be a flow network. Let <math>f</math> be an arbitrary flow in <math>G</math>, and let <math>(S,\bar{S})</math> be an arbitrary <math>s</math>-<math>t</math> cut. Then
::<math>\sum_{v:(s,v)}f_{sv}\le \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>,
:that is, the value of any flow is no greater than the value of any cut.
}}
{{Proof|By the definition of <math>s</math>-<math>t</math> cut, <math>s\in S</math> and <math>t\not\in S</math>.

Due to the conservation of flow,
:<math>\sum_{u\in S}\left(\sum_{v:(u,v)\in E}f_{uv}-\sum_{v:(v,u)\in E}f_{vu}\right)=\sum_{v:(s,v)\in E}f_{sv}+\sum_{u\in S\setminus\{s\}}\left(\sum_{v:(u,v)\in E}f_{uv}-\sum_{v:(v,u)\in E}f_{vu}\right)=\sum_{v:(s,v)\in E}f_{sv}\,.</math>
On the other hand, summing flow over edges,
:<math>\sum_{v\in S}\left(\sum_{u:(u,v)\in E}f_{uv}-\sum_{u:(v,u)\in E}f_{vu}\right)=\sum_{u\in S,v\in S\atop (u,v)\in E}\left(f_{uv}-f_{uv}\right)+\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}=\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}\,.</math>
Therefore,
:<math>\sum_{v:(s,v)\in E}f_{sv}=\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}\le\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}\le \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}\,,</math>
}}

=== Augmenting paths ===
{{Theorem|Definition (Augmenting path)|
:Let <math>f</math> be a flow in <math>G</math>. An '''augmenting path to <math>u_k</math>''' is a sequence of distinct vertices <math>P=(u_0,u_1,\cdots, u_k)</math>, such that
:* <math>u_0=s\,</math>;
:and each pair of consecutive vertices <math>u_{i}u_{i+1}\,</math> in <math>P</math> corresponds to either a '''forward edge''' <math>(u_{i},u_{i+1})\in E</math> or a '''reverse edge''' <math>(u_{i+1},u_{i})\in E</math>, and
:* <math>f(u_i,u_{i+1})<c(u_i,u_{i+1})\,</math> when <math>u_{i}u_{i+1}\,</math> corresponds to a forward edge <math>(u_{i},u_{i+1})\in E</math>, and
:* <math>f(u_{i+1},u_i)>0\,</math> when <math>u_{i}u_{i+1}\,</math> corresponds to a reverse edge <math>(u_{i+1},u_{i})\in E</math>.
:If <math>u_k=t\,</math>, we simply call <math>P</math> an '''augmenting path'''.
}}

Let <math>f</math> be a flow in <math>G</math>. Suppose there is an augmenting path <math>P=u_0u_1\cdots u_k</math>, where <math>u_0=s</math> and <math>u_k=t</math>. Let <math>\epsilon>0</math> be a positive constant satisfying
*<math>\epsilon \le c(u_{i},u_{i+1})-f(u_i,u_{i+1})</math> for all forward edges <math>(u_{i},u_{i+1})\in E</math> in <math>P</math>;
*<math>\epsilon \le f(u_{i+1},u_i)</math> for all reverse edges <math>(u_{i+1},u_i)\in E</math> in <math>P</math>.
Due to the definition of augmenting path, we can always find such a positive <math>\epsilon</math>.

Increase <math>f(u_i,u_{i+1})</math> by <math>\epsilon</math> for all forward edges <math>(u_{i},u_{i+1})\in E</math> in <math>P</math> and decrease <math>f(u_{i+1},u_i)</math> by <math>\epsilon</math> for all reverse edges <math>(u_{i+1},u_i)\in E</math> in <math>P</math>. Denote the modified flow by <math>f'</math>. It can be verified that <math>f'</math> satisfies the capacity constraint and conservation constraint thus is still a valid flow. On the other hand, the value of the new flow <math>f'</math>
:<math>\sum_{v:(s,v)\in E}f_{sv}'=\epsilon+\sum_{v:(s,v)\in E}f_{sv}>\sum_{v:(s,v)\in E}f_{sv}</math>.
Therefore, the value of the flow can be "augmented" by adjusting the flow on the augmenting path. This immediately implies that if a flow is maximum, then there is no augmenting path. Surprisingly, the converse is also true, thus maximum flows are "characterized" by augmenting paths.

{{Theorem|Lemma|
:A flow <math>f</math> is maximum if and only if there are no augmenting paths.
}}
{{Proof|We have already proved the "only if" direction above. Now we prove the "if" direction.

Let <math>S=\{u\in V\mid \exists\text{an augmenting path to }u\}</math>. Clearly <math>s\in S</math>, and since there is no augmenting path <math>t\not\in S</math>. Therefore, <math>(S,\bar{S})</math> defines an <math>s</math>-<math>t</math> cut.

We claim that
:<math>\sum_{v:(s,v)}f_{sv}= \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>,
that is, the value of flow <math>f</math> approach the value of the cut <math>(S,\bar{S})</math> defined above. By the above lemma, this will imply that the current flow <math>f</math> is maximum.

To prove this claim, we first observe that
:<math>\sum_{v:(s,v)}f_{sv}= \sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}</math>.
This identity is implied by the flow conservation constraint, and holds for any <math>s</math>-<math>t</math> cut <math>(S,\bar{S})</math>.

We then claim that
*<math>f_{uv}=c_{uv}</math> for all <math>u\in S,v\not\in S, (u,v)\in E</math>; and
*<math>f_{vu}=0</math> for all <math>u\in S,v\not\in S, (v,u)\in E</math>.
If otherwise, then the augmenting path to <math>u</math> apending <math>uv</math> becomes a new augmenting path to <math>v</math>, which contradicts that <math>S</math> includes all vertices to which there exist augmenting paths.

Therefore,
:<math>\sum_{v:(s,v)}f_{sv}= \sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu} = \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>.
As discussed above, this proves the theorem.
}}

=== The max-flow min-cut theorem ===
{{Theorem|Max-Flow Min-Cut Theorem|
:In a flow network, the maximum value of any <math>s</math>-<math>t</math> flow equals the minimum value of any <math>s</math>-<math>t</math> cut.
}}

{{Proof|
Let <math>f</math> be a flow with maximum value, so there is no augmenting path.

Again, let <math>S=\{u\in V\mid \exists\text{an augmenting path to }u\}</math>. As proved above, <math>(S,\bar{S})</math> forms an <math>s</math>-<math>t</math> cut, and
:<math>\sum_{v:(s,v)}f_{sv}= \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>,
that is, the value of flow <math>f</math> equals the value of cut <math>(S,\bar{S})</math>.

Since we know that all <math>s</math>-<math>t</math> flows are not greater than any <math>s</math>-<math>t</math> cut, the value of flow <math>f</math> equals the minimum value of any <math>s</math>-<math>t</math> cut.
}}

{{Theorem|Flow Integrality Theorem|
:Let <math>(G,c,s,t)</math> be a flow network with integral capacity <math>c</math>. There exists an integral flow which is maximum.
}}
{{Proof|
Let <math>f</math> be an integral flow of maximum value. If there is an augmenting path, since both <math>c</math> and <math>f</math> are integral, a new flow can be constructed of value 1+the value of <math>f</math>, contradicting that <math>f</math> is maximum over all integral flows. Therefore, there is no augmenting path, which means that <math>f</math> is maximum over all flows, integral or not.
}}

== Unimodularity ==

=== Integer Programming===

=== Integrality of polytopes ===

=== Unimodularity and total unimodularity ===
{{Theorem|Definition (Unimodularity)|
:An <math>n\times n</math> integer matrix <math>A</math> is called '''unimodular''' if <math>\det(A)=\pm1</math>.
:An <math>m\times n</math> integer matrix <math>A</math> is called '''total unimodular''' if every square submatrix <math>B</math> of <math>A</math> has <math>\det(B)\in\{1,-1,0\}</math>, that is, every square, nonsingular submatrix of <math>A</math> is unimodular.
}}

{{Theorem|Theorem|
:Let <math>A</math> be an <math>m\times n</math> integer matrix.
:If <math>A</math> is totally unimodualr, then for any integer vector <math>\boldsymbol{b}\in\mathbb{Z}^n</math> the polyhedron <math>\{\boldsymbol{x}\in\mathbb{R}^n\mid A\boldsymbol{x}=\boldsymbol{b}, \boldsymbol{x}\ge \boldsymbol{0}\}</math> is integral.
}}
{{Proof|
Let <math>B</math> be a basis of <math>A</math>, and let <math>\boldsymbol{b}'</math> be the corresponding coordinates in <math>\boldsymbol{b}</math>. A basic solution is formed by <math>B^{-1}\boldsymbol{b}'</math> and zeros. Since <math>A</math> is totally unimodular and <math>B</math> is a basis thus nonsingular, <math>\det(B)\in\{1,-1,0\}</math>. By [http://en.wikipedia.org/wiki/Cramer's_rule Cramer's rule], <math>B^{-1}</math> has integer entries, thus <math>B^{-1}\boldsymbol{b}'</math> is integral. Therefore, any basic solution of <math>A\boldsymbol{x}=\boldsymbol{b}, \boldsymbol{x}\ge \boldsymbol{0}</math> is integral, which means the polyhedron <math>\{\boldsymbol{x}\in\mathbb{R}^n\mid A\boldsymbol{x}=\boldsymbol{b}, \boldsymbol{x}\ge \boldsymbol{0}\}</math> is integral.
}}

{{Theorem|Theorem (Hoffman-Kruskal 1956)|
:Let <math>A</math> be an <math>m\times n</math> integer matrix.
:If <math>A</math> is totally unimodualr, then for any integer vector <math>\boldsymbol{b}\in\mathbb{Z}^n</math> the polyhedron <math>\{\boldsymbol{x}\in\mathbb{R}^n\mid A\boldsymbol{x}\ge\boldsymbol{b}, \boldsymbol{x}\ge \boldsymbol{0}\}</math> is integral.
}}
{{Proof|
Let <math>A'=\begin{bmatrix}A & -I\end{bmatrix}</math>. We claim that <math>A'</math> is also totally unimodular. Any square submatrix <math>B</math> of <math>A</math> can be written in the following form after permutation:
:<math>B=\begin{bmatrix}
C & 0\\
D & I
\end{bmatrix}</math>
where <math>C</math> is a square submatrix of <math>A</math> and <math>I</math> is identity matrix. Therefore,
:<math>\det(B)=\det(C)\in\{1,-1,0\}</math>,
thus <math>A'</math> is totally unimodular.

Add slack variables to transform the constraints to the standard form <math>A'\boldsymbol{z}=\boldsymbol{b},\boldsymbol{z}\ge\boldsymbol{0}</math>. The polyhedron <math>\{\boldsymbol{x}\mid A\boldsymbol{x}\ge\boldsymbol{b}, \boldsymbol{x}\ge \boldsymbol{0}\}</math> is integral if the polyhedron <math>\{\boldsymbol{z}\mid A'\boldsymbol{z}=\boldsymbol{b}, \boldsymbol{z}\ge \boldsymbol{0}\}</math> is integral, which is implied by the total unimodularity of <math>A'\,</math>.
}}

Combinatorics (Fall 2010)/Flow and matching

2010-12-26T07:09:55Z

172.21.5.190: /* Integer Programmings */

== Flow and Cut==

=== Flows ===
An instance of the maximum flow problem consists of:
* a directed graph <math>G(V,E)</math>;
* two distinguished vertices <math>s</math> (the '''source''') and <math>t</math> (the '''sink'''), where the in-degree of <math>s</math> and the out-degree of <math>t</math> are both 0;
* the '''capacity function''' <math>c:E\rightarrow\mathbb{R}^+</math> which associates each directed edge <math>(u,v)\in E</math> a nonnegative real number <math>c_{uv}</math> called the '''capacity''' of the edge.

The quadruple <math>(G,c,s,t)</math> is called a '''flow network'''.

A function <math>f:E\rightarrow\mathbb{R}^+</math> is called a '''flow''' (to be specific an '''<math>s</math>-<math>t</math> flow''') in the network <math>G(V,E)</math> if it satisfies:
* '''Capacity constraint:''' <math>f_{uv}\le c_{uv}</math> for all <math>(u,v)\in E</math>.
* '''Conservation constraint:''' <math>\sum_{u:(u,v)\in E}f_{uv}=\sum_{w:(v,w)\in E}f_{vw}</math> for all <math>v\in V\setminus\{s,t\}</math>.

The '''value''' of the flow <math>f</math> is <math>\sum_{v:(s,v)\in E}f_{sv}</math>.

Given a flow network, the maximum flow problem asks to find the flow of the maximum value.

The maximum flow problem can be described as the following linear program.
:<math>
\begin{align}
\text{maximize} \quad& \sum_{v:(s,v)\in E}f_{sv}\\
\begin{align}
\text{subject to} \\
\\
\\
\\
\end{align}
\quad &
\begin{align} f_{uv}&\le c_{uv} &\quad& \forall (u,v)\in E\\
\sum_{u:(u,v)\in E}f_{uv}-\sum_{w:(v,w)\in E}f_{vw} &=0 &\quad& \forall v\in V\setminus\{s,t\}\\
f_{uv}&\ge 0 &\quad& \forall (u,v)\in E
\end{align}
\end{align}
</math>

=== Cuts ===
{{Theorem|Definition|
:Let <math>(G(V,E),c,s,t)</math> be a flow network. Let <math>S\subset V</math>. We call <math>(S,\bar{S})</math> an '''<math>s</math>-<math>t</math> cut''' if <math>s\in S</math> and <math>t\not\in S</math>.
:The '''value''' of the cut (also called the '''capacity''' of the cut) is defined as <math>\sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>.
}}

A fundamental fact in the theory of flow is that cuts always upper bound flows.

{{Theorem|Lemma|
:Let <math>(G(V,E),c,s,t)</math> be a flow network. Let <math>f</math> be an arbitrary flow in <math>G</math>, and let <math>(S,\bar{S})</math> be an arbitrary <math>s</math>-<math>t</math> cut. Then
::<math>\sum_{v:(s,v)}f_{sv}\le \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>,
:that is, the value of any flow is no greater than the value of any cut.
}}
{{Proof|By the definition of <math>s</math>-<math>t</math> cut, <math>s\in S</math> and <math>t\not\in S</math>.

Due to the conservation of flow,
:<math>\sum_{u\in S}\left(\sum_{v:(u,v)\in E}f_{uv}-\sum_{v:(v,u)\in E}f_{vu}\right)=\sum_{v:(s,v)\in E}f_{sv}+\sum_{u\in S\setminus\{s\}}\left(\sum_{v:(u,v)\in E}f_{uv}-\sum_{v:(v,u)\in E}f_{vu}\right)=\sum_{v:(s,v)\in E}f_{sv}\,.</math>
On the other hand, summing flow over edges,
:<math>\sum_{v\in S}\left(\sum_{u:(u,v)\in E}f_{uv}-\sum_{u:(v,u)\in E}f_{vu}\right)=\sum_{u\in S,v\in S\atop (u,v)\in E}\left(f_{uv}-f_{uv}\right)+\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}=\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}\,.</math>
Therefore,
:<math>\sum_{v:(s,v)\in E}f_{sv}=\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}\le\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}\le \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}\,,</math>
}}

=== Augmenting paths ===
{{Theorem|Definition (Augmenting path)|
:Let <math>f</math> be a flow in <math>G</math>. An '''augmenting path to <math>u_k</math>''' is a sequence of distinct vertices <math>P=(u_0,u_1,\cdots, u_k)</math>, such that
:* <math>u_0=s\,</math>;
:and each pair of consecutive vertices <math>u_{i}u_{i+1}\,</math> in <math>P</math> corresponds to either a '''forward edge''' <math>(u_{i},u_{i+1})\in E</math> or a '''reverse edge''' <math>(u_{i+1},u_{i})\in E</math>, and
:* <math>f(u_i,u_{i+1})<c(u_i,u_{i+1})\,</math> when <math>u_{i}u_{i+1}\,</math> corresponds to a forward edge <math>(u_{i},u_{i+1})\in E</math>, and
:* <math>f(u_{i+1},u_i)>0\,</math> when <math>u_{i}u_{i+1}\,</math> corresponds to a reverse edge <math>(u_{i+1},u_{i})\in E</math>.
:If <math>u_k=t\,</math>, we simply call <math>P</math> an '''augmenting path'''.
}}

Let <math>f</math> be a flow in <math>G</math>. Suppose there is an augmenting path <math>P=u_0u_1\cdots u_k</math>, where <math>u_0=s</math> and <math>u_k=t</math>. Let <math>\epsilon>0</math> be a positive constant satisfying
*<math>\epsilon \le c(u_{i},u_{i+1})-f(u_i,u_{i+1})</math> for all forward edges <math>(u_{i},u_{i+1})\in E</math> in <math>P</math>;
*<math>\epsilon \le f(u_{i+1},u_i)</math> for all reverse edges <math>(u_{i+1},u_i)\in E</math> in <math>P</math>.
Due to the definition of augmenting path, we can always find such a positive <math>\epsilon</math>.

Increase <math>f(u_i,u_{i+1})</math> by <math>\epsilon</math> for all forward edges <math>(u_{i},u_{i+1})\in E</math> in <math>P</math> and decrease <math>f(u_{i+1},u_i)</math> by <math>\epsilon</math> for all reverse edges <math>(u_{i+1},u_i)\in E</math> in <math>P</math>. Denote the modified flow by <math>f'</math>. It can be verified that <math>f'</math> satisfies the capacity constraint and conservation constraint thus is still a valid flow. On the other hand, the value of the new flow <math>f'</math>
:<math>\sum_{v:(s,v)\in E}f_{sv}'=\epsilon+\sum_{v:(s,v)\in E}f_{sv}>\sum_{v:(s,v)\in E}f_{sv}</math>.
Therefore, the value of the flow can be "augmented" by adjusting the flow on the augmenting path. This immediately implies that if a flow is maximum, then there is no augmenting path. Surprisingly, the converse is also true, thus maximum flows are "characterized" by augmenting paths.

{{Theorem|Lemma|
:A flow <math>f</math> is maximum if and only if there are no augmenting paths.
}}
{{Proof|We have already proved the "only if" direction above. Now we prove the "if" direction.

Let <math>S=\{u\in V\mid \exists\text{an augmenting path to }u\}</math>. Clearly <math>s\in S</math>, and since there is no augmenting path <math>t\not\in S</math>. Therefore, <math>(S,\bar{S})</math> defines an <math>s</math>-<math>t</math> cut.

We claim that
:<math>\sum_{v:(s,v)}f_{sv}= \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>,
that is, the value of flow <math>f</math> approach the value of the cut <math>(S,\bar{S})</math> defined above. By the above lemma, this will imply that the current flow <math>f</math> is maximum.

To prove this claim, we first observe that
:<math>\sum_{v:(s,v)}f_{sv}= \sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}</math>.
This identity is implied by the flow conservation constraint, and holds for any <math>s</math>-<math>t</math> cut <math>(S,\bar{S})</math>.

We then claim that
*<math>f_{uv}=c_{uv}</math> for all <math>u\in S,v\not\in S, (u,v)\in E</math>; and
*<math>f_{vu}=0</math> for all <math>u\in S,v\not\in S, (v,u)\in E</math>.
If otherwise, then the augmenting path to <math>u</math> apending <math>uv</math> becomes a new augmenting path to <math>v</math>, which contradicts that <math>S</math> includes all vertices to which there exist augmenting paths.

Therefore,
:<math>\sum_{v:(s,v)}f_{sv}= \sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu} = \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>.
As discussed above, this proves the theorem.
}}

=== The max-flow min-cut theorem ===
{{Theorem|Max-Flow Min-Cut Theorem|
:In a flow network, the maximum value of any <math>s</math>-<math>t</math> flow equals the minimum value of any <math>s</math>-<math>t</math> cut.
}}

{{Proof|
Let <math>f</math> be a flow with maximum value, so there is no augmenting path.

Again, let <math>S=\{u\in V\mid \exists\text{an augmenting path to }u\}</math>. As proved above, <math>(S,\bar{S})</math> forms an <math>s</math>-<math>t</math> cut, and
:<math>\sum_{v:(s,v)}f_{sv}= \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>,
that is, the value of flow <math>f</math> equals the value of cut <math>(S,\bar{S})</math>.

Since we know that all <math>s</math>-<math>t</math> flows are not greater than any <math>s</math>-<math>t</math> cut, the value of flow <math>f</math> equals the minimum value of any <math>s</math>-<math>t</math> cut.
}}

{{Theorem|Flow Integrality Theorem|
:Let <math>(G,c,s,t)</math> be a flow network with integral capacity <math>c</math>. There exists an integral flow which is maximum.
}}
{{Proof|
Let <math>f</math> be an integral flow of maximum value. If there is an augmenting path, since both <math>c</math> and <math>f</math> are integral, a new flow can be constructed of value 1+the value of <math>f</math>, contradicting that <math>f</math> is maximum over all integral flows. Therefore, there is no augmenting path, which means that <math>f</math> is maximum over all flows, integral or not.
}}

== Unimodularity ==

=== Integer Programming===

=== Integrality of polytopes ===

=== Unimodularity and total unimodularity ===
{{Theorem|Definition (Unimodularity)|
:An <math>n\times n</math> integer matrix <math>A</math> is called '''unimodular''' if <math>\det(A)=\pm1</math>.
:An <math>m\times n</math> integer matrix <math>A</math> is called '''total unimodular''' if every square submatrix <math>B</math> of <math>A</math> has <math>\det(B)\in\{1,-1,0\}</math>, that is, every square, nonsingular submatrix of <math>A</math> is unimodular.
}}

{{Theorem|Theorem|
:Let <math>A</math> be an <math>m\times n</math> integer matrix.
:If <math>A</math> is totally unimodualr, then for any integer vector <math>\boldsymbol{b}\in\mathbb{Z}^n</math> the polyhedron <math>\{\boldsymbol{x}\in\mathbb{R}^n\mid A\boldsymbol{x}=\boldsymbol{b}, \boldsymbol{x}\ge \boldsymbol{0}\}</math> is integral.
}}
{{Proof|
Let <math>B</math> be a basis of <math>A</math>, and let <math>\boldsymbol{b}'</math> be the corresponding coordinates in <math>\boldsymbol{b}</math>. A basic solution is formed by <math>B^{-1}\boldsymbol{b}'</math> and zeros. Since <math>A</math> is totally unimodular and <math>B</math> is a basis thus nonsingular, <math>\det(B)\in\{1,-1,0\}</math>. By [http://en.wikipedia.org/wiki/Cramer's_rule Cramer's rule], <math>B^{-1}</math> has integer entries, thus <math>B^{-1}\boldsymbol{b}'</math> is integral. Therefore, any basic solution of <math>A\boldsymbol{x}=\boldsymbol{b}, \boldsymbol{x}\ge \boldsymbol{0}</math> is integral, which means the polyhedron <math>\{\boldsymbol{x}\in\mathbb{R}^n\mid A\boldsymbol{x}=\boldsymbol{b}, \boldsymbol{x}\ge \boldsymbol{0}\}</math> is integral.
}}

{{Theorem|Theorem (Hoffman-Kruskal 1956)|
:Let <math>A</math> be an <math>m\times n</math> integer matrix.
:If <math>A</math> is totally unimodualr, then for any integer vector <math>\boldsymbol{b}\in\mathbb{Z}^n</math> the polyhedron <math>\{\boldsymbol{x}\in\mathbb{R}^n\mid A\boldsymbol{x}\ge\boldsymbol{b}, \boldsymbol{x}\ge \boldsymbol{0}\}</math> is integral.
}}
{{Proof|
Let <math>A'=\begin{bmatrix}A & -I\end{bmatrix}</math>. We claim that <math>A'</math> is also totally unimodular. Any square submatrix <math>B</math> of <math>A</math> can be written in the following form after permutation:
:<math>B=\begin{bmatrix}
C & 0\\
D & I
\end{bmatrix}</math>
where <math>C</math> is a square submatrix of <math>A</math> and <math>I</math> is identity matrix. Therefore,
:<math>\det(B)=\det(C)\in\{1,-1,0\}</math>,
thus <math>A'</math> is totally unimodular.

Add slack variables to transform the constraints to the standard form <math>A'\boldsymbol{z}=\boldsymbol{b},\boldsymbol{z}\ge\boldsymbol{0}</math>. The polyhedron <math>\{\boldsymbol{x}\mid A\boldsymbol{x}\ge\boldsymbol{b}, \boldsymbol{x}\ge \boldsymbol{0}\}</math> is integral if the polyhedron <math>\{\boldsymbol{z}\mid A'\boldsymbol{z}=\boldsymbol{b}, \boldsymbol{z}\ge \boldsymbol{0}\}</math> is integral, which is implied by the total unimodularity of <math>A'\,</math>.
}}

Combinatorics (Fall 2010)/Flow and matching

2010-12-26T07:04:57Z

172.21.5.190: /* Unimodularity and total unimodularity */

== Flow and Cut==

=== Flows ===
An instance of the maximum flow problem consists of:
* a directed graph <math>G(V,E)</math>;
* two distinguished vertices <math>s</math> (the '''source''') and <math>t</math> (the '''sink'''), where the in-degree of <math>s</math> and the out-degree of <math>t</math> are both 0;
* the '''capacity function''' <math>c:E\rightarrow\mathbb{R}^+</math> which associates each directed edge <math>(u,v)\in E</math> a nonnegative real number <math>c_{uv}</math> called the '''capacity''' of the edge.

The quadruple <math>(G,c,s,t)</math> is called a '''flow network'''.

A function <math>f:E\rightarrow\mathbb{R}^+</math> is called a '''flow''' (to be specific an '''<math>s</math>-<math>t</math> flow''') in the network <math>G(V,E)</math> if it satisfies:
* '''Capacity constraint:''' <math>f_{uv}\le c_{uv}</math> for all <math>(u,v)\in E</math>.
* '''Conservation constraint:''' <math>\sum_{u:(u,v)\in E}f_{uv}=\sum_{w:(v,w)\in E}f_{vw}</math> for all <math>v\in V\setminus\{s,t\}</math>.

The '''value''' of the flow <math>f</math> is <math>\sum_{v:(s,v)\in E}f_{sv}</math>.

Given a flow network, the maximum flow problem asks to find the flow of the maximum value.

The maximum flow problem can be described as the following linear program.
:<math>
\begin{align}
\text{maximize} \quad& \sum_{v:(s,v)\in E}f_{sv}\\
\begin{align}
\text{subject to} \\
\\
\\
\\
\end{align}
\quad &
\begin{align} f_{uv}&\le c_{uv} &\quad& \forall (u,v)\in E\\
\sum_{u:(u,v)\in E}f_{uv}-\sum_{w:(v,w)\in E}f_{vw} &=0 &\quad& \forall v\in V\setminus\{s,t\}\\
f_{uv}&\ge 0 &\quad& \forall (u,v)\in E
\end{align}
\end{align}
</math>

=== Cuts ===
{{Theorem|Definition|
:Let <math>(G(V,E),c,s,t)</math> be a flow network. Let <math>S\subset V</math>. We call <math>(S,\bar{S})</math> an '''<math>s</math>-<math>t</math> cut''' if <math>s\in S</math> and <math>t\not\in S</math>.
:The '''value''' of the cut (also called the '''capacity''' of the cut) is defined as <math>\sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>.
}}

A fundamental fact in the theory of flow is that cuts always upper bound flows.

{{Theorem|Lemma|
:Let <math>(G(V,E),c,s,t)</math> be a flow network. Let <math>f</math> be an arbitrary flow in <math>G</math>, and let <math>(S,\bar{S})</math> be an arbitrary <math>s</math>-<math>t</math> cut. Then
::<math>\sum_{v:(s,v)}f_{sv}\le \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>,
:that is, the value of any flow is no greater than the value of any cut.
}}
{{Proof|By the definition of <math>s</math>-<math>t</math> cut, <math>s\in S</math> and <math>t\not\in S</math>.

Due to the conservation of flow,
:<math>\sum_{u\in S}\left(\sum_{v:(u,v)\in E}f_{uv}-\sum_{v:(v,u)\in E}f_{vu}\right)=\sum_{v:(s,v)\in E}f_{sv}+\sum_{u\in S\setminus\{s\}}\left(\sum_{v:(u,v)\in E}f_{uv}-\sum_{v:(v,u)\in E}f_{vu}\right)=\sum_{v:(s,v)\in E}f_{sv}\,.</math>
On the other hand, summing flow over edges,
:<math>\sum_{v\in S}\left(\sum_{u:(u,v)\in E}f_{uv}-\sum_{u:(v,u)\in E}f_{vu}\right)=\sum_{u\in S,v\in S\atop (u,v)\in E}\left(f_{uv}-f_{uv}\right)+\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}=\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}\,.</math>
Therefore,
:<math>\sum_{v:(s,v)\in E}f_{sv}=\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}\le\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}\le \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}\,,</math>
}}

=== Augmenting paths ===
{{Theorem|Definition (Augmenting path)|
:Let <math>f</math> be a flow in <math>G</math>. An '''augmenting path to <math>u_k</math>''' is a sequence of distinct vertices <math>P=(u_0,u_1,\cdots, u_k)</math>, such that
:* <math>u_0=s\,</math>;
:and each pair of consecutive vertices <math>u_{i}u_{i+1}\,</math> in <math>P</math> corresponds to either a '''forward edge''' <math>(u_{i},u_{i+1})\in E</math> or a '''reverse edge''' <math>(u_{i+1},u_{i})\in E</math>, and
:* <math>f(u_i,u_{i+1})<c(u_i,u_{i+1})\,</math> when <math>u_{i}u_{i+1}\,</math> corresponds to a forward edge <math>(u_{i},u_{i+1})\in E</math>, and
:* <math>f(u_{i+1},u_i)>0\,</math> when <math>u_{i}u_{i+1}\,</math> corresponds to a reverse edge <math>(u_{i+1},u_{i})\in E</math>.
:If <math>u_k=t\,</math>, we simply call <math>P</math> an '''augmenting path'''.
}}

Let <math>f</math> be a flow in <math>G</math>. Suppose there is an augmenting path <math>P=u_0u_1\cdots u_k</math>, where <math>u_0=s</math> and <math>u_k=t</math>. Let <math>\epsilon>0</math> be a positive constant satisfying
*<math>\epsilon \le c(u_{i},u_{i+1})-f(u_i,u_{i+1})</math> for all forward edges <math>(u_{i},u_{i+1})\in E</math> in <math>P</math>;
*<math>\epsilon \le f(u_{i+1},u_i)</math> for all reverse edges <math>(u_{i+1},u_i)\in E</math> in <math>P</math>.
Due to the definition of augmenting path, we can always find such a positive <math>\epsilon</math>.

Increase <math>f(u_i,u_{i+1})</math> by <math>\epsilon</math> for all forward edges <math>(u_{i},u_{i+1})\in E</math> in <math>P</math> and decrease <math>f(u_{i+1},u_i)</math> by <math>\epsilon</math> for all reverse edges <math>(u_{i+1},u_i)\in E</math> in <math>P</math>. Denote the modified flow by <math>f'</math>. It can be verified that <math>f'</math> satisfies the capacity constraint and conservation constraint thus is still a valid flow. On the other hand, the value of the new flow <math>f'</math>
:<math>\sum_{v:(s,v)\in E}f_{sv}'=\epsilon+\sum_{v:(s,v)\in E}f_{sv}>\sum_{v:(s,v)\in E}f_{sv}</math>.
Therefore, the value of the flow can be "augmented" by adjusting the flow on the augmenting path. This immediately implies that if a flow is maximum, then there is no augmenting path. Surprisingly, the converse is also true, thus maximum flows are "characterized" by augmenting paths.

{{Theorem|Lemma|
:A flow <math>f</math> is maximum if and only if there are no augmenting paths.
}}
{{Proof|We have already proved the "only if" direction above. Now we prove the "if" direction.

Let <math>S=\{u\in V\mid \exists\text{an augmenting path to }u\}</math>. Clearly <math>s\in S</math>, and since there is no augmenting path <math>t\not\in S</math>. Therefore, <math>(S,\bar{S})</math> defines an <math>s</math>-<math>t</math> cut.

We claim that
:<math>\sum_{v:(s,v)}f_{sv}= \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>,
that is, the value of flow <math>f</math> approach the value of the cut <math>(S,\bar{S})</math> defined above. By the above lemma, this will imply that the current flow <math>f</math> is maximum.

To prove this claim, we first observe that
:<math>\sum_{v:(s,v)}f_{sv}= \sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}</math>.
This identity is implied by the flow conservation constraint, and holds for any <math>s</math>-<math>t</math> cut <math>(S,\bar{S})</math>.

We then claim that
*<math>f_{uv}=c_{uv}</math> for all <math>u\in S,v\not\in S, (u,v)\in E</math>; and
*<math>f_{vu}=0</math> for all <math>u\in S,v\not\in S, (v,u)\in E</math>.
If otherwise, then the augmenting path to <math>u</math> apending <math>uv</math> becomes a new augmenting path to <math>v</math>, which contradicts that <math>S</math> includes all vertices to which there exist augmenting paths.

Therefore,
:<math>\sum_{v:(s,v)}f_{sv}= \sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu} = \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>.
As discussed above, this proves the theorem.
}}

=== The max-flow min-cut theorem ===
{{Theorem|Max-Flow Min-Cut Theorem|
:In a flow network, the maximum value of any <math>s</math>-<math>t</math> flow equals the minimum value of any <math>s</math>-<math>t</math> cut.
}}

{{Proof|
Let <math>f</math> be a flow with maximum value, so there is no augmenting path.

Again, let <math>S=\{u\in V\mid \exists\text{an augmenting path to }u\}</math>. As proved above, <math>(S,\bar{S})</math> forms an <math>s</math>-<math>t</math> cut, and
:<math>\sum_{v:(s,v)}f_{sv}= \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>,
that is, the value of flow <math>f</math> equals the value of cut <math>(S,\bar{S})</math>.

Since we know that all <math>s</math>-<math>t</math> flows are not greater than any <math>s</math>-<math>t</math> cut, the value of flow <math>f</math> equals the minimum value of any <math>s</math>-<math>t</math> cut.
}}

{{Theorem|Flow Integrality Theorem|
:Let <math>(G,c,s,t)</math> be a flow network with integral capacity <math>c</math>. There exists an integral flow which is maximum.
}}
{{Proof|
Let <math>f</math> be an integral flow of maximum value. If there is an augmenting path, since both <math>c</math> and <math>f</math> are integral, a new flow can be constructed of value 1+the value of <math>f</math>, contradicting that <math>f</math> is maximum over all integral flows. Therefore, there is no augmenting path, which means that <math>f</math> is maximum over all flows, integral or not.
}}

== Unimodularity ==

=== Integer Programmings===

=== Integrality of polytopes ===

=== Unimodularity and total unimodularity ===
{{Theorem|Definition (Unimodularity)|
:An <math>n\times n</math> integer matrix <math>A</math> is called '''unimodular''' if <math>\det(A)=\pm1</math>.
:An <math>m\times n</math> integer matrix <math>A</math> is called '''total unimodular''' if every square submatrix <math>B</math> of <math>A</math> has <math>\det(B)\in\{1,-1,0\}</math>, that is, every square, nonsingular submatrix of <math>A</math> is unimodular.
}}

{{Theorem|Theorem|
:Let <math>A</math> be an <math>m\times n</math> integer matrix.
:If <math>A</math> is totally unimodualr, then for any integer vector <math>\boldsymbol{b}\in\mathbb{Z}^n</math> the polyhedron <math>\{\boldsymbol{x}\in\mathbb{R}^n\mid A\boldsymbol{x}=\boldsymbol{b}, \boldsymbol{x}\ge \boldsymbol{0}\}</math> is integral.
}}
{{Proof|
Let <math>B</math> be a basis of <math>A</math>, and let <math>\boldsymbol{b}'</math> be the corresponding coordinates in <math>\boldsymbol{b}</math>. A basic solution is formed by <math>B^{-1}\boldsymbol{b}'</math> and zeros. Since <math>A</math> is totally unimodular and <math>B</math> is a basis thus nonsingular, <math>\det(B)\in\{1,-1,0\}</math>. By [http://en.wikipedia.org/wiki/Cramer's_rule Cramer's rule], <math>B^{-1}</math> has integer entries, thus <math>B^{-1}\boldsymbol{b}'</math> is integral. Therefore, any basic solution of <math>A\boldsymbol{x}=\boldsymbol{b}, \boldsymbol{x}\ge \boldsymbol{0}</math> is integral, which means the polyhedron <math>\{\boldsymbol{x}\in\mathbb{R}^n\mid A\boldsymbol{x}=\boldsymbol{b}, \boldsymbol{x}\ge \boldsymbol{0}\}</math> is integral.
}}

{{Theorem|Theorem (Hoffman-Kruskal 1956)|
:Let <math>A</math> be an <math>m\times n</math> integer matrix.
:If <math>A</math> is totally unimodualr, then for any integer vector <math>\boldsymbol{b}\in\mathbb{Z}^n</math> the polyhedron <math>\{\boldsymbol{x}\in\mathbb{R}^n\mid A\boldsymbol{x}\ge\boldsymbol{b}, \boldsymbol{x}\ge \boldsymbol{0}\}</math> is integral.
}}
{{Proof|
Let <math>A'=\begin{bmatrix}A & -I\end{bmatrix}</math>. We claim that <math>A'</math> is also totally unimodular. Any square submatrix <math>B</math> of <math>A</math> can be written in the following form after permutation:
:<math>B=\begin{bmatrix}
C & 0\\
D & I
\end{bmatrix}</math>
where <math>C</math> is a square submatrix of <math>A</math> and <math>I</math> is identity matrix. Therefore,
:<math>\det(B)=\det(C)\in\{1,-1,0\}</math>,
thus <math>A'</math> is totally unimodular.

Add slack variables to transform the constraints to the standard form <math>A'\boldsymbol{z}=\boldsymbol{b},\boldsymbol{z}\ge\boldsymbol{0}</math>. The polyhedron <math>\{\boldsymbol{x}\mid A\boldsymbol{x}\ge\boldsymbol{b}, \boldsymbol{x}\ge \boldsymbol{0}\}</math> is integral if the polyhedron <math>\{\boldsymbol{z}\mid A'\boldsymbol{z}=\boldsymbol{b}, \boldsymbol{z}\ge \boldsymbol{0}\}</math> is integral, which is implied by the total unimodularity of <math>A'\,</math>.
}}

Combinatorics (Fall 2010)/Flow and matching

2010-12-24T12:43:13Z

172.21.3.81: /* The augmenting paths */

== Flow ==

=== The maximum flow problem ===
An instance of the maximum flow problem consists of:
* a directed graph <math>G(V,E)</math>;
* two distinguished vertices <math>s</math> (the '''source''') and <math>t</math> (the '''sink'''), where the in-degree of <math>s</math> and the out-degree of <math>t</math> are both 0;
* the '''capacity function''' <math>c:E\rightarrow\mathbb{R}^+</math> which associates each directed edge <math>(u,v)\in E</math> a nonnegative real number <math>c_{uv}</math> called the '''capacity''' of the edge.

The quadruple <math>(G,c,s,t)</math> is called a '''flow network'''.

A function <math>f:E\rightarrow\mathbb{R}^+</math> is called a '''flow''' (or an '''<math>s</math>-<math>t</math> flow''') in the network <math>G(V,E)</math> if it satisfies:
* '''Capacity constraint:''' <math>f_{uv}\le c_{uv}</math> for all <math>(u,v)\in E</math>.
* '''Conservation constraint:''' <math>\sum_{u:(u,v)\in E}f_{uv}=\sum_{w:(v,w)\in E}f_{vw}</math> for all <math>v\in V\setminus\{s,t\}</math>.

The '''value''' of the flow <math>f</math> is <math>\sum_{v:(s,v)\in E}f_{sv}</math>.

Given a flow network, the maximum flow problem asks to find the flow of the maximum value.

The maximum flow problem can be described as the following linear program.
:<math>
\begin{align}
\text{maximize} \quad& \sum_{v:(s,v)\in E}f_{sv}\\
\begin{align}
\text{subject to} \\
\\
\\
\\
\end{align}
\quad &
\begin{align} f_{uv}&\le c_{uv} &\quad& \forall (u,v)\in E\\
\sum_{u:(u,v)\in E}f_{uv}-\sum_{w:(v,w)\in E}f_{vw} &=0 &\quad& \forall v\in V\setminus\{s,t\}\\
f_{uv}&\ge 0 &\quad& \forall (u,v)\in E
\end{align}
\end{align}
</math>

=== Cuts ===
{{Theorem|Definition|
:Let <math>(G(V,E),c,s,t)</math> be a flow network. Let <math>S\subset V</math>. We call <math>(S,\bar{S})</math> an '''<math>s</math>-<math>t</math> cut''' if <math>s\in S</math> and <math>t\not\in S</math>.
:The '''value''' of the cut (also called the '''capacity''' of the cut) is defined as <math>\sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>.
}}

A fundamental fact in the theory of flow is that cuts always upper bound flows.

{{Theorem|Lemma|
:Let <math>(G(V,E),c,s,t)</math> be a flow network. Let <math>f</math> be an arbitrary flow in <math>G</math>, and let <math>(S,\bar{S})</math> be an arbitrary <math>s</math>-<math>t</math> cut. Then
::<math>\sum_{v:(s,v)}f_{sv}\le \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>,
:that is, the value of any flow is no greater than the value of any cut.
}}
{{Proof|By the definition of <math>s</math>-<math>t</math> cut, <math>s\in S</math> and <math>t\not\in S</math>.

Due to the conservation of flow,
:<math>\sum_{u\in S}\left(\sum_{v:(u,v)\in E}f_{uv}-\sum_{v:(v,u)\in E}f_{vu}\right)=\sum_{v:(s,v)\in E}f_{sv}+\sum_{u\in S\setminus\{s\}}\left(\sum_{v:(u,v)\in E}f_{uv}-\sum_{v:(v,u)\in E}f_{vu}\right)=\sum_{v:(s,v)\in E}f_{sv}\,.</math>
On the other hand, summing flow over edges,
:<math>\sum_{v\in S}\left(\sum_{u:(u,v)\in E}f_{uv}-\sum_{u:(v,u)\in E}f_{vu}\right)=\sum_{u\in S,v\in S\atop (u,v)\in E}\left(f_{uv}-f_{uv}\right)+\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}=\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}\,.</math>
Therefore,
:<math>\sum_{v:(s,v)\in E}f_{sv}=\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}\le\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}\le \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}\,,</math>
}}

=== The augmenting paths ===
{{Theorem|Definition (Augmenting path)|
:Let <math>f</math> be a flow in <math>G</math>. An '''augmenting path to <math>u_k</math>''' is a sequence of distinct vertices <math>P=(u_0,u_1,\cdots, u_k)</math>, such that
:* <math>u_0=s\,</math>;
:and each pair of consecutive vertices <math>u_{i}u_{i+1}\,</math> in <math>P</math> corresponds to either a '''forward edge''' <math>(u_{i},u_{i+1})\in E</math> or a '''reverse edge''' <math>(u_{i+1},u_{i})\in E</math>, and
:* <math>f(u_i,u_{i+1})<c(u_i,u_{i+1})\,</math> when <math>u_{i}u_{i+1}\,</math> corresponds to a forward edge <math>(u_{i},u_{i+1})\in E</math>, and
:* <math>f(u_{i+1},u_i)>0\,</math> when <math>u_{i}u_{i+1}\,</math> corresponds to a reverse edge <math>(u_{i+1},u_{i})\in E</math>.
:If <math>u_k=t\,</math>, we simply call <math>P</math> an '''augmenting path'''.
}}

Let <math>f</math> be a flow in <math>G</math>. Suppose there is an augmenting path <math>P=u_0u_1\cdots u_k</math>, where <math>u_0=s</math> and <math>u_k=t</math>. Let <math>\epsilon>0</math> be a positive constant satisfying
*<math>\epsilon \le c(u_{i},u_{i+1})-f(u_i,u_{i+1})</math> for all forward edges <math>(u_{i},u_{i+1})\in E</math> in <math>P</math>;
*<math>\epsilon \le f(u_{i+1},u_i)</math> for all reverse edges <math>(u_{i+1},u_i)\in E</math> in <math>P</math>.
Due to the definition of augmenting path, we can always find such a positive <math>\epsilon</math>.

Increase <math>f(u_i,u_{i+1})</math> by <math>\epsilon</math> for all forward edges <math>(u_{i},u_{i+1})\in E</math> in <math>P</math> and decrease <math>f(u_{i+1},u_i)</math> by <math>\epsilon</math> for all reverse edges <math>(u_{i+1},u_i)\in E</math> in <math>P</math>. Denote the modified flow by <math>f'</math>. It is easy to see that <math>f'</math> satisfies the capacity constraint and conservation constraint thus is still a valid flow. On the other hand, the value of the new flow <math>f'</math>
:<math>\sum_{v:(s,v)\in E}f_{sv}'=\epsilon+\sum_{v:(s,v)\in E}f_{sv}>\sum_{v:(s,v)\in E}f_{sv}</math>.
Therefore, the value of the flow can be "augmented" by adjusting the flow on the augmenting path. This immediately implies that if a flow is maximum, then there is no augmenting path. Surprisingly, the converse is also true, thus maximum flows are "characterized" by augmenting paths.

{{Theorem|Lemma|
:A flow <math>f</math> is maximum if and only if there are no augmenting paths.
}}
{{Proof|We have already proved the "only if" direction above. Now we prove the "if" direction.

Let <math>S=\{u\in V\mid \exists\text{an augmenting path to }u\}</math>. Clearly <math>s\in S</math>, and since there is no augmenting path <math>t\not\in S</math>. Therefore, <math>(S,\bar{S})</math> defines an <math>s</math>-<math>t</math> cut.

We claim that
:<math>\sum_{v:(s,v)}f_{sv}= \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>,
that is, the value of flow <math>f</math> approach the value of the cut <math>(S,\bar{S})</math> defined above. By the above lemma, this will imply that the current flow <math>f</math> is maximum.

To prove this claim, we first observe that
:<math>\sum_{v:(s,v)}f_{sv}= \sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}</math>.
This identity is implied by the flow conservation constraint, and holds for any <math>s</math>-<math>t</math> cut <math>(S,\bar{S})</math>.

We then claim that
*<math>f_{uv}=c_{uv}</math> for all <math>u\in S,v\not\in S, (u,v)\in E</math>; and
*<math>f_{vu}=0</math> for all <math>u\in S,v\not\in S, (v,u)\in E</math>.
If otherwise, then the augmenting path to <math>u</math> apending <math>uv</math> becomes a new augmenting path to <math>v</math>, which contradicts that <math>S</math> includes all vertices to which there exist augmenting paths.

Therefore,
:<math>\sum_{v:(s,v)}f_{sv}= \sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu} = \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>.
As discussed above, this proves the theorem.
}}

=== The max-flow min-cut theorem ===

== Unimodularity ==

=== Integrality of polytopes ===

=== Unimodularity and total unimodularity ===

Combinatorics (Fall 2010)/Flow and matching

2010-12-24T12:19:41Z

172.21.3.81: /* The augmenting paths */

== Flow ==

=== The maximum flow problem ===
An instance of the maximum flow problem consists of:
* a directed graph <math>G(V,E)</math>;
* two distinguished vertices <math>s</math> (the '''source''') and <math>t</math> (the '''sink'''), where the in-degree of <math>s</math> and the out-degree of <math>t</math> are both 0;
* the '''capacity function''' <math>c:E\rightarrow\mathbb{R}^+</math> which associates each directed edge <math>(u,v)\in E</math> a nonnegative real number <math>c_{uv}</math> called the '''capacity''' of the edge.

The quadruple <math>(G,c,s,t)</math> is called a '''flow network'''.

A function <math>f:E\rightarrow\mathbb{R}^+</math> is called a '''flow''' (or an '''<math>s</math>-<math>t</math> flow''') in the network <math>G(V,E)</math> if it satisfies:
* '''Capacity constraint:''' <math>f_{uv}\le c_{uv}</math> for all <math>(u,v)\in E</math>.
* '''Conservation constraint:''' <math>\sum_{u:(u,v)\in E}f_{uv}=\sum_{w:(v,w)\in E}f_{vw}</math> for all <math>v\in V\setminus\{s,t\}</math>.

The '''value''' of the flow <math>f</math> is <math>\sum_{v:(s,v)\in E}f_{sv}</math>.

Given a flow network, the maximum flow problem asks to find the flow of the maximum value.

The maximum flow problem can be described as the following linear program.
:<math>
\begin{align}
\text{maximize} \quad& \sum_{v:(s,v)\in E}f_{sv}\\
\begin{align}
\text{subject to} \\
\\
\\
\\
\end{align}
\quad &
\begin{align} f_{uv}&\le c_{uv} &\quad& \forall (u,v)\in E\\
\sum_{u:(u,v)\in E}f_{uv}-\sum_{w:(v,w)\in E}f_{vw} &=0 &\quad& \forall v\in V\setminus\{s,t\}\\
f_{uv}&\ge 0 &\quad& \forall (u,v)\in E
\end{align}
\end{align}
</math>

=== Cuts ===
{{Theorem|Definition|
:Let <math>(G(V,E),c,s,t)</math> be a flow network. Let <math>S\subset V</math>. We call <math>(S,\bar{S})</math> an '''<math>s</math>-<math>t</math> cut''' if <math>s\in S</math> and <math>t\not\in S</math>.
:The '''value''' of the cut (also called the '''capacity''' of the cut) is defined as <math>\sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>.
}}

A fundamental fact in the theory of flow is that cuts always upper bound flows.

{{Theorem|Lemma|
:Let <math>(G(V,E),c,s,t)</math> be a flow network. Let <math>f</math> be an arbitrary flow in <math>G</math>, and let <math>(S,\bar{S})</math> be an arbitrary <math>s</math>-<math>t</math> cut. Then
::<math>\sum_{v:(s,v)}f_{sv}\le \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>,
:that is, the value of any flow is no greater than the value of any cut.
}}
{{Proof|By the definition of <math>s</math>-<math>t</math> cut, <math>s\in S</math> and <math>t\not\in S</math>.

Due to the conservation of flow,
:<math>\sum_{u\in S}\left(\sum_{v:(u,v)\in E}f_{uv}-\sum_{v:(v,u)\in E}f_{vu}\right)=\sum_{v:(s,v)\in E}f_{sv}+\sum_{u\in S\setminus\{s\}}\left(\sum_{v:(u,v)\in E}f_{uv}-\sum_{v:(v,u)\in E}f_{vu}\right)=\sum_{v:(s,v)\in E}f_{sv}\,.</math>
On the other hand, summing flow over edges,
:<math>\sum_{v\in S}\left(\sum_{u:(u,v)\in E}f_{uv}-\sum_{u:(v,u)\in E}f_{vu}\right)=\sum_{u\in S,v\in S\atop (u,v)\in E}\left(f_{uv}-f_{uv}\right)+\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}=\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}\,.</math>
Therefore,
:<math>\sum_{v:(s,v)\in E}f_{sv}=\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}\le\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}\le \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}\,,</math>
}}

=== The augmenting paths ===
{{Theorem|Definition (Augmenting path)|
:Let <math>f</math> be a flow in <math>G</math>. An '''augmenting path to <math>u_k</math>''' is a sequence of distinct vertices <math>P=(u_0,u_1,\cdots, u_k)</math>, such that
:* <math>u_0=s\,</math>;
:and each pair of consecutive vertices <math>u_{i}u_{i+1}\,</math> in <math>P</math> corresponds to either a '''forward edge''' <math>(u_{i},u_{i+1})\in E</math> or a '''reverse edge''' <math>(u_{i+1},u_{i})\in E</math>, and
:* <math>f(u_i,u_{i+1})<c(u_i,u_{i+1})\,</math> when <math>u_{i}u_{i+1}\,</math> corresponds to a forward edge <math>(u_{i},u_{i+1})\in E</math>, and
:* <math>f(u_{i+1},u_i)>0\,</math> when <math>u_{i}u_{i+1}\,</math> corresponds to a reverse edge <math>(u_{i+1},u_{i})\in E</math>.
:If <math>u_k=t\,</math>, we simply call <math>P</math> an '''augmenting path'''.
}}

Let <math>f</math> be a flow in <math>G</math>. Suppose there is an augmenting path <math>P=u_0u_1\cdots u_k</math>, where <math>u_0=s</math> and <math>u_k=t</math>. Let <math>\epsilon>0</math> be a positive constant satisfying
*<math>\epsilon \le c(u_{i},u_{i+1})-f(u_i,u_{i+1})</math> for all forward edges <math>(u_{i},u_{i+1})\in E</math> in <math>P</math>;
*<math>\epsilon \le f(u_{i+1},u_i)</math> for all reverse edges <math>(u_{i+1},u_i)\in E</math> in <math>P</math>.
Due to the definition of augmenting path, we can always find such a positive <math>\epsilon</math>.

Increase <math>f(u_i,u_{i+1})</math> by <math>\epsilon</math> for all forward edges <math>(u_{i},u_{i+1})\in E</math> in <math>P</math> and decrease <math>f(u_{i+1},u_i)</math> by <math>\epsilon</math> for all reverse edges <math>(u_{i+1},u_i)\in E</math> in <math>P</math>. Denote the modified flow by <math>f'</math>. It is easy to see that <math>f'</math> satisfies the capacity constraint and conservation constraint thus is still a valid flow. On the other hand, the value of the new flow <math>f'</math>
:<math>\sum_{v:(s,v)\in E}f_{sv}'=\epsilon+\sum_{v:(s,v)\in E}f_{sv}>\sum_{v:(s,v)\in E}f_{sv}</math>.
Therefore, the value of the flow can be "augmented" by adjusting the flow on the augmenting path. This immediately implies that if a flow is maximum, then there is no augmenting path. Surprisingly, the converse is also true, thus maximum flows are "characterized" by augmenting paths.

{{Theorem|Lemma|
:A flow <math>f</math> is maximum if and only if there are no augmenting paths.
}}
{{Proof|We have already proved the "only if" direction above. Now we prove the "if" direction.

Let <math>S=\{u\in V\mid \exists\text{an augmenting path to }u\}</math>. Clearly <math>s\in S</math>, and since there is no augmenting path <math>t\not\in S</math>. Therefore, <math>(S,\bar{S})</math> defines an <math>s</math>-<math>t</math> cut.

We claim that
:<math>\sum_{v:(s,v)}f_{sv}= \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>,
that is, the value of flow <math>f</math> approach the value of the cut <math>(S,\bar{S})</math> defined above. By the above lemma, this will imply that the current flow <math>f</math> is maximum.

}}

=== The max-flow min-cut theorem ===

== Unimodularity ==

=== Integrality of polytopes ===

=== Unimodularity and total unimodularity ===

Combinatorics (Fall 2010)/Flow and matching

2010-12-24T12:06:22Z

172.21.3.81: /* The augmenting paths */

== Flow ==

=== The maximum flow problem ===
An instance of the maximum flow problem consists of:
* a directed graph <math>G(V,E)</math>;
* two distinguished vertices <math>s</math> (the '''source''') and <math>t</math> (the '''sink'''), where the in-degree of <math>s</math> and the out-degree of <math>t</math> are both 0;
* the '''capacity function''' <math>c:E\rightarrow\mathbb{R}^+</math> which associates each directed edge <math>(u,v)\in E</math> a nonnegative real number <math>c_{uv}</math> called the '''capacity''' of the edge.

The quadruple <math>(G,c,s,t)</math> is called a '''flow network'''.

A function <math>f:E\rightarrow\mathbb{R}^+</math> is called a '''flow''' (or an '''<math>s</math>-<math>t</math> flow''') in the network <math>G(V,E)</math> if it satisfies:
* '''Capacity constraint:''' <math>f_{uv}\le c_{uv}</math> for all <math>(u,v)\in E</math>.
* '''Conservation constraint:''' <math>\sum_{u:(u,v)\in E}f_{uv}=\sum_{w:(v,w)\in E}f_{vw}</math> for all <math>v\in V\setminus\{s,t\}</math>.

The '''value''' of the flow <math>f</math> is <math>\sum_{v:(s,v)\in E}f_{sv}</math>.

Given a flow network, the maximum flow problem asks to find the flow of the maximum value.

The maximum flow problem can be described as the following linear program.
:<math>
\begin{align}
\text{maximize} \quad& \sum_{v:(s,v)\in E}f_{sv}\\
\begin{align}
\text{subject to} \\
\\
\\
\\
\end{align}
\quad &
\begin{align} f_{uv}&\le c_{uv} &\quad& \forall (u,v)\in E\\
\sum_{u:(u,v)\in E}f_{uv}-\sum_{w:(v,w)\in E}f_{vw} &=0 &\quad& \forall v\in V\setminus\{s,t\}\\
f_{uv}&\ge 0 &\quad& \forall (u,v)\in E
\end{align}
\end{align}
</math>

=== Cuts ===
{{Theorem|Definition|
:Let <math>(G(V,E),c,s,t)</math> be a flow network. Let <math>S\subset V</math>. We call <math>(S,\bar{S})</math> an '''<math>s</math>-<math>t</math> cut''' if <math>s\in S</math> and <math>t\not\in S</math>.
:The '''value''' of the cut (also called the '''capacity''' of the cut) is defined as <math>\sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>.
}}

A fundamental fact in the theory of flow is that cuts always upper bound flows.

{{Theorem|Lemma|
:Let <math>(G(V,E),c,s,t)</math> be a flow network. Let <math>f</math> be an arbitrary flow in <math>G</math>, and let <math>(S,\bar{S})</math> be an arbitrary <math>s</math>-<math>t</math> cut. Then
::<math>\sum_{v:(s,v)}f_{sv}\le \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>,
:that is, the value of any flow is no greater than the value of any cut.
}}
{{Proof|By the definition of <math>s</math>-<math>t</math> cut, <math>s\in S</math> and <math>t\not\in S</math>.

Due to the conservation of flow,
:<math>\sum_{u\in S}\left(\sum_{v:(u,v)\in E}f_{uv}-\sum_{v:(v,u)\in E}f_{vu}\right)=\sum_{v:(s,v)\in E}f_{sv}+\sum_{u\in S\setminus\{s\}}\left(\sum_{v:(u,v)\in E}f_{uv}-\sum_{v:(v,u)\in E}f_{vu}\right)=\sum_{v:(s,v)\in E}f_{sv}\,.</math>
On the other hand, summing flow over edges,
:<math>\sum_{v\in S}\left(\sum_{u:(u,v)\in E}f_{uv}-\sum_{u:(v,u)\in E}f_{vu}\right)=\sum_{u\in S,v\in S\atop (u,v)\in E}\left(f_{uv}-f_{uv}\right)+\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}=\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}\,.</math>
Therefore,
:<math>\sum_{v:(s,v)\in E}f_{sv}=\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}\le\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}\le \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}\,,</math>
}}

=== The augmenting paths ===
{{Theorem|Definition (Augmenting path)|
:Let <math>f</math> be a flow in <math>G</math>. An '''augmenting path to <math>u_k</math>''' is a sequence of distinct vertices <math>P=(u_0,u_1,\cdots, u_k)</math>, such that
:* <math>u_0=s\,</math>;
:and each pair of consecutive vertices <math>u_{i}u_{i+1}\,</math> in <math>P</math> corresponds to either a '''forward edge''' <math>(u_{i},u_{i+1})\in E</math> or a '''reverse edge''' <math>(u_{i+1},u_{i})\in E</math>, and
:* <math>f(u_i,u_{i+1})<c(u_i,u_{i+1})\,</math> when <math>u_{i}u_{i+1}\,</math> corresponds to a forward edge <math>(u_{i},u_{i+1})\in E</math>, and
:* <math>f(u_{i+1},u_i)>0\,</math> when <math>u_{i}u_{i+1}\,</math> corresponds to a reverse edge <math>(u_{i+1},u_{i})\in E</math>.
:If <math>u_k=t\,</math>, we simply call <math>P</math> an '''augmenting path'''.
}}

Let <math>f</math> be a flow in <math>G</math>. Suppose there is an augmenting path <math>P=u_0u_1\cdots u_k</math>, where <math>u_0=s</math> and <math>u_k=t</math>. Let <math>\epsilon>0</math> be a positive constant satisfying
*<math>\epsilon \le c(u_{i},u_{i+1})-f(u_i,u_{i+1})</math> for all forward edges <math>(u_{i},u_{i+1})\in E</math> in <math>P</math>;
*<math>\epsilon \le f(u_{i+1},u_i)</math> for all reverse edges <math>(u_{i+1},u_i)\in E</math> in <math>P</math>.
Due to the definition of augmenting path, we can always find such a positive <math>\epsilon</math>.

Increase <math>f(u_i,u_{i+1})</math> by <math>\epsilon</math> for all forward edges <math>(u_{i},u_{i+1})\in E</math> in <math>P</math> and decrease <math>f(u_{i+1},u_i)</math> by <math>\epsilon</math> for all reverse edges <math>(u_{i+1},u_i)\in E</math> in <math>P</math>. Denote the modified flow by <math>f'</math>. It is easy to see that <math>f'</math> satisfies the capacity constraint and conservation constraint thus is still a valid flow. On the other hand, the value of the new flow <math>f'</math>
:<math>\sum_{v:(s,v)\in E}f_{sv}'=\epsilon+\sum_{v:(s,v)\in E}f_{sv}>\sum_{v:(s,v)\in E}f_{sv}</math>.
Therefore, the value of the flow can be "augmented" by adjusting the flow on the augmenting path. This immediately implies that if a flow is maximum, then there is no augmenting path. Surprisingly, the converse is also true, thus maximum flows are "characterized" by augmenting paths.

{{Theorem|Lemma|
:A flow <math>f</math> is maximum if and only if there are no augmenting paths.
}}
{{Proof|We have already proved the "only if" direction above. Now we prove the "if" direction.

Let <math>S=\{u\in V\mid \exists\text{an augmenting path to }u\}</math>. Clearly <math>s\in S</math>, and since there is no augmenting path <math>t\not\in S</math>. Therefore, <math>(S,\bar{S})</math> defines an <math>s</math>-<math>t</math> cut.
}}

=== The max-flow min-cut theorem ===

== Unimodularity ==

=== Integrality of polytopes ===

=== Unimodularity and total unimodularity ===

Combinatorics (Fall 2010)/Flow and matching

2010-12-24T12:05:34Z

172.21.3.81: /* The augmenting paths */

== Flow ==

=== The maximum flow problem ===
An instance of the maximum flow problem consists of:
* a directed graph <math>G(V,E)</math>;
* two distinguished vertices <math>s</math> (the '''source''') and <math>t</math> (the '''sink'''), where the in-degree of <math>s</math> and the out-degree of <math>t</math> are both 0;
* the '''capacity function''' <math>c:E\rightarrow\mathbb{R}^+</math> which associates each directed edge <math>(u,v)\in E</math> a nonnegative real number <math>c_{uv}</math> called the '''capacity''' of the edge.

The quadruple <math>(G,c,s,t)</math> is called a '''flow network'''.

A function <math>f:E\rightarrow\mathbb{R}^+</math> is called a '''flow''' (or an '''<math>s</math>-<math>t</math> flow''') in the network <math>G(V,E)</math> if it satisfies:
* '''Capacity constraint:''' <math>f_{uv}\le c_{uv}</math> for all <math>(u,v)\in E</math>.
* '''Conservation constraint:''' <math>\sum_{u:(u,v)\in E}f_{uv}=\sum_{w:(v,w)\in E}f_{vw}</math> for all <math>v\in V\setminus\{s,t\}</math>.

The '''value''' of the flow <math>f</math> is <math>\sum_{v:(s,v)\in E}f_{sv}</math>.

Given a flow network, the maximum flow problem asks to find the flow of the maximum value.

The maximum flow problem can be described as the following linear program.
:<math>
\begin{align}
\text{maximize} \quad& \sum_{v:(s,v)\in E}f_{sv}\\
\begin{align}
\text{subject to} \\
\\
\\
\\
\end{align}
\quad &
\begin{align} f_{uv}&\le c_{uv} &\quad& \forall (u,v)\in E\\
\sum_{u:(u,v)\in E}f_{uv}-\sum_{w:(v,w)\in E}f_{vw} &=0 &\quad& \forall v\in V\setminus\{s,t\}\\
f_{uv}&\ge 0 &\quad& \forall (u,v)\in E
\end{align}
\end{align}
</math>

=== Cuts ===
{{Theorem|Definition|
:Let <math>(G(V,E),c,s,t)</math> be a flow network. Let <math>S\subset V</math>. We call <math>(S,\bar{S})</math> an '''<math>s</math>-<math>t</math> cut''' if <math>s\in S</math> and <math>t\not\in S</math>.
:The '''value''' of the cut (also called the '''capacity''' of the cut) is defined as <math>\sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>.
}}

A fundamental fact in the theory of flow is that cuts always upper bound flows.

{{Theorem|Lemma|
:Let <math>(G(V,E),c,s,t)</math> be a flow network. Let <math>f</math> be an arbitrary flow in <math>G</math>, and let <math>(S,\bar{S})</math> be an arbitrary <math>s</math>-<math>t</math> cut. Then
::<math>\sum_{v:(s,v)}f_{sv}\le \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>,
:that is, the value of any flow is no greater than the value of any cut.
}}
{{Proof|By the definition of <math>s</math>-<math>t</math> cut, <math>s\in S</math> and <math>t\not\in S</math>.

Due to the conservation of flow,
:<math>\sum_{u\in S}\left(\sum_{v:(u,v)\in E}f_{uv}-\sum_{v:(v,u)\in E}f_{vu}\right)=\sum_{v:(s,v)\in E}f_{sv}+\sum_{u\in S\setminus\{s\}}\left(\sum_{v:(u,v)\in E}f_{uv}-\sum_{v:(v,u)\in E}f_{vu}\right)=\sum_{v:(s,v)\in E}f_{sv}\,.</math>
On the other hand, summing flow over edges,
:<math>\sum_{v\in S}\left(\sum_{u:(u,v)\in E}f_{uv}-\sum_{u:(v,u)\in E}f_{vu}\right)=\sum_{u\in S,v\in S\atop (u,v)\in E}\left(f_{uv}-f_{uv}\right)+\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}=\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}\,.</math>
Therefore,
:<math>\sum_{v:(s,v)\in E}f_{sv}=\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}\le\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}\le \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}\,,</math>
}}

=== The augmenting paths ===
{{Theorem|Definition (Augmenting path)|
:Let <math>f</math> be a flow in <math>G</math>. An '''augmenting path to <math>u_k</math>''' is a sequence of distinct vertices <math>P=(u_0,u_1,\cdots, u_k)</math>, such that
:* <math>u_0=s\,</math>;
:and each pair of consecutive vertices <math>u_{i}u_{i+1}\,</math> in <math>P</math> corresponds to either a '''forward edge''' <math>(u_{i},u_{i+1})\in E</math> or a '''reverse edge''' <math>(u_{i+1},u_{i})\in E</math>, and
:* <math>f(u_i,u_{i+1})<c(u_i,u_{i+1})\,</math> when <math>u_{i}u_{i+1}\,</math> corresponds to a forward edge <math>(u_{i},u_{i+1})\in E</math>, and
:* <math>f(u_{i+1},u_i)>0\,</math> when <math>u_{i}u_{i+1}\,</math> corresponds to a reverse edge <math>(u_{i+1},u_{i})\in E</math>.
:If <math>u_k=t\,</math>, we simply call <math>P</math> an '''augmenting path'''.
}}

Let <math>f</math> be a flow in <math>G</math>. Suppose there is an augmenting path <math>P=u_0u_1\cdots u_k</math>, where <math>u_0=s</math> and <math>u_k=t</math>. Let <math>\epsilon>0</math> be a positive constant satisfying
*<math>\epsilon \le c(u_{i},u_{i+1})-f(u_i,u_{i+1})</math> for all forward edges <math>(u_{i},u_{i+1})\in E</math> in <math>P</math>;
*<math>\epsilon \le f(u_{i+1},u_i)</math> for all reverse edges <math>(u_{i+1},u_i)\in E</math> in <math>P</math>.
Due to the definition of augmenting path, we can always find such a positive <math>\epsilon</math>.

Increase <math>f(u_i,u_{i+1})</math> by <math>\epsilon</math> for all forward edges <math>(u_{i},u_{i+1})\in E</math> in <math>P</math> and decrease <math>f(u_{i+1},u_i)</math> by <math>\epsilon</math> for all reverse edges <math>(u_{i+1},u_i)\in E</math> in <math>P</math>. Denote the modified flow by <math>f'</math>. It is easy to see that <math>f'</math> satisfies the capacity constraint and conservation constraint thus is still a valid flow. On the other hand, the value of the new flow <math>f'</math>
:<math>\sum_{v:(s,v)\in E}f_{sv}'=\epsilon+\sum_{v:(s,v)\in E}f_{sv}>\sum_{v:(s,v)\in E}f_{sv}</math>.
Therefore, the value of the flow can be "augmented" by adjusting the flow on the augmenting path. This immediately implies that if a flow is maximum, then there is no augmenting path. Surprisingly, the converse is also true, thus maximum flows are "characterized" by augmenting paths.

{{Theorem|Lemma|
:A flow <math>f</math> is maximum if and only if there are no augmenting paths.
}}
{{Proof|We have already proved the "only if" direction above. Now we prove the "if" direction.

Let <math>S=\{u\in V\mid \exists\text{an augmenting path to }u\}</math>. Clearly <math>s\in S</math>, and since there is no augmenting path <math>t\not\in S</math>. Therefore,
}}

=== The max-flow min-cut theorem ===

== Unimodularity ==

=== Integrality of polytopes ===

=== Unimodularity and total unimodularity ===

Combinatorics (Fall 2010)/Flow and matching

2010-12-24T12:02:56Z

172.21.3.81: /* Cuts */

== Flow ==

=== The maximum flow problem ===
An instance of the maximum flow problem consists of:
* a directed graph <math>G(V,E)</math>;
* two distinguished vertices <math>s</math> (the '''source''') and <math>t</math> (the '''sink'''), where the in-degree of <math>s</math> and the out-degree of <math>t</math> are both 0;
* the '''capacity function''' <math>c:E\rightarrow\mathbb{R}^+</math> which associates each directed edge <math>(u,v)\in E</math> a nonnegative real number <math>c_{uv}</math> called the '''capacity''' of the edge.

The quadruple <math>(G,c,s,t)</math> is called a '''flow network'''.

A function <math>f:E\rightarrow\mathbb{R}^+</math> is called a '''flow''' (or an '''<math>s</math>-<math>t</math> flow''') in the network <math>G(V,E)</math> if it satisfies:
* '''Capacity constraint:''' <math>f_{uv}\le c_{uv}</math> for all <math>(u,v)\in E</math>.
* '''Conservation constraint:''' <math>\sum_{u:(u,v)\in E}f_{uv}=\sum_{w:(v,w)\in E}f_{vw}</math> for all <math>v\in V\setminus\{s,t\}</math>.

The '''value''' of the flow <math>f</math> is <math>\sum_{v:(s,v)\in E}f_{sv}</math>.

Given a flow network, the maximum flow problem asks to find the flow of the maximum value.

The maximum flow problem can be described as the following linear program.
:<math>
\begin{align}
\text{maximize} \quad& \sum_{v:(s,v)\in E}f_{sv}\\
\begin{align}
\text{subject to} \\
\\
\\
\\
\end{align}
\quad &
\begin{align} f_{uv}&\le c_{uv} &\quad& \forall (u,v)\in E\\
\sum_{u:(u,v)\in E}f_{uv}-\sum_{w:(v,w)\in E}f_{vw} &=0 &\quad& \forall v\in V\setminus\{s,t\}\\
f_{uv}&\ge 0 &\quad& \forall (u,v)\in E
\end{align}
\end{align}
</math>

=== Cuts ===
{{Theorem|Definition|
:Let <math>(G(V,E),c,s,t)</math> be a flow network. Let <math>S\subset V</math>. We call <math>(S,\bar{S})</math> an '''<math>s</math>-<math>t</math> cut''' if <math>s\in S</math> and <math>t\not\in S</math>.
:The '''value''' of the cut (also called the '''capacity''' of the cut) is defined as <math>\sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>.
}}

A fundamental fact in the theory of flow is that cuts always upper bound flows.

{{Theorem|Lemma|
:Let <math>(G(V,E),c,s,t)</math> be a flow network. Let <math>f</math> be an arbitrary flow in <math>G</math>, and let <math>(S,\bar{S})</math> be an arbitrary <math>s</math>-<math>t</math> cut. Then
::<math>\sum_{v:(s,v)}f_{sv}\le \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}</math>,
:that is, the value of any flow is no greater than the value of any cut.
}}
{{Proof|By the definition of <math>s</math>-<math>t</math> cut, <math>s\in S</math> and <math>t\not\in S</math>.

Due to the conservation of flow,
:<math>\sum_{u\in S}\left(\sum_{v:(u,v)\in E}f_{uv}-\sum_{v:(v,u)\in E}f_{vu}\right)=\sum_{v:(s,v)\in E}f_{sv}+\sum_{u\in S\setminus\{s\}}\left(\sum_{v:(u,v)\in E}f_{uv}-\sum_{v:(v,u)\in E}f_{vu}\right)=\sum_{v:(s,v)\in E}f_{sv}\,.</math>
On the other hand, summing flow over edges,
:<math>\sum_{v\in S}\left(\sum_{u:(u,v)\in E}f_{uv}-\sum_{u:(v,u)\in E}f_{vu}\right)=\sum_{u\in S,v\in S\atop (u,v)\in E}\left(f_{uv}-f_{uv}\right)+\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}=\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}\,.</math>
Therefore,
:<math>\sum_{v:(s,v)\in E}f_{sv}=\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}-\sum_{u\in S,v\not\in S\atop (v,u)\in E}f_{vu}\le\sum_{u\in S,v\not\in S\atop (u,v)\in E}f_{uv}\le \sum_{u\in S,v\not\in S\atop (u,v)\in E}c_{uv}\,,</math>
}}

=== The augmenting paths ===
{{Theorem|Definition (Augmenting path)|
:Let <math>f</math> be a flow in <math>G</math>. An '''augmenting path to <math>u_k</math>''' is a sequence of distinct vertices <math>P=(u_0,u_1,\cdots, u_k)</math>, such that
:* <math>u_0=s\,</math>;
:and each pair of consecutive vertices <math>u_{i}u_{i+1}\,</math> in <math>P</math> corresponds to either a '''forward edge''' <math>(u_{i},u_{i+1})\in E</math> or a '''reverse edge''' <math>(u_{i+1},u_{i})\in E</math>, and
:* <math>f(u_i,u_{i+1})<c(u_i,u_{i+1})\,</math> when <math>u_{i}u_{i+1}\,</math> corresponds to a forward edge <math>(u_{i},u_{i+1})\in E</math>, and
:* <math>f(u_{i+1},u_i)>0\,</math> when <math>u_{i}u_{i+1}\,</math> corresponds to a reverse edge <math>(u_{i+1},u_{i})\in E</math>.
:If <math>u_k=t\,</math>, we simply call <math>P</math> an '''augmenting path'''.
}}

Let <math>f</math> be a flow in <math>G</math>. Suppose there is an augmenting path <math>P=u_0u_1\cdots u_k</math>, where <math>u_0=s</math> and <math>u_k=t</math>. Let <math>\epsilon>0</math> be a positive constant satisfying
*<math>\epsilon \le c(u_{i},u_{i+1})-f(u_i,u_{i+1})</math> for all forward edges <math>(u_{i},u_{i+1})\in E</math> in <math>P</math>;
*<math>\epsilon \le f(u_{i+1},u_i)</math> for all reverse edges <math>(u_{i+1},u_i)\in E</math> in <math>P</math>.
Due to the definition of augmenting path, we can always find such a positive <math>\epsilon</math>.

Increase <math>f(u_i,u_{i+1})</math> by <math>\epsilon</math> for all forward edges <math>(u_{i},u_{i+1})\in E</math> in <math>P</math> and decrease <math>f(u_{i+1},u_i)</math> by <math>\epsilon</math> for all reverse edges <math>(u_{i+1},u_i)\in E</math> in <math>P</math>. Denote the modified flow by <math>f'</math>. It is easy to see that <math>f'</math> satisfies the capacity constraint and conservation constraint thus is still a valid flow. On the other hand, the value of the new flow <math>f'</math>
:<math>\sum_{v:(s,v)\in E}f_{sv}'=\epsilon+\sum_{v:(s,v)\in E}f_{sv}>\sum_{v:(s,v)\in E}f_{sv}</math>.
Therefore, the value of the flow can be "augmented" by adjusting the flow on the augmenting path. This immediately implies that if a flow is maximum, then there is no augmenting path. Surprisingly, the converse is also true, thus maximum flows are "characterized" by augmenting paths.

{{Theorem|Lemma|
:A flow <math>f</math> is maximum if and only if there are no augmenting paths.
}}
{{Proof|We have already proved the "only if" direction above. Now we prove the "if" direction.

}}

=== The max-flow min-cut theorem ===

== Unimodularity ==

=== Integrality of polytopes ===

=== Unimodularity and total unimodularity ===

Combinatorics (Fall 2010)/Extremal set theory

2010-12-05T07:53:38Z

172.21.15.93: /* First proof (shadows) */

== Sperner system ==
A set family <math>\mathcal{F}\subseteq 2^X</math> with the relation <math>\subseteq</math> define a poset. Thus, a '''chain''' is a sequence <math>S_1\subseteq S_2\subseteq\cdots\subseteq S_k</math>.

A set family <math>\mathcal{F}\subseteq 2^X</math> is an '''antichain''' (also called a '''Sperner system''') if for all <math>S,T\in\mathcal{F}</math> that <math>S\neq T</math>, we have <math>S\not\subseteq T</math>.

The <math>k</math>-uniform <math>{X\choose k}</math> is an antichain. Let <math>n=|X|</math>. The size of <math>{X\choose k}</math> is maximized when <math>k=\lfloor n/2\rfloor</math>. We wonder whether this is also the largest possible size of any antichain <math>\mathcal{F}\subseteq 2^X</math>.

In 1928, Emanuel Sperner proved a theorem saying that it is indeed the largest possible antichain. This result, called Sperner's theorem today, initiated the studies of extremal set theory.

{{Theorem|Theorem (Sperner 1928)|
:Let <math>\mathcal{F}\subseteq 2^X</math> where <math>|X|=n</math>. If <math>\mathcal{F}</math> is an antichain, then
::<math>|\mathcal{F}|\le{n\choose \lfloor n/2\rfloor}</math>.
}}

=== First proof (shadows)===
We first introduce the original proof by Sperner, which uses concepts called '''shadows''' and '''shades''' of set systems.

{{Theorem|Definition|
:Let <math>|X|=n\,</math> and <math>\mathcal{F}\subseteq {X\choose k}</math>, <math>k<n\,</math>.
:The '''shade''' of <math>\mathcal{F}</math> is defined to be
::<math>\nabla\mathcal{F}=\left\{T\in {X\choose k+1}\,\,\bigg|\,\, \exists S\in\mathcal{F}\mbox{ such that } S\subset T\right\}</math>.
:Thus the shade <math>\nabla\mathcal{F}</math> of <math>\mathcal{F}</math> consists of all subsets of <math>X</math> which can be obtained by adding an element to a set in <math>\mathcal{F}</math>.
:Similarly, the '''shadow''' of <math>\mathcal{F}</math> is defined to be
::<math>\Delta\mathcal{F}=\left\{T\in {X\choose k-1}\,\,\bigg|\,\, \exists S\in\mathcal{F}\mbox{ such that } T\subset S\right\}</math>.
:Thus the shadow <math>\Delta\mathcal{F}</math> of <math>\mathcal{F}</math> consists of all subsets of <math>X</math> which can be obtained by removing an element from a set in <math>\mathcal{F}</math>.
}}

Next lemma bounds the effects of shadows and shades on the sizes of set systems.

{{Theorem|Lemma (Sperner)|
:Let <math>|X|=n\,</math> and <math>\mathcal{F}\subseteq {X\choose k}</math>. Then
::<math>
\begin{align}
&|\nabla\mathcal{F}|\ge\frac{n-k}{k+1}|\mathcal{F}| &\text{ if } k<n\\
&|\Delta\mathcal{F}|\ge\frac{k}{n-k+1}|\mathcal{F}| &\text{ if } k>0.
\end{align}
</math>
}}

{{Proof|
The lemma is proved by double counting. We prove the inequality of <math>|\nabla\mathcal{F}|</math>. Assume that <math>0\le k<n</math>.

Define
:<math>\mathcal{R}=\{(S,T)\mid S\in\mathcal{F}, T\in\nabla\mathcal{F}, S\subset T\}</math>.
We estimate <math>|\mathcal{R}|</math> in two ways.

For each <math>S\in\mathcal{F}</math>, there are <math>n-k</math> different <math>T\in\nabla\mathcal{F}</math> that <math>S\subset T</math>.
:<math>|\mathcal{R}|=(n-k)|\mathcal{F}|</math>.
For each <math>T\in\nabla\mathcal{F}</math>, there are <math>k+1</math> ways to choose an <math>S\subset T</math> with <math>|S|=k</math>, some of which may not be in <math>\mathcal{F}</math>.
:<math>|\mathcal{R}|\le (k+1)|\nabla\mathcal{F}|</math>.

Altogether, we show that <math>|\nabla\mathcal{F}|\ge\frac{n-k}{k+1}|\mathcal{F}|</math>.

The inequality of <math>|\Delta\mathcal{F}|</math> can be proved in the same way.
}}

An immediate corollary of the previous lemma is as follows.

{{Theorem|Proposition 1|
:If <math>k\le \frac{n-1}{2}</math>, then <math>|\nabla\mathcal{F}|\ge|\mathcal{F}|</math>.
:If <math>k\ge \frac{n-1}{2}</math>, then <math>|\Delta\mathcal{F}|\ge|\mathcal{F}|</math>.
}}

The idea of Sperner's proof is pretty clear:
* we "push up" all the sets in <math>\mathcal{F}</math> of size <math><\frac{n-1}{2}</math> replacing them by their shades;
* and also "push down" all the sets in <math>\mathcal{F}</math> of size <math>\ge\frac{n+1}{2}</math> replacing them by their shadows.
Repeat this process we end up with a set system <math>\mathcal{F}\subseteq{X\choose \lfloor n/2\rfloor}</math>. We need to show that this process does not decrease the size of <math>\mathcal{F}</math>.

{{Theorem|Proposition 2|
:Suppose that <math>\mathcal{F}\subseteq2^X</math> where <math>|X|=n</math>. Let <math>k_\min</math> be the smallest <math>k</math> that <math>|\mathcal{F}_k|>0</math>, and let
::<math>
\mathcal{F}'=\begin{cases}
\mathcal{F}\setminus\mathcal{F}_k\cup \nabla\mathcal{F}_k & \mbox{if }k_\min<\frac{n-1}{2},\\
\mathcal{F} & \mbox{otherwise.}
\end{cases}
</math>
:Similarly, let <math>k_\max</math> be the largest <math>k</math> that <math>|\mathcal{F}_k|>0</math>, and let
::<math>
\mathcal{F}''=\begin{cases}
\mathcal{F}\setminus\mathcal{F}_k\cup \Delta\mathcal{F}_k & \mbox{if }k_\max\ge\frac{n+1}{2},\\
\mathcal{F} & \mbox{otherwise.}
\end{cases}
</math>
:If <math>\mathcal{F}</math> is an antichain, <math>\mathcal{F}'</math> and <math>\mathcal{F}''</math> are antichains, and we have <math>|\mathcal{F}'|\ge|\mathcal{F}|</math> and <math>|\mathcal{F}''|\ge|\mathcal{F}|</math>.
}}
{{Proof|
We show that <math>\mathcal{F}'</math> is an antichain and <math>|\mathcal{F}'|\ge|\mathcal{F}|</math>.

First, observe that <math>\nabla\mathcal{F}_k\cap\mathcal{F}=\emptyset</math>, otherwise <math>\mathcal{F}</math> cannot be an antichain, and due to Proposition 1, <math>|\nabla\mathcal{F}_k|\ge|\mathcal{F}_k|</math> when <math>k\le \frac{n-1}{2}</math>, so <math>|\mathcal{F}'|=|\mathcal{F}|-|\mathcal{F}_k|+|\nabla\mathcal{F}_k|\ge |\mathcal{F}|</math>.

Now we prove that <math>\mathcal{F}'</math> is an antichain . By contradiction, assume that there are <math>S, T\in \mathcal{F}'</math>, such that <math>S\subset T</math>. One of the <math>S,T</math> must be in <math>\nabla\mathcal{F}_k</math>, or otherwise <math>\mathcal{F}</math> cannot be an antichain. Recall that <math>k_\min</math> is the smallest <math>k</math> that <math>|\mathcal{F}_k|>0</math>, thus it must be <math>S\in \nabla\mathcal{F}_k</math>, and <math>T\in\mathcal{F}</math>. This implies that there is an <math>R\in \mathcal{F}_k\subseteq \mathcal{F}</math> such that <math>R\subset S\subset T</math>, which contradicts that <math>\mathcal{F}</math> is an antichain.

The statement for <math>\mathcal{F}''</math> can be proved in the same way.
}}

Applying the above process, we prove the Sperner's theorem.
{{Prooftitle|Proof of Sperner's theorem | (original proof of Sperner)
Let <math>\mathcal{F}_k=\{S\in\mathcal{F}\mid |S|=k\}</math>, where <math>0\le k\le n</math>.

We change <math>\mathcal{F}</math> as follows:
* for the smallest <math>k</math> that <math>|\mathcal{F}_k|>0</math>, if <math>k<\frac{n-1}{2}</math>, replace <math>\mathcal{F}_k</math> by <math>\nabla\mathcal{F}_k</math>.

Due to Proposition 2, this procedure preserves <math>\mathcal{F}</math> as an antichain and does not decrease <math>|\mathcal{F}|</math>. Repeat this procedure, until <math>|\mathcal{F}_k|=0</math> for all <math>k<\frac{n-1}{2}</math>, that is, there is no member set of <math>\mathcal{F}</math> has size less than <math>\frac{n-1}{2}</math>.

We then define another symmetric procedure:
* for the largest <math>k</math> that <math>|\mathcal{F}_k|>0</math>, if <math>k\ge\frac{n+1}{2}</math>, replace <math>\mathcal{F}_k</math> by <math>\Delta\mathcal{F}_k</math>.
Also due to Proposition 2, this procedure preserves <math>\mathcal{F}</math> as an antichain and does not decrease <math>|\mathcal{F}|</math>. After repeatedly applying this procedure, <math>|\mathcal{F}_k|=0</math> for all <math>k\ge\frac{n+1}{2}</math>.

The resulting <math>\mathcal{F}</math> has <math>\mathcal{F}\subseteq{X\choose \lfloor n/2\rfloor}</math>, and since <math>|\mathcal{F}|</math> is never decreased, for the original <math>\mathcal{F}</math>, we have
:<math>|\mathcal{F}|\le {n\choose \lfloor n/2\rfloor}</math>.
}}

=== Second proof (counting)===
We now introduce an elegant proof due to Lubell. The proof uses a counting argument, and tells more information than just the size of the set system.

{{Prooftitle|Proof of Sperner's theorem | (Lubell 1966)
Let <math>\pi</math> be a permutation of <math>X</math>. We say that an <math>S\subseteq X</math> '''prefixes''' <math>\pi</math>, if <math>S=\{\pi_1,\pi_2,\ldots, \pi_{|S|}\}</math>, that is, <math>S</math> is precisely the set of the first <math>|S|</math> elements in the permutation <math>\pi</math>.

Fix an <math>S\subseteq X</math>. It is easy to see that the number of permutations <math>\pi</math> of <math>X</math> prefixed by <math>S</math> is <math>|S|!(n-|S|)!</math>. Also, since <math>\mathcal{F}</math> is an antichain, no permutation <math>\pi</math> of <math>X</math> can be prefixed by more than one members of <math>\mathcal{F}</math>, otherwise one of the member sets must contain the other, which contradicts that <math>\mathcal{F}</math> is an antichain. Thus, the number of permutations <math>\pi</math> prefixed by some <math>S\in\mathcal{F}</math> is
:<math>\sum_{S\in\mathcal{F}}|S|!(n-|S|)!</math>,
which cannot be larger than the total number of permutations, <math>n!</math>, therefore,
:<math>\sum_{S\in\mathcal{F}}|S|!(n-|S|)!\le n!</math>.
Dividing both sides by <math>n!</math>, we have
:<math>\sum_{S\in\mathcal{F}}\frac{1}{{n\choose |S|}}=\sum_{S\in\mathcal{F}}\frac{|S|!(n-|S|)!}{n!}\le 1</math>,
where <math>{n\choose |S|}\le {n\choose \lfloor n/2\rfloor}</math>, so
:<math>\sum_{S\in\mathcal{F}}\frac{1}{{n\choose |S|}}\ge \frac{|\mathcal{F}|}{{n\choose \lfloor n/2\rfloor}}</math>.
Combining this with the above inequality, we prove the Sperner's theorem.
}}

=== The LYM inequality ===
Lubell's proof proves the following inequality:
:<math>\sum_{S\in\mathcal{F}}\frac{1}{{n\choose |S|}}\le 1</math>
which is actually stronger than Sperner's original statement that <math>|\mathcal{F}|\le{n\choose \lfloor n/2\rfloor}</math>.

This inequality is independently discovered by Lubell-Yamamoto, Meschalkin, and Bollobás, and is called the LYM inequality today.

{{Theorem|Theorem (Lubell, Yamamoto 1954; Meschalkin 1963)|
:Let <math>\mathcal{F}\subseteq 2^X</math> where <math>|X|=n</math>. If <math>\mathcal{F}</math> is an antichain, then
::<math>\sum_{S\in\mathcal{F}}\frac{1}{{n\choose |S|}}\le 1</math>.
}}

In Lubell's counting argument proves the LYM inequality, which implies the Sperner's theorem. Here we give another proof of the LYM inequality by the probabilistic method, due to Noga Alon.

{{Prooftitle|Third proof (the probabilistic method)| (Due to Alon.)
Let <math>\pi</math> be a uniformly random permutation of <math>X</math>. Define a random maximal chain by
:<math>\mathcal{C}_\pi=\{\{\pi_i\mid 1\le i\le k\}\mid 0\le k\le n\}</math>.
For any <math>S\in\mathcal{F}</math>, let <math>X_S</math> be the 0-1 random variable which indicates whether <math>S\in\mathcal{C}_\pi</math>, that is
:<math>
X_S=\begin{cases}
1 & \mbox{if }S\in\mathcal{C}_\pi,\\
0 & \mbox{otherwise.}
\end{cases}
</math>
Note that for a uniformly random <math>\pi</math>, <math>\mathcal{C}_\pi</math> has exact one member set of size <math>|S|</math>, uniformly distributed over <math>{X\choose |S|}</math>, thus
:<math>\mathbf{E}[X_S]=\Pr[S\in\mathcal{C}_\pi]=\frac{1}{{n\choose |S|}}</math>.
Let <math>X=\sum_{S\in\mathcal{F}}X_S</math>. Note that <math>X=|\mathcal{F}\cap\mathcal{C}_\pi|</math>. By the linearity of expectation,
:<math>\mathbf{E}[X]=\sum_{S\in\mathcal{F}}\mathbf{E}[X_S]=\sum_{S\in\mathcal{F}}\frac{1}{{n\choose |S|}}</math>.
On the other hand, since <math>\mathcal{F}</math> is an antichain, it can never intersect a chain at more than one elements, thus we always have <math>X=|\mathcal{F}\cap\mathcal{C}_\pi|\le 1</math>. Therefore,
:<math>\sum_{S\in\mathcal{F}}\frac{1}{{n\choose |S|}}\le \mathbf{E}[X] \le 1</math>.
}}

The Sperner's theorem is an immediate consequence of the LYM inequality.

{{Theorem|Proposition|
:<math>\sum_{S\in\mathcal{F}}\frac{1}{{n\choose |S|}}\le 1</math> implies that <math>|\mathcal{F}|\le{n\choose \lfloor n/2\rfloor}</math>.
}}
{{Proof|
It holds that <math>{n\choose k}\le {n\choose \lfloor n/2\rfloor}</math> for any <math>k</math>. Thus,
:<math>1\ge \sum_{S\in\mathcal{F}}\frac{1}{{n\choose |S|}}\ge \frac{|\mathcal{F}|}{{n\choose \lfloor n/2\rfloor}}</math>,
which implies that <math>|\mathcal{F}|\le {n\choose \lfloor n/2\rfloor}</math>.
}}

== Sunflowers ==
An set system is a '''sunflower''' if all its member sets intersect at the same set of elements.
{{Theorem|Definition (sunflower)|
: A set family <math>\mathcal{F}\subseteq 2^X</math> is a '''sunflower''' of size <math>r</math> with a '''core''' <math>C\subseteq X</math> if
::<math>\forall S,T\in\mathcal{F}</math> that <math>S\neq T</math>, <math>S\cap T=C</math>.
}}
Note that we do not require the core to be nonempty, thus a family of disjoint sets is also a sunflower (with the core <math>\emptyset</math>).

The next result due to Erdős and Rado, called the sunflower lemma, is a famous result in extremal set theory, and has some important applications in Boolean circuit complexity.
{{Theorem|Sunflower Lemma (Erdős-Rado)|
:Let <math>\mathcal{F}\subseteq {X\choose k}</math>. If <math>|\mathcal{F}|>k!(r-1)^k</math>, then <math>\mathcal{F}</math> contains a sunflower of size <math>r</math>.
}}
{{Proof|
We proceed by induction on <math>k</math>. For <math>k=1</math>, <math>\mathcal{F}\subseteq{X\choose 1}</math>, thus all sets in <math>\mathcal{F}</math> are disjoint. And since <math>|\mathcal{F}|>r-1</math>, we can choose <math>r</math> of these sets and form a sunflower.

Now let <math>k\ge 2</math> and assume the lemma holds for all smaller <math>k</math>. Take a maximal family <math>\mathcal{G}\subseteq \mathcal{F}</math> whose members are disjoint, i.e. for any <math>S,T\in \mathcal{G}</math> that <math>S\neq T</math>, <math>S\cap T=\emptyset</math>.

If <math>|\mathcal{G}|\ge r</math>, then <math>\mathcal{G}</math> is a sunflower of size at least <math>r</math> and we are done.

Assume that <math>|\mathcal{G}|\le r-1</math>, and let <math>Y=\bigcup_{S\in\mathcal{G}}S</math>. Then <math>|Y|=k|\mathcal{G}|\le k(r-1)</math> (since all members of <math>\mathcal{G}</math>) are disjoint). We claim that <math>Y</math> intersets all members of <math>\mathcal{F}</math>, since if otherwise, there exists an <math>S\in\mathcal{F}</math> such that <math>S\cap Y=\emptyset</math>, then we can enlarge <math>\mathcal{G}</math> by adding <math>S</math> into <math>\mathcal{G}</math> and still have all members of <math>\mathcal{G}</math> disjoint, which contradicts the assumption that <math>\mathcal{G}</math> is the maximum of such families.

By the pigeonhole principle, some elements <math>y\in Y</math> must contained in at least
:<math>\frac{|\mathcal{F}|}{|Y|}>\frac{k!(r-1)^k}{k(r-1)}=(k-1)!(r-1)^{k-1}</math>
members of <math>\mathcal{F}</math>. We delete this <math>y</math> from these sets and consider the family
:<math>\mathcal{H}=\{S\setminus\{y\}\mid S\in\mathcal{F}\wedge y\in S\}</math>.
We have <math>\mathcal{H}\subseteq {X\choose k-1}</math> and <math>|\mathcal{H}|>(k-1)!(r-1)^{k-1}</math>, thus by the induction hypothesis, <math>\mathcal{H}</math>contains a sunflower of size <math>r</math>. Adding <math>y</math> to the members of this sunflower, we get the desired sunflower in the original family <math>\mathcal{F}</math>.
}}

==The Erdős–Ko–Rado Theorem ==
A set family <math>\mathcal{F}\subseteq 2^X</math> is called '''intersecting''', if for any <math>S,T\in\mathcal{F}</math>, <math>S\cap T\neq\emptyset</math>. A natural question of extremal favor is: "how large can an intersecting family be?"

Assume <math>|X|=n</math>. When <math>n<2k</math>, every pair of <math>k</math>-subsets of <math>X</math> intersects. So the non-trivial case is when <math>n\le 2k</math>. The famous Erdős–Ko–Rado theorem gives the largest possible cardinality of a nontrivially intersecting family.

According to Erdős, the theorem itself was proved in 1938, but was not published until 23 years later.
{{Theorem|Erdős–Ko–Rado theorem (proved in 1938, published in 1961)|
:Let <math>\mathcal{F}\subseteq {X\choose k}</math> where <math>|X|=n</math> and <math>n\ge 2k</math>. If <math>\mathcal{F}</math> is intersecting, then
::<math>|\mathcal{F}|\le{n-1\choose k-1}</math>.
}}

=== Katona's proof ===
We first introduce a proof discovered by Katona in 1972. The proof uses double counting.

Let <math>\pi</math> be a '''cyclic permutation''' of <math>X</math>, that is, we think of assigning <math>X</math> in a circle and ignore the rotations of the circle. It is easy to see that there are <math>(n-1)!</math> cyclic permutations of an <math>n</math>-set (each cyclic permutation corresponds to <math>n</math> permutations).
Let
:<math>\mathcal{G}_\pi=\{\{\pi_{(i+j)\bmod n}\mid j\in[k]\}\mid i\in [n]\}</math>.

The next lemma states the following observation: in a circle of <math>n</math> points, supposed <math>n\ge 2k</math>, there can be at most <math>k</math> arcs, each consisting of <math>k</math> points, such that every pair of arcs share at least one point.
{{Theorem|Lemma|
:Let <math>\mathcal{F}\subseteq {X\choose k}</math> where <math>|X|=n</math> and <math>n\ge 2k</math>. If <math>\mathcal{F}</math> is intersecting, then for any cyclic permutation <math>\pi</math> of <math>X</math>, it holds that <math>|\mathcal{G}_\pi\cap\mathcal{F}|\le k</math>.
}}
{{Proof|
Fix a cyclic permutation <math>\pi</math> of <math>X</math>. Let <math>A_i=\{\pi_{(i+j+n)\bmod n}\mid j\in[k]\}</math>. Then <math>\mathcal{G}_\pi</math> can be written as <math>\mathcal{G}_\pi=\{A_i\mid i\in [n]\}</math>.

Suppose that <math>A_t\in\mathcal{F}</math>. Since <math>\mathcal{F}</math> is intersecting, the only sets <math>A_i</math> that can be in <math>\mathcal{F}</math> other than <math>A_t</math> itself are the <math>2k-2</math> sets <math>A_i</math> with <math>t-(k-1)\le i\le t+k-1, i\neq t</math>. We partition these sets into <math>k-1</math> pairs <math>\{A_i,A_{i+k}\}</math>, where <math>t-(k-1)\le i\le t-1</math>.

Note that for <math>n\ge 2k</math>, it holds that <math>A_i\cap C_{i+k}=\emptyset</math>. Since <math>\mathcal{F}</math> is intersecting, <math>\mathcal{F}</math> can contain at most one set of each such pair. The lemma follows.
}}

The Katona's proof of Erdős–Ko–Rado theorem is done by counting in two ways the pairs of member <math>S</math> of <math>\mathcal{F}</math> and cyclic permutation <math>\pi</math> which contain <math>S</math> as a continuous path on the circle (i.e., an arc).

{{Prooftitle|Katona's proof of Erdős–Ko–Rado theorem|(double counting)
Let
:<math>\mathcal{R}=\{(S,\pi)\mid \pi \text{ is a cyclic permutation of }X, \text{and }S\in\mathcal{F}\cap\mathcal{G}_\pi\}</math>.
We count <math>\mathcal{R}</math> in two ways.

First, due to the lemma, <math>|\mathcal{F}\cap\mathcal{G}_\pi|\le k</math> for any cyclic permutation <math>\pi</math>. There are <math>(n-1)!</math> cyclic permutations in total. Thus,
:<math>|\mathcal{R}|=\sum_{\text{cyclic }\pi}|\mathcal{F}\cap\mathcal{G}_\pi|\le k(n-1)!</math>.

Next, for each <math>S\in\mathcal{F}</math>, the number of cyclic permutations <math>\pi</math> in which <math>S</math> is continuous is <math>|S|!(n-|S|)!=k!(n-k)!</math>. Thus,
:<math>|\mathcal{R}|=\sum_{S\in\mathcal{F}}k!(n-k)!=|\mathcal{F}|k!(n-k)!</math>.

Altogether, we have
:<math>|\mathcal{F}|\le\frac{k(n-1)!}{k!(n-k)!}=\frac{(n-1)!}{(k-1)!(n-k)!}={n-1\choose k-1}</math>.
}}

=== Erdős' shifting technique ===
We now introduce the original proof of the Erdős–Ko–Rado theorem, which uses a technique called '''shifting''' (originally called '''compression''').

Without loss of generality, we assume <math>X=[n]</math>, and restate the Erdős–Ko–Rado theorem as follows.
{{Theorem|Erdős–Ko–Rado theorem|
:Let <math>\mathcal{F}\subseteq {[n]\choose k}</math> and <math>n\ge 2k</math>. If <math>\mathcal{F}</math> is intersecting, then <math>|\mathcal{F}|\le{n-1\choose k-1}</math>.
}}

We define a '''shift operator''' for the set family.
{{Theorem|Definition (shift operator)|
: Assume <math>\mathcal{F}\subseteq 2^{[n]}</math>, and <math>0\le i<j\le n-1</math>. Define the '''<math>(i,j)</math>-shift''' <math>S_{ij}</math> as an operator on <math>\mathcal{F}</math> as follows:
:*for each <math>T\in\mathcal{F}</math>, write <math>T_{ij}=(T\setminus\{j\})\cup\{i\} </math>, and let
::<math>S_{ij}(T)=
\begin{cases}
T_{ij} & \mbox{if }j\in T, i\not\in T, \mbox{ and }T_{ij} \not\in\mathcal{F},\\
T & \mbox{otherwise;}
\end{cases}</math>
:* let <math>S_{ij}(\mathcal{F})=\{S_{ij}(T)\mid T\in \mathcal{F}\}</math>.
}}

It is easy to verify the following propositions of shifts.

{{Theorem|Proposition|
# <math>|S_{ij}(T)|=|T|\,</math> and <math>|S_{ij}(\mathcal{F})|=\mathcal{F}</math>;
# if <math>\mathcal{F}</math> is intersecting, then so is <math>S_{ij}(\mathcal{F})</math>.
}}
{{Proof|
(1) is immediate. Now we prove (2).

Let <math>A,B\in\mathcal{F}</math>. All the cases are easy to dealt with except when <math>A\cap B=\{j\}</math>, <math>i\in A</math>, and <math>i\not\in B</math>. Denote <math>A_{ij}=A\setminus\{j\}\cup\{i\}</math>. It holds that <math>A_{ij}\cap B=(A\cap B)\setminus\{j\}=\emptyset</math>. Since <math>\mathcal{F}</math> is intersecting, it must hold that <math>A_{ij}\not\in\mathcal{F}</math>. Thus, <math>S_{ij}(A)=A_{ij}</math> and clearly <math>S_{ij}(B)=B_{ij}=B\setminus\{j\}\cup\{i\}</math>. Therefore, <math>i\in S_{ij}(A)\cap S_{ij}(B)</math> thus <math>S_{ij}(A)\cap S_{ij}(B)\neq \emptyset</math>.
}}

Repeatedly applying <math>S_{ij}(\mathcal{F})</math> for any <math>0\le i<j\le n-1</math>, since we only replace elements by smaller elements, eventually <math>\mathcal{F}</math> will stop changing, that is, <math>S_{ij}(\mathcal{F})=\mathcal{F}</math> for all <math>0\le i<j\le n-1</math>. We call such an <math>\mathcal{F}</math> '''shifted'''.

The idea behind the shifting technique is very natural: by applying shifting, all intersecting families are transformed to some ''special forms'', and we only need to prove the theorem for these special form of intersecting families.

{{Prooftitle|Proof of Erdős-Ko-Rado theorem| (The original proof of Erdős-Ko-Rado by shifting)
By the above lemma, it is sufficient to prove the Erdős-Ko-Rado theorem holds for shifted <math>\mathcal{F}</math>. We assume that <math>\mathcal{F}</math> is shifted.

First, it is trivial to see that the theorem holds for <math>k=1</math> (no matter whether shifted).

Next, we show that the theorem holds when <math>n=2k</math> (no matter whether shifted). For any <math>S\in{X\choose k}</math>, both <math>S</math> and <math>X\setminus S</math> are in <math>{X\choose k}</math>, but at most one of them can be in <math>\mathcal{F}</math>. Thus,
:<math>|\mathcal{F}|\le\frac{1}{2}{n\choose k}=\frac{n!}{2k!(n-k)!}=\frac{(n-1)!}{(k-1)!(n-k)!}={n-1\choose k-1}</math>.

We then apply the induction on <math>n</math>. For <math>n> 2k</math>, the induction hypothesis is stated as:
* the Erdős-Ko-Rado theorem holds for any smaller <math>n</math>.
Define
:<math>\mathcal{F}_0=\{S\in\mathcal{F}\mid n\not\in S\}</math> and <math>\mathcal{F}_1=\{S\in\mathcal{F}\mid n\in S\}</math>.
Clearly, <math>\mathcal{F}_0\subseteq{[n-1]\choose k}</math> and <math>\mathcal{F}_0</math> is intersecting. Due to the induction hypothesis, <math>|\mathcal{F}_0|\le{n-2\choose k-1}</math>.

In order to apply the induction, we let
:<math>\mathcal{F}_1'=\{S\setminus\{n\}\mid S\in\mathcal{F}_1\}</math>.
Clearly, <math>\mathcal{F}_1'\subseteq{[n-1]\choose k-1}</math>. If only it is also intersecting, we can apply the induction hypothesis, and indeed it is. To see this, by contradiction we assume that <math>\mathcal{F}_1'</math> is not intersecting. Then there must exist <math>A,B\in\mathcal{F}</math> such that <math>A\cap B=\{n\}</math>, which means that <math>|A\cup B|\le 2k-1<n-1</math>. Thus, there is some <math>0\le i\le n-1</math> such that <math>i\not\in A\cup B</math>. Since <math>\mathcal{F}</math> is shifted, <math>A_{in}=A\setminus\{n\}\cup\{i\}\in\mathcal{F}</math>. On the other hand it can be verified that <math>A_{in}\cap B=\emptyset</math>, which contradicts that <math>\mathcal{F}</math> is intersecting.

Thus, <math>\mathcal{F}_1'\subseteq{[n-1]\choose k-1}</math> and <math>\mathcal{F}_1'</math> is intersecting. Due to the induction hypothesis, <math>|\mathcal{F}_1'|\le{n-2\choose k-2}</math>.

Combining these together,
:<math>|\mathcal{F}|=|\mathcal{F}_0|+|\mathcal{F}_1|=|\mathcal{F}_0|+|\mathcal{F}_1'|\le {n-2\choose k-1}+{n-2\choose k-2}={n-1\choose k-1}</math>.
}}

== References ==
:('''声明:''' 资料受版权保护, 仅用于教学.)
:('''Disclaimer:''' The following copyrighted materials are meant for educational uses only.)

* van Lin and Wilson. ''A course in combinatorics.'' Cambridge Press. Chapter 6.
* Aigner and Ziegler. ''Proofs from THE BOOK, 4th Edition.'' Springer-Verlag. [[media:PFTB_chap27.pdf| Chapter 27]].

Combinatorics (Fall 2010)/Extremal graphs

2010-10-14T07:14:17Z

172.21.1.240: /* Erdős–Stone theorem */

== Extremal Graph Theory ==

=== Mantel's theorem ===
We consider a typical extremal problem for graphs: the largest possible number of edges of '''triangle-free''' graphs, i.e. graphs contains no <math>K_3</math>.

{{Theorem|Theorem (Mantel 1907)|
:Suppose <math>G(V,E)</math> is graph on <math>n</math> vertice without triangles. Then <math>|E|\le\frac{n^2}{4}</math>.
}}

{{Prooftitle|First proof. (pigeonhole principle)|
We prove an equivalent theorem: Any <math>G(V,E)</math> with <math>|V|=n</math> and <math>|E|>\frac{n^2}{4}</math> must have a triangle.

Use induction on <math>n</math>. The theorem holds trivially for <math>n\le 3</math>.

Induction hypothesis: assume the theorem hold for <math>|V|\le n-1</math>.

For <math>G</math> with <math>n</math> vertices, without loss of generality, assume that <math>|E|=\frac{n^2}{4}+1</math>, we will show that <math>G</math> must contain a triangle. Take a <math>uv\in E</math>, and let <math>H</math> be the subgraph of <math>G</math> induced by <math>V\setminus \{u,v\}</math>. Clearly, <math>H</math> has <math>n-2</math> vertices.
:'''Case.1:''' If <math>H</math> has <math>>\frac{(n-2)^2}{4}</math> edges, then by the induction hypothesis, <math>H</math> has a triangle.
:'''Case.2:''' If <math>H</math> has <math>\le\frac{(n-2)^2}{4}</math> edges, then at least <math>\left(\frac{n^2}{4}+1\right)-\frac{(n-2)^2}{4}-1=n-1</math> edges are between <math>H</math> and <math>\{u,v\}</math>. By pigeonhole principle, there must be a vertex in <math>H</math> that is adjacent to both <math>u</math> and <math>v</math>. Thus, <math>G</math> has a triangle.
}}

{{Prooftitle|Second proof. (Cauchy-Schwarz inequality)|(Mantel's original proof)
For any edge <math>uv\in E</math>, no vertex can be a neighbor of both <math>u</math> and <math>v</math>, or otherwise there will be a triangle. Thus, for any edge <math>uv\in E</math>, <math>d_u+d_v\le n</math>. It follows that
:<math>\sum_{uv\in E}(d_u+d_v)\le n|E|</math>.
Note that <math>d(v)</math> appears exactly <math>d_v</math> times in the sum, so that
:<math>\sum_{uv\in E}(d_u+d_v)=\sum_{v\in V}d_v^2</math>.
Applying Chauchy-Schwarz inequality,
:<math>
n|E|\ge\sum_{v\in V}d_v^2\ge\frac{\left(\sum_{v\in V}d_v\right)^2}{n}=\frac{4|E|^2}{n},
</math>
where the last equation is due to Euler's equality <math>\sum_{v\in V}d_v=2|E|</math>. The theorem follows.
}}

{{Prooftitle|Third proof. (inequality of the arithmetic and geometric mean)|
Assume that <math>G(V,E)</math> has <math>|V|=n</math> vertices and is triangle-free.

Let <math>A</math> be the largest independent set in <math>G</math> and let <math>\alpha=|A|</math>.
Since <math>G</math> is triangle-free, for very vertex <math>v</math>, all its neighbors must form an independent set, thus <math>d(v)\le \alpha</math> for all <math>v\in V</math>.

Take <math>B=V\setminus A</math> and let <math>\beta=|B|</math>.
Since <math>A</math> is an independent set, all edges in <math>E</math> must have at least one endpoint in <math>B</math>. Counting the edges in <math>E</math> according to their endpoints in <math>B</math>, we obtain <math>|E|\le\sum_{v\in B}d_v</math>. By the inequality of the arithmetic and geometric mean,
:<math>|E|\le\sum_{v\in B}d_v\le\alpha\beta\le\left(\frac{\alpha+\beta}{2}\right)^2=\frac{n^2}{4}</math>.
}}

=== Turán's theorem ===
{{Theorem|Theorem (Turán 1941)|
:Let <math>G(V,E)</math> be a graph with <math>|V|=n</math>. If <math>G</math> has no <math>r</math>-clique, <math>k\ge 2</math>, then
::<math>|E|\le\frac{r-2}{2(r-1)}n^2</math>.
}}

{{Prooftitle|First proof. (induction)|(Turán's original proof)
}}

{{Prooftitle|Second proof. (weight shifting)|(due to Motzkin and Straus)
}}

{{Prooftitle|Third proof. (the probabilistic method)|(due to Alon and Spencer)
}}

{{Prooftitle|Fourth proof.|
Let <math>G(V,E)</math> be a <math>r</math>-clique-free graph on <math>n</math> vertices with a maximum number of edges.
:'''Claim:''' <math>G</math> does not contain three vertices <math>u,v,w</math> such that <math>uv\in E</math> but <math>uw\not\in E, vw\not\in E</math>.
Suppose otherwise. There are two cases.
* '''Case.1:''' <math>d(w)<d(u)</math> or <math>d(w)<d(v)</math>. Without loss of generality, suppose that <math>d(w)<d(u)</math>. We duplicate <math>u</math> by creating a new vertex <math>u'</math> which has exactly the same neighbors as <math>u</math> (but <math>uu'</math> is not an edge). Such duplication will not increase the clique size. We then remove <math>w</math>. The resulting graph <math>G'</math> is still <math>r</math>-clique-free, and has <math>n</math> vertices. The number of edges in <math>G'</math> is
::<math>|E(G')|=|E(G)|+d(u)-d(w)>|E(G)|\,</math>,
:which contradicts the assumption that <math>|E(G)|</math> is maximal.
* '''Case.2:''' <math>d(w)\ge d(u)</math> and <math>d(w)\ge d(v)</math>. Duplicate <math>w</math> twice and delete <math>u</math> and <math>v</math>. The new graph <math>G'</math> has no <math>r</math>-clique, and the number of edges is
::<math>|E(G')|=|E(G)|+2d(w)-(d(u)+d(v)+1)>|E(G)|\,</math>.
:Contradiction again.

The claim implies that <math>uv\not\in E</math> defines an equivalence relation on vertices (to be more precise, it guarantees the transitivity of the relation, while the reflexivity and symmetry hold directly). Graph <math>G</math> must be a complete multipartite graph <math>K_{n_1,n_2,\ldots,n_{r-1}}</math> with <math>n_1+n_2+\cdots +n_{r-1}=n</math>. Optimize the edge number, we have the Turán graph.
}}

=== Erdős–Stone theorem ===
Let <math>K_s^r=K_{\underbrace{s,s,\cdots,s}_{r}}</math> be the complete <math>r</math>-partite graph with <math>s</math> vertices in each class, i.e., the Turán graph <math>T(rs,r)</math>.

{{Theorem|Fundamental theorem of extremal graph theory (Erdős–Stone 1946)|
:For any integers <math>r\ge 2</math> and <math>s\ge 1</math>, and any <math>\epsilon>0</math>, if <math>n</math> is sufficiently large then every graph on <math>n</math> vertices and with at least <math>\left(\frac{r-2}{2(r-1)}+\epsilon\right)n^2</math> edges contains <math>K_{r,s}</math> as a subgraph, i.e.,
:::<math>\mathrm{ex}(n,K_s^r)= \left(\frac{r-2}{2(r-1)}+o(1)\right)n^2</math>.
}}

{{Theorem|Corollary|
:For every nonempty graph <math>H</math>,
::<math>\lim_{n\rightarrow\infty}\frac{\mathrm{ex}(n,H)}{{n\choose 2}}=\frac{\chi(H)-2}{\chi(H)-1}</math>.
}}

== Cycle Structures ==
=== Girth ===

=== Hamiltonian cycle ===

Combinatorics (Fall 2010)/Extremal graphs

2010-10-14T07:04:04Z

172.21.1.240: /* Erdős–Stone theorem */

== Extremal Graph Theory ==

=== Mantel's theorem ===
We consider a typical extremal problem for graphs: the largest possible number of edges of '''triangle-free''' graphs, i.e. graphs contains no <math>K_3</math>.

{{Theorem|Theorem (Mantel 1907)|
:Suppose <math>G(V,E)</math> is graph on <math>n</math> vertice without triangles. Then <math>|E|\le\frac{n^2}{4}</math>.
}}

{{Prooftitle|First proof. (pigeonhole principle)|
We prove an equivalent theorem: Any <math>G(V,E)</math> with <math>|V|=n</math> and <math>|E|>\frac{n^2}{4}</math> must have a triangle.

Use induction on <math>n</math>. The theorem holds trivially for <math>n\le 3</math>.

Induction hypothesis: assume the theorem hold for <math>|V|\le n-1</math>.

For <math>G</math> with <math>n</math> vertices, without loss of generality, assume that <math>|E|=\frac{n^2}{4}+1</math>, we will show that <math>G</math> must contain a triangle. Take a <math>uv\in E</math>, and let <math>H</math> be the subgraph of <math>G</math> induced by <math>V\setminus \{u,v\}</math>. Clearly, <math>H</math> has <math>n-2</math> vertices.
:'''Case.1:''' If <math>H</math> has <math>>\frac{(n-2)^2}{4}</math> edges, then by the induction hypothesis, <math>H</math> has a triangle.
:'''Case.2:''' If <math>H</math> has <math>\le\frac{(n-2)^2}{4}</math> edges, then at least <math>\left(\frac{n^2}{4}+1\right)-\frac{(n-2)^2}{4}-1=n-1</math> edges are between <math>H</math> and <math>\{u,v\}</math>. By pigeonhole principle, there must be a vertex in <math>H</math> that is adjacent to both <math>u</math> and <math>v</math>. Thus, <math>G</math> has a triangle.
}}

{{Prooftitle|Second proof. (Cauchy-Schwarz inequality)|(Mantel's original proof)
For any edge <math>uv\in E</math>, no vertex can be a neighbor of both <math>u</math> and <math>v</math>, or otherwise there will be a triangle. Thus, for any edge <math>uv\in E</math>, <math>d_u+d_v\le n</math>. It follows that
:<math>\sum_{uv\in E}(d_u+d_v)\le n|E|</math>.
Note that <math>d(v)</math> appears exactly <math>d_v</math> times in the sum, so that
:<math>\sum_{uv\in E}(d_u+d_v)=\sum_{v\in V}d_v^2</math>.
Applying Chauchy-Schwarz inequality,
:<math>
n|E|\ge\sum_{v\in V}d_v^2\ge\frac{\left(\sum_{v\in V}d_v\right)^2}{n}=\frac{4|E|^2}{n},
</math>
where the last equation is due to Euler's equality <math>\sum_{v\in V}d_v=2|E|</math>. The theorem follows.
}}

{{Prooftitle|Third proof. (inequality of the arithmetic and geometric mean)|
Assume that <math>G(V,E)</math> has <math>|V|=n</math> vertices and is triangle-free.

Let <math>A</math> be the largest independent set in <math>G</math> and let <math>\alpha=|A|</math>.
Since <math>G</math> is triangle-free, for very vertex <math>v</math>, all its neighbors must form an independent set, thus <math>d(v)\le \alpha</math> for all <math>v\in V</math>.

Take <math>B=V\setminus A</math> and let <math>\beta=|B|</math>.
Since <math>A</math> is an independent set, all edges in <math>E</math> must have at least one endpoint in <math>B</math>. Counting the edges in <math>E</math> according to their endpoints in <math>B</math>, we obtain <math>|E|\le\sum_{v\in B}d_v</math>. By the inequality of the arithmetic and geometric mean,
:<math>|E|\le\sum_{v\in B}d_v\le\alpha\beta\le\left(\frac{\alpha+\beta}{2}\right)^2=\frac{n^2}{4}</math>.
}}

=== Turán's theorem ===
{{Theorem|Theorem (Turán 1941)|
:Let <math>G(V,E)</math> be a graph with <math>|V|=n</math>. If <math>G</math> has no <math>r</math>-clique, <math>k\ge 2</math>, then
::<math>|E|\le\frac{r-2}{2(r-1)}n^2</math>.
}}

{{Prooftitle|First proof. (induction)|(Turán's original proof)
}}

{{Prooftitle|Second proof. (weight shifting)|(due to Motzkin and Straus)
}}

{{Prooftitle|Third proof. (the probabilistic method)|(due to Alon and Spencer)
}}

{{Prooftitle|Fourth proof.|
Let <math>G(V,E)</math> be a <math>r</math>-clique-free graph on <math>n</math> vertices with a maximum number of edges.
:'''Claim:''' <math>G</math> does not contain three vertices <math>u,v,w</math> such that <math>uv\in E</math> but <math>uw\not\in E, vw\not\in E</math>.
Suppose otherwise. There are two cases.
* '''Case.1:''' <math>d(w)<d(u)</math> or <math>d(w)<d(v)</math>. Without loss of generality, suppose that <math>d(w)<d(u)</math>. We duplicate <math>u</math> by creating a new vertex <math>u'</math> which has exactly the same neighbors as <math>u</math> (but <math>uu'</math> is not an edge). Such duplication will not increase the clique size. We then remove <math>w</math>. The resulting graph <math>G'</math> is still <math>r</math>-clique-free, and has <math>n</math> vertices. The number of edges in <math>G'</math> is
::<math>|E(G')|=|E(G)|+d(u)-d(w)>|E(G)|\,</math>,
:which contradicts the assumption that <math>|E(G)|</math> is maximal.
* '''Case.2:''' <math>d(w)\ge d(u)</math> and <math>d(w)\ge d(v)</math>. Duplicate <math>w</math> twice and delete <math>u</math> and <math>v</math>. The new graph <math>G'</math> has no <math>r</math>-clique, and the number of edges is
::<math>|E(G')|=|E(G)|+2d(w)-(d(u)+d(v)+1)>|E(G)|\,</math>.
:Contradiction again.

The claim implies that <math>uv\not\in E</math> defines an equivalence relation on vertices (to be more precise, it guarantees the transitivity of the relation, while the reflexivity and symmetry hold directly). Graph <math>G</math> must be a complete multipartite graph <math>K_{n_1,n_2,\ldots,n_{r-1}}</math> with <math>n_1+n_2+\cdots +n_{r-1}=n</math>. Optimize the edge number, we have the Turán graph.
}}

=== Erdős–Stone theorem ===

{{Theorem|Fundamental theorem of extremal graph theory (Erdős–Stone 1946)|
:For any integers <math>r\ge 2</math> and <math>s\ge 1</math>, and any <math>\epsilon>0</math>, if <math>n</math> is sufficiently large then every graph on <math>n</math> vertices and with at least <math>\left(\frac{r-2}{2(r-1)}+\epsilon\right)n^2</math> edges contains <math>K_{r,s}</math> as a subgraph, i.e.,
:::<math>\mathrm{ex}(n,K_{r,s})= \left(\frac{r-2}{2(r-1)}+o(1)\right)n^2</math>.
}}

{{Theorem|Corollary|
:For every nonempty graph <math>H</math>,
::<math>\lim_{n\rightarrow\infty}\frac{\mathrm{ex}(n,H)}{{n\choose 2}}=\frac{\chi(H)-2}{\chi(H)-1}</math>.
}}

== Cycle Structures ==
=== Girth ===

=== Hamiltonian cycle ===

Combinatorics (Fall 2010)/Extremal graphs

2010-10-14T07:00:05Z

172.21.1.240: /* Erdős–Stone theorem */

== Extremal Graph Theory ==

=== Mantel's theorem ===
We consider a typical extremal problem for graphs: the largest possible number of edges of '''triangle-free''' graphs, i.e. graphs contains no <math>K_3</math>.

{{Theorem|Theorem (Mantel 1907)|
:Suppose <math>G(V,E)</math> is graph on <math>n</math> vertice without triangles. Then <math>|E|\le\frac{n^2}{4}</math>.
}}

{{Prooftitle|First proof. (pigeonhole principle)|
We prove an equivalent theorem: Any <math>G(V,E)</math> with <math>|V|=n</math> and <math>|E|>\frac{n^2}{4}</math> must have a triangle.

Use induction on <math>n</math>. The theorem holds trivially for <math>n\le 3</math>.

Induction hypothesis: assume the theorem hold for <math>|V|\le n-1</math>.

For <math>G</math> with <math>n</math> vertices, without loss of generality, assume that <math>|E|=\frac{n^2}{4}+1</math>, we will show that <math>G</math> must contain a triangle. Take a <math>uv\in E</math>, and let <math>H</math> be the subgraph of <math>G</math> induced by <math>V\setminus \{u,v\}</math>. Clearly, <math>H</math> has <math>n-2</math> vertices.
:'''Case.1:''' If <math>H</math> has <math>>\frac{(n-2)^2}{4}</math> edges, then by the induction hypothesis, <math>H</math> has a triangle.
:'''Case.2:''' If <math>H</math> has <math>\le\frac{(n-2)^2}{4}</math> edges, then at least <math>\left(\frac{n^2}{4}+1\right)-\frac{(n-2)^2}{4}-1=n-1</math> edges are between <math>H</math> and <math>\{u,v\}</math>. By pigeonhole principle, there must be a vertex in <math>H</math> that is adjacent to both <math>u</math> and <math>v</math>. Thus, <math>G</math> has a triangle.
}}

{{Prooftitle|Second proof. (Cauchy-Schwarz inequality)|(Mantel's original proof)
For any edge <math>uv\in E</math>, no vertex can be a neighbor of both <math>u</math> and <math>v</math>, or otherwise there will be a triangle. Thus, for any edge <math>uv\in E</math>, <math>d_u+d_v\le n</math>. It follows that
:<math>\sum_{uv\in E}(d_u+d_v)\le n|E|</math>.
Note that <math>d(v)</math> appears exactly <math>d_v</math> times in the sum, so that
:<math>\sum_{uv\in E}(d_u+d_v)=\sum_{v\in V}d_v^2</math>.
Applying Chauchy-Schwarz inequality,
:<math>
n|E|\ge\sum_{v\in V}d_v^2\ge\frac{\left(\sum_{v\in V}d_v\right)^2}{n}=\frac{4|E|^2}{n},
</math>
where the last equation is due to Euler's equality <math>\sum_{v\in V}d_v=2|E|</math>. The theorem follows.
}}

{{Prooftitle|Third proof. (inequality of the arithmetic and geometric mean)|
Assume that <math>G(V,E)</math> has <math>|V|=n</math> vertices and is triangle-free.

Let <math>A</math> be the largest independent set in <math>G</math> and let <math>\alpha=|A|</math>.
Since <math>G</math> is triangle-free, for very vertex <math>v</math>, all its neighbors must form an independent set, thus <math>d(v)\le \alpha</math> for all <math>v\in V</math>.

Take <math>B=V\setminus A</math> and let <math>\beta=|B|</math>.
Since <math>A</math> is an independent set, all edges in <math>E</math> must have at least one endpoint in <math>B</math>. Counting the edges in <math>E</math> according to their endpoints in <math>B</math>, we obtain <math>|E|\le\sum_{v\in B}d_v</math>. By the inequality of the arithmetic and geometric mean,
:<math>|E|\le\sum_{v\in B}d_v\le\alpha\beta\le\left(\frac{\alpha+\beta}{2}\right)^2=\frac{n^2}{4}</math>.
}}

=== Turán's theorem ===
{{Theorem|Theorem (Turán 1941)|
:Let <math>G(V,E)</math> be a graph with <math>|V|=n</math>. If <math>G</math> has no <math>r</math>-clique, <math>k\ge 2</math>, then
::<math>|E|\le\frac{r-2}{2(r-1)}n^2</math>.
}}

{{Prooftitle|First proof. (induction)|(Turán's original proof)
}}

{{Prooftitle|Second proof. (weight shifting)|(due to Motzkin and Straus)
}}

{{Prooftitle|Third proof. (the probabilistic method)|(due to Alon and Spencer)
}}

{{Prooftitle|Fourth proof.|
Let <math>G(V,E)</math> be a <math>r</math>-clique-free graph on <math>n</math> vertices with a maximum number of edges.
:'''Claim:''' <math>G</math> does not contain three vertices <math>u,v,w</math> such that <math>uv\in E</math> but <math>uw\not\in E, vw\not\in E</math>.
Suppose otherwise. There are two cases.
* '''Case.1:''' <math>d(w)<d(u)</math> or <math>d(w)<d(v)</math>. Without loss of generality, suppose that <math>d(w)<d(u)</math>. We duplicate <math>u</math> by creating a new vertex <math>u'</math> which has exactly the same neighbors as <math>u</math> (but <math>uu'</math> is not an edge). Such duplication will not increase the clique size. We then remove <math>w</math>. The resulting graph <math>G'</math> is still <math>r</math>-clique-free, and has <math>n</math> vertices. The number of edges in <math>G'</math> is
::<math>|E(G')|=|E(G)|+d(u)-d(w)>|E(G)|\,</math>,
:which contradicts the assumption that <math>|E(G)|</math> is maximal.
* '''Case.2:''' <math>d(w)\ge d(u)</math> and <math>d(w)\ge d(v)</math>. Duplicate <math>w</math> twice and delete <math>u</math> and <math>v</math>. The new graph <math>G'</math> has no <math>r</math>-clique, and the number of edges is
::<math>|E(G')|=|E(G)|+2d(w)-(d(u)+d(v)+1)>|E(G)|\,</math>.
:Contradiction again.

The claim implies that <math>uv\not\in E</math> defines an equivalence relation on vertices (to be more precise, it guarantees the transitivity of the relation, while the reflexivity and symmetry hold directly). Graph <math>G</math> must be a complete multipartite graph <math>K_{n_1,n_2,\ldots,n_{r-1}}</math> with <math>n_1+n_2+\cdots +n_{r-1}=n</math>. Optimize the edge number, we have the Turán graph.
}}

=== Erdős–Stone theorem ===

{{Theorem|Fundamental theorem of extremal graph theory (Erdős–Stone 1946)|
:For any integers <math>r\ge 2</math> and <math>s\ge 1</math>, and any <math>\epsilon>0</math>, there exists an <math>N_0</math> such that every graph with <math>n\ge N_0</math> vertices and at least <math>\left(\frac{r-2}{2(r-1)}+\epsilon\right)n^2</math> edges contains <math>K_{r,s}</math> as a subgraph, i.e.,
:::<math>\mathrm{ex}(n,K_{r,s})= \left(\frac{r-2}{2(r-1)}+o(1)\right)n^2</math>.
}}

{{Theorem|Corollary|
:For every nonempty graph <math>H</math>,
::<math>\lim_{n\rightarrow\infty}\frac{\mathrm{ex}(n,H)}{{n\choose 2}}=\frac{\chi(H)-2}{\chi(H)-1}</math>.
}}

== Cycle Structures ==
=== Girth ===

=== Hamiltonian cycle ===

Combinatorics (Fall 2010)/Extremal graphs

2010-10-14T06:59:33Z

172.21.1.240: /* Erdős–Stone theorem */

== Extremal Graph Theory ==

=== Mantel's theorem ===
We consider a typical extremal problem for graphs: the largest possible number of edges of '''triangle-free''' graphs, i.e. graphs contains no <math>K_3</math>.

{{Theorem|Theorem (Mantel 1907)|
:Suppose <math>G(V,E)</math> is graph on <math>n</math> vertice without triangles. Then <math>|E|\le\frac{n^2}{4}</math>.
}}

{{Prooftitle|First proof. (pigeonhole principle)|
We prove an equivalent theorem: Any <math>G(V,E)</math> with <math>|V|=n</math> and <math>|E|>\frac{n^2}{4}</math> must have a triangle.

Use induction on <math>n</math>. The theorem holds trivially for <math>n\le 3</math>.

Induction hypothesis: assume the theorem hold for <math>|V|\le n-1</math>.

For <math>G</math> with <math>n</math> vertices, without loss of generality, assume that <math>|E|=\frac{n^2}{4}+1</math>, we will show that <math>G</math> must contain a triangle. Take a <math>uv\in E</math>, and let <math>H</math> be the subgraph of <math>G</math> induced by <math>V\setminus \{u,v\}</math>. Clearly, <math>H</math> has <math>n-2</math> vertices.
:'''Case.1:''' If <math>H</math> has <math>>\frac{(n-2)^2}{4}</math> edges, then by the induction hypothesis, <math>H</math> has a triangle.
:'''Case.2:''' If <math>H</math> has <math>\le\frac{(n-2)^2}{4}</math> edges, then at least <math>\left(\frac{n^2}{4}+1\right)-\frac{(n-2)^2}{4}-1=n-1</math> edges are between <math>H</math> and <math>\{u,v\}</math>. By pigeonhole principle, there must be a vertex in <math>H</math> that is adjacent to both <math>u</math> and <math>v</math>. Thus, <math>G</math> has a triangle.
}}

{{Prooftitle|Second proof. (Cauchy-Schwarz inequality)|(Mantel's original proof)
For any edge <math>uv\in E</math>, no vertex can be a neighbor of both <math>u</math> and <math>v</math>, or otherwise there will be a triangle. Thus, for any edge <math>uv\in E</math>, <math>d_u+d_v\le n</math>. It follows that
:<math>\sum_{uv\in E}(d_u+d_v)\le n|E|</math>.
Note that <math>d(v)</math> appears exactly <math>d_v</math> times in the sum, so that
:<math>\sum_{uv\in E}(d_u+d_v)=\sum_{v\in V}d_v^2</math>.
Applying Chauchy-Schwarz inequality,
:<math>
n|E|\ge\sum_{v\in V}d_v^2\ge\frac{\left(\sum_{v\in V}d_v\right)^2}{n}=\frac{4|E|^2}{n},
</math>
where the last equation is due to Euler's equality <math>\sum_{v\in V}d_v=2|E|</math>. The theorem follows.
}}

{{Prooftitle|Third proof. (inequality of the arithmetic and geometric mean)|
Assume that <math>G(V,E)</math> has <math>|V|=n</math> vertices and is triangle-free.

Let <math>A</math> be the largest independent set in <math>G</math> and let <math>\alpha=|A|</math>.
Since <math>G</math> is triangle-free, for very vertex <math>v</math>, all its neighbors must form an independent set, thus <math>d(v)\le \alpha</math> for all <math>v\in V</math>.

Take <math>B=V\setminus A</math> and let <math>\beta=|B|</math>.
Since <math>A</math> is an independent set, all edges in <math>E</math> must have at least one endpoint in <math>B</math>. Counting the edges in <math>E</math> according to their endpoints in <math>B</math>, we obtain <math>|E|\le\sum_{v\in B}d_v</math>. By the inequality of the arithmetic and geometric mean,
:<math>|E|\le\sum_{v\in B}d_v\le\alpha\beta\le\left(\frac{\alpha+\beta}{2}\right)^2=\frac{n^2}{4}</math>.
}}

=== Turán's theorem ===
{{Theorem|Theorem (Turán 1941)|
:Let <math>G(V,E)</math> be a graph with <math>|V|=n</math>. If <math>G</math> has no <math>r</math>-clique, <math>k\ge 2</math>, then
::<math>|E|\le\frac{r-2}{2(r-1)}n^2</math>.
}}

{{Prooftitle|First proof. (induction)|(Turán's original proof)
}}

{{Prooftitle|Second proof. (weight shifting)|(due to Motzkin and Straus)
}}

{{Prooftitle|Third proof. (the probabilistic method)|(due to Alon and Spencer)
}}

{{Prooftitle|Fourth proof.|
Let <math>G(V,E)</math> be a <math>r</math>-clique-free graph on <math>n</math> vertices with a maximum number of edges.
:'''Claim:''' <math>G</math> does not contain three vertices <math>u,v,w</math> such that <math>uv\in E</math> but <math>uw\not\in E, vw\not\in E</math>.
Suppose otherwise. There are two cases.
* '''Case.1:''' <math>d(w)<d(u)</math> or <math>d(w)<d(v)</math>. Without loss of generality, suppose that <math>d(w)<d(u)</math>. We duplicate <math>u</math> by creating a new vertex <math>u'</math> which has exactly the same neighbors as <math>u</math> (but <math>uu'</math> is not an edge). Such duplication will not increase the clique size. We then remove <math>w</math>. The resulting graph <math>G'</math> is still <math>r</math>-clique-free, and has <math>n</math> vertices. The number of edges in <math>G'</math> is
::<math>|E(G')|=|E(G)|+d(u)-d(w)>|E(G)|\,</math>,
:which contradicts the assumption that <math>|E(G)|</math> is maximal.
* '''Case.2:''' <math>d(w)\ge d(u)</math> and <math>d(w)\ge d(v)</math>. Duplicate <math>w</math> twice and delete <math>u</math> and <math>v</math>. The new graph <math>G'</math> has no <math>r</math>-clique, and the number of edges is
::<math>|E(G')|=|E(G)|+2d(w)-(d(u)+d(v)+1)>|E(G)|\,</math>.
:Contradiction again.

The claim implies that <math>uv\not\in E</math> defines an equivalence relation on vertices (to be more precise, it guarantees the transitivity of the relation, while the reflexivity and symmetry hold directly). Graph <math>G</math> must be a complete multipartite graph <math>K_{n_1,n_2,\ldots,n_{r-1}}</math> with <math>n_1+n_2+\cdots +n_{r-1}=n</math>. Optimize the edge number, we have the Turán graph.
}}

=== Erdős–Stone theorem ===

{{Theorem|Fundamental theorem of extremal graph theory (Erdős–Stone 1946)|
:For any integers <math>r\ge 2</math> and <math>s\ge 1</math>, and any <math>\epsilon>0</math>, there exists an <math>N_0</math> such that every graph with <math>n\ge N_0</math> vertices and at least <math>\left(\frac{r-2}{2(r-1)}+\epsilon\right)n^2</math> edges contains <math>K_{r,s}</math> as a subgraph, i.e.,
:::<math>\mathrm{ex}(n,K_{s,r})= \left(\frac{r-2}{2(r-1)}+o(1)\right)n^2</math>.
}}

{{Theorem|Corollary|
:For every nonempty graph <math>H</math>,
::<math>\lim_{n\rightarrow\infty}\frac{\mathrm{ex}(n,H)}{{n\choose 2}}=\frac{\chi(H)-2}{\chi(H)-1}</math>.
}}

== Cycle Structures ==
=== Girth ===

=== Hamiltonian cycle ===

Combinatorics (Fall 2010)/Extremal graphs

2010-10-14T06:58:37Z

172.21.1.240: /* Erdős–Stone theorem */

== Extremal Graph Theory ==

=== Mantel's theorem ===
We consider a typical extremal problem for graphs: the largest possible number of edges of '''triangle-free''' graphs, i.e. graphs contains no <math>K_3</math>.

{{Theorem|Theorem (Mantel 1907)|
:Suppose <math>G(V,E)</math> is graph on <math>n</math> vertice without triangles. Then <math>|E|\le\frac{n^2}{4}</math>.
}}

{{Prooftitle|First proof. (pigeonhole principle)|
We prove an equivalent theorem: Any <math>G(V,E)</math> with <math>|V|=n</math> and <math>|E|>\frac{n^2}{4}</math> must have a triangle.

Use induction on <math>n</math>. The theorem holds trivially for <math>n\le 3</math>.

Induction hypothesis: assume the theorem hold for <math>|V|\le n-1</math>.

For <math>G</math> with <math>n</math> vertices, without loss of generality, assume that <math>|E|=\frac{n^2}{4}+1</math>, we will show that <math>G</math> must contain a triangle. Take a <math>uv\in E</math>, and let <math>H</math> be the subgraph of <math>G</math> induced by <math>V\setminus \{u,v\}</math>. Clearly, <math>H</math> has <math>n-2</math> vertices.
:'''Case.1:''' If <math>H</math> has <math>>\frac{(n-2)^2}{4}</math> edges, then by the induction hypothesis, <math>H</math> has a triangle.
:'''Case.2:''' If <math>H</math> has <math>\le\frac{(n-2)^2}{4}</math> edges, then at least <math>\left(\frac{n^2}{4}+1\right)-\frac{(n-2)^2}{4}-1=n-1</math> edges are between <math>H</math> and <math>\{u,v\}</math>. By pigeonhole principle, there must be a vertex in <math>H</math> that is adjacent to both <math>u</math> and <math>v</math>. Thus, <math>G</math> has a triangle.
}}

{{Prooftitle|Second proof. (Cauchy-Schwarz inequality)|(Mantel's original proof)
For any edge <math>uv\in E</math>, no vertex can be a neighbor of both <math>u</math> and <math>v</math>, or otherwise there will be a triangle. Thus, for any edge <math>uv\in E</math>, <math>d_u+d_v\le n</math>. It follows that
:<math>\sum_{uv\in E}(d_u+d_v)\le n|E|</math>.
Note that <math>d(v)</math> appears exactly <math>d_v</math> times in the sum, so that
:<math>\sum_{uv\in E}(d_u+d_v)=\sum_{v\in V}d_v^2</math>.
Applying Chauchy-Schwarz inequality,
:<math>
n|E|\ge\sum_{v\in V}d_v^2\ge\frac{\left(\sum_{v\in V}d_v\right)^2}{n}=\frac{4|E|^2}{n},
</math>
where the last equation is due to Euler's equality <math>\sum_{v\in V}d_v=2|E|</math>. The theorem follows.
}}

{{Prooftitle|Third proof. (inequality of the arithmetic and geometric mean)|
Assume that <math>G(V,E)</math> has <math>|V|=n</math> vertices and is triangle-free.

Let <math>A</math> be the largest independent set in <math>G</math> and let <math>\alpha=|A|</math>.
Since <math>G</math> is triangle-free, for very vertex <math>v</math>, all its neighbors must form an independent set, thus <math>d(v)\le \alpha</math> for all <math>v\in V</math>.

Take <math>B=V\setminus A</math> and let <math>\beta=|B|</math>.
Since <math>A</math> is an independent set, all edges in <math>E</math> must have at least one endpoint in <math>B</math>. Counting the edges in <math>E</math> according to their endpoints in <math>B</math>, we obtain <math>|E|\le\sum_{v\in B}d_v</math>. By the inequality of the arithmetic and geometric mean,
:<math>|E|\le\sum_{v\in B}d_v\le\alpha\beta\le\left(\frac{\alpha+\beta}{2}\right)^2=\frac{n^2}{4}</math>.
}}

=== Turán's theorem ===
{{Theorem|Theorem (Turán 1941)|
:Let <math>G(V,E)</math> be a graph with <math>|V|=n</math>. If <math>G</math> has no <math>r</math>-clique, <math>k\ge 2</math>, then
::<math>|E|\le\frac{r-2}{2(r-1)}n^2</math>.
}}

{{Prooftitle|First proof. (induction)|(Turán's original proof)
}}

{{Prooftitle|Second proof. (weight shifting)|(due to Motzkin and Straus)
}}

{{Prooftitle|Third proof. (the probabilistic method)|(due to Alon and Spencer)
}}

{{Prooftitle|Fourth proof.|
Let <math>G(V,E)</math> be a <math>r</math>-clique-free graph on <math>n</math> vertices with a maximum number of edges.
:'''Claim:''' <math>G</math> does not contain three vertices <math>u,v,w</math> such that <math>uv\in E</math> but <math>uw\not\in E, vw\not\in E</math>.
Suppose otherwise. There are two cases.
* '''Case.1:''' <math>d(w)<d(u)</math> or <math>d(w)<d(v)</math>. Without loss of generality, suppose that <math>d(w)<d(u)</math>. We duplicate <math>u</math> by creating a new vertex <math>u'</math> which has exactly the same neighbors as <math>u</math> (but <math>uu'</math> is not an edge). Such duplication will not increase the clique size. We then remove <math>w</math>. The resulting graph <math>G'</math> is still <math>r</math>-clique-free, and has <math>n</math> vertices. The number of edges in <math>G'</math> is
::<math>|E(G')|=|E(G)|+d(u)-d(w)>|E(G)|\,</math>,
:which contradicts the assumption that <math>|E(G)|</math> is maximal.
* '''Case.2:''' <math>d(w)\ge d(u)</math> and <math>d(w)\ge d(v)</math>. Duplicate <math>w</math> twice and delete <math>u</math> and <math>v</math>. The new graph <math>G'</math> has no <math>r</math>-clique, and the number of edges is
::<math>|E(G')|=|E(G)|+2d(w)-(d(u)+d(v)+1)>|E(G)|\,</math>.
:Contradiction again.

The claim implies that <math>uv\not\in E</math> defines an equivalence relation on vertices (to be more precise, it guarantees the transitivity of the relation, while the reflexivity and symmetry hold directly). Graph <math>G</math> must be a complete multipartite graph <math>K_{n_1,n_2,\ldots,n_{r-1}}</math> with <math>n_1+n_2+\cdots +n_{r-1}=n</math>. Optimize the edge number, we have the Turán graph.
}}

=== Erdős–Stone theorem ===

{{Theorem|Fundamental theorem of extremal graph theory (Erdős–Stone 1946)|
:For any integers <math>r\ge 2</math> and <math>s\ge 1</math>, and any <math>\epsilon>0</math>, there exists an <math>N_0</math> such that every graph with <math>n\ge N_0</math> vertices and at least <math>\left(\frac{r-2}{2(r-1)}+\epsilon\right)n^2</math> edges contains <math>K_{r,s}</math> as a subgraph, i.e.,
:::<math>\mathrm{ex}(n,K_{s,r})\le \left(\frac{r-2}{2(r-1)}+o(1)\right)n^2</math>.
}}

{{Theorem|Corollary|
:For every nonempty graph <math>H</math>,
::<math>\lim_{n\rightarrow\infty}\frac{\mathrm{ex}(n,H)}{{n\choose 2}}=\frac{\chi(H)-2}{\chi(H)-1}</math>.
}}

== Cycle Structures ==
=== Girth ===

=== Hamiltonian cycle ===

Combinatorics (Fall 2010)/Extremal graphs

2010-10-14T06:50:52Z

172.21.1.240: /* Erdős–Stone theorem */

== Extremal Graph Theory ==

=== Mantel's theorem ===
We consider a typical extremal problem for graphs: the largest possible number of edges of '''triangle-free''' graphs, i.e. graphs contains no <math>K_3</math>.

{{Theorem|Theorem (Mantel 1907)|
:Suppose <math>G(V,E)</math> is graph on <math>n</math> vertice without triangles. Then <math>|E|\le\frac{n^2}{4}</math>.
}}

{{Prooftitle|First proof. (pigeonhole principle)|
We prove an equivalent theorem: Any <math>G(V,E)</math> with <math>|V|=n</math> and <math>|E|>\frac{n^2}{4}</math> must have a triangle.

Use induction on <math>n</math>. The theorem holds trivially for <math>n\le 3</math>.

Induction hypothesis: assume the theorem hold for <math>|V|\le n-1</math>.

For <math>G</math> with <math>n</math> vertices, without loss of generality, assume that <math>|E|=\frac{n^2}{4}+1</math>, we will show that <math>G</math> must contain a triangle. Take a <math>uv\in E</math>, and let <math>H</math> be the subgraph of <math>G</math> induced by <math>V\setminus \{u,v\}</math>. Clearly, <math>H</math> has <math>n-2</math> vertices.
:'''Case.1:''' If <math>H</math> has <math>>\frac{(n-2)^2}{4}</math> edges, then by the induction hypothesis, <math>H</math> has a triangle.
:'''Case.2:''' If <math>H</math> has <math>\le\frac{(n-2)^2}{4}</math> edges, then at least <math>\left(\frac{n^2}{4}+1\right)-\frac{(n-2)^2}{4}-1=n-1</math> edges are between <math>H</math> and <math>\{u,v\}</math>. By pigeonhole principle, there must be a vertex in <math>H</math> that is adjacent to both <math>u</math> and <math>v</math>. Thus, <math>G</math> has a triangle.
}}

{{Prooftitle|Second proof. (Cauchy-Schwarz inequality)|(Mantel's original proof)
For any edge <math>uv\in E</math>, no vertex can be a neighbor of both <math>u</math> and <math>v</math>, or otherwise there will be a triangle. Thus, for any edge <math>uv\in E</math>, <math>d_u+d_v\le n</math>. It follows that
:<math>\sum_{uv\in E}(d_u+d_v)\le n|E|</math>.
Note that <math>d(v)</math> appears exactly <math>d_v</math> times in the sum, so that
:<math>\sum_{uv\in E}(d_u+d_v)=\sum_{v\in V}d_v^2</math>.
Applying Chauchy-Schwarz inequality,
:<math>
n|E|\ge\sum_{v\in V}d_v^2\ge\frac{\left(\sum_{v\in V}d_v\right)^2}{n}=\frac{4|E|^2}{n},
</math>
where the last equation is due to Euler's equality <math>\sum_{v\in V}d_v=2|E|</math>. The theorem follows.
}}

{{Prooftitle|Third proof. (inequality of the arithmetic and geometric mean)|
Assume that <math>G(V,E)</math> has <math>|V|=n</math> vertices and is triangle-free.

Let <math>A</math> be the largest independent set in <math>G</math> and let <math>\alpha=|A|</math>.
Since <math>G</math> is triangle-free, for very vertex <math>v</math>, all its neighbors must form an independent set, thus <math>d(v)\le \alpha</math> for all <math>v\in V</math>.

Take <math>B=V\setminus A</math> and let <math>\beta=|B|</math>.
Since <math>A</math> is an independent set, all edges in <math>E</math> must have at least one endpoint in <math>B</math>. Counting the edges in <math>E</math> according to their endpoints in <math>B</math>, we obtain <math>|E|\le\sum_{v\in B}d_v</math>. By the inequality of the arithmetic and geometric mean,
:<math>|E|\le\sum_{v\in B}d_v\le\alpha\beta\le\left(\frac{\alpha+\beta}{2}\right)^2=\frac{n^2}{4}</math>.
}}

=== Turán's theorem ===
{{Theorem|Theorem (Turán 1941)|
:Let <math>G(V,E)</math> be a graph with <math>|V|=n</math>. If <math>G</math> has no <math>r</math>-clique, <math>k\ge 2</math>, then
::<math>|E|\le\frac{r-2}{2(r-1)}n^2</math>.
}}

{{Prooftitle|First proof. (induction)|(Turán's original proof)
}}

{{Prooftitle|Second proof. (weight shifting)|(due to Motzkin and Straus)
}}

{{Prooftitle|Third proof. (the probabilistic method)|(due to Alon and Spencer)
}}

{{Prooftitle|Fourth proof.|
Let <math>G(V,E)</math> be a <math>r</math>-clique-free graph on <math>n</math> vertices with a maximum number of edges.
:'''Claim:''' <math>G</math> does not contain three vertices <math>u,v,w</math> such that <math>uv\in E</math> but <math>uw\not\in E, vw\not\in E</math>.
Suppose otherwise. There are two cases.
* '''Case.1:''' <math>d(w)<d(u)</math> or <math>d(w)<d(v)</math>. Without loss of generality, suppose that <math>d(w)<d(u)</math>. We duplicate <math>u</math> by creating a new vertex <math>u'</math> which has exactly the same neighbors as <math>u</math> (but <math>uu'</math> is not an edge). Such duplication will not increase the clique size. We then remove <math>w</math>. The resulting graph <math>G'</math> is still <math>r</math>-clique-free, and has <math>n</math> vertices. The number of edges in <math>G'</math> is
::<math>|E(G')|=|E(G)|+d(u)-d(w)>|E(G)|\,</math>,
:which contradicts the assumption that <math>|E(G)|</math> is maximal.
* '''Case.2:''' <math>d(w)\ge d(u)</math> and <math>d(w)\ge d(v)</math>. Duplicate <math>w</math> twice and delete <math>u</math> and <math>v</math>. The new graph <math>G'</math> has no <math>r</math>-clique, and the number of edges is
::<math>|E(G')|=|E(G)|+2d(w)-(d(u)+d(v)+1)>|E(G)|\,</math>.
:Contradiction again.

The claim implies that <math>uv\not\in E</math> defines an equivalence relation on vertices (to be more precise, it guarantees the transitivity of the relation, while the reflexivity and symmetry hold directly). Graph <math>G</math> must be a complete multipartite graph <math>K_{n_1,n_2,\ldots,n_{r-1}}</math> with <math>n_1+n_2+\cdots +n_{r-1}=n</math>. Optimize the edge number, we have the Turán graph.
}}

=== Erdős–Stone theorem ===

{{Theorem|Fundamental theorem of extremal graph theory (Erdős–Stone 1946)|
:For any <math>r\ge 2</math> and <math>s\ge 1</math>, and every sufficiently large <math>n</math>, every graph with <math>n</math> vertices and at least <math>\left(\frac{r-2}{2(r-1)}+\epsilon\right)n^2</math> edges contains <math>K_{r,s}</math> as a subgraph.
}}

{{Theorem|Corollary|

}}

== Cycle Structures ==
=== Girth ===

=== Hamiltonian cycle ===

Combinatorics (Fall 2010)/Existence, the probabilistic method

2010-09-20T08:56:23Z

172.21.6.120: /* Dirichlet's theorem */

== Counting arguments ==
;Circuit complexity

This is a fundamental problem in in Computer Science.

A '''boolean function''' is a function is the form <math>f:\{0,1\}^n\rightarrow \{0,1\}</math>.

Formally, a boolean circuit is a directed acyclic graph. Nodes with indegree zero are input nodes, labeled <math>x_1, x_2, \ldots , x_n</math>. A circuit has a unique node with outdegree zero, called the output node. Every other node is a gate. There are three types of gates: AND, OR (both with indegree two), and NOT (with indegree one).

Computations in Turing machines can be simulated by circuits, and any boolean function in '''P''' can be computed by a circuit with polynomially many gates. Thus, if we can find a function in '''NP''' that cannot be computed by any circuit with polynomially many gates, then '''NP'''<math>\neq</math>'''P'''.

The following theorem due to Shannon says that functions with exponentially large circuit complexity do exist.

{{Theorem
|Theorem (Shannon 1949)|
:There is a boolean function <math>f:\{0,1\}^n\rightarrow \{0,1\}</math> with circuit complexity greater than <math>\frac{2^n}{3n}</math>.
}}
{{Proof|
We first count the number of boolean functions <math>f:\{0,1\}^n\rightarrow \{0,1\}</math>. There are <math>2^{2^n}</math> boolean functions <math>f:\{0,1\}^n\rightarrow \{0,1\}</math>.

Then we count the number of boolean circuit with fixed number of gates.
Fix an integer <math>t</math>, we count the number of circuits with <math>t</math> gates. By the [http://en.wikipedia.org/wiki/De_Morgan's_laws De Morgan's laws], we can assume that all NOTs are pushed back to the inputs. Each gate has one of the two types (AND or OR), and has two inputs. Each of the inputs to a gate is either a constant 0 or 1, an input variable <math>x_i</math>, an inverted input variable <math>\neg x_i</math>, or the output of another gate; thus, there are at most <math>2+2n+t-1</math> possible gate inputs. It follows that the number of circuits with <math>t</math> gates is at most <math>2^t(t+2n+1)^{2t}</math>.

If <math>t=2^n/3n</math>, then
:<math>
\frac{2^t(t+2n+1)^{2t}}{2^{2^n}}=o(1)<1,</math> thus, <math>2^t(t+2n+1)^{2t} < 2^{2^n}.</math>

Each boolean circuit computes one boolean function. Therefore, there must exist a boolean function <math>f</math> which cannot be computed by any circuits with <math>2^n/3n</math> gates.
}}

Note that by Shannon's theorem, not only there exists a boolean function with exponentially large circuit complexity, but ''almost all'' boolean functions have exponentially large circuit complexity.

=== Double counting ===
;Handshaking lemma
{{Theorem|Handshaking Lemma|
:At a party, the number of guests who shake hands an odd number of times is even.
}}

We model this scenario as an undirected graph <math>G(V,E)</math> with <math>|V|=n</math> standing for the <math>n</math> guests. There is an edge <math>uv\in E</math> if <math>u</math> and <math>v</math> shake hands. Let <math>d(v)</math> be the degree of vertex <math>v</math>, which represents the number of times that <math>v</math> shakes hand. The handshaking lemma states that in any undirected graph, the sum of odd degrees is even.

The handshaking lemma is a direct consequence of the following lemma, which is proved by Euler in a 1736 paper that began the study of graph theory.

{{Theorem|Lemma (Euler 1736)|
:<math>\sum_{v\in V}d(v)=2|E|</math>
}}
{{Proof|
We count the number of '''directed''' edges. A directed edge is an ordered pair <math>(u,v)</math> such that <math>\{u,v\}\in E</math>. There are two ways to count the directed edges.

First, we can enumerate by edges. Pick every edge <math>uv\in E</math> and apply two directions <math>(u,v)</math> and <math>(v,u)</math> to the edge. This gives us <math>2|E|</math> directed edges.

On the other hand, we can enumerate by vertices. Pick every vertex <math>v\in V</math> and for each of its <math>d(v)</math> neighbors, say <math>u</math>, generate a directed edge <math>(v,u)</math>. This gives us <math>\sum_{v\in V}d(v)</math> directed edges.

It is obvious that the two terms are equal, since we just count the same thing twice with different methods. The lemma follows.
}}

The handshaking lemma is implied directly by the above lemma, since the sum of even degrees is even.

;Cayley's formula
{{Theorem|Caylay's formula for trees|
: There are <math>n^{n-2}</math> different trees on <math>n</math> distinct vertices.
}}

== The Pigeonhole Principle ==

=== Monotonic subsequences ===

{{Theorem|Theorem (Erdős-Szekeres 1935)|
:A sequence of more than <math>mn</math> different real numbers must contain either an increasing subsequence of length <math>m+1</math>, or a decreasing subsequence of length <math>n+1</math>.
}}
{{Proof|(due to Seidenberg 1959)
}}

=== Dirichlet's theorem ===

{{Theorem|Theorem (Dirichlet 1879)|
:Let <math>x</math> be a real number. For any natural number <math>n</math>, there is a rational number <math>\frac{p}{q}</math> such that <math>1\le q\le n</math> and
::<math>\left|x-\frac{p}{q}\right|<\frac{1}{nq}</math>.
}}

== The Probabilistic Method ==

Suppose we want prove the existence of mathematic objects with certain properties. One way to do so is to explicitly construct such an object. This kind of proofs can be interpreted as ''deterministic algorithms'' which find the object with desirable properties.

The probabilistic method provides another way of proving the existence of objects: instead of explicitly constructing an object, we define a probability space of objects in which the probability is positive that a randomly selected object has the required property.

The basic principle of the probabilistic method is very simple, and can be stated in intuitive ways:
*If an object chosen randomly from a universe satisfies a property with positive probability, then there must be an object in the universe that satisfies that property.
:For example, for a ball(the object) randomly chosen from a box(the universe) of balls, if the probability that the chosen ball is blue(the property) is >0, then there must be a blue ball in the box.
*Any random variable assumes at least one value that is no smaller than its expectation, and at least one value that is no greater than the expectation.
:For example, if we know the average height of the students in the class is <math>\ell</math>, then we know there is a students whose height is at least <math>\ell</math>, and there is a student whose height is at most <math>\ell</math>.

Although the idea of the probabilistic method is simple, it provides us a powerful tool for existential proof. In same cases, the proof itself is a ''randomized algorithm'', and if we are lucky, the algorithm could be very efficient.

=== Sampling ===
;Ramsey number

Recall the Ramsey theorem which states that in a meeting of at least six people, there are either three people knowing each other or three people not knowing each other. In graph theoretical terms, this means that no matter how we color the edges of <math>K_6</math> (the complete graph on six vertices), there must be a '''monochromatic''' <math>K_3</math> (a triangle whose edges have the same color).

Generally, the '''Ramsey number''' <math>R(k,\ell)</math> is the smallest integer <math>n</math> such that in any two-coloring of the edges of a complete graph on <math>n</math> vertices <math>K_n</math> by red and blue, either there is a red <math>K_k</math> or there is a blue <math>K_\ell</math>.

Ramsey showed in 1929 that <math>R(k,\ell)</math> is finite for any <math>k</math> and <math>\ell</math>. It is extremely hard to compute the exact value of <math>R(k,\ell)</math>. Here we give a lower bound of <math>R(k,k)</math> by the probabilistic method.

{{Theorem
|Theorem (Erdős 1947)|
:If <math>{n\choose k}\cdot 2^{1-{k\choose 2}}<1</math> then it is possible to color the edges of <math>K_n</math> with two colors so that there is no monochromatic <math>K_k</math> subgraph.
}}
{{Proof| Consider a random two-coloring of edges of <math>K_n</math> obtained as follows:
* For each edge of <math>K_n</math>, independently flip a fair coin to decide the color of the edge.

For any fixed set <math>S</math> of <math>k</math> vertices, let <math>\mathcal{E}_S</math> be the event that the <math>K_k</math> subgraph induced by <math>S</math> is monochromatic. There are <math>{k\choose 2}</math> many edges in <math>K_k</math>, therefore
:<math>\Pr[\mathcal{E}_S]=2\cdot 2^{-{k\choose 2}}=2^{1-{k\choose 2}}.</math>

Since there are <math>{n\choose k}</math> possible choices of <math>S</math>, by the union bound
:<math>
\Pr[\exists S, \mathcal{E}_S]\le {n\choose k}\cdot\Pr[\mathcal{E}_S]={n\choose k}\cdot 2^{1-{k\choose 2}}.
</math>
Due to the assumption, <math>{n\choose k}\cdot 2^{1-{k\choose 2}}<1</math>, thus there exists a two coloring that none of <math>\mathcal{E}_S</math> occurs, which means there is no monochromatic <math>K_k</math> subgraph.
}}

For <math>k\ge 3</math> and we take <math>n=\lfloor2^{k/2}\rfloor</math>, then
:<math>
\begin{align}
{n\choose k}\cdot 2^{1-{k\choose 2}}
&<
\frac{n^k}{k!}\cdot\frac{2^{1+\frac{k}{2}}}{2^{k^2/2}}\\
&\le
\frac{2^{k^2/2}}{k!}\cdot\frac{2^{1+\frac{k}{2}}}{2^{k^2/2}}\\
&=
\frac{2^{1+\frac{k}{2}}}{k!}\\
&<1.
\end{align}
</math>
By the above theorem, there exists a two-coloring of <math>K_n</math> that there is no monochromatic <math>K_k</math>. Therefore, the Ramsey number <math>R(k,k)>\lfloor2^{k/2}\rfloor</math> for all <math>k\ge 3</math>.

Note that for sufficiently large <math>k</math>, if <math>n= \lfloor 2^{k/2}\rfloor</math>, then the probability that there exists a monochromatic <math>K_k</math> is bounded by
:<math>
{n\choose k}\cdot 2^{1-{k\choose 2}}
<
\frac{2^{1+\frac{k}{2}}}{k!}
\ll 1,
</math>
which means that a random two-coloring of <math>K_n</math> is very likely not to contain a monochromatic <math>K_{2\log n}</math>. This gives us a very simple randomized algorithm for finding a two-coloring of <math>K_n</math> without monochromatic <math>K_{2\log n}</math>.

=== Linearity of expectation ===

;Maximum cut

Given an undirected graph <math>G(V,E)</math>, a set <math>C</math> of edges of <math>G</math> is called a '''cut''' if <math>G</math> is disconnected after removing the edges in <math>C</math>. We can represent a cut by <math>c(S,T)</math> where <math>(S,T)</math> is a bipartition of the vertex set <math>V</math>, and <math>c(S,T)=\{uv\in E\mid u\in S,v\in T\}</math> is the set of edges crossing between <math>S</math> and <math>T</math>.

We have seen how to compute min-cut: either by deterministic max-flow algorithm, or by Karger's randomized algorithm. On the other hand, max-cut is hard to compute, because it is '''NP-complete'''. Actually, the weighted version of max-cut is among the [http://en.wikipedia.org/wiki/Karp's_21_NP-complete_problems Karp's 21 NP-complete problems].

We now show by the probabilistic method that a max-cut always has at least half the edges.

{{Theorem
|Theorem|
:Given an undirected graph <math>G</math> with <math>n</math> vertices and <math>m</math> edges, there is a cut of size at least <math>\frac{m}{2}</math>.
}}
{{Proof| Enumerate the vertices in an arbitrary order. Partition the vertex set <math>V</math> into two disjoint sets <math>S</math> and <math>T</math> as follows.
:For each vertex <math>v\in V</math>,
:* independently choose one of <math>S</math> and <math>T</math> with equal probability, and let <math>v</math> join the chosen set.

For each vertex <math>v\in V</math>, let <math>X_v\in\{S,T\}</math> be the random variable which represents the set that <math>v</math> joins. For each edge <math>uv\in E</math>, let <math>Y_{uv}</math> be the 0-1 random variable which indicates whether <math>uv</math> crosses between <math>S</math> and <math>T</math>. Clearly,
:<math>
\Pr[Y_{uv}=1]=\Pr[X_u\neq X_v]=\frac{1}{2}.
</math>

The size of <math>c(S,T)</math> is given by <math>Y=\sum_{uv\in E}Y_{uv}</math>. By the linearity of expectation,
:<math>
\mathbf{E}[Y]=\sum_{uv\in E}\mathbf{E}[Y_{uv}]=\sum_{uv\in E}\Pr[Y_{uv}=1]=\frac{m}{2}.
</math>
Therefore, there exist a bipartition <math>(S,T)</math> of <math>V</math> such that <math>|c(S,T)|\ge\frac{m}{2}</math>, i.e. there exists a cut of <math>G</math> which contains at least <math>\frac{m}{2}</math> edges.
}}

;Maximum satisfiability

Suppose that we have a number of boolean variables <math>x_1,x_2,\ldots,\in\{\mathrm{true},\mathrm{false}\}</math>. A '''literal''' is either a variable <math>x_i</math> itself or its negation <math>\neg x_i</math>. A logic expression is a '''conjunctive normal form (CNF)''' if it is written as the conjunction(AND) of a set of '''clauses''', where each clause is a disjunction(OR) of literals. For example:
:<math>
(x_1\vee \neg x_2 \vee \neg x_3)\wedge (\neg x_1\vee \neg x_3)\wedge (x_1\vee x_2\vee x_4)\wedge (x_4\vee \neg x_3)\wedge (x_4\vee \neg x_1).
</math>

The satisfiability (SAT) problem ask whether the CNF is satisfiable, i.e. there exists an assignment of variables to the values of true and false so that all clauses are true. The maximum satisfiability (MAXSAT) is the optimization version of SAT, which ask for an assignment that the number of satisfied clauses is maximized.

SAT is the first problem known to be '''NP-complete''' (the Cook-Levin theorem). MAXSAT is also '''NP-complete'''. We then see that there always exists a roughly good truth assignment which satisfies half the clauses.

{{Theorem
|Theorem|
:For any set of <math>m</math> clauses, there is a truth assignment that satisfies at least <math>\frac{m}{2}</math> clauses.
}}
{{Proof| For each variable, independently assign a random value in <math>\{\mathrm{true},\mathrm{false}\}</math> with equal probability. For the <math>i</math>th clause, let <math>X_i</math> be the random variable which indicates whether the <math>i</math>th clause is satisfied. Suppose that there are <math>k</math> literals in the clause. The probability that the clause is satisfied is
:<math>\Pr[X_k=1]\ge(1-2^{-k})\ge\frac{1}{2}</math>.

Let <math>X=\sum_{i=1}^m X_i</math> be the number of satisfied clauses. By the linearity of expectation,
:<math>
\mathbf{E}[X]=\sum_{i=1}^{m}\mathbf{E}[X_i]\ge \frac{m}{2}.
</math>
Therefore, there exists an assignment such that at least <math>\frac{m}{2}</math> clauses are satisfied.
}}

=== Alterations ===
;Independent sets
An independent set of a graph is a set of vertices with no edges between them. The following theorem gives a lower bound on the size of the largest independent set.
{{Theorem
|Theorem|
:Let <math>G(V,E)</math> be a graph on <math>n</math> vertices with <math>m</math> edges. Then <math>G</math> has an independent set with at least <math>\frac{n^2}{4m}</math> vertices.
}}
{{Proof| Let <math>S</math> be a set of vertices constructed as follows:
:For each vertex <math>v\in V</math>:
:* <math>v</math> is included in <math>S</math> independently with probability <math>p</math>,
<math>p</math> to be determined.

Let <math>X=|S|</math>. It is obvious that <math>\mathbf{E}[X]=np</math>.

For each edge <math>e\in E</math>, let <math>Y_{e}</math> be the random variable which indicates whether both endpoints of <math></math> are in <math>S</math>.
:<math>
\mathbf{E}[Y_{uv}]=\Pr[u\in S\wedge v\in S]=p^2.
</math>
Let <math>Y</math> be the number of edges in the subgraph of <math>G</math> induced by <math>S</math>. It holds that <math>Y=\sum_{e\in E}Y_e</math>. By linearity of expectation,
:<math>\mathbf{E}[Y]=\sum_{e\in E}\mathbf{E}[Y_e]=mp^2</math>.

Note that although <math>S</math> is not necessary an independent set, it can be modified to one if for each edge <math>e</math> of the induced subgraph <math>G(S)</math>, we delete one of the endpoint of <math>e</math> from <math>S</math>. Let <math>S^*</math> be the resulting set. It is obvious that <math>S^*</math> is an independent set since there is no edge left in the induced subgraph <math>G(S^*)</math>.

Since there are <math>Y</math> edges in <math>G(S)</math>, there are at most <math>Y</math> vertices in <math>S</math> are deleted to make it become <math>S^*</math>. Therefore, <math>|S^*|\ge X-Y</math>. By linearity of expectation,
:<math>
\mathbf{E}[|S^*|]\ge\mathbf{E}[X-Y]=\mathbf{E}[X]-\mathbf{E}[Y]=np-mp^2.
</math>
The expectation is maximized when <math>p=\frac{n}{2m}</math>, thus
:<math>
\mathbf{E}[|S^*|]\ge n\cdot\frac{n}{2m}-m\left(\frac{n}{2m}\right)^2=\frac{n^2}{4m}.
</math>
There exists an independent set which contains at least <math>\frac{n^2}{4m}</math> vertices.
}}

The proof actually propose a randomized algorithm for constructing large independent set:

{{Theorem
|Algorithm|
Given a graph on <math>n</math> vertices with <math>m</math> edges, let <math>d=\frac{2m}{n}</math> be the average degree.
#For each vertex <math>v\in V</math>, <math>v</math> is included in <math>S</math> independently with probability <math>\frac{1}{d}</math>.
#For each remaining edge in the induced subgraph <math>G(S)</math>, remove one of the endpoints from <math>S</math>.
}}

Let <math>S^*</math> be the resulting set. We have shown that <math>S^*</math> is an independent set and <math>\mathbf{E}[|S^*|]\ge\frac{n^2}{4m}</math>.

Combinatorics (Fall 2010)/Existence, the probabilistic method

2010-09-20T08:54:33Z

172.21.6.120: /* Monotonic subsequences */

== Counting arguments ==
;Circuit complexity

This is a fundamental problem in in Computer Science.

A '''boolean function''' is a function is the form <math>f:\{0,1\}^n\rightarrow \{0,1\}</math>.

Formally, a boolean circuit is a directed acyclic graph. Nodes with indegree zero are input nodes, labeled <math>x_1, x_2, \ldots , x_n</math>. A circuit has a unique node with outdegree zero, called the output node. Every other node is a gate. There are three types of gates: AND, OR (both with indegree two), and NOT (with indegree one).

Computations in Turing machines can be simulated by circuits, and any boolean function in '''P''' can be computed by a circuit with polynomially many gates. Thus, if we can find a function in '''NP''' that cannot be computed by any circuit with polynomially many gates, then '''NP'''<math>\neq</math>'''P'''.

The following theorem due to Shannon says that functions with exponentially large circuit complexity do exist.

{{Theorem
|Theorem (Shannon 1949)|
:There is a boolean function <math>f:\{0,1\}^n\rightarrow \{0,1\}</math> with circuit complexity greater than <math>\frac{2^n}{3n}</math>.
}}
{{Proof|
We first count the number of boolean functions <math>f:\{0,1\}^n\rightarrow \{0,1\}</math>. There are <math>2^{2^n}</math> boolean functions <math>f:\{0,1\}^n\rightarrow \{0,1\}</math>.

Then we count the number of boolean circuit with fixed number of gates.
Fix an integer <math>t</math>, we count the number of circuits with <math>t</math> gates. By the [http://en.wikipedia.org/wiki/De_Morgan's_laws De Morgan's laws], we can assume that all NOTs are pushed back to the inputs. Each gate has one of the two types (AND or OR), and has two inputs. Each of the inputs to a gate is either a constant 0 or 1, an input variable <math>x_i</math>, an inverted input variable <math>\neg x_i</math>, or the output of another gate; thus, there are at most <math>2+2n+t-1</math> possible gate inputs. It follows that the number of circuits with <math>t</math> gates is at most <math>2^t(t+2n+1)^{2t}</math>.

If <math>t=2^n/3n</math>, then
:<math>
\frac{2^t(t+2n+1)^{2t}}{2^{2^n}}=o(1)<1,</math> thus, <math>2^t(t+2n+1)^{2t} < 2^{2^n}.</math>

Each boolean circuit computes one boolean function. Therefore, there must exist a boolean function <math>f</math> which cannot be computed by any circuits with <math>2^n/3n</math> gates.
}}

Note that by Shannon's theorem, not only there exists a boolean function with exponentially large circuit complexity, but ''almost all'' boolean functions have exponentially large circuit complexity.

=== Double counting ===
;Handshaking lemma
{{Theorem|Handshaking Lemma|
:At a party, the number of guests who shake hands an odd number of times is even.
}}

We model this scenario as an undirected graph <math>G(V,E)</math> with <math>|V|=n</math> standing for the <math>n</math> guests. There is an edge <math>uv\in E</math> if <math>u</math> and <math>v</math> shake hands. Let <math>d(v)</math> be the degree of vertex <math>v</math>, which represents the number of times that <math>v</math> shakes hand. The handshaking lemma states that in any undirected graph, the sum of odd degrees is even.

The handshaking lemma is a direct consequence of the following lemma, which is proved by Euler in a 1736 paper that began the study of graph theory.

{{Theorem|Lemma (Euler 1736)|
:<math>\sum_{v\in V}d(v)=2|E|</math>
}}
{{Proof|
We count the number of '''directed''' edges. A directed edge is an ordered pair <math>(u,v)</math> such that <math>\{u,v\}\in E</math>. There are two ways to count the directed edges.

First, we can enumerate by edges. Pick every edge <math>uv\in E</math> and apply two directions <math>(u,v)</math> and <math>(v,u)</math> to the edge. This gives us <math>2|E|</math> directed edges.

On the other hand, we can enumerate by vertices. Pick every vertex <math>v\in V</math> and for each of its <math>d(v)</math> neighbors, say <math>u</math>, generate a directed edge <math>(v,u)</math>. This gives us <math>\sum_{v\in V}d(v)</math> directed edges.

It is obvious that the two terms are equal, since we just count the same thing twice with different methods. The lemma follows.
}}

The handshaking lemma is implied directly by the above lemma, since the sum of even degrees is even.

;Cayley's formula
{{Theorem|Caylay's formula for trees|
: There are <math>n^{n-2}</math> different trees on <math>n</math> distinct vertices.
}}

== The Pigeonhole Principle ==

=== Monotonic subsequences ===

{{Theorem|Theorem (Erdős-Szekeres 1935)|
:A sequence of more than <math>mn</math> different real numbers must contain either an increasing subsequence of length <math>m+1</math>, or a decreasing subsequence of length <math>n+1</math>.
}}
{{Proof|(due to Seidenberg 1959)
}}

=== Dirichlet's theorem ===

{{Theorem|Theorem (Dirichlet 1879)|
:Let <math>x</math> be a real number. For any natural number <math>n</math>, there is a rational number <math>\frac{p}{q}</math> such that <math>1\le q\le n</math> and
::<math>\left|x-\frac{p}{q}\right|<\frac{1}{nq}\le\frac{1}{q^2}</math>.
}}

== The Probabilistic Method ==

Suppose we want prove the existence of mathematic objects with certain properties. One way to do so is to explicitly construct such an object. This kind of proofs can be interpreted as ''deterministic algorithms'' which find the object with desirable properties.

The probabilistic method provides another way of proving the existence of objects: instead of explicitly constructing an object, we define a probability space of objects in which the probability is positive that a randomly selected object has the required property.

The basic principle of the probabilistic method is very simple, and can be stated in intuitive ways:
*If an object chosen randomly from a universe satisfies a property with positive probability, then there must be an object in the universe that satisfies that property.
:For example, for a ball(the object) randomly chosen from a box(the universe) of balls, if the probability that the chosen ball is blue(the property) is >0, then there must be a blue ball in the box.
*Any random variable assumes at least one value that is no smaller than its expectation, and at least one value that is no greater than the expectation.
:For example, if we know the average height of the students in the class is <math>\ell</math>, then we know there is a students whose height is at least <math>\ell</math>, and there is a student whose height is at most <math>\ell</math>.

Although the idea of the probabilistic method is simple, it provides us a powerful tool for existential proof. In same cases, the proof itself is a ''randomized algorithm'', and if we are lucky, the algorithm could be very efficient.

=== Sampling ===
;Ramsey number

Recall the Ramsey theorem which states that in a meeting of at least six people, there are either three people knowing each other or three people not knowing each other. In graph theoretical terms, this means that no matter how we color the edges of <math>K_6</math> (the complete graph on six vertices), there must be a '''monochromatic''' <math>K_3</math> (a triangle whose edges have the same color).

Generally, the '''Ramsey number''' <math>R(k,\ell)</math> is the smallest integer <math>n</math> such that in any two-coloring of the edges of a complete graph on <math>n</math> vertices <math>K_n</math> by red and blue, either there is a red <math>K_k</math> or there is a blue <math>K_\ell</math>.

Ramsey showed in 1929 that <math>R(k,\ell)</math> is finite for any <math>k</math> and <math>\ell</math>. It is extremely hard to compute the exact value of <math>R(k,\ell)</math>. Here we give a lower bound of <math>R(k,k)</math> by the probabilistic method.

{{Theorem
|Theorem (Erdős 1947)|
:If <math>{n\choose k}\cdot 2^{1-{k\choose 2}}<1</math> then it is possible to color the edges of <math>K_n</math> with two colors so that there is no monochromatic <math>K_k</math> subgraph.
}}
{{Proof| Consider a random two-coloring of edges of <math>K_n</math> obtained as follows:
* For each edge of <math>K_n</math>, independently flip a fair coin to decide the color of the edge.

For any fixed set <math>S</math> of <math>k</math> vertices, let <math>\mathcal{E}_S</math> be the event that the <math>K_k</math> subgraph induced by <math>S</math> is monochromatic. There are <math>{k\choose 2}</math> many edges in <math>K_k</math>, therefore
:<math>\Pr[\mathcal{E}_S]=2\cdot 2^{-{k\choose 2}}=2^{1-{k\choose 2}}.</math>

Since there are <math>{n\choose k}</math> possible choices of <math>S</math>, by the union bound
:<math>
\Pr[\exists S, \mathcal{E}_S]\le {n\choose k}\cdot\Pr[\mathcal{E}_S]={n\choose k}\cdot 2^{1-{k\choose 2}}.
</math>
Due to the assumption, <math>{n\choose k}\cdot 2^{1-{k\choose 2}}<1</math>, thus there exists a two coloring that none of <math>\mathcal{E}_S</math> occurs, which means there is no monochromatic <math>K_k</math> subgraph.
}}

For <math>k\ge 3</math> and we take <math>n=\lfloor2^{k/2}\rfloor</math>, then
:<math>
\begin{align}
{n\choose k}\cdot 2^{1-{k\choose 2}}
&<
\frac{n^k}{k!}\cdot\frac{2^{1+\frac{k}{2}}}{2^{k^2/2}}\\
&\le
\frac{2^{k^2/2}}{k!}\cdot\frac{2^{1+\frac{k}{2}}}{2^{k^2/2}}\\
&=
\frac{2^{1+\frac{k}{2}}}{k!}\\
&<1.
\end{align}
</math>
By the above theorem, there exists a two-coloring of <math>K_n</math> that there is no monochromatic <math>K_k</math>. Therefore, the Ramsey number <math>R(k,k)>\lfloor2^{k/2}\rfloor</math> for all <math>k\ge 3</math>.

Note that for sufficiently large <math>k</math>, if <math>n= \lfloor 2^{k/2}\rfloor</math>, then the probability that there exists a monochromatic <math>K_k</math> is bounded by
:<math>
{n\choose k}\cdot 2^{1-{k\choose 2}}
<
\frac{2^{1+\frac{k}{2}}}{k!}
\ll 1,
</math>
which means that a random two-coloring of <math>K_n</math> is very likely not to contain a monochromatic <math>K_{2\log n}</math>. This gives us a very simple randomized algorithm for finding a two-coloring of <math>K_n</math> without monochromatic <math>K_{2\log n}</math>.

=== Linearity of expectation ===

;Maximum cut

Given an undirected graph <math>G(V,E)</math>, a set <math>C</math> of edges of <math>G</math> is called a '''cut''' if <math>G</math> is disconnected after removing the edges in <math>C</math>. We can represent a cut by <math>c(S,T)</math> where <math>(S,T)</math> is a bipartition of the vertex set <math>V</math>, and <math>c(S,T)=\{uv\in E\mid u\in S,v\in T\}</math> is the set of edges crossing between <math>S</math> and <math>T</math>.

We have seen how to compute min-cut: either by deterministic max-flow algorithm, or by Karger's randomized algorithm. On the other hand, max-cut is hard to compute, because it is '''NP-complete'''. Actually, the weighted version of max-cut is among the [http://en.wikipedia.org/wiki/Karp's_21_NP-complete_problems Karp's 21 NP-complete problems].

We now show by the probabilistic method that a max-cut always has at least half the edges.

{{Theorem
|Theorem|
:Given an undirected graph <math>G</math> with <math>n</math> vertices and <math>m</math> edges, there is a cut of size at least <math>\frac{m}{2}</math>.
}}
{{Proof| Enumerate the vertices in an arbitrary order. Partition the vertex set <math>V</math> into two disjoint sets <math>S</math> and <math>T</math> as follows.
:For each vertex <math>v\in V</math>,
:* independently choose one of <math>S</math> and <math>T</math> with equal probability, and let <math>v</math> join the chosen set.

For each vertex <math>v\in V</math>, let <math>X_v\in\{S,T\}</math> be the random variable which represents the set that <math>v</math> joins. For each edge <math>uv\in E</math>, let <math>Y_{uv}</math> be the 0-1 random variable which indicates whether <math>uv</math> crosses between <math>S</math> and <math>T</math>. Clearly,
:<math>
\Pr[Y_{uv}=1]=\Pr[X_u\neq X_v]=\frac{1}{2}.
</math>

The size of <math>c(S,T)</math> is given by <math>Y=\sum_{uv\in E}Y_{uv}</math>. By the linearity of expectation,
:<math>
\mathbf{E}[Y]=\sum_{uv\in E}\mathbf{E}[Y_{uv}]=\sum_{uv\in E}\Pr[Y_{uv}=1]=\frac{m}{2}.
</math>
Therefore, there exist a bipartition <math>(S,T)</math> of <math>V</math> such that <math>|c(S,T)|\ge\frac{m}{2}</math>, i.e. there exists a cut of <math>G</math> which contains at least <math>\frac{m}{2}</math> edges.
}}

;Maximum satisfiability

Suppose that we have a number of boolean variables <math>x_1,x_2,\ldots,\in\{\mathrm{true},\mathrm{false}\}</math>. A '''literal''' is either a variable <math>x_i</math> itself or its negation <math>\neg x_i</math>. A logic expression is a '''conjunctive normal form (CNF)''' if it is written as the conjunction(AND) of a set of '''clauses''', where each clause is a disjunction(OR) of literals. For example:
:<math>
(x_1\vee \neg x_2 \vee \neg x_3)\wedge (\neg x_1\vee \neg x_3)\wedge (x_1\vee x_2\vee x_4)\wedge (x_4\vee \neg x_3)\wedge (x_4\vee \neg x_1).
</math>

The satisfiability (SAT) problem ask whether the CNF is satisfiable, i.e. there exists an assignment of variables to the values of true and false so that all clauses are true. The maximum satisfiability (MAXSAT) is the optimization version of SAT, which ask for an assignment that the number of satisfied clauses is maximized.

SAT is the first problem known to be '''NP-complete''' (the Cook-Levin theorem). MAXSAT is also '''NP-complete'''. We then see that there always exists a roughly good truth assignment which satisfies half the clauses.

{{Theorem
|Theorem|
:For any set of <math>m</math> clauses, there is a truth assignment that satisfies at least <math>\frac{m}{2}</math> clauses.
}}
{{Proof| For each variable, independently assign a random value in <math>\{\mathrm{true},\mathrm{false}\}</math> with equal probability. For the <math>i</math>th clause, let <math>X_i</math> be the random variable which indicates whether the <math>i</math>th clause is satisfied. Suppose that there are <math>k</math> literals in the clause. The probability that the clause is satisfied is
:<math>\Pr[X_k=1]\ge(1-2^{-k})\ge\frac{1}{2}</math>.

Let <math>X=\sum_{i=1}^m X_i</math> be the number of satisfied clauses. By the linearity of expectation,
:<math>
\mathbf{E}[X]=\sum_{i=1}^{m}\mathbf{E}[X_i]\ge \frac{m}{2}.
</math>
Therefore, there exists an assignment such that at least <math>\frac{m}{2}</math> clauses are satisfied.
}}

=== Alterations ===
;Independent sets
An independent set of a graph is a set of vertices with no edges between them. The following theorem gives a lower bound on the size of the largest independent set.
{{Theorem
|Theorem|
:Let <math>G(V,E)</math> be a graph on <math>n</math> vertices with <math>m</math> edges. Then <math>G</math> has an independent set with at least <math>\frac{n^2}{4m}</math> vertices.
}}
{{Proof| Let <math>S</math> be a set of vertices constructed as follows:
:For each vertex <math>v\in V</math>:
:* <math>v</math> is included in <math>S</math> independently with probability <math>p</math>,
<math>p</math> to be determined.

Let <math>X=|S|</math>. It is obvious that <math>\mathbf{E}[X]=np</math>.

For each edge <math>e\in E</math>, let <math>Y_{e}</math> be the random variable which indicates whether both endpoints of <math></math> are in <math>S</math>.
:<math>
\mathbf{E}[Y_{uv}]=\Pr[u\in S\wedge v\in S]=p^2.
</math>
Let <math>Y</math> be the number of edges in the subgraph of <math>G</math> induced by <math>S</math>. It holds that <math>Y=\sum_{e\in E}Y_e</math>. By linearity of expectation,
:<math>\mathbf{E}[Y]=\sum_{e\in E}\mathbf{E}[Y_e]=mp^2</math>.

Note that although <math>S</math> is not necessary an independent set, it can be modified to one if for each edge <math>e</math> of the induced subgraph <math>G(S)</math>, we delete one of the endpoint of <math>e</math> from <math>S</math>. Let <math>S^*</math> be the resulting set. It is obvious that <math>S^*</math> is an independent set since there is no edge left in the induced subgraph <math>G(S^*)</math>.

Since there are <math>Y</math> edges in <math>G(S)</math>, there are at most <math>Y</math> vertices in <math>S</math> are deleted to make it become <math>S^*</math>. Therefore, <math>|S^*|\ge X-Y</math>. By linearity of expectation,
:<math>
\mathbf{E}[|S^*|]\ge\mathbf{E}[X-Y]=\mathbf{E}[X]-\mathbf{E}[Y]=np-mp^2.
</math>
The expectation is maximized when <math>p=\frac{n}{2m}</math>, thus
:<math>
\mathbf{E}[|S^*|]\ge n\cdot\frac{n}{2m}-m\left(\frac{n}{2m}\right)^2=\frac{n^2}{4m}.
</math>
There exists an independent set which contains at least <math>\frac{n^2}{4m}</math> vertices.
}}

The proof actually propose a randomized algorithm for constructing large independent set:

{{Theorem
|Algorithm|
Given a graph on <math>n</math> vertices with <math>m</math> edges, let <math>d=\frac{2m}{n}</math> be the average degree.
#For each vertex <math>v\in V</math>, <math>v</math> is included in <math>S</math> independently with probability <math>\frac{1}{d}</math>.
#For each remaining edge in the induced subgraph <math>G(S)</math>, remove one of the endpoints from <math>S</math>.
}}

Let <math>S^*</math> be the resulting set. We have shown that <math>S^*</math> is an independent set and <math>\mathbf{E}[|S^*|]\ge\frac{n^2}{4m}</math>.

Combinatorics (Fall 2010)/Existence, the probabilistic method

2010-09-20T06:20:28Z

172.21.6.120: /* Monotonic subsequences */

== Counting arguments ==
;Circuit complexity

This is a fundamental problem in in Computer Science.

A '''boolean function''' is a function is the form <math>f:\{0,1\}^n\rightarrow \{0,1\}</math>.

Formally, a boolean circuit is a directed acyclic graph. Nodes with indegree zero are input nodes, labeled <math>x_1, x_2, \ldots , x_n</math>. A circuit has a unique node with outdegree zero, called the output node. Every other node is a gate. There are three types of gates: AND, OR (both with indegree two), and NOT (with indegree one).

Computations in Turing machines can be simulated by circuits, and any boolean function in '''P''' can be computed by a circuit with polynomially many gates. Thus, if we can find a function in '''NP''' that cannot be computed by any circuit with polynomially many gates, then '''NP'''<math>\neq</math>'''P'''.

The following theorem due to Shannon says that functions with exponentially large circuit complexity do exist.

{{Theorem
|Theorem (Shannon 1949)|
:There is a boolean function <math>f:\{0,1\}^n\rightarrow \{0,1\}</math> with circuit complexity greater than <math>\frac{2^n}{3n}</math>.
}}
{{Proof|
We first count the number of boolean functions <math>f:\{0,1\}^n\rightarrow \{0,1\}</math>. There are <math>2^{2^n}</math> boolean functions <math>f:\{0,1\}^n\rightarrow \{0,1\}</math>.

Then we count the number of boolean circuit with fixed number of gates.
Fix an integer <math>t</math>, we count the number of circuits with <math>t</math> gates. By the [http://en.wikipedia.org/wiki/De_Morgan's_laws De Morgan's laws], we can assume that all NOTs are pushed back to the inputs. Each gate has one of the two types (AND or OR), and has two inputs. Each of the inputs to a gate is either a constant 0 or 1, an input variable <math>x_i</math>, an inverted input variable <math>\neg x_i</math>, or the output of another gate; thus, there are at most <math>2+2n+t-1</math> possible gate inputs. It follows that the number of circuits with <math>t</math> gates is at most <math>2^t(t+2n+1)^{2t}</math>.

If <math>t=2^n/3n</math>, then
:<math>
\frac{2^t(t+2n+1)^{2t}}{2^{2^n}}=o(1)<1,</math> thus, <math>2^t(t+2n+1)^{2t} < 2^{2^n}.</math>

Each boolean circuit computes one boolean function. Therefore, there must exist a boolean function <math>f</math> which cannot be computed by any circuits with <math>2^n/3n</math> gates.
}}

Note that by Shannon's theorem, not only there exists a boolean function with exponentially large circuit complexity, but ''almost all'' boolean functions have exponentially large circuit complexity.

=== Double counting ===
;Handshaking lemma
{{Theorem|Handshaking Lemma|
:At a party, the number of guests who shake hands an odd number of times is even.
}}

We model this scenario as an undirected graph <math>G(V,E)</math> with <math>|V|=n</math> standing for the <math>n</math> guests. There is an edge <math>uv\in E</math> if <math>u</math> and <math>v</math> shake hands. Let <math>d(v)</math> be the degree of vertex <math>v</math>, which represents the number of times that <math>v</math> shakes hand. The handshaking lemma states that in any undirected graph, the sum of odd degrees is even.

The handshaking lemma is a direct consequence of the following lemma, which is proved by Euler in a 1736 paper that began the study of graph theory.

{{Theorem|Lemma (Euler 1736)|
:<math>\sum_{v\in V}d(v)=2|E|</math>
}}
{{Proof|
We count the number of '''directed''' edges. A directed edge is an ordered pair <math>(u,v)</math> such that <math>\{u,v\}\in E</math>. There are two ways to count the directed edges.

First, we can enumerate by edges. Pick every edge <math>uv\in E</math> and apply two directions <math>(u,v)</math> and <math>(v,u)</math> to the edge. This gives us <math>2|E|</math> directed edges.

On the other hand, we can enumerate by vertices. Pick every vertex <math>v\in V</math> and for each of its <math>d(v)</math> neighbors, say <math>u</math>, generate a directed edge <math>(v,u)</math>. This gives us <math>\sum_{v\in V}d(v)</math> directed edges.

It is obvious that the two terms are equal, since we just count the same thing twice with different methods. The lemma follows.
}}

The handshaking lemma is implied directly by the above lemma, since the sum of even degrees is even.

;Cayley's formula
{{Theorem|Caylay's formula for trees|
: There are <math>n^{n-2}</math> different trees on <math>n</math> distinct vertices.
}}

== The Pigeonhole Principle ==

=== Monotonic subsequences ===

{{Theorem|Theorem (Erdős-Szekeres 1935)|
:A sequence of more than <math>mn</math> different real numbers must contain either an increasing subsequence of length <math>m+1</math>, or a decreasing subsequence of length <math>n+1</math>.
}}
{{Proof|(due to Seidenberg 1959)
}}

== The Probabilistic Method ==

Suppose we want prove the existence of mathematic objects with certain properties. One way to do so is to explicitly construct such an object. This kind of proofs can be interpreted as ''deterministic algorithms'' which find the object with desirable properties.

The probabilistic method provides another way of proving the existence of objects: instead of explicitly constructing an object, we define a probability space of objects in which the probability is positive that a randomly selected object has the required property.

The basic principle of the probabilistic method is very simple, and can be stated in intuitive ways:
*If an object chosen randomly from a universe satisfies a property with positive probability, then there must be an object in the universe that satisfies that property.
:For example, for a ball(the object) randomly chosen from a box(the universe) of balls, if the probability that the chosen ball is blue(the property) is >0, then there must be a blue ball in the box.
*Any random variable assumes at least one value that is no smaller than its expectation, and at least one value that is no greater than the expectation.
:For example, if we know the average height of the students in the class is <math>\ell</math>, then we know there is a students whose height is at least <math>\ell</math>, and there is a student whose height is at most <math>\ell</math>.

Although the idea of the probabilistic method is simple, it provides us a powerful tool for existential proof. In same cases, the proof itself is a ''randomized algorithm'', and if we are lucky, the algorithm could be very efficient.

=== Sampling ===
;Ramsey number

Recall the Ramsey theorem which states that in a meeting of at least six people, there are either three people knowing each other or three people not knowing each other. In graph theoretical terms, this means that no matter how we color the edges of <math>K_6</math> (the complete graph on six vertices), there must be a '''monochromatic''' <math>K_3</math> (a triangle whose edges have the same color).

Generally, the '''Ramsey number''' <math>R(k,\ell)</math> is the smallest integer <math>n</math> such that in any two-coloring of the edges of a complete graph on <math>n</math> vertices <math>K_n</math> by red and blue, either there is a red <math>K_k</math> or there is a blue <math>K_\ell</math>.

Ramsey showed in 1929 that <math>R(k,\ell)</math> is finite for any <math>k</math> and <math>\ell</math>. It is extremely hard to compute the exact value of <math>R(k,\ell)</math>. Here we give a lower bound of <math>R(k,k)</math> by the probabilistic method.

{{Theorem
|Theorem (Erdős 1947)|
:If <math>{n\choose k}\cdot 2^{1-{k\choose 2}}<1</math> then it is possible to color the edges of <math>K_n</math> with two colors so that there is no monochromatic <math>K_k</math> subgraph.
}}
{{Proof| Consider a random two-coloring of edges of <math>K_n</math> obtained as follows:
* For each edge of <math>K_n</math>, independently flip a fair coin to decide the color of the edge.

For any fixed set <math>S</math> of <math>k</math> vertices, let <math>\mathcal{E}_S</math> be the event that the <math>K_k</math> subgraph induced by <math>S</math> is monochromatic. There are <math>{k\choose 2}</math> many edges in <math>K_k</math>, therefore
:<math>\Pr[\mathcal{E}_S]=2\cdot 2^{-{k\choose 2}}=2^{1-{k\choose 2}}.</math>

Since there are <math>{n\choose k}</math> possible choices of <math>S</math>, by the union bound
:<math>
\Pr[\exists S, \mathcal{E}_S]\le {n\choose k}\cdot\Pr[\mathcal{E}_S]={n\choose k}\cdot 2^{1-{k\choose 2}}.
</math>
Due to the assumption, <math>{n\choose k}\cdot 2^{1-{k\choose 2}}<1</math>, thus there exists a two coloring that none of <math>\mathcal{E}_S</math> occurs, which means there is no monochromatic <math>K_k</math> subgraph.
}}

For <math>k\ge 3</math> and we take <math>n=\lfloor2^{k/2}\rfloor</math>, then
:<math>
\begin{align}
{n\choose k}\cdot 2^{1-{k\choose 2}}
&<
\frac{n^k}{k!}\cdot\frac{2^{1+\frac{k}{2}}}{2^{k^2/2}}\\
&\le
\frac{2^{k^2/2}}{k!}\cdot\frac{2^{1+\frac{k}{2}}}{2^{k^2/2}}\\
&=
\frac{2^{1+\frac{k}{2}}}{k!}\\
&<1.
\end{align}
</math>
By the above theorem, there exists a two-coloring of <math>K_n</math> that there is no monochromatic <math>K_k</math>. Therefore, the Ramsey number <math>R(k,k)>\lfloor2^{k/2}\rfloor</math> for all <math>k\ge 3</math>.

Note that for sufficiently large <math>k</math>, if <math>n= \lfloor 2^{k/2}\rfloor</math>, then the probability that there exists a monochromatic <math>K_k</math> is bounded by
:<math>
{n\choose k}\cdot 2^{1-{k\choose 2}}
<
\frac{2^{1+\frac{k}{2}}}{k!}
\ll 1,
</math>
which means that a random two-coloring of <math>K_n</math> is very likely not to contain a monochromatic <math>K_{2\log n}</math>. This gives us a very simple randomized algorithm for finding a two-coloring of <math>K_n</math> without monochromatic <math>K_{2\log n}</math>.

=== Linearity of expectation ===

;Maximum cut

Given an undirected graph <math>G(V,E)</math>, a set <math>C</math> of edges of <math>G</math> is called a '''cut''' if <math>G</math> is disconnected after removing the edges in <math>C</math>. We can represent a cut by <math>c(S,T)</math> where <math>(S,T)</math> is a bipartition of the vertex set <math>V</math>, and <math>c(S,T)=\{uv\in E\mid u\in S,v\in T\}</math> is the set of edges crossing between <math>S</math> and <math>T</math>.

We have seen how to compute min-cut: either by deterministic max-flow algorithm, or by Karger's randomized algorithm. On the other hand, max-cut is hard to compute, because it is '''NP-complete'''. Actually, the weighted version of max-cut is among the [http://en.wikipedia.org/wiki/Karp's_21_NP-complete_problems Karp's 21 NP-complete problems].

We now show by the probabilistic method that a max-cut always has at least half the edges.

{{Theorem
|Theorem|
:Given an undirected graph <math>G</math> with <math>n</math> vertices and <math>m</math> edges, there is a cut of size at least <math>\frac{m}{2}</math>.
}}
{{Proof| Enumerate the vertices in an arbitrary order. Partition the vertex set <math>V</math> into two disjoint sets <math>S</math> and <math>T</math> as follows.
:For each vertex <math>v\in V</math>,
:* independently choose one of <math>S</math> and <math>T</math> with equal probability, and let <math>v</math> join the chosen set.

For each vertex <math>v\in V</math>, let <math>X_v\in\{S,T\}</math> be the random variable which represents the set that <math>v</math> joins. For each edge <math>uv\in E</math>, let <math>Y_{uv}</math> be the 0-1 random variable which indicates whether <math>uv</math> crosses between <math>S</math> and <math>T</math>. Clearly,
:<math>
\Pr[Y_{uv}=1]=\Pr[X_u\neq X_v]=\frac{1}{2}.
</math>

The size of <math>c(S,T)</math> is given by <math>Y=\sum_{uv\in E}Y_{uv}</math>. By the linearity of expectation,
:<math>
\mathbf{E}[Y]=\sum_{uv\in E}\mathbf{E}[Y_{uv}]=\sum_{uv\in E}\Pr[Y_{uv}=1]=\frac{m}{2}.
</math>
Therefore, there exist a bipartition <math>(S,T)</math> of <math>V</math> such that <math>|c(S,T)|\ge\frac{m}{2}</math>, i.e. there exists a cut of <math>G</math> which contains at least <math>\frac{m}{2}</math> edges.
}}

;Maximum satisfiability

Suppose that we have a number of boolean variables <math>x_1,x_2,\ldots,\in\{\mathrm{true},\mathrm{false}\}</math>. A '''literal''' is either a variable <math>x_i</math> itself or its negation <math>\neg x_i</math>. A logic expression is a '''conjunctive normal form (CNF)''' if it is written as the conjunction(AND) of a set of '''clauses''', where each clause is a disjunction(OR) of literals. For example:
:<math>
(x_1\vee \neg x_2 \vee \neg x_3)\wedge (\neg x_1\vee \neg x_3)\wedge (x_1\vee x_2\vee x_4)\wedge (x_4\vee \neg x_3)\wedge (x_4\vee \neg x_1).
</math>

The satisfiability (SAT) problem ask whether the CNF is satisfiable, i.e. there exists an assignment of variables to the values of true and false so that all clauses are true. The maximum satisfiability (MAXSAT) is the optimization version of SAT, which ask for an assignment that the number of satisfied clauses is maximized.

SAT is the first problem known to be '''NP-complete''' (the Cook-Levin theorem). MAXSAT is also '''NP-complete'''. We then see that there always exists a roughly good truth assignment which satisfies half the clauses.

{{Theorem
|Theorem|
:For any set of <math>m</math> clauses, there is a truth assignment that satisfies at least <math>\frac{m}{2}</math> clauses.
}}
{{Proof| For each variable, independently assign a random value in <math>\{\mathrm{true},\mathrm{false}\}</math> with equal probability. For the <math>i</math>th clause, let <math>X_i</math> be the random variable which indicates whether the <math>i</math>th clause is satisfied. Suppose that there are <math>k</math> literals in the clause. The probability that the clause is satisfied is
:<math>\Pr[X_k=1]\ge(1-2^{-k})\ge\frac{1}{2}</math>.

Let <math>X=\sum_{i=1}^m X_i</math> be the number of satisfied clauses. By the linearity of expectation,
:<math>
\mathbf{E}[X]=\sum_{i=1}^{m}\mathbf{E}[X_i]\ge \frac{m}{2}.
</math>
Therefore, there exists an assignment such that at least <math>\frac{m}{2}</math> clauses are satisfied.
}}

=== Alterations ===
;Independent sets
An independent set of a graph is a set of vertices with no edges between them. The following theorem gives a lower bound on the size of the largest independent set.
{{Theorem
|Theorem|
:Let <math>G(V,E)</math> be a graph on <math>n</math> vertices with <math>m</math> edges. Then <math>G</math> has an independent set with at least <math>\frac{n^2}{4m}</math> vertices.
}}
{{Proof| Let <math>S</math> be a set of vertices constructed as follows:
:For each vertex <math>v\in V</math>:
:* <math>v</math> is included in <math>S</math> independently with probability <math>p</math>,
<math>p</math> to be determined.

Let <math>X=|S|</math>. It is obvious that <math>\mathbf{E}[X]=np</math>.

For each edge <math>e\in E</math>, let <math>Y_{e}</math> be the random variable which indicates whether both endpoints of <math></math> are in <math>S</math>.
:<math>
\mathbf{E}[Y_{uv}]=\Pr[u\in S\wedge v\in S]=p^2.
</math>
Let <math>Y</math> be the number of edges in the subgraph of <math>G</math> induced by <math>S</math>. It holds that <math>Y=\sum_{e\in E}Y_e</math>. By linearity of expectation,
:<math>\mathbf{E}[Y]=\sum_{e\in E}\mathbf{E}[Y_e]=mp^2</math>.

Note that although <math>S</math> is not necessary an independent set, it can be modified to one if for each edge <math>e</math> of the induced subgraph <math>G(S)</math>, we delete one of the endpoint of <math>e</math> from <math>S</math>. Let <math>S^*</math> be the resulting set. It is obvious that <math>S^*</math> is an independent set since there is no edge left in the induced subgraph <math>G(S^*)</math>.

Since there are <math>Y</math> edges in <math>G(S)</math>, there are at most <math>Y</math> vertices in <math>S</math> are deleted to make it become <math>S^*</math>. Therefore, <math>|S^*|\ge X-Y</math>. By linearity of expectation,
:<math>
\mathbf{E}[|S^*|]\ge\mathbf{E}[X-Y]=\mathbf{E}[X]-\mathbf{E}[Y]=np-mp^2.
</math>
The expectation is maximized when <math>p=\frac{n}{2m}</math>, thus
:<math>
\mathbf{E}[|S^*|]\ge n\cdot\frac{n}{2m}-m\left(\frac{n}{2m}\right)^2=\frac{n^2}{4m}.
</math>
There exists an independent set which contains at least <math>\frac{n^2}{4m}</math> vertices.
}}

The proof actually propose a randomized algorithm for constructing large independent set:

{{Theorem
|Algorithm|
Given a graph on <math>n</math> vertices with <math>m</math> edges, let <math>d=\frac{2m}{n}</math> be the average degree.
#For each vertex <math>v\in V</math>, <math>v</math> is included in <math>S</math> independently with probability <math>\frac{1}{d}</math>.
#For each remaining edge in the induced subgraph <math>G(S)</math>, remove one of the endpoints from <math>S</math>.
}}

Let <math>S^*</math> be the resulting set. We have shown that <math>S^*</math> is an independent set and <math>\mathbf{E}[|S^*|]\ge\frac{n^2}{4m}</math>.

Combinatorics (Fall 2010)/Existence, the probabilistic method

2010-09-19T13:32:00Z

172.21.8.48: /* Counting arguments */

== Counting arguments ==
;Circuit complexity

This is a fundamental problem in in Computer Science.

A '''boolean function''' is a function is the form <math>f:\{0,1\}^n\rightarrow \{0,1\}</math>.

Formally, a boolean circuit is a directed acyclic graph. Nodes with indegree zero are input nodes, labeled <math>x_1, x_2, \ldots , x_n</math>. A circuit has a unique node with outdegree zero, called the output node. Every other node is a gate. There are three types of gates: AND, OR (both with indegree two), and NOT (with indegree one).

Computations in Turing machines can be simulated by circuits, and any boolean function in '''P''' can be computed by a circuit with polynomially many gates. Thus, if we can find a function in '''NP''' that cannot be computed by any circuit with polynomially many gates, then '''NP'''<math>\neq</math>'''P'''.

The following theorem due to Shannon says that functions with exponentially large circuit complexity do exist.

{{Theorem
|Theorem (Shannon 1949)|
:There is a boolean function <math>f:\{0,1\}^n\rightarrow \{0,1\}</math> with circuit complexity greater than <math>\frac{2^n}{3n}</math>.
}}
{{Proof|
We first count the number of boolean functions <math>f:\{0,1\}^n\rightarrow \{0,1\}</math>. There are <math>2^{2^n}</math> boolean functions <math>f:\{0,1\}^n\rightarrow \{0,1\}</math>.

Then we count the number of boolean circuit with fixed number of gates.
Fix an integer <math>t</math>, we count the number of circuits with <math>t</math> gates. By the [http://en.wikipedia.org/wiki/De_Morgan's_laws De Morgan's laws], we can assume that all NOTs are pushed back to the inputs. Each gate has one of the two types (AND or OR), and has two inputs. Each of the inputs to a gate is either a constant 0 or 1, an input variable <math>x_i</math>, an inverted input variable <math>\neg x_i</math>, or the output of another gate; thus, there are at most <math>2+2n+t-1</math> possible gate inputs. It follows that the number of circuits with <math>t</math> gates is at most <math>2^t(t+2n+1)^{2t}</math>.

If <math>t=2^n/3n</math>, then
:<math>
\frac{2^t(t+2n+1)^{2t}}{2^{2^n}}=o(1)<1,</math> thus, <math>2^t(t+2n+1)^{2t} < 2^{2^n}.</math>

Each boolean circuit computes one boolean function. Therefore, there must exist a boolean function <math>f</math> which cannot be computed by any circuits with <math>2^n/3n</math> gates.
}}

Note that by Shannon's theorem, not only there exists a boolean function with exponentially large circuit complexity, but ''almost all'' boolean functions have exponentially large circuit complexity.

=== Double counting ===

{{Theorem|Handshaking Lemma|
:At a party, the number of guests who shake hands an odd number of times is even.
}}

{{Theorem|Caylay's formula for trees|
: There are <math>n^{n-2}</math> different trees on <math>n</math> distinct vertices.
}}

== The Pigeonhole Principle ==

== The Probabilistic Method ==

Suppose we want prove the existence of mathematic objects with certain properties. One way to do so is to explicitly construct such an object. This kind of proofs can be interpreted as ''deterministic algorithms'' which find the object with desirable properties.

The probabilistic method provides another way of proving the existence of objects: instead of explicitly constructing an object, we define a probability space of objects in which the probability is positive that a randomly selected object has the required property.

The basic principle of the probabilistic method is very simple, and can be stated in intuitive ways:
*If an object chosen randomly from a universe satisfies a property with positive probability, then there must be an object in the universe that satisfies that property.
:For example, for a ball(the object) randomly chosen from a box(the universe) of balls, if the probability that the chosen ball is blue(the property) is >0, then there must be a blue ball in the box.
*Any random variable assumes at least one value that is no smaller than its expectation, and at least one value that is no greater than the expectation.
:For example, if we know the average height of the students in the class is <math>\ell</math>, then we know there is a students whose height is at least <math>\ell</math>, and there is a student whose height is at most <math>\ell</math>.

Although the idea of the probabilistic method is simple, it provides us a powerful tool for existential proof. In same cases, the proof itself is a ''randomized algorithm'', and if we are lucky, the algorithm could be very efficient.

=== Sampling ===
;Ramsey number

Recall the Ramsey theorem which states that in a meeting of at least six people, there are either three people knowing each other or three people not knowing each other. In graph theoretical terms, this means that no matter how we color the edges of <math>K_6</math> (the complete graph on six vertices), there must be a '''monochromatic''' <math>K_3</math> (a triangle whose edges have the same color).

Generally, the '''Ramsey number''' <math>R(k,\ell)</math> is the smallest integer <math>n</math> such that in any two-coloring of the edges of a complete graph on <math>n</math> vertices <math>K_n</math> by red and blue, either there is a red <math>K_k</math> or there is a blue <math>K_\ell</math>.

Ramsey showed in 1929 that <math>R(k,\ell)</math> is finite for any <math>k</math> and <math>\ell</math>. It is extremely hard to compute the exact value of <math>R(k,\ell)</math>. Here we give a lower bound of <math>R(k,k)</math> by the probabilistic method.

{{Theorem
|Theorem (Erdős 1947)|
:If <math>{n\choose k}\cdot 2^{1-{k\choose 2}}<1</math> then it is possible to color the edges of <math>K_n</math> with two colors so that there is no monochromatic <math>K_k</math> subgraph.
}}
{{Proof| Consider a random two-coloring of edges of <math>K_n</math> obtained as follows:
* For each edge of <math>K_n</math>, independently flip a fair coin to decide the color of the edge.

For any fixed set <math>S</math> of <math>k</math> vertices, let <math>\mathcal{E}_S</math> be the event that the <math>K_k</math> subgraph induced by <math>S</math> is monochromatic. There are <math>{k\choose 2}</math> many edges in <math>K_k</math>, therefore
:<math>\Pr[\mathcal{E}_S]=2\cdot 2^{-{k\choose 2}}=2^{1-{k\choose 2}}.</math>

Since there are <math>{n\choose k}</math> possible choices of <math>S</math>, by the union bound
:<math>
\Pr[\exists S, \mathcal{E}_S]\le {n\choose k}\cdot\Pr[\mathcal{E}_S]={n\choose k}\cdot 2^{1-{k\choose 2}}.
</math>
Due to the assumption, <math>{n\choose k}\cdot 2^{1-{k\choose 2}}<1</math>, thus there exists a two coloring that none of <math>\mathcal{E}_S</math> occurs, which means there is no monochromatic <math>K_k</math> subgraph.
}}

For <math>k\ge 3</math> and we take <math>n=\lfloor2^{k/2}\rfloor</math>, then
:<math>
\begin{align}
{n\choose k}\cdot 2^{1-{k\choose 2}}
&<
\frac{n^k}{k!}\cdot\frac{2^{1+\frac{k}{2}}}{2^{k^2/2}}\\
&\le
\frac{2^{k^2/2}}{k!}\cdot\frac{2^{1+\frac{k}{2}}}{2^{k^2/2}}\\
&=
\frac{2^{1+\frac{k}{2}}}{k!}\\
&<1.
\end{align}
</math>
By the above theorem, there exists a two-coloring of <math>K_n</math> that there is no monochromatic <math>K_k</math>. Therefore, the Ramsey number <math>R(k,k)>\lfloor2^{k/2}\rfloor</math> for all <math>k\ge 3</math>.

Note that for sufficiently large <math>k</math>, if <math>n= \lfloor 2^{k/2}\rfloor</math>, then the probability that there exists a monochromatic <math>K_k</math> is bounded by
:<math>
{n\choose k}\cdot 2^{1-{k\choose 2}}
<
\frac{2^{1+\frac{k}{2}}}{k!}
\ll 1,
</math>
which means that a random two-coloring of <math>K_n</math> is very likely not to contain a monochromatic <math>K_{2\log n}</math>. This gives us a very simple randomized algorithm for finding a two-coloring of <math>K_n</math> without monochromatic <math>K_{2\log n}</math>.

=== Linearity of expectation ===

;Maximum cut

Given an undirected graph <math>G(V,E)</math>, a set <math>C</math> of edges of <math>G</math> is called a '''cut''' if <math>G</math> is disconnected after removing the edges in <math>C</math>. We can represent a cut by <math>c(S,T)</math> where <math>(S,T)</math> is a bipartition of the vertex set <math>V</math>, and <math>c(S,T)=\{uv\in E\mid u\in S,v\in T\}</math> is the set of edges crossing between <math>S</math> and <math>T</math>.

We have seen how to compute min-cut: either by deterministic max-flow algorithm, or by Karger's randomized algorithm. On the other hand, max-cut is hard to compute, because it is '''NP-complete'''. Actually, the weighted version of max-cut is among the [http://en.wikipedia.org/wiki/Karp's_21_NP-complete_problems Karp's 21 NP-complete problems].

We now show by the probabilistic method that a max-cut always has at least half the edges.

{{Theorem
|Theorem|
:Given an undirected graph <math>G</math> with <math>n</math> vertices and <math>m</math> edges, there is a cut of size at least <math>\frac{m}{2}</math>.
}}
{{Proof| Enumerate the vertices in an arbitrary order. Partition the vertex set <math>V</math> into two disjoint sets <math>S</math> and <math>T</math> as follows.
:For each vertex <math>v\in V</math>,
:* independently choose one of <math>S</math> and <math>T</math> with equal probability, and let <math>v</math> join the chosen set.

For each vertex <math>v\in V</math>, let <math>X_v\in\{S,T\}</math> be the random variable which represents the set that <math>v</math> joins. For each edge <math>uv\in E</math>, let <math>Y_{uv}</math> be the 0-1 random variable which indicates whether <math>uv</math> crosses between <math>S</math> and <math>T</math>. Clearly,
:<math>
\Pr[Y_{uv}=1]=\Pr[X_u\neq X_v]=\frac{1}{2}.
</math>

The size of <math>c(S,T)</math> is given by <math>Y=\sum_{uv\in E}Y_{uv}</math>. By the linearity of expectation,
:<math>
\mathbf{E}[Y]=\sum_{uv\in E}\mathbf{E}[Y_{uv}]=\sum_{uv\in E}\Pr[Y_{uv}=1]=\frac{m}{2}.
</math>
Therefore, there exist a bipartition <math>(S,T)</math> of <math>V</math> such that <math>|c(S,T)|\ge\frac{m}{2}</math>, i.e. there exists a cut of <math>G</math> which contains at least <math>\frac{m}{2}</math> edges.
}}

;Maximum satisfiability

Suppose that we have a number of boolean variables <math>x_1,x_2,\ldots,\in\{\mathrm{true},\mathrm{false}\}</math>. A '''literal''' is either a variable <math>x_i</math> itself or its negation <math>\neg x_i</math>. A logic expression is a '''conjunctive normal form (CNF)''' if it is written as the conjunction(AND) of a set of '''clauses''', where each clause is a disjunction(OR) of literals. For example:
:<math>
(x_1\vee \neg x_2 \vee \neg x_3)\wedge (\neg x_1\vee \neg x_3)\wedge (x_1\vee x_2\vee x_4)\wedge (x_4\vee \neg x_3)\wedge (x_4\vee \neg x_1).
</math>

The satisfiability (SAT) problem ask whether the CNF is satisfiable, i.e. there exists an assignment of variables to the values of true and false so that all clauses are true. The maximum satisfiability (MAXSAT) is the optimization version of SAT, which ask for an assignment that the number of satisfied clauses is maximized.

SAT is the first problem known to be '''NP-complete''' (the Cook-Levin theorem). MAXSAT is also '''NP-complete'''. We then see that there always exists a roughly good truth assignment which satisfies half the clauses.

{{Theorem
|Theorem|
:For any set of <math>m</math> clauses, there is a truth assignment that satisfies at least <math>\frac{m}{2}</math> clauses.
}}
{{Proof| For each variable, independently assign a random value in <math>\{\mathrm{true},\mathrm{false}\}</math> with equal probability. For the <math>i</math>th clause, let <math>X_i</math> be the random variable which indicates whether the <math>i</math>th clause is satisfied. Suppose that there are <math>k</math> literals in the clause. The probability that the clause is satisfied is
:<math>\Pr[X_k=1]\ge(1-2^{-k})\ge\frac{1}{2}</math>.

Let <math>X=\sum_{i=1}^m X_i</math> be the number of satisfied clauses. By the linearity of expectation,
:<math>
\mathbf{E}[X]=\sum_{i=1}^{m}\mathbf{E}[X_i]\ge \frac{m}{2}.
</math>
Therefore, there exists an assignment such that at least <math>\frac{m}{2}</math> clauses are satisfied.
}}

=== Alterations ===
;Independent sets
An independent set of a graph is a set of vertices with no edges between them. The following theorem gives a lower bound on the size of the largest independent set.
{{Theorem
|Theorem|
:Let <math>G(V,E)</math> be a graph on <math>n</math> vertices with <math>m</math> edges. Then <math>G</math> has an independent set with at least <math>\frac{n^2}{4m}</math> vertices.
}}
{{Proof| Let <math>S</math> be a set of vertices constructed as follows:
:For each vertex <math>v\in V</math>:
:* <math>v</math> is included in <math>S</math> independently with probability <math>p</math>,
<math>p</math> to be determined.

Let <math>X=|S|</math>. It is obvious that <math>\mathbf{E}[X]=np</math>.

For each edge <math>e\in E</math>, let <math>Y_{e}</math> be the random variable which indicates whether both endpoints of <math></math> are in <math>S</math>.
:<math>
\mathbf{E}[Y_{uv}]=\Pr[u\in S\wedge v\in S]=p^2.
</math>
Let <math>Y</math> be the number of edges in the subgraph of <math>G</math> induced by <math>S</math>. It holds that <math>Y=\sum_{e\in E}Y_e</math>. By linearity of expectation,
:<math>\mathbf{E}[Y]=\sum_{e\in E}\mathbf{E}[Y_e]=mp^2</math>.

Note that although <math>S</math> is not necessary an independent set, it can be modified to one if for each edge <math>e</math> of the induced subgraph <math>G(S)</math>, we delete one of the endpoint of <math>e</math> from <math>S</math>. Let <math>S^*</math> be the resulting set. It is obvious that <math>S^*</math> is an independent set since there is no edge left in the induced subgraph <math>G(S^*)</math>.

Since there are <math>Y</math> edges in <math>G(S)</math>, there are at most <math>Y</math> vertices in <math>S</math> are deleted to make it become <math>S^*</math>. Therefore, <math>|S^*|\ge X-Y</math>. By linearity of expectation,
:<math>
\mathbf{E}[|S^*|]\ge\mathbf{E}[X-Y]=\mathbf{E}[X]-\mathbf{E}[Y]=np-mp^2.
</math>
The expectation is maximized when <math>p=\frac{n}{2m}</math>, thus
:<math>
\mathbf{E}[|S^*|]\ge n\cdot\frac{n}{2m}-m\left(\frac{n}{2m}\right)^2=\frac{n^2}{4m}.
</math>
There exists an independent set which contains at least <math>\frac{n^2}{4m}</math> vertices.
}}

The proof actually propose a randomized algorithm for constructing large independent set:

{{Theorem
|Algorithm|
Given a graph on <math>n</math> vertices with <math>m</math> edges, let <math>d=\frac{2m}{n}</math> be the average degree.
#For each vertex <math>v\in V</math>, <math>v</math> is included in <math>S</math> independently with probability <math>\frac{1}{d}</math>.
#For each remaining edge in the induced subgraph <math>G(S)</math>, remove one of the endpoints from <math>S</math>.
}}

Let <math>S^*</math> be the resulting set. We have shown that <math>S^*</math> is an independent set and <math>\mathbf{E}[|S^*|]\ge\frac{n^2}{4m}</math>.

Combinatorics (Fall 2010)/Partitions, sieve methods

2010-09-12T06:33:00Z

172.21.1.108: /* Reference */

== Partitions ==
We count the ways of partitioning <math>n</math> ''identical'' objects into <math>k</math> ''unordered'' groups. This is equivalent to counting the ways partitioning a number <math>n</math> into <math>k</math> unordered parts.

A '''<math>k</math>-partition''' of a number <math>n</math> is a multiset <math>\{x_1,x_2,\ldots,x_k\}</math> with <math>x_i\ge 1</math> for every element <math>x_i</math> and <math>x_1+x_2+\cdots+x_k=n</math>.

We define <math>p_k(n)</math> as the number of <math>k</math>-partitions of <math>n</math>.

For example, number 7 has the following partitions:
<div class="center"><math>
\begin{align}
&\{7\}
& p_1(7)=1\\
&\{1,6\},\{2,5\},\{3,4\}
& p_2(7)=3\\
&\{1,1,5\}, \{1,2,4\}, \{1,3,3\}, \{2,2,3\}
& p_3(7)=4\\
&\{1,1,1,4\},\{1,1,2,3\}, \{1,2,2,2\}
& p_4(7)=3\\
&\{1,1,1,1,3\},\{1,1,1,2,2\}
& p_5(7)=2\\
&\{1,1,1,1,1,2\}
& p_6(7)=1\\
&\{1,1,1,1,1,1,1\}
& p_7(7)=1
\end{align}
</math></div>

Equivalently, we can also define that A <math>k</math>-partition of a number <math>n</math> is a <math>k</math>-tuple <math>(x_1,x_2,\ldots,x_k)</math> with:
* <math>x_1\ge x_2\ge\cdots\ge x_k\ge 1</math>;
* <math>x_1+x_2+\cdots+x_k=n</math>.

<math>p_k(n)</math> the number of integral solutions to the above system.

Let <math>p(n)=\sum_{k=1}^n p_k(n)</math> be the total number of partitions of <math>n</math>. The function <math>p(n)</math> is called the '''partition number'''.

=== Counting <math>p_k(n)</math>===
We now try to determine <math>p_k(n)</math>. Unlike most problems we learned in the last lecture, <math>p_k(n)</math> does not have a nice closed form formula. We now give a recurrence for <math>p_k(n)</math>.

{{Theorem|Proposition|
:<math>p_k(n)=p_{k-1}(n-1)+p_k(n-k)\,</math>.
}}
{{Proof|
Suppose that <math>(x_1,\ldots,x_k)</math> is a <math>k</math>-partition of <math>n</math>. Note that it must hold that
:<math>x_1\ge x_2\ge \cdots \ge x_k\ge 1</math>.
There are two cases: <math>x_k=1</math> or <math>x_k>1</math>.
;Case 1.
:If <math>x_k=1</math>, then <math>(x_1,\cdots,x_{k-1})</math> is a distinct <math>(k-1)</math>-partition of <math>n-1</math>. And every <math>(k-1)</math>-partition of <math>n-1</math> can be obtained in this way. Thus the number of <math>k</math>-partitions of <math>n</math> in this case is <math>p_{k-1}(n-1)</math>.
;Case 2.
:If <math>x_k>1</math>, then <math>(x_1-1,\cdots,x_{k}-1)</math> is a distinct <math>k</math>-partition of <math>n-k</math>. And every <math>k</math>-partition of <math>n-k</math> can be obtained in this way. Thus the number of <math>k</math>-partitions of <math>n</math> in this case is <math>p_{k}(n-k)</math>.
In conclusion, the number of <math>k</math>-partitions of <math>n</math> is <math>p_{k-1}(n-1)+p_k(n-k)</math>, i.e.
:<math>p_k(n)=p_{k-1}(n-1)+p_k(n-k)\,</math>.
}}

Use the above recurrence, we can compute the <math>p_k(n)</math> for some decent <math>n</math> and <math>k</math> by computer simulation.

If we are not restricted ourselves to the precise estimation of <math>p_k(n)</math>, the next theorem gives an asymptotic estimation of <math>p_k(n)</math>. Note that it only holds for '''constant''' <math>k</math>, i.e. <math>k</math> does not depend on <math>n</math>.

{{Theorem|Theorem|
For any fixed <math>k</math>,
:<math>p_k(n)\sim\frac{n^{k-1}}{k!(k-1)!}</math>,
as <math>n\rightarrow \infty</math>.
}}
{{Proof|
Suppose that <math>(x_1,\ldots,x_k)</math> is a <math>k</math>-partition of <math>n</math>. Then <math>x_1+x_2+\cdots+x_k=n</math> and <math>x_1\ge x_2\ge \cdots \ge x_k\ge 1</math>.

The <math>k!</math> permutations of <math>(x_1,\ldots,x_k)</math> yield at most <math>k!</math> many <math>k</math>-compositions (the ''ordered'' sum of <math>k</math> positive integers). There are <math>{n-1\choose k-1}</math> many <math>k</math>-compositions of <math>n</math>, every one of which can be yielded in this way by permuting a partition. Thus,
:<math>k!p_k(n)\ge{n-1\choose k-1}</math>.

Let <math>y_i=x_i+k-i</math>. That is, <math>y_k=x_k, y_{k-1}=x_k+1, y_{k-2}=x_k+2,\ldots, y_{1}=x_k+k-1</math>. Then, it holds that
* <math>y_1>y_2>\cdots>y_k\ge 1</math>; and
* <math>y_1+y_2+\cdots+y_k=n+\frac{k(k-1)}{2}</math>.
Each permutation of <math>(y_1,y_2,\ldots,y_k)</math> yields a '''distinct''' <math>k</math>-composition of <math>n+\frac{k(k-1)}{2}</math>, because all <math>y_i</math> are distinct.
Thus,
:<math>k!p_k(n)\le {n+\frac{k(k-1)}{2}-1\choose k-1}</math>.

Combining the two inequalities, we have
:<math>\frac{{n-1\choose k-1}}{k!}\le p_k(n)\le \frac{{n+\frac{k(k-1)}{2}-1\choose k-1}}{k!}</math>.
The theorem follows.
}}

=== Ferrers diagram ===
A partition of a number <math>n</math> can be represented as a diagram of dots (or squares), called a '''Ferrers diagram''' (the square version of Ferrers diagram is also called a '''Young diagram''', named after a structured called Young tableaux).

Let <math>(x_1,x_2,\ldots,x_k)</math> with that <math>x_1\ge x_2\ge \cdots x_k\ge 1</math> be a partition of <math>n</math>. Its Ferrers diagram consists of <math>k</math> rows, where the <math>i</math>-th row contains <math>x_i</math> dots (or squares).

<div class="center">
{|border="0"
|
{|border="0"
|[[File:Chess xot45.svg|22px]]||[[File:Chess xot45.svg|22px]]||[[File:Chess xot45.svg|22px]]||[[File:Chess xot45.svg|22px]]||[[File:Chess xot45.svg|22px]]
|-
|[[File:Chess xot45.svg|22px]]||[[File:Chess xot45.svg|22px]]||[[File:Chess xot45.svg|22px]]||[[File:Chess xot45.svg|22px]]
|-
|[[File:Chess xot45.svg|22px]]||[[File:Chess xot45.svg|22px]]
|-
|[[File:Chess xot45.svg|22px]]
|}
|
[[File:Chess t45.svg|120px]]
|align=center|
{|border="2" cellspacing="4" cellpadding="3" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse;empty-cells:show;"
|[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]
|-
|[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]
|-
|[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]
|-
|[[File:Chess t45.svg|22px]]
|}
|-
|align=center|Ferrers diagram (''dot version'') of (5,4,2,1)||
|align=center|Ferrers diagram (''square version'') of (5,4,2,1)
|}
</div>

;Conjugate partition
The partition we get by reading the Ferrers diagram by column instead of rows is called the '''conjugate''' of the original partition.
<div class="center">
{|border="0"
|align=center|
{|border="2" cellspacing="4" cellpadding="3" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse;empty-cells:show;"
|[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]
|-
|[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]
|-
|[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]
|-
|[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]
|-
|[[File:Chess t45.svg|22px]]
|}
|
[[File:Chess t45.svg|120px]]
|align=center|
{|border="2" cellspacing="4" cellpadding="3" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse;empty-cells:show;"
|[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]
|-
|[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]
|-
|[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]
|-
|[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]||[[File:Chess t45.svg|22px]]
|-
|[[File:Chess t45.svg|22px]]
|-
|[[File:Chess t45.svg|22px]]
|}
|-
|align=center|<math>(6,4,4,2,1)</math>||
|align=center|conjugate: <math>(5,4,3,3,1,1)</math>
|}
</div>

Clearly,
* different partitions cannot have the same conjugate, and
* every partition of <math>n</math> is the conjugate of some partition of <math>n</math>,
so the conjugation mapping is a permutation on the set of partitions of <math>n</math>. This fact is very useful in proving theorems for partitions numbers.

Some theorems of partitions can be easily proved by representing partitions in Ferrers diagrams.

{{Theorem|Proposition|
# The number of partitions of <math>n</math> which have largest summand <math>k</math>, is <math>p_k(n)</math>.
# The number of <math>n</math> into <math>k</math> parts equals the number of partitions of <math>n-k</math> into at most <math>k</math> parts. Formally,
::<math>p_k(n)=\sum_{j=1}^k p_j(n-k)</math>.
}}
{{Proof|
# For every <math>k</math>-partition, the conjugate partition has largest part <math>k</math>. And vice versa.
# For a <math>k</math>-partition of <math>n</math>, remove the leftmost cell of every row of the Ferrers diagram. Totally <math>k</math> cells are removed and the remaining diagram is a partition of <math>n-k</math> into at most <math>k</math> parts. And for a partition of <math>n-k</math> into at most <math>k</math> parts, add a cell to each of the <math>k</math> rows (including the empty ones). This will give us a <math>k</math>-partition of <math>n</math>. It is easy to see the above mappings are 1-1 correspondences. Thus, the number of <math>n</math> into <math>k</math> parts equals the number of partitions of <math>n-k</math> into at most <math>k</math> parts.
}}

== Principle of Inclusion-Exclusion ==
Let <math>A</math> and <math>B</math> be two finite sets. The cardinality of their union is
:<math>|A\cup B|=|A|+|B|-{\color{Blue}|A\cap B|}</math>.
For three sets <math>A</math>, <math>B</math>, and <math>C</math>, the cardinality of the union of these three sets is computed as
:<math>|A\cup B\cup C|=|A|+|B|+|C|-{\color{Blue}|A\cap B|}-{\color{Blue}|A\cap C|}-{\color{Blue}|B\cap C|}+{\color{Red}|A\cap B\cap C|}</math>.
This is illustrated by the following figure.
::[[Image:Inclusion-exclusion.png|200px|border|center]]

Generally, the '''Principle of Inclusion-Exclusion''' states the rule for computing the union of <math>n</math> finite sets <math>A_1,A_2,\ldots,A_n</math>, such that
{{Equation|
<math>
\begin{align}
\left|\bigcup_{i=1}^nA_i\right|
&=
\sum_{I\subseteq\{1,\ldots,n\}}(-1)^{|I|-1}\left|\bigcap_{i\in I}A_i\right|.
\end{align}
</math>
}}
-----

In combinatorial enumeration, the Principle of Inclusion-Exclusion is usually applied in its complement form.

Let <math>A_1,A_2,\ldots,A_n\subseteq U</math> be subsets of some finite set <math>U</math>. Here <math>U</math> is some universe of combinatorial objects, whose cardinality is easy to calculate (e.g. all strings, tuples, permutations), and each <math>A_i</math> contains the objects with some specific property (e.g. a "pattern") which we want to avoid. The problem is to count the number of objects without any of the <math>n</math> properties. We write <math>\bar{A_i}=U-A</math>. The number of objects without any of the properties <math>A_1,A_2,\ldots,A_n</math> is
{{Equation|
<math>
\begin{align}
\left|\bar{A_1}\cap\bar{A_2}\cap\cdots\cap\bar{A_n}\right|=\left|U-\bigcup_{i=1}^nA_i\right|
&=
|U|-\sum_{I\subseteq\{1,\ldots,n\}}(-1)^{|I|}\left|\bigcap_{i\in I}A_i\right|.
\end{align}
</math>
}}
For an <math>I\subseteq\{1,2,\ldots,n\}</math>, we denote
:<math>A_I=\bigcap_{i\in I}A_i</math>
with the convention that <math>A_\emptyset=U</math>. The above equation is stated as:
{{Theorem|Principle of Inclusion-Exclusion|
:Let <math>A_1,A_2,\ldots,A_n</math> be a family of subsets of <math>U</math>. Then the number of elements of <math>U</math> which lie in none of the subsets <math>A_i</math> is
::<math>\sum_{I\subseteq\{1,\ldots, n\}}(-1)^{|I|}|A_I|</math>.
}}

Let <math>S_k=\sum_{|I|=k}|A_I|\,</math>. Conventionally, <math>S_0=|A_\emptyset|=|U|</math>. The principle of inclusion-exclusion can be expressed as
{{Equation|<math>
S_0-S_1+S_2+\cdots+(-1)^nS_n.
</math>
}}

=== Surjections ===
In the twelvefold way, we discuss the counting problems incurred by the mappings <math>f:N\rightarrow M</math>. The basic case is that elements from both <math>N</math> and <math>M</math> are distinguishable. In this case, it is easy to count the number of arbitrary mappings (which is <math>m^n</math>) and the number of injective (one-to-one) mappings (which is <math>(m)_n</math>), but the number of surjective is difficult. Here we apply the principle of inclusion-exclusion to count the number of surjective (onto) mappings.
{{Theorem|Theorem|
:The number of surjective mappings from an <math>n</math>-set to an <math>m</math>-set is given by
::<math>\sum_{k=1}^m(-1)^{m-k}{m\choose k}k^n</math>.
}}
{{Proof|
Let <math>U=\{f:[n]\rightarrow[m]\}</math> be the set of mappings from <math>[n]</math> to <math>[m]</math>. Then <math>|U|=m^n</math>.

For <math>i\in[m]</math>, let <math>A_i</math> be the set of mappings <math>f:[n]\rightarrow[m]</math> that none of <math>j\in[n]</math> is mapped to <math>i</math>, i.e. <math>A_i=\{f:[n]\rightarrow[m]\setminus\{i\}\}</math>, thus <math>|A_i|=(m-1)^n</math>.

More generally, for <math>I\subseteq [m]</math>, <math>A_I=\bigcap_{i\in I}A_i</math> contains the mappings <math>f:[n]\rightarrow[m]\setminus I</math>. And <math>|A_I|=(m-|I|)^n\,</math>.

A mapping <math>f:[n]\rightarrow[m]</math> is surjective if <math>f</math> lies in none of <math>A_i</math>. By the principle of inclusion-exclusion, the number of surjective <math>f:[n]\rightarrow[m]</math> is
:<math>\sum_{I\subseteq[m]}(-1)^{|I|}\left|A_I\right|=\sum_{I\subseteq[m]}(-1)^{|I|}(m-|I|)^n=\sum_{j=0}^m(-1)^j{m\choose j}(m-j)^n</math>.
Let <math>k=m-j</math>. The theorem is proved.
}}

Recall that, in the twelvefold way, we establish a relation between surjections and partitions.

* Surjection to ordered partition:
:For a surjective <math>f:[n]\rightarrow[m]</math>, <math>(f^{-1}(0),f^{-1}(1),\ldots,f^{-1}(m-1))</math> is an '''ordered partition''' of <math>[n]</math>.
* Ordered partition to surjection:
:For an ordered <math>m</math>-partition <math>(B_0,B_1,\ldots, B_{m-1})</math> of <math>[n]</math>, we can define a function <math>f:[n]\rightarrow[m]</math> by letting <math>f(i)=j</math> if and only if <math>i\in B_j</math>. <math>f</math> is surjective since as a partition, none of <math>B_i</math> is empty.

Therefore, we have a one-to-one correspondence between surjective mappings from an <math>n</math>-set to an <math>m</math>-set and the ordered <math>m</math>-partitions of an <math>n</math>-set.

The Stirling number of the second kind <math>S(n,m)</math> is the number of <math>m</math>-partitions of an <math>n</math>-set. There are <math>m!</math> ways to order an <math>m</math>-partition, thus the number of surjective mappings <math>f:[n]\rightarrow[m]</math> is <math>m! S(n,m)</math>. Combining with what we have proved for surjections, we give the following result for the Stirling number of the second kind.

{{Theorem|Proposition|
:<math>S(n,m)=\frac{1}{m!}\sum_{k=1}^m(-1)^{m-k}{m\choose k}k^n</math>.
}}

=== Derangements ===
We now count the number of bijections from a set to itself with no fixed points. This is the '''derangement problem'''.

For a permutation <math>\pi</math> of <math>\{1,2,\ldots,n\}</math>, a '''fixed point''' is such an <math>i\in\{1,2,\ldots,n\}</math> that <math>\pi(i)=i</math>.
A [http://en.wikipedia.org/wiki/Derangement '''derangement'''] of <math>\{1,2,\ldots,n\}</math> is a permutation of <math>\{1,2,\ldots,n\}</math> that has no fixed points.

{{Theorem|Theorem|
:The number of derangements of <math>\{1,2,\ldots,n\}</math> given by
::<math>n!\sum_{k=0}^n\frac{(-1)^k}{k!}\approx \frac{n!}{\mathrm{e}}</math>.
}}
{{Proof|
Let <math>U</math> be the set of all permutations of <math>\{1,2,\ldots,n\}</math>. So <math>|U|=n!</math>.

Let <math>A_i</math> be the set of permutations with fixed point <math>i</math>; so <math>|A_i|=(n-1)!</math>. More generally, for any <math>I\subseteq \{1,2,\ldots,n\}</math>, <math>A_I=\bigcap_{i\in I}A_i</math>, and <math>|A_I|=(n-|I|)!</math>, since permutations in <math>A_I</math> fix every point in <math>I</math> and permute the remaining points arbitrarily. A permutation is a derangement if and only if it lies in none of the sets <math>A_i</math>. So the number of derangements is
:<math>\sum_{I\subseteq\{1,2,\ldots,n\}}(-1)^{|I|}(n-|I|)!=\sum_{k=0}^n(-1)^k{n\choose k}(n-k)!=n!\sum_{k=0}^n\frac{(-1)^k}{k!}.</math>
By Taylor's series,
:<math>\frac{1}{\mathrm{e}}=\sum_{k=0}^\infty\frac{(-1)^k}{k!}=\sum_{k=0}^n\frac{(-1)^k}{k!}\pm o\left(\frac{1}{n!}\right)</math>.
It is not hard to see that <math>n!\sum_{k=0}^n\frac{(-1)^k}{k!}</math> is the closest integer to <math>\frac{n!}{\mathrm{e}}</math>.
}}

Therefore, there are about <math>\frac{1}{\mathrm{e}}</math> fraction of all permutations with no fixed points.

=== Permutations with restricted positions ===
We introduce a general theory of counting permutations with restricted positions. In the derangement problem, we count the number of permutations that <math>\pi(i)\neq i</math>. We now generalize to the problem of counting permutations which avoid a set of arbitrarily specified positions.

It is traditionally described using terminology from the game of chess. Let <math>B\subseteq \{1,\ldots,n\}\times \{1,\ldots,n\}</math>, called a '''board'''. As illustrated below, we can think of <math>B</math> as a chess board, with the positions in <math>B</math> marked by "<math>\times</math>".
{{Chess diagram small
|
|
|=
8 |__|xx|xx|__|xx|__|__|xx|=
7 |xx|__|__|xx|__|__|xx|__|=
6 |xx|__|xx|xx|__|xx|xx|__|=
5 |__|xx|__|__|xx|__|xx|__|=
4 |xx|__|__|__|xx|xx|xx|__|=
3 |__|xx|__|xx|__|__|__|xx|=
2 |__|__|xx|__|xx|__|__|xx|=
1 |xx|__|__|xx|__|xx|__|__|=
a b c d e f g h
|
}}
For a permutation <math>\pi</math> of <math>\{1,\ldots,n\}</math>, define the '''graph''' <math>G_\pi(V,E)</math> as
:<math>
\begin{align}
G_\pi &= \{(i,\pi(i))\mid i\in \{1,2,\ldots,n\}\}.
\end{align}
</math>
This can also be viewed as a set of marked positions on a chess board. Each row and each column has only one marked position, because <math>\pi</math> is a permutation. Thus, we can identify each <math>G_\pi</math> as a placement of <math>n</math> rooks (“城堡”，规则同中国象棋里的“车”) without attacking each other.

For example, the following is the <math>G_\pi</math> of such <math>\pi</math> that <math>\pi(i)=i</math>.
{{Chess diagram small
|
|
|=
8 |rl|__|__|__|__|__|__|__|=
7 |__|rl|__|__|__|__|__|__|=
6 |__|__|rl|__|__|__|__|__|=
5 |__|__|__|rl|__|__|__|__|=
4 |__|__|__|__|rl|__|__|__|=
3 |__|__|__|__|__|rl|__|__|=
2 |__|__|__|__|__|__|rl|__|=
1 |__|__|__|__|__|__|__|rl|=
a b c d e f g h
|
}}
Now define
:<math>\begin{align}
N_0 &= \left|\left\{\pi\mid B\cap G_\pi=\emptyset\right\}\right|\\
r_k &= \mbox{number of }k\mbox{-subsets of }B\mbox{ such that no two elements have a common coordinate}\\
&=\left|\left\{S\in{B\choose k} \,\bigg|\, \forall (i_1,j_1),(i_2,j_2)\in S, i_1\neq i_2, j_1\neq j_2 \right\}\right|
\end{align}
</math>
Interpreted in chess game,
* <math>B</math>: a set of marked positions in an <math>[n]\times [n]</math> chess board.
* <math>N_0</math>: the number of ways of placing <math>n</math> non-attacking rooks on the chess board such that none of these rooks lie in <math>B</math>.
* <math>r_k</math>: number of ways of placing <math>k</math> non-attacking rooks on <math>B</math>.

Our goal is to count <math>N_0</math> in terms of <math>r_k</math>. This gives the number of permutations avoid all positions in a <math>B</math>.

{{Theorem|Theorem|
:<math>N_0=\sum_{k=0}^n(-1)^kr_k(n-k)!</math>.
}}
{{Proof|
For each <math>i\in[n]</math>, let <math>A_i=\{\pi\mid (i,\pi(i))\in B\}</math> be the set of permutations <math>\pi</math> whose <math>i</math>-th position is in <math>B</math>.

<math>N_0</math> is the number of permutations avoid all positions in <math>B</math>. Thus, our goal is to count the number of permutations <math>\pi</math> in none of <math>A_i</math> for <math>i\in [n]</math>.

For each <math>I\subseteq [n]</math>, let <math>A_I=\bigcap_{i\in I}A_i</math>, which is the set of permutations <math>\pi</math> such that <math>(i,\pi(i))\in B</math> for all <math>i\in I</math>. Due to the principle of inclusion-exclusion,
:<math>N_0=\sum_{I\subseteq [n]} (-1)^{|I|}|A_I|=\sum_{k=0}^n(-1)^k\sum_{I\in{[n]\choose k}}|A_I|</math>.

The next observation is that
:<math>\sum_{I\in{[n]\choose k}}|A_I|=r_k(n-k)!</math>,
because we can count both sides by first placing <math>k</math> non-attacking rooks on <math>B</math> and placing <math>n-k</math> additional non-attacking rooks on <math>[n]\times [n]</math> in <math>(n-k)!</math> ways.

Therefore,
:<math>N_0=\sum_{k=0}^n(-1)^kr_k(n-k)!</math>.
}}

====Derangement problem====
We use the above general method to solve the derange problem again.

Take <math>B=\{(1,1),(2,2),\ldots,(n,n)\}</math> as the chess board. A derangement <math>\pi</math> is a placement of <math>n</math> non-attacking rooks such that none of them is in <math>B</math>.
{{Chess diagram small
|
|
|=
8 |xx|__|__|__|__|__|__|__|=
7 |__|xx|__|__|__|__|__|__|=
6 |__|__|xx|__|__|__|__|__|=
5 |__|__|__|xx|__|__|__|__|=
4 |__|__|__|__|xx|__|__|__|=
3 |__|__|__|__|__|xx|__|__|=
2 |__|__|__|__|__|__|xx|__|=
1 |__|__|__|__|__|__|__|xx|=
a b c d e f g h
|
}}
Clearly, the number of ways of placing <math>k</math> non-attacking rooks on <math>B</math> is <math>r_k={n\choose k}</math>. We want to count <math>N_0</math>, which gives the number of ways of placing <math>n</math> non-attacking rooks such that none of these rooks lie in <math>B</math>.

By the above theorem
:<math>
N_0=\sum_{k=0}^n(-1)^kr_k(n-k)!=\sum_{k=0}^n(-1)^k{n\choose k}(n-k)!=\sum_{k=0}^n(-1)^k\frac{n!}{k!}=n!\sum_{k=0}^n(-1)^k\frac{1}{k!}\approx\frac{n!}{e}.
</math>

====Problème des ménages====
Suppose that in a banquet, we want to seat <math>n</math> couples at a circular table, satisfying the following constraints:
* Men and women are in alternate places.
* No one sits next to his/her spouse.

In how many ways can this be done?

(For convenience, we assume that every seat at the table marked differently so that rotating the seats clockwise or anti-clockwise will end up with a '''different''' solution.)

First, let the <math>n</math> ladies find their seats. They may either sit at the odd numbered seats or even numbered seats, in either case, there are <math>n!</math> different orders. Thus, there are <math>2(n!)</math> ways to seat the <math>n</math> ladies.

After sitting the wives, we label the remaining <math>n</math> places clockwise as <math>0,1,\ldots, n-1</math>. And a seating of the <math>n</math> husbands is given by a permutation <math>\pi</math> of <math>[n]</math> defined as follows. Let <math>\pi(i)</math> be the seat of the husband of he lady sitting at the <math>i</math>-th place.

It is easy to see that <math>\pi</math> satisfies that <math>\pi(i)\neq i</math> and <math>\pi(i)\not\equiv i+1\pmod n</math>, and every permutation <math>\pi</math> with these properties gives a feasible seating of the <math>n</math> husbands. Thus, we only need to count the number of permutations <math>\pi</math> such that <math>\pi(i)\not\equiv i, i+1\pmod n</math>.

Take <math>B=\{(0,0),(1,1),\ldots,(n-1,n-1), (0,1),(1,2),\ldots,(n-2,n-1),(n-1,0)\}</math> as the chess board. A permutation <math>\pi</math> which defines a way of seating the husbands, is a placement of <math>n</math> non-attacking rooks such that none of them is in <math>B</math>.
{{Chess diagram small
|
|
|=
8 |xx|xx|__|__|__|__|__|__|=
7 |__|xx|xx|__|__|__|__|__|=
6 |__|__|xx|xx|__|__|__|__|=
5 |__|__|__|xx|xx|__|__|__|=
4 |__|__|__|__|xx|xx|__|__|=
3 |__|__|__|__|__|xx|xx|__|=
2 |__|__|__|__|__|__|xx|xx|=
1 |xx|__|__|__|__|__|__|xx|=
a b c d e f g h
|
}}
We need to compute <math>r_k</math>, the number of ways of placing <math>k</math> non-attacking rooks on <math>B</math>. For our choice of <math>B</math>, <math>r_k</math> is the number of ways of choosing <math>k</math> points, no two consecutive, from a collection of <math>2n</math> points arranged in a circle.

We first see how to do this in a ''line''.
{{Theorem|Lemma|
:The number of ways of choosing <math>k</math> ''non-consecutive'' objects from a collection of <math>m</math> objects arranged in a ''line'', is <math>{m-k+1\choose k}</math>.
}}
{{Proof|
We draw a line of <math>m-k</math> black points, and then insert <math>k</math> red points into the <math>m-k+1</math> spaces between the black points (including the beginning and end).
::<math>
\begin{align}
&\sqcup \, \bullet \, \sqcup \, \bullet \, \sqcup \, \bullet \, \sqcup \, \bullet \, \sqcup \, \bullet \, \sqcup \, \bullet \, \sqcup \, \bullet \, \sqcup \\
&\qquad\qquad\qquad\quad\Downarrow\\
&\sqcup \, \bullet \,\, {\color{Red}\bullet} \, \bullet \,\, {\color{Red}\bullet} \, \bullet \, \sqcup \, \bullet \,\, {\color{Red}\bullet}\, \, \bullet \, \sqcup \, \bullet \, \sqcup \, \bullet \,\, {\color{Red}\bullet}
\end{align}
</math>
This gives us a line of <math>m</math> points, and the red points specifies the chosen objects, which are non-consecutive. The mapping is 1-1 correspondence.
There are <math>{m-k+1\choose k}</math> ways of placing <math>k</math> red points into <math>m-k+1</math> spaces.
}}

The problem of choosing non-consecutive objects in a circle can be reduced to the case that the objects are in a line.

{{Theorem|Lemma|
:The number of ways of choosing <math>k</math> ''non-consecutive'' objects from a collection of <math>m</math> objects arranged in a ''circle'', is <math>\frac{m}{m-k}{m-k\choose k}</math>.
}}
{{Proof|
Let <math>f(m,k)</math> be the desired number; and let <math>g(m,k)</math> be the number of ways of choosing <math>k</math> non-consecutive points from <math>m</math> points arranged in a circle, next coloring the <math>k</math> points red, and then coloring one of the uncolored point blue.

Clearly, <math>g(m,k)=(m-k)f(m,k)</math>.

But we can also compute <math>g(m,k)</math> as follows:
* Choose one of the <math>m</math> points and color it blue. This gives us <math>m</math> ways.
* Cut the circle to make a line of <math>m-1</math> points by removing the blue point.
* Choose <math>k</math> non-consecutive points from the line of <math>m-1</math> points and color them red. This gives <math>{m-k\choose k}</math> ways due to the previous lemma.

Thus, <math>g(m,k)=m{m-k\choose k}</math>. Therefore we have the desired number <math>f(m,k)=\frac{m}{m-k}{m-k\choose k}</math>.
}}

By the above lemma, we have that <math>r_k=\frac{2n}{2n-k}{2n-k\choose k}</math>. Then apply the theorem of counting permutations with restricted positions,
:<math>
N_0=\sum_{k=0}^n(-1)^kr_k(n-k)!=\sum_{k=0}^n(-1)^k\frac{2n}{2n-k}{2n-k\choose k}(n-k)!.
</math>

This gives the number of ways of seating the <math>n</math> husbands ''after the ladies are seated''. Recall that there are <math>2n!</math> ways of seating the <math>n</math> ladies. Thus, the total number of ways of seating <math>n</math> couples as required by problème des ménages is
:<math>
2n!\sum_{k=0}^n(-1)^k\frac{2n}{2n-k}{2n-k\choose k}(n-k)!.
</math>

=== The Euler totient function ===
Two integers <math>m, n</math> are said to be '''relatively prime''' if their greatest common diviser <math>\mathrm{gcd}(m,n)=1</math>. For a positive integer <math>n</math>, let <math>\phi(n)</math> be the number of positive integers from <math>\{1,2,\ldots,n\}</math> that are relative prime to <math>n</math>. This function, called the Euler <math>\phi</math> function or '''the Euler totient function''', is fundamental in number theory.

We know derive a formula for this function by using the principle of inclusion-exclusion.
{{Theorem|Theorem (The Euler totient function)|
Suppose <math>n</math> is divisible by precisely <math>r</math> different primes, denoted <math>p_1,\ldots,p_r</math>. Then
:<math>\phi(n)=n\prod_{i=1}^r\left(1-\frac{1}{p_i}\right)</math>.
}}
{{Proof|
Let <math>U=\{1,2,\ldots,n\}</math> be the universe. The number of positive integers from <math>U</math> which is divisible by some <math>p_{i_1},p_{i_2},\ldots,p_{i_s}\in\{p_1,\ldots,p_r\}</math>, is <math>\frac{n}{p_{i_1}p_{i_2}\cdots p_{i_s}}</math>.

<math>\phi(n)</math> is the number of integers from <math>U</math> which is not divisible by any <math>p_1,\ldots,p_r</math>.
By principle of inclusion-exclusion,
:<math>
\begin{align}
\phi(n)
&=n+\sum_{k=1}^r(-1)^k\sum_{1\le i_1<i_2<\cdots <i_k\le n}\frac{n}{p_{i_1}p_{i_2}\cdots p_{i_k}}\\
&=n-\sum_{1\le i\le n}\frac{n}{p_i}+\sum_{1\le i<j\le n}\frac{n}{p_i p_j}-\sum_{1\le i<j<k\le n}\frac{n}{p_{i} p_{j} p_{k}}+\cdots + (-1)^r\frac{n}{p_{1}p_{2}\cdots p_{r}}\\
&=n\left(1-\sum_{1\le i\le n}\frac{1}{p_i}+\sum_{1\le i<j\le n}\frac{1}{p_i p_j}-\sum_{1\le i<j<k\le n}\frac{1}{p_{i} p_{j} p_{k}}+\cdots + (-1)^r\frac{1}{p_{1}p_{2}\cdots p_{r}}\right)\\
&=n\prod_{i=1}^n\left(1-\frac{1}{p_i}\right).
\end{align}
</math>
}}

== Reference ==
* ''Stanley,'' Enumerative Combinatorics, Volume 1, Chapter 2.
* ''van Lin and Wilson'', A course in combinatorics, Chapter 10, 15.

Combinatorics (Fall 2010)/Generating functions

2010-09-12T06:32:36Z

172.21.1.108: /* Reference */

== Generating Functions ==
In Stanley's magnificent book ''Enumerative Combinatorics'', he comments the generating function as "the most useful but most difficult to understand method (for counting)".

The solution to a counting problem is usually represented as some <math>a_n</math> depending a parameter <math>n</math>. Sometimes this <math>a_n</math> is called a ''counting function'' as it is a function of the parameter <math>n</math>. <math>a_n</math> can also be treated as a infinite series:
:<math>a_0,a_1,a_2,\ldots</math>

The '''ordinary generating function (OGF)''' defined by <math>a_n</math> is
:<math>
G(x)=\sum_{n\ge 0} a_nx^n.
</math>

So <math>G(x)=a_0+a_1x+a_2x^2+\cdots</math>. An expression in this form is called a [http://en.wikipedia.org/wiki/Formal_power_series '''formal power series'''], and <math>a_0,a_1,a_2,\ldots</math> is the sequence of '''coefficients'''.

Furthermore, the generating function can be expanded as
:G(x)=<math>(\underbrace{1+\cdots+1}_{a_0})+(\underbrace{x+\cdots+x}_{a_1})+(\underbrace{x^2+\cdots+x^2}_{a_2})+\cdots+(\underbrace{x^n+\cdots+x^n}_{a_n})+\cdots</math>
so it indeed "generates" all the possible instances of the objects we want to count.

Usually, we do not evaluate the generating function <math>GF(x)</math> on any particular value. <math>x</math> remains as a '''formal variable''' without assuming any value. The numbers that we want to count are the coefficients carried by the terms in the formal power series. So far the generating function is just another way to represent the sequence
:<math>(a_0,a_1,a_2,\ldots\ldots)</math>.

The true power of generating functions comes from the various algebraic operations that we can perform on these generating functions. We use an example to demonstrate this.

=== Combinations ===
Suppose we wish to enumerate all subsets of an <math>n</math>-set. To construct a subset, we specifies for every element of the <math>n</math>-set whether the element is chosen or not. Let us denote the choice to omit an element by <math>x_0</math>, and the choice to include it by <math>x_1</math>. Using "<math>+</math>" to represent "OR", and using the multiplication to denote "AND", the choices of subsets of the <math>n</math>-set are expressed as
:<math>\underbrace{(x_0+x_1)(x_0+x_1)\cdots (x_0+x_1)}_{n\mbox{ elements}}=(x_0+x_1)^n</math>.

For example, when <math>n=3</math>, we have
:<math>\begin{align}
(x_0+x_1)^3
&=x_0x_0x_0+x_0x_0x_1+x_0x_1x_0+x_0x_1x_1\\
&\quad +x_1x_0x_0+x_1x_0x_1+x_1x_1x_0+x_1x_1x_1
\end{align}</math>.

So it "generate" all subsets of the 3-set. Writing <math>1</math> for <math>x_0</math> and <math>x</math> for <math>x_1</math>, we have <math>(1+x)^3=1+3x+3x^2+x^3</math>. The coefficient of <math>x^k</math> is the number of <math>k</math>-subsets of a 3-element set.

In general, <math>(1+x)^n</math> has the coefficients which are the number of subsets of fixed sizes of an <math>n</math>-element set.

-----

Suppose that we have twelve balls: <font color="red">3 red</font>, <font color="blue">4 blue</font>, and <font color="green">5 green</font>. Balls with the same color are indistinguishable.

We want to determine the number of ways to select <math>k</math> balls from these twelve balls, for some <math>0\le k\le 12</math>.

The generating function of this sequence is
:<math>\begin{align}
&\quad {\color{Red}(1+x+x^2+x^3)}{\color{Blue}(1+x+x^2+x^3+x^4)}{\color{OliveGreen}(1+x+x^2+x^3+x^4+x^5)}\\
&=1+3x+6x^2+10x^3+14x^4+17x^5+18x^6+17x^7+14x^8+10x^9+6x^{10}+3x^{11}+x^{12}.
\end{align}</math>
The coefficient of <math>x^k</math> gives the number of ways to select <math>k</math> balls.

=== Fibonacci numbers ===
Consider the following counting problems.
* Count the number of ways that the nonnegative integer <math>n</math> can be written as a sum of ones and twos (in order).
: The problem asks for the number of compositions of <math>n</math> with summands from <math>\{1,2\}</math>. Formally, we are counting the number of tuples <math>(x_1,x_2,\ldots,x_k)</math> for some <math>k\le n</math> such that <math>x_i\in\{1,2\}</math> and <math>x_1+x_2+\cdots+x_k=n</math>.
: Let <math>F_n</math> be the solution. We observe that a composition either starts with a 1, in which case the rest is a composition of <math>n-1</math>; or starts with a 2, in which case the rest is a composition of <math>n-2</math>. So we have the recursion for <math>F_n</math> that
::<math>F_n=F_{n-1}+F_{n-2}</math>.
* Count the ways to completely cover a <math>2\times n</math> rectangle with <math>2\times 1</math> dominos without any overlaps.
: Dominos are identical <math>2\times 1</math> rectangles, so that only their orientations --- vertical or horizontal matter.
: Let <math>F_n</math> be the solution. It also holds that <math>F_n=F_{n-1}+F_{n-2}</math>. The proof is left as an exercise.

In both problems, the solution is given by <math>F_n</math> which satisfies the following recursion.
:<math>F_n=\begin{cases}
F_{n-1}+F_{n-2} & \mbox{if }n\ge 2,\\
1 & \mbox{if }n=1\\
0 & \mbox{if }n=0.
\end{cases}</math>

<math>F_n</math> is called the [http://en.wikipedia.org/wiki/Fibonacci_number Fibonacci number].

{{Theorem|Theorem|
::<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)</math>,
:where <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>.
}}
The quantity <math>\phi=\frac{1+\sqrt{5}}{2}</math> is the so-called [http://en.wikipedia.org/wiki/Golden_ratio golden ratio], a constant with some significance in mathematics and aesthetics.

We now prove this theorem by using generating functions.
The ordinary generating function for the Fibonacci number <math>F_{n}</math> is
:<math>G(x)=\sum_{n\ge 0}F_n x^n</math>.
We have that <math>F_{n}=F_{n-1}+F_{n-2}</math> for <math>n\ge 2</math>, thus
:<math>\begin{align}
G(x)
&=
\sum_{n\ge 0}F_n x^n
&=
x+\sum_{n\ge 2}(F_{n-1}+F_{n-2})x^n.
\end{align}
</math>
For generating functions, there are general ways to generate <math>F_{n-1}</math> and <math>F_{n-2}</math>, or the coefficients with any smaller indices.
:<math>
\begin{align}
xG(x)
&=\sum_{n\ge 0}F_n x^{n+1}=\sum_{n\ge 1}F_{n-1} x^n=\sum_{n\ge 2}F_{n-1} x^n\\
x^2G(x)
&=\sum_{n\ge 0}F_n x^{n+2}=\sum_{n\ge 2}F_{n-2} x^n.
\end{align}
</math>
So we have
:<math>G(x)=x+(x+x^2)G(x)\,</math>,
hence
:<math>G(x)=\frac{x}{1-x-x^2}</math>.
The value of <math>F_n</math> is the coefficient of <math>x^n</math> in the Taylor series for this formular, which is <math>\frac{G^{(n)}(0)}{n!}=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>. Although this expansion works in principle, the detailed calculus is rather painful.

----
It is easier to expand the generating function by breaking it into two geometric series.
{{Theorem|Proposition|
:Let <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>. It holds that
::<math>\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
}}

It is easy to verify the above equation, but to deduce it, we need some (high school) calculation.

{|border="2" width="100%" cellspacing="4" cellpadding="3" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse;empty-cells:show;"
|
:{|
|
<math>1-x-x^2</math> has two roots <math>\frac{-1\pm\sqrt{5}}{2}</math>.

Denote that <math>\phi=\frac{2}{-1+\sqrt{5}}=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{2}{-1-\sqrt{5}}=\frac{1-\sqrt{5}}{2}</math>.

Then <math>(1-x-x^2)=(1-\phi x)(1-\hat{\phi}x)</math>, so we can write
:<math>
\begin{align}
\frac{x}{1-x-x^2}
&=\frac{x}{(1-\phi x)(1-\hat{\phi} x)}\\
&=\frac{\alpha}{(1-\phi x)}+\frac{\beta}{(1-\hat{\phi} x)},
\end{align}
</math>
where <math>\alpha</math> and <math>\beta</math> satisfying that
:<math>\begin{cases}
\alpha+\beta=0\\
\alpha\phi+\beta\hat{\phi}= -1.
\end{cases}</math>
Solving this we have that <math>\alpha=\frac{1}{\sqrt{5}}</math> and <math>\beta=-\frac{1}{\sqrt{5}}</math>. Thus,
:<math>G(x)=\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
|}
:<math>\square</math>
|}

Note that the expression <math>\frac{1}{1-z}</math> has a well known geometric expansion:
:<math>\frac{1}{1-z}=\sum_{n\ge 0}z^n</math>.

Therefore, <math>G(x)</math> can be expanded as
:<math>
\begin{align}
G(x)
&=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}\\
&=\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\phi x)^n-\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\hat{\phi} x)^n\\
&=\sum_{n\ge 0}\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)x^n.
\end{align}</math>
So the <math>n</math>th Fibonacci number is given by
:<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>.

== Solving recurrences ==
The following steps describe a general methodology of solving recurrences by generating functions.
:1. Give a recursion that computes <math>a_n</math>. In the case of Fibonacci sequence
::<math>a_n=a_{n-1}+a_{n-2}</math>.
:2. Multiply both sides of the equation by <math>x^n</math> and sum over all <math>n</math>. This gives the generating function
::<math>G(x)=\sum_{n\ge 0}a_nx^n=\sum_{n\ge 0}(a_{n-1}+a_{n-2})x^n</math>.
:: And manipulate the right hand side of the equation so that it becomes some other expression involving <math>G(x)</math>.
::<math>G(x)=x+(x+x^2)G(x)\,</math>.
:3. Solve the resulting equation to derive an explicit formula for <math>G(x)</math>.
::<math>G(x)=\frac{x}{1-x-x^2}</math>.
:4. Expand <math>G(x)</math> into a power series and read off the coefficient of <math>x^n</math>, which is a closed form for <math>a_n</math>.

The first step is usually established by combinatorial observations, or explicitly given by the problem. The third step is trivial.

The second and the forth steps need some non-trivial analytic techniques.

=== Algebraic operations on generating functions ===
The second step in the above methodology is somehow tricky. It involves first applying the recurrence to the coefficients of <math>G(x)</math>, which is easy; and then manipulating the resulting formal power series to express it in terms of <math>G(x)</math>, which is more difficult (because it works backwards).

We can apply several natural algebraic operations on the formal power series.

{{Theorem|Generating function manipulation|
:Let <math>G(x)=\sum_{n\ge 0}g_nx^n</math> and <math>F(x)=\sum_{n\ge 0}f_nx^n</math>.
----

::<math>
\begin{align}
x^k G(x)
&= \sum_{n\ge k}g_{n-k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\frac{G(x)-\sum_{i=0}^{k-1}g_iz^i}{x^k}
&=\sum_{n\ge 0}g_{n+k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\alpha F(x)+\beta G(x)
&= \sum_{n\ge 0} (\alpha f_n+\beta g_n)x^n\\
F(x)G(x)
&= \sum_{n\ge 0}\sum_{k=0}^nf_kg_{n-k}x^n\\
G(cx)
&= \sum_{n\ge 0} c^ng_n x^n\\
G'(x)
&=
\sum_{n\ge 0}(n+1)g_{n+1}x^n
\end{align}
</math>
}}

When manipulating generating functions, these rules are applied backwards; that is, from the right-hand-side to the left-hand-side.

=== Expanding generating functions ===
The last step of solving recurrences by generating function is expanding the closed form generating function <math>G(x)</math> to evaluate its <math>n</math>-th coefficient. In principle, we can always use the [http://en.wikipedia.org/wiki/Taylor_series Taylor series]
:<math>G(x)=\sum_{n\ge 0}\frac{G^{(n)}(0)}{n!}x^n</math>,
where <math>G^{(n)}(0)</math> is the value of the <math>n</math>-th derivative of <math>G(x)</math> evaluated at <math>x=0</math>.

Some interesting special cases are very useful.

====Geometric sequence====
In the example of Fibonacci numbers, we use the well known geometric series:
:<math>\frac{1}{1-x}=\sum_{n\ge 0}x^n</math>.
It is useful when we can express the generating function in the form of <math>G(x)=\frac{a_1}{1-b_1x}+\frac{a_2}{1-b_2x}+\cdots+\frac{a_k}{1-b_kx}</math>. The coefficient of <math>x^n</math> in such <math>G(x)</math> is <math>a_1b_1^n+a_2b_2^n+\cdots+a_kb_k^n</math>.

====Binomial theorem====
The <math>n</math>-th derivative of <math>(1+x)^\alpha</math> for some real <math>\alpha</math> is
:<math>\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)(1+x)^{\alpha-n}</math>.
By Taylor series, we get a generalized version of the binomial theorem known as [http://en.wikipedia.org/wiki/Binomial_coefficient#Newton.27s_binomial_series '''Newton's formula''']:
{{Theorem|Newton's formular (generalized binomial theorem)|
If <math>|x|<1</math>, then
:<math>(1+x)^\alpha=\sum_{n\ge 0}{\alpha\choose n}x^{n}</math>,
where <math>{\alpha\choose n}</math> is the '''generalized binomial coefficient''' defined by
:<math>{\alpha\choose n}=\frac{\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)}{n!}</math>.
}}

=== Example: multisets ===
In the last lecture we gave a combinatorial proof of the number of <math>k</math>-multisets on an <math>n</math>-set. Now we give a generating function approach to the problem.

Let <math>S=\{x_1,x_2,\ldots,x_n\}</math> be an <math>n</math>-element set. We have
:<math>(1+x_1+x_1^2+\cdots)(1+x_2+x_2^2+\cdots)\cdots(1+x_n+x_n^2+\cdots)=\sum_{m:S\rightarrow\mathbb{N}} \prod_{x_i\in S}x_i^{m(x_i)}</math>,
where each <math>m:S\rightarrow\mathbb{N}</math> species a possible multiset on <math>S</math> with multiplicity function <math>m</math>.

Let all <math>x_i=x</math>. Then
:<math>
\begin{align}
(1+x+x^2+\cdots)^n
&=
\sum_{m:S\rightarrow\mathbb{N}}x^{m(x_1)+\cdots+m(x_n)}\\
&=
\sum_{\text{multiset }M\text{ on }S}x^{|M|}\\
&=
\sum_{k\ge 0}\left({n\choose k}\right)x^k.
\end{align}
</math>
The last equation is due to the the definition of <math>\left({n\choose k}\right)</math>. Our task is to evaluate <math>\left({n\choose k}\right)</math>.

Due to the geometric sequence and the Newton's formula
:<math>
(1+x+x^2+\cdots)^n=(1-x)^{-n}=\sum_{k\ge 0}{-n\choose k}(-x)^k.
</math>
So
:<math>
\left({n\choose k}\right)=(-1)^k{-n\choose k}={n+k-1\choose k}.
</math>
The last equation is due to the definition of the generalized binomial coefficient. We use an analytic (generating function) proof to get the same result of <math>\left({n\choose k}\right)</math> as the combinatorial proof.

=== Example: Quicksort ===

== Catalan Number ==
We now introduce a class of counting problems, all with the same solution, called [http://en.wikipedia.org/wiki/Catalan_number '''Catalan number'''].

The <math>n</math>th Catalan number is denoted as <math>C_n</math>.
In Volume 2 of Stanley's ''Enumerative Combinatorics'', a set of exercises describe 66 different interpretations of the Catalan numbers. We give a few examples, cited from Wikipedia.
* ''C''<sub>''n''</sub> is the number of '''Dyck words''' of length 2''n''. A Dyck word is a string consisting of ''n'' X's and ''n'' Y's such that no initial segment of the string has more Y's than X's (see also [http://en.wikipedia.org/wiki/Dyck_language Dyck language]). For example, the following are the Dyck words of length 6:
<div class="center"><big> XXXYYY     XYXXYY     XYXYXY     XXYYXY     XXYXYY.</big></div>

* Re-interpreting the symbol X as an open parenthesis and Y as a close parenthesis, ''C''<sub>''n''</sub> counts the number of expressions containing ''n'' pairs of parentheses which are correctly matched:
<div class="center"><big> ((()))     ()(())     ()()()     (())()     (()()) </big></div>

* ''C''<sub>''n''</sub> is the number of different ways ''n'' + 1 factors can be completely parenthesized (or the number of ways of associating ''n'' applications of a '''binary operator'''). For ''n'' = 3, for example, we have the following five different parenthesizations of four factors:
<div class="center"><math>((ab)c)d \quad (a(bc))d \quad(ab)(cd) \quad a((bc)d) \quad a(b(cd))</math></div>

* Successive applications of a binary operator can be represented in terms of a '''full binary tree'''. (A rooted binary tree is ''full'' if every vertex has either two children or no children.) It follows that ''C''<sub>''n''</sub> is the number of full binary trees with ''n'' + 1 leaves:
[[Image:Catalan number binary tree example.png|center]]

* ''C''<sub>''n''</sub> is the number of '''monotonic paths''' along the edges of a grid with ''n'' × ''n'' square cells, which do not pass above the diagonal. A monotonic path is one which starts in the lower left corner, finishes in the upper right corner, and consists entirely of edges pointing rightwards or upwards. Counting such paths is equivalent to counting Dyck words: X stands for "move right" and Y stands for "move up". The following diagrams show the case ''n'' = 4:
[[Image:Catalan number 4x4 grid example.svg.png|450px|center]]

* ''C''<sub>''n''</sub> is the number of different ways a [http://en.wikipedia.org/wiki/Convex_polygon '''convex polygon'''] with ''n'' + 2 sides can be cut into '''triangles''' by connecting vertices with straight lines. The following hexagons illustrate the case ''n'' = 4:
[[Image:Catalan-Hexagons-example.png|400px|center]]

* ''C''<sub>''n''</sub> is the number of [http://en.wikipedia.org/wiki/Stack_(data_structure) '''stack''']-sortable permutations of {1, ..., ''n''}. A permutation ''w'' is called '''stack-sortable''' if ''S''(''w'') = (1, ..., ''n''), where ''S''(''w'') is defined recursively as follows: write ''w'' = ''unv'' where ''n'' is the largest element in ''w'' and ''u'' and ''v'' are shorter sequences, and set ''S''(''w'') = ''S''(''u'')''S''(''v'')''n'', with ''S'' being the identity for one-element sequences.

* ''C''<sub>''n''</sub> is the number of ways to tile a stairstep shape of height ''n'' with ''n'' rectangles. The following figure illustrates the case ''n'' = 4:
[[Image:Catalan stairsteps 4.png|400px|center]]

{{Theorem|Recurrence relation for Catalan numbers|
:<math>C_0=0</math>, <math>C_1=1</math>, and for <math>n>1</math>,
::<math>
C_n=\sum_{i=1}^{n-1}C_iC_{n-i}.
</math>
}}

Let <math>G(x)=\sum_{n\ge 0}C_nx^n</math> be the generating function. Apply the product rule,
:<math>G(x)^2=\sum_{n\ge 0}\sum_{k=0}^{n}C_kC_{n-k}x^n=\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n</math>.
Due to the recurrence,
:<math>G(x)=\sum_{n\ge 0}C_nx^n=x+\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n=x+G(x)^2</math>.
Solving this, we obtain
:<math>G(x)=\frac{1\pm(1-4x)^{1/2}}{2}</math>.
Because <math>C_0=0</math>, it must hold that <math>G(x)=\frac{1-(1-4x)^{1/2}}{2}</math>, or otherwise the constant term is not zero. Expanding <math>(1-4x)^{1/2}</math> by Newton's formula, we have
:<math>
\begin{align}
G(x)
&=
\frac{1-(1-4x)^{1/2}}{2}\\
&=
1-\frac{1}{2}\sum_{n\ge 0}{1/2\choose n}(-4x)^n
\end{align}
</math>
Thus,
:<math>
\begin{align}
C_n
&=-\frac{1}{2}{1/2\choose n}(-4)^n\\
&=-\frac{1}{2}\cdot\frac{1}{2}\cdot\frac{-1}{2}\cdot\frac{-3}{2}\cdots\frac{-(2n-3)}{2}\cdot(-4)^n/n!\\
&=\frac{(2n-2)!}{(n-1)!n!}\\
&=\frac{1}{n}{2n-2\choose n-1}.
\end{align}
</math>
So we prove the following closed form for Catalan number.
{{Theorem|Theorem|
:<math>C_n=\frac{1}{n}{2n-2\choose n-1}</math>.
}}

== Reference ==
* ''Graham, Knuth, and Patashnik'', Concrete Mathematics: A Foundation for Computer Science, Chapter 7.
* ''Cameron'', Combinatorics: Topics, Techniques, Algorithms, Chapter 4.
* ''van Lin and Wilson'', A course in combinatorics, Chapter 14.

Combinatorics (Fall 2010)/Generating functions

2010-09-12T06:32:05Z

172.21.1.108:

== Generating Functions ==
In Stanley's magnificent book ''Enumerative Combinatorics'', he comments the generating function as "the most useful but most difficult to understand method (for counting)".

The solution to a counting problem is usually represented as some <math>a_n</math> depending a parameter <math>n</math>. Sometimes this <math>a_n</math> is called a ''counting function'' as it is a function of the parameter <math>n</math>. <math>a_n</math> can also be treated as a infinite series:
:<math>a_0,a_1,a_2,\ldots</math>

The '''ordinary generating function (OGF)''' defined by <math>a_n</math> is
:<math>
G(x)=\sum_{n\ge 0} a_nx^n.
</math>

So <math>G(x)=a_0+a_1x+a_2x^2+\cdots</math>. An expression in this form is called a [http://en.wikipedia.org/wiki/Formal_power_series '''formal power series'''], and <math>a_0,a_1,a_2,\ldots</math> is the sequence of '''coefficients'''.

Furthermore, the generating function can be expanded as
:G(x)=<math>(\underbrace{1+\cdots+1}_{a_0})+(\underbrace{x+\cdots+x}_{a_1})+(\underbrace{x^2+\cdots+x^2}_{a_2})+\cdots+(\underbrace{x^n+\cdots+x^n}_{a_n})+\cdots</math>
so it indeed "generates" all the possible instances of the objects we want to count.

Usually, we do not evaluate the generating function <math>GF(x)</math> on any particular value. <math>x</math> remains as a '''formal variable''' without assuming any value. The numbers that we want to count are the coefficients carried by the terms in the formal power series. So far the generating function is just another way to represent the sequence
:<math>(a_0,a_1,a_2,\ldots\ldots)</math>.

The true power of generating functions comes from the various algebraic operations that we can perform on these generating functions. We use an example to demonstrate this.

=== Combinations ===
Suppose we wish to enumerate all subsets of an <math>n</math>-set. To construct a subset, we specifies for every element of the <math>n</math>-set whether the element is chosen or not. Let us denote the choice to omit an element by <math>x_0</math>, and the choice to include it by <math>x_1</math>. Using "<math>+</math>" to represent "OR", and using the multiplication to denote "AND", the choices of subsets of the <math>n</math>-set are expressed as
:<math>\underbrace{(x_0+x_1)(x_0+x_1)\cdots (x_0+x_1)}_{n\mbox{ elements}}=(x_0+x_1)^n</math>.

For example, when <math>n=3</math>, we have
:<math>\begin{align}
(x_0+x_1)^3
&=x_0x_0x_0+x_0x_0x_1+x_0x_1x_0+x_0x_1x_1\\
&\quad +x_1x_0x_0+x_1x_0x_1+x_1x_1x_0+x_1x_1x_1
\end{align}</math>.

So it "generate" all subsets of the 3-set. Writing <math>1</math> for <math>x_0</math> and <math>x</math> for <math>x_1</math>, we have <math>(1+x)^3=1+3x+3x^2+x^3</math>. The coefficient of <math>x^k</math> is the number of <math>k</math>-subsets of a 3-element set.

In general, <math>(1+x)^n</math> has the coefficients which are the number of subsets of fixed sizes of an <math>n</math>-element set.

-----

Suppose that we have twelve balls: <font color="red">3 red</font>, <font color="blue">4 blue</font>, and <font color="green">5 green</font>. Balls with the same color are indistinguishable.

We want to determine the number of ways to select <math>k</math> balls from these twelve balls, for some <math>0\le k\le 12</math>.

The generating function of this sequence is
:<math>\begin{align}
&\quad {\color{Red}(1+x+x^2+x^3)}{\color{Blue}(1+x+x^2+x^3+x^4)}{\color{OliveGreen}(1+x+x^2+x^3+x^4+x^5)}\\
&=1+3x+6x^2+10x^3+14x^4+17x^5+18x^6+17x^7+14x^8+10x^9+6x^{10}+3x^{11}+x^{12}.
\end{align}</math>
The coefficient of <math>x^k</math> gives the number of ways to select <math>k</math> balls.

=== Fibonacci numbers ===
Consider the following counting problems.
* Count the number of ways that the nonnegative integer <math>n</math> can be written as a sum of ones and twos (in order).
: The problem asks for the number of compositions of <math>n</math> with summands from <math>\{1,2\}</math>. Formally, we are counting the number of tuples <math>(x_1,x_2,\ldots,x_k)</math> for some <math>k\le n</math> such that <math>x_i\in\{1,2\}</math> and <math>x_1+x_2+\cdots+x_k=n</math>.
: Let <math>F_n</math> be the solution. We observe that a composition either starts with a 1, in which case the rest is a composition of <math>n-1</math>; or starts with a 2, in which case the rest is a composition of <math>n-2</math>. So we have the recursion for <math>F_n</math> that
::<math>F_n=F_{n-1}+F_{n-2}</math>.
* Count the ways to completely cover a <math>2\times n</math> rectangle with <math>2\times 1</math> dominos without any overlaps.
: Dominos are identical <math>2\times 1</math> rectangles, so that only their orientations --- vertical or horizontal matter.
: Let <math>F_n</math> be the solution. It also holds that <math>F_n=F_{n-1}+F_{n-2}</math>. The proof is left as an exercise.

In both problems, the solution is given by <math>F_n</math> which satisfies the following recursion.
:<math>F_n=\begin{cases}
F_{n-1}+F_{n-2} & \mbox{if }n\ge 2,\\
1 & \mbox{if }n=1\\
0 & \mbox{if }n=0.
\end{cases}</math>

<math>F_n</math> is called the [http://en.wikipedia.org/wiki/Fibonacci_number Fibonacci number].

{{Theorem|Theorem|
::<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)</math>,
:where <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>.
}}
The quantity <math>\phi=\frac{1+\sqrt{5}}{2}</math> is the so-called [http://en.wikipedia.org/wiki/Golden_ratio golden ratio], a constant with some significance in mathematics and aesthetics.

We now prove this theorem by using generating functions.
The ordinary generating function for the Fibonacci number <math>F_{n}</math> is
:<math>G(x)=\sum_{n\ge 0}F_n x^n</math>.
We have that <math>F_{n}=F_{n-1}+F_{n-2}</math> for <math>n\ge 2</math>, thus
:<math>\begin{align}
G(x)
&=
\sum_{n\ge 0}F_n x^n
&=
x+\sum_{n\ge 2}(F_{n-1}+F_{n-2})x^n.
\end{align}
</math>
For generating functions, there are general ways to generate <math>F_{n-1}</math> and <math>F_{n-2}</math>, or the coefficients with any smaller indices.
:<math>
\begin{align}
xG(x)
&=\sum_{n\ge 0}F_n x^{n+1}=\sum_{n\ge 1}F_{n-1} x^n=\sum_{n\ge 2}F_{n-1} x^n\\
x^2G(x)
&=\sum_{n\ge 0}F_n x^{n+2}=\sum_{n\ge 2}F_{n-2} x^n.
\end{align}
</math>
So we have
:<math>G(x)=x+(x+x^2)G(x)\,</math>,
hence
:<math>G(x)=\frac{x}{1-x-x^2}</math>.
The value of <math>F_n</math> is the coefficient of <math>x^n</math> in the Taylor series for this formular, which is <math>\frac{G^{(n)}(0)}{n!}=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>. Although this expansion works in principle, the detailed calculus is rather painful.

----
It is easier to expand the generating function by breaking it into two geometric series.
{{Theorem|Proposition|
:Let <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>. It holds that
::<math>\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
}}

It is easy to verify the above equation, but to deduce it, we need some (high school) calculation.

{|border="2" width="100%" cellspacing="4" cellpadding="3" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse;empty-cells:show;"
|
:{|
|
<math>1-x-x^2</math> has two roots <math>\frac{-1\pm\sqrt{5}}{2}</math>.

Denote that <math>\phi=\frac{2}{-1+\sqrt{5}}=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{2}{-1-\sqrt{5}}=\frac{1-\sqrt{5}}{2}</math>.

Then <math>(1-x-x^2)=(1-\phi x)(1-\hat{\phi}x)</math>, so we can write
:<math>
\begin{align}
\frac{x}{1-x-x^2}
&=\frac{x}{(1-\phi x)(1-\hat{\phi} x)}\\
&=\frac{\alpha}{(1-\phi x)}+\frac{\beta}{(1-\hat{\phi} x)},
\end{align}
</math>
where <math>\alpha</math> and <math>\beta</math> satisfying that
:<math>\begin{cases}
\alpha+\beta=0\\
\alpha\phi+\beta\hat{\phi}= -1.
\end{cases}</math>
Solving this we have that <math>\alpha=\frac{1}{\sqrt{5}}</math> and <math>\beta=-\frac{1}{\sqrt{5}}</math>. Thus,
:<math>G(x)=\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
|}
:<math>\square</math>
|}

Note that the expression <math>\frac{1}{1-z}</math> has a well known geometric expansion:
:<math>\frac{1}{1-z}=\sum_{n\ge 0}z^n</math>.

Therefore, <math>G(x)</math> can be expanded as
:<math>
\begin{align}
G(x)
&=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}\\
&=\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\phi x)^n-\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\hat{\phi} x)^n\\
&=\sum_{n\ge 0}\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)x^n.
\end{align}</math>
So the <math>n</math>th Fibonacci number is given by
:<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>.

== Solving recurrences ==
The following steps describe a general methodology of solving recurrences by generating functions.
:1. Give a recursion that computes <math>a_n</math>. In the case of Fibonacci sequence
::<math>a_n=a_{n-1}+a_{n-2}</math>.
:2. Multiply both sides of the equation by <math>x^n</math> and sum over all <math>n</math>. This gives the generating function
::<math>G(x)=\sum_{n\ge 0}a_nx^n=\sum_{n\ge 0}(a_{n-1}+a_{n-2})x^n</math>.
:: And manipulate the right hand side of the equation so that it becomes some other expression involving <math>G(x)</math>.
::<math>G(x)=x+(x+x^2)G(x)\,</math>.
:3. Solve the resulting equation to derive an explicit formula for <math>G(x)</math>.
::<math>G(x)=\frac{x}{1-x-x^2}</math>.
:4. Expand <math>G(x)</math> into a power series and read off the coefficient of <math>x^n</math>, which is a closed form for <math>a_n</math>.

The first step is usually established by combinatorial observations, or explicitly given by the problem. The third step is trivial.

The second and the forth steps need some non-trivial analytic techniques.

=== Algebraic operations on generating functions ===
The second step in the above methodology is somehow tricky. It involves first applying the recurrence to the coefficients of <math>G(x)</math>, which is easy; and then manipulating the resulting formal power series to express it in terms of <math>G(x)</math>, which is more difficult (because it works backwards).

We can apply several natural algebraic operations on the formal power series.

{{Theorem|Generating function manipulation|
:Let <math>G(x)=\sum_{n\ge 0}g_nx^n</math> and <math>F(x)=\sum_{n\ge 0}f_nx^n</math>.
----

::<math>
\begin{align}
x^k G(x)
&= \sum_{n\ge k}g_{n-k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\frac{G(x)-\sum_{i=0}^{k-1}g_iz^i}{x^k}
&=\sum_{n\ge 0}g_{n+k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\alpha F(x)+\beta G(x)
&= \sum_{n\ge 0} (\alpha f_n+\beta g_n)x^n\\
F(x)G(x)
&= \sum_{n\ge 0}\sum_{k=0}^nf_kg_{n-k}x^n\\
G(cx)
&= \sum_{n\ge 0} c^ng_n x^n\\
G'(x)
&=
\sum_{n\ge 0}(n+1)g_{n+1}x^n
\end{align}
</math>
}}

When manipulating generating functions, these rules are applied backwards; that is, from the right-hand-side to the left-hand-side.

=== Expanding generating functions ===
The last step of solving recurrences by generating function is expanding the closed form generating function <math>G(x)</math> to evaluate its <math>n</math>-th coefficient. In principle, we can always use the [http://en.wikipedia.org/wiki/Taylor_series Taylor series]
:<math>G(x)=\sum_{n\ge 0}\frac{G^{(n)}(0)}{n!}x^n</math>,
where <math>G^{(n)}(0)</math> is the value of the <math>n</math>-th derivative of <math>G(x)</math> evaluated at <math>x=0</math>.

Some interesting special cases are very useful.

====Geometric sequence====
In the example of Fibonacci numbers, we use the well known geometric series:
:<math>\frac{1}{1-x}=\sum_{n\ge 0}x^n</math>.
It is useful when we can express the generating function in the form of <math>G(x)=\frac{a_1}{1-b_1x}+\frac{a_2}{1-b_2x}+\cdots+\frac{a_k}{1-b_kx}</math>. The coefficient of <math>x^n</math> in such <math>G(x)</math> is <math>a_1b_1^n+a_2b_2^n+\cdots+a_kb_k^n</math>.

====Binomial theorem====
The <math>n</math>-th derivative of <math>(1+x)^\alpha</math> for some real <math>\alpha</math> is
:<math>\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)(1+x)^{\alpha-n}</math>.
By Taylor series, we get a generalized version of the binomial theorem known as [http://en.wikipedia.org/wiki/Binomial_coefficient#Newton.27s_binomial_series '''Newton's formula''']:
{{Theorem|Newton's formular (generalized binomial theorem)|
If <math>|x|<1</math>, then
:<math>(1+x)^\alpha=\sum_{n\ge 0}{\alpha\choose n}x^{n}</math>,
where <math>{\alpha\choose n}</math> is the '''generalized binomial coefficient''' defined by
:<math>{\alpha\choose n}=\frac{\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)}{n!}</math>.
}}

=== Example: multisets ===
In the last lecture we gave a combinatorial proof of the number of <math>k</math>-multisets on an <math>n</math>-set. Now we give a generating function approach to the problem.

Let <math>S=\{x_1,x_2,\ldots,x_n\}</math> be an <math>n</math>-element set. We have
:<math>(1+x_1+x_1^2+\cdots)(1+x_2+x_2^2+\cdots)\cdots(1+x_n+x_n^2+\cdots)=\sum_{m:S\rightarrow\mathbb{N}} \prod_{x_i\in S}x_i^{m(x_i)}</math>,
where each <math>m:S\rightarrow\mathbb{N}</math> species a possible multiset on <math>S</math> with multiplicity function <math>m</math>.

Let all <math>x_i=x</math>. Then
:<math>
\begin{align}
(1+x+x^2+\cdots)^n
&=
\sum_{m:S\rightarrow\mathbb{N}}x^{m(x_1)+\cdots+m(x_n)}\\
&=
\sum_{\text{multiset }M\text{ on }S}x^{|M|}\\
&=
\sum_{k\ge 0}\left({n\choose k}\right)x^k.
\end{align}
</math>
The last equation is due to the the definition of <math>\left({n\choose k}\right)</math>. Our task is to evaluate <math>\left({n\choose k}\right)</math>.

Due to the geometric sequence and the Newton's formula
:<math>
(1+x+x^2+\cdots)^n=(1-x)^{-n}=\sum_{k\ge 0}{-n\choose k}(-x)^k.
</math>
So
:<math>
\left({n\choose k}\right)=(-1)^k{-n\choose k}={n+k-1\choose k}.
</math>
The last equation is due to the definition of the generalized binomial coefficient. We use an analytic (generating function) proof to get the same result of <math>\left({n\choose k}\right)</math> as the combinatorial proof.

=== Example: Quicksort ===

== Catalan Number ==
We now introduce a class of counting problems, all with the same solution, called [http://en.wikipedia.org/wiki/Catalan_number '''Catalan number'''].

The <math>n</math>th Catalan number is denoted as <math>C_n</math>.
In Volume 2 of Stanley's ''Enumerative Combinatorics'', a set of exercises describe 66 different interpretations of the Catalan numbers. We give a few examples, cited from Wikipedia.
* ''C''<sub>''n''</sub> is the number of '''Dyck words''' of length 2''n''. A Dyck word is a string consisting of ''n'' X's and ''n'' Y's such that no initial segment of the string has more Y's than X's (see also [http://en.wikipedia.org/wiki/Dyck_language Dyck language]). For example, the following are the Dyck words of length 6:
<div class="center"><big> XXXYYY     XYXXYY     XYXYXY     XXYYXY     XXYXYY.</big></div>

* Re-interpreting the symbol X as an open parenthesis and Y as a close parenthesis, ''C''<sub>''n''</sub> counts the number of expressions containing ''n'' pairs of parentheses which are correctly matched:
<div class="center"><big> ((()))     ()(())     ()()()     (())()     (()()) </big></div>

* ''C''<sub>''n''</sub> is the number of different ways ''n'' + 1 factors can be completely parenthesized (or the number of ways of associating ''n'' applications of a '''binary operator'''). For ''n'' = 3, for example, we have the following five different parenthesizations of four factors:
<div class="center"><math>((ab)c)d \quad (a(bc))d \quad(ab)(cd) \quad a((bc)d) \quad a(b(cd))</math></div>

* Successive applications of a binary operator can be represented in terms of a '''full binary tree'''. (A rooted binary tree is ''full'' if every vertex has either two children or no children.) It follows that ''C''<sub>''n''</sub> is the number of full binary trees with ''n'' + 1 leaves:
[[Image:Catalan number binary tree example.png|center]]

* ''C''<sub>''n''</sub> is the number of '''monotonic paths''' along the edges of a grid with ''n'' × ''n'' square cells, which do not pass above the diagonal. A monotonic path is one which starts in the lower left corner, finishes in the upper right corner, and consists entirely of edges pointing rightwards or upwards. Counting such paths is equivalent to counting Dyck words: X stands for "move right" and Y stands for "move up". The following diagrams show the case ''n'' = 4:
[[Image:Catalan number 4x4 grid example.svg.png|450px|center]]

* ''C''<sub>''n''</sub> is the number of different ways a [http://en.wikipedia.org/wiki/Convex_polygon '''convex polygon'''] with ''n'' + 2 sides can be cut into '''triangles''' by connecting vertices with straight lines. The following hexagons illustrate the case ''n'' = 4:
[[Image:Catalan-Hexagons-example.png|400px|center]]

* ''C''<sub>''n''</sub> is the number of [http://en.wikipedia.org/wiki/Stack_(data_structure) '''stack''']-sortable permutations of {1, ..., ''n''}. A permutation ''w'' is called '''stack-sortable''' if ''S''(''w'') = (1, ..., ''n''), where ''S''(''w'') is defined recursively as follows: write ''w'' = ''unv'' where ''n'' is the largest element in ''w'' and ''u'' and ''v'' are shorter sequences, and set ''S''(''w'') = ''S''(''u'')''S''(''v'')''n'', with ''S'' being the identity for one-element sequences.

* ''C''<sub>''n''</sub> is the number of ways to tile a stairstep shape of height ''n'' with ''n'' rectangles. The following figure illustrates the case ''n'' = 4:
[[Image:Catalan stairsteps 4.png|400px|center]]

{{Theorem|Recurrence relation for Catalan numbers|
:<math>C_0=0</math>, <math>C_1=1</math>, and for <math>n>1</math>,
::<math>
C_n=\sum_{i=1}^{n-1}C_iC_{n-i}.
</math>
}}

Let <math>G(x)=\sum_{n\ge 0}C_nx^n</math> be the generating function. Apply the product rule,
:<math>G(x)^2=\sum_{n\ge 0}\sum_{k=0}^{n}C_kC_{n-k}x^n=\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n</math>.
Due to the recurrence,
:<math>G(x)=\sum_{n\ge 0}C_nx^n=x+\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n=x+G(x)^2</math>.
Solving this, we obtain
:<math>G(x)=\frac{1\pm(1-4x)^{1/2}}{2}</math>.
Because <math>C_0=0</math>, it must hold that <math>G(x)=\frac{1-(1-4x)^{1/2}}{2}</math>, or otherwise the constant term is not zero. Expanding <math>(1-4x)^{1/2}</math> by Newton's formula, we have
:<math>
\begin{align}
G(x)
&=
\frac{1-(1-4x)^{1/2}}{2}\\
&=
1-\frac{1}{2}\sum_{n\ge 0}{1/2\choose n}(-4x)^n
\end{align}
</math>
Thus,
:<math>
\begin{align}
C_n
&=-\frac{1}{2}{1/2\choose n}(-4)^n\\
&=-\frac{1}{2}\cdot\frac{1}{2}\cdot\frac{-1}{2}\cdot\frac{-3}{2}\cdots\frac{-(2n-3)}{2}\cdot(-4)^n/n!\\
&=\frac{(2n-2)!}{(n-1)!n!}\\
&=\frac{1}{n}{2n-2\choose n-1}.
\end{align}
</math>
So we prove the following closed form for Catalan number.
{{Theorem|Theorem|
:<math>C_n=\frac{1}{n}{2n-2\choose n-1}</math>.
}}

== Reference ==
* ''Graham, Knuth, and Patashnik'', Concrete Mathematics: A Foundation for Computer Science, Chapter 7.
* ''Cameron'', Combinatorics: Topics, Techniques, Algorithms, Chapter 4.
* "van Lin and Wilson", A course in combinatorics, Chapter 14.

Combinatorics (Fall 2010)/Generating functions

2010-09-12T06:24:37Z

172.21.1.108: /* Combinations */

Combinatorics (Fall 2010)/Generating functions

2010-09-12T06:19:53Z

172.21.1.108: /* Combinations */

== Generating Functions ==
In Stanley's magnificent book ''Enumerative Combinatorics'', he comments the generating function as "the most useful but most difficult to understand method (for counting)".

The solution to a counting problem is usually represented as some <math>a_n</math> depending a parameter <math>n</math>. Sometimes this <math>a_n</math> is called a ''counting function'' as it is a function of the parameter <math>n</math>. <math>a_n</math> can also be treated as a infinite series:
:<math>a_0,a_1,a_2,\ldots</math>

The '''ordinary generating function (OGF)''' defined by <math>a_n</math> is
:<math>
G(x)=\sum_{n\ge 0} a_nx^n.
</math>

So <math>G(x)=a_0+a_1x+a_2x^2+\cdots</math>. An expression in this form is called a [http://en.wikipedia.org/wiki/Formal_power_series '''formal power series'''], and <math>a_0,a_1,a_2,\ldots</math> is the sequence of '''coefficients'''.

Furthermore, the generating function can be expanded as
:G(x)=<math>(\underbrace{1+\cdots+1}_{a_0})+(\underbrace{x+\cdots+x}_{a_1})+(\underbrace{x^2+\cdots+x^2}_{a_2})+\cdots+(\underbrace{x^n+\cdots+x^n}_{a_n})+\cdots</math>
so it indeed "generates" all the possible instances of the objects we want to count.

Usually, we do not evaluate the generating function <math>GF(x)</math> on any particular value. <math>x</math> remains as a '''formal variable''' without assuming any value. The numbers that we want to count are the coefficients carried by the terms in the formal power series. So far the generating function is just another way to represent the sequence
:<math>(a_0,a_1,a_2,\ldots\ldots)</math>.

The true power of generating functions comes from the various algebraic operations that we can perform on these generating functions. We use an example to demonstrate this.

=== Combinations ===
Suppose we wish to enumerate all subsets of an <math>n</math>-set. To construct a subset, we specifies for every element of the <math>n</math>-set whether the element is chosen or not. Let us denote the choice to omit an element by <math>x_0</math>, and the choice to include it by <math>x_1</math>. Using "<math>+</math>" to represent "OR", and using the multiplication to denote "AND", the choices of subsets of the <math>n</math>-set are expressed as
:<math>\underbrace{(x_0+x_1)(x_0+x_1)\cdots (x_0+x_1)}_{n\mbox{ elements}}=(x_0+x_1)^n</math>.

For example, when <math>n=3</math>, we have
:<math>\begin{align}
(x_0+x_1)^3
&=x_0x_0x_0+x_0x_0x_1+x_0x_1x_0+x_0x_1x_1\\
&\quad +x_1x_0x_0+x_1x_0x_1+x_1x_1x_0+x_1x_1x_1
\end{align}</math>.

So it "generate" all subsets of the 3-set. Writing <math>1</math> for <math>x_0</math> and <math>x</math> for <math>x_1</math>, we have <math>(1+x)^3=1+3x+3x^2+x^3</math>. The coefficient of <math>x^k</math> is the number of <math>k</math>-subsets of a 3-element set.

In general, <math>(1+x)^n</math> has the coefficients which are the number of subsets of fixed sizes of an <math>n</math>-element set.

-----

Suppose that we have twelve balls: <font color="red">3 red</font>, <font color="blue">4 blue</font>, and <font color="green">5 green</font>. Balls with the same color are indistinguishable.

We want to determine the number of ways to select <math>k</math> balls from these twelve balls, for some <math>0\le k\le 12</math>.

:<math>{\color{Red}(1+x+x^2+x^3)}{\color{Blue}(1+x+x^2+x^3+x^4)}{\color{OliveGreen}(1+x+x^2+x^3+x^4+x^5)}</math>

=== Fibonacci numbers ===
Consider the following counting problems.
* Count the number of ways that the nonnegative integer <math>n</math> can be written as a sum of ones and twos (in order).
: The problem asks for the number of compositions of <math>n</math> with summands from <math>\{1,2\}</math>. Formally, we are counting the number of tuples <math>(x_1,x_2,\ldots,x_k)</math> for some <math>k\le n</math> such that <math>x_i\in\{1,2\}</math> and <math>x_1+x_2+\cdots+x_k=n</math>.
: Let <math>F_n</math> be the solution. We observe that a composition either starts with a 1, in which case the rest is a composition of <math>n-1</math>; or starts with a 2, in which case the rest is a composition of <math>n-2</math>. So we have the recursion for <math>F_n</math> that
::<math>F_n=F_{n-1}+F_{n-2}</math>.
* Count the ways to completely cover a <math>2\times n</math> rectangle with <math>2\times 1</math> dominos without any overlaps.
: Dominos are identical <math>2\times 1</math> rectangles, so that only their orientations --- vertical or horizontal matter.
: Let <math>F_n</math> be the solution. It also holds that <math>F_n=F_{n-1}+F_{n-2}</math>. The proof is left as an exercise.

In both problems, the solution is given by <math>F_n</math> which satisfies the following recursion.
:<math>F_n=\begin{cases}
F_{n-1}+F_{n-2} & \mbox{if }n\ge 2,\\
1 & \mbox{if }n=1\\
0 & \mbox{if }n=0.
\end{cases}</math>

<math>F_n</math> is called the [http://en.wikipedia.org/wiki/Fibonacci_number Fibonacci number].

{{Theorem|Theorem|
::<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)</math>,
:where <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>.
}}
The quantity <math>\phi=\frac{1+\sqrt{5}}{2}</math> is the so-called [http://en.wikipedia.org/wiki/Golden_ratio golden ratio], a constant with some significance in mathematics and aesthetics.

We now prove this theorem by using generating functions.
The ordinary generating function for the Fibonacci number <math>F_{n}</math> is
:<math>G(x)=\sum_{n\ge 0}F_n x^n</math>.
We have that <math>F_{n}=F_{n-1}+F_{n-2}</math> for <math>n\ge 2</math>, thus
:<math>\begin{align}
G(x)
&=
\sum_{n\ge 0}F_n x^n
&=
x+\sum_{n\ge 2}(F_{n-1}+F_{n-2})x^n.
\end{align}
</math>
For generating functions, there are general ways to generate <math>F_{n-1}</math> and <math>F_{n-2}</math>, or the coefficients with any smaller indices.
:<math>
\begin{align}
xG(x)
&=\sum_{n\ge 0}F_n x^{n+1}=\sum_{n\ge 1}F_{n-1} x^n=\sum_{n\ge 2}F_{n-1} x^n\\
x^2G(x)
&=\sum_{n\ge 0}F_n x^{n+2}=\sum_{n\ge 2}F_{n-2} x^n.
\end{align}
</math>
So we have
:<math>G(x)=x+(x+x^2)G(x)\,</math>,
hence
:<math>G(x)=\frac{x}{1-x-x^2}</math>.
The value of <math>F_n</math> is the coefficient of <math>x^n</math> in the Taylor series for this formular, which is <math>\frac{G^{(n)}(0)}{n!}=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>. Although this expansion works in principle, the detailed calculus is rather painful.

----
It is easier to expand the generating function by breaking it into two geometric series.
{{Theorem|Proposition|
:Let <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>. It holds that
::<math>\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
}}

It is easy to verify the above equation, but to deduce it, we need some (high school) calculation.

{|border="2" width="100%" cellspacing="4" cellpadding="3" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse;empty-cells:show;"
|
:{|
|
<math>1-x-x^2</math> has two roots <math>\frac{-1\pm\sqrt{5}}{2}</math>.

Denote that <math>\phi=\frac{2}{-1+\sqrt{5}}=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{2}{-1-\sqrt{5}}=\frac{1-\sqrt{5}}{2}</math>.

Then <math>(1-x-x^2)=(1-\phi x)(1-\hat{\phi}x)</math>, so we can write
:<math>
\begin{align}
\frac{x}{1-x-x^2}
&=\frac{x}{(1-\phi x)(1-\hat{\phi} x)}\\
&=\frac{\alpha}{(1-\phi x)}+\frac{\beta}{(1-\hat{\phi} x)},
\end{align}
</math>
where <math>\alpha</math> and <math>\beta</math> satisfying that
:<math>\begin{cases}
\alpha+\beta=0\\
\alpha\phi+\beta\hat{\phi}= -1.
\end{cases}</math>
Solving this we have that <math>\alpha=\frac{1}{\sqrt{5}}</math> and <math>\beta=-\frac{1}{\sqrt{5}}</math>. Thus,
:<math>G(x)=\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
|}
:<math>\square</math>
|}

Note that the expression <math>\frac{1}{1-z}</math> has a well known geometric expansion:
:<math>\frac{1}{1-z}=\sum_{n\ge 0}z^n</math>.

Therefore, <math>G(x)</math> can be expanded as
:<math>
\begin{align}
G(x)
&=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}\\
&=\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\phi x)^n-\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\hat{\phi} x)^n\\
&=\sum_{n\ge 0}\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)x^n.
\end{align}</math>
So the <math>n</math>th Fibonacci number is given by
:<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>.

== Solving recurrences ==
The following steps describe a general methodology of solving recurrences by generating functions.
:1. Give a recursion that computes <math>a_n</math>. In the case of Fibonacci sequence
::<math>a_n=a_{n-1}+a_{n-2}</math>.
:2. Multiply both sides of the equation by <math>x^n</math> and sum over all <math>n</math>. This gives the generating function
::<math>G(x)=\sum_{n\ge 0}a_nx^n=\sum_{n\ge 0}(a_{n-1}+a_{n-2})x^n</math>.
:: And manipulate the right hand side of the equation so that it becomes some other expression involving <math>G(x)</math>.
::<math>G(x)=x+(x+x^2)G(x)\,</math>.
:3. Solve the resulting equation to derive an explicit formula for <math>G(x)</math>.
::<math>G(x)=\frac{x}{1-x-x^2}</math>.
:4. Expand <math>G(x)</math> into a power series and read off the coefficient of <math>x^n</math>, which is a closed form for <math>a_n</math>.

The first step is usually established by combinatorial observations, or explicitly given by the problem. The third step is trivial.

The second and the forth steps need some non-trivial analytic techniques.

=== Algebraic operations on generating functions ===
The second step in the above methodology is somehow tricky. It involves first applying the recurrence to the coefficients of <math>G(x)</math>, which is easy; and then manipulating the resulting formal power series to express it in terms of <math>G(x)</math>, which is more difficult (because it works backwards).

We can apply several natural algebraic operations on the formal power series.

{{Theorem|Generating function manipulation|
:Let <math>G(x)=\sum_{n\ge 0}g_nx^n</math> and <math>F(x)=\sum_{n\ge 0}f_nx^n</math>.
----

::<math>
\begin{align}
x^k G(x)
&= \sum_{n\ge k}g_{n-k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\frac{G(x)-\sum_{i=0}^{k-1}g_iz^i}{x^k}
&=\sum_{n\ge 0}g_{n+k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\alpha F(x)+\beta G(x)
&= \sum_{n\ge 0} (\alpha f_n+\beta g_n)x^n\\
F(x)G(x)
&= \sum_{n\ge 0}\sum_{k=0}^nf_kg_{n-k}x^n\\
G(cx)
&= \sum_{n\ge 0} c^ng_n x^n\\
G'(x)
&=
\sum_{n\ge 0}(n+1)g_{n+1}x^n
\end{align}
</math>
}}

When manipulating generating functions, these rules are applied backwards; that is, from the right-hand-side to the left-hand-side.

=== Expanding generating functions ===
The last step of solving recurrences by generating function is expanding the closed form generating function <math>G(x)</math> to evaluate its <math>n</math>-th coefficient. In principle, we can always use the [http://en.wikipedia.org/wiki/Taylor_series Taylor series]
:<math>G(x)=\sum_{n\ge 0}\frac{G^{(n)}(0)}{n!}x^n</math>,
where <math>G^{(n)}(0)</math> is the value of the <math>n</math>-th derivative of <math>G(x)</math> evaluated at <math>x=0</math>.

Some interesting special cases are very useful.

====Geometric sequence====
In the example of Fibonacci numbers, we use the well known geometric series:
:<math>\frac{1}{1-x}=\sum_{n\ge 0}x^n</math>.
It is useful when we can express the generating function in the form of <math>G(x)=\frac{a_1}{1-b_1x}+\frac{a_2}{1-b_2x}+\cdots+\frac{a_k}{1-b_kx}</math>. The coefficient of <math>x^n</math> in such <math>G(x)</math> is <math>a_1b_1^n+a_2b_2^n+\cdots+a_kb_k^n</math>.

====Binomial theorem====
The <math>n</math>-th derivative of <math>(1+x)^\alpha</math> for some real <math>\alpha</math> is
:<math>\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)(1+x)^{\alpha-n}</math>.
By Taylor series, we get a generalized version of the binomial theorem known as [http://en.wikipedia.org/wiki/Binomial_coefficient#Newton.27s_binomial_series '''Newton's formula''']:
{{Theorem|Newton's formular (generalized binomial theorem)|
If <math>|x|<1</math>, then
:<math>(1+x)^\alpha=\sum_{n\ge 0}{\alpha\choose n}x^{n}</math>,
where <math>{\alpha\choose n}</math> is the '''generalized binomial coefficient''' defined by
:<math>{\alpha\choose n}=\frac{\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)}{n!}</math>.
}}

=== Example: multisets ===
In the last lecture we gave a combinatorial proof of the number of <math>k</math>-multisets on an <math>n</math>-set. Now we give a generating function approach to the problem.

Let <math>S=\{x_1,x_2,\ldots,x_n\}</math> be an <math>n</math>-element set. We have
:<math>(1+x_1+x_1^2+\cdots)(1+x_2+x_2^2+\cdots)\cdots(1+x_n+x_n^2+\cdots)=\sum_{m:S\rightarrow\mathbb{N}} \prod_{x_i\in S}x_i^{m(x_i)}</math>,
where each <math>m:S\rightarrow\mathbb{N}</math> species a possible multiset on <math>S</math> with multiplicity function <math>m</math>.

Let all <math>x_i=x</math>. Then
:<math>
\begin{align}
(1+x+x^2+\cdots)^n
&=
\sum_{m:S\rightarrow\mathbb{N}}x^{m(x_1)+\cdots+m(x_n)}\\
&=
\sum_{\text{multiset }M\text{ on }S}x^{|M|}\\
&=
\sum_{k\ge 0}\left({n\choose k}\right)x^k.
\end{align}
</math>
The last equation is due to the the definition of <math>\left({n\choose k}\right)</math>. Our task is to evaluate <math>\left({n\choose k}\right)</math>.

Due to the geometric sequence and the Newton's formula
:<math>
(1+x+x^2+\cdots)^n=(1-x)^{-n}=\sum_{k\ge 0}{-n\choose k}(-x)^k.
</math>
So
:<math>
\left({n\choose k}\right)=(-1)^k{-n\choose k}={n+k-1\choose k}.
</math>
The last equation is due to the definition of the generalized binomial coefficient. We use an analytic (generating function) proof to get the same result of <math>\left({n\choose k}\right)</math> as the combinatorial proof.

=== Example: Quicksort ===

== Catalan Number ==
We now introduce a class of counting problems, all with the same solution, called [http://en.wikipedia.org/wiki/Catalan_number '''Catalan number'''].

The <math>n</math>th Catalan number is denoted as <math>C_n</math>.
In Volume 2 of Stanley's ''Enumerative Combinatorics'', a set of exercises describe 66 different interpretations of the Catalan numbers. We give a few examples, cited from Wikipedia.
* ''C''<sub>''n''</sub> is the number of '''Dyck words''' of length 2''n''. A Dyck word is a string consisting of ''n'' X's and ''n'' Y's such that no initial segment of the string has more Y's than X's (see also [http://en.wikipedia.org/wiki/Dyck_language Dyck language]). For example, the following are the Dyck words of length 6:
<div class="center"><big> XXXYYY     XYXXYY     XYXYXY     XXYYXY     XXYXYY.</big></div>

* Re-interpreting the symbol X as an open parenthesis and Y as a close parenthesis, ''C''<sub>''n''</sub> counts the number of expressions containing ''n'' pairs of parentheses which are correctly matched:
<div class="center"><big> ((()))     ()(())     ()()()     (())()     (()()) </big></div>

* ''C''<sub>''n''</sub> is the number of different ways ''n'' + 1 factors can be completely parenthesized (or the number of ways of associating ''n'' applications of a '''binary operator'''). For ''n'' = 3, for example, we have the following five different parenthesizations of four factors:
<div class="center"><math>((ab)c)d \quad (a(bc))d \quad(ab)(cd) \quad a((bc)d) \quad a(b(cd))</math></div>

* Successive applications of a binary operator can be represented in terms of a '''full binary tree'''. (A rooted binary tree is ''full'' if every vertex has either two children or no children.) It follows that ''C''<sub>''n''</sub> is the number of full binary trees with ''n'' + 1 leaves:
[[Image:Catalan number binary tree example.png|center]]

* ''C''<sub>''n''</sub> is the number of '''monotonic paths''' along the edges of a grid with ''n'' × ''n'' square cells, which do not pass above the diagonal. A monotonic path is one which starts in the lower left corner, finishes in the upper right corner, and consists entirely of edges pointing rightwards or upwards. Counting such paths is equivalent to counting Dyck words: X stands for "move right" and Y stands for "move up". The following diagrams show the case ''n'' = 4:
[[Image:Catalan number 4x4 grid example.svg.png|450px|center]]

* ''C''<sub>''n''</sub> is the number of different ways a [http://en.wikipedia.org/wiki/Convex_polygon '''convex polygon'''] with ''n'' + 2 sides can be cut into '''triangles''' by connecting vertices with straight lines. The following hexagons illustrate the case ''n'' = 4:
[[Image:Catalan-Hexagons-example.png|400px|center]]

* ''C''<sub>''n''</sub> is the number of [http://en.wikipedia.org/wiki/Stack_(data_structure) '''stack''']-sortable permutations of {1, ..., ''n''}. A permutation ''w'' is called '''stack-sortable''' if ''S''(''w'') = (1, ..., ''n''), where ''S''(''w'') is defined recursively as follows: write ''w'' = ''unv'' where ''n'' is the largest element in ''w'' and ''u'' and ''v'' are shorter sequences, and set ''S''(''w'') = ''S''(''u'')''S''(''v'')''n'', with ''S'' being the identity for one-element sequences.

* ''C''<sub>''n''</sub> is the number of ways to tile a stairstep shape of height ''n'' with ''n'' rectangles. The following figure illustrates the case ''n'' = 4:
[[Image:Catalan stairsteps 4.png|400px|center]]

{{Theorem|Recurrence relation for Catalan numbers|
:<math>C_0=0</math>, <math>C_1=1</math>, and for <math>n>1</math>,
::<math>
C_n=\sum_{i=1}^{n-1}C_iC_{n-i}.
</math>
}}

Let <math>G(x)=\sum_{n\ge 0}C_nx^n</math> be the generating function. Apply the product rule,
:<math>G(x)^2=\sum_{n\ge 0}\sum_{k=0}^{n}C_kC_{n-k}x^n=\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n</math>.
Due to the recurrence,
:<math>G(x)=\sum_{n\ge 0}C_nx^n=x+\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n=x+G(x)^2</math>.
Solving this, we obtain
:<math>G(x)=\frac{1\pm(1-4x)^{1/2}}{2}</math>.
Because <math>C_0=0</math>, it must hold that <math>G(x)=\frac{1-(1-4x)^{1/2}}{2}</math>, or otherwise the constant term is not zero. Expanding <math>(1-4x)^{1/2}</math> by Newton's formula, we have
:<math>
\begin{align}
G(x)
&=
\frac{1-(1-4x)^{1/2}}{2}\\
&=
1-\frac{1}{2}\sum_{n\ge 0}{1/2\choose n}(-4x)^n
\end{align}
</math>
Thus,
:<math>
\begin{align}
C_n
&=-\frac{1}{2}{1/2\choose n}(-4)^n\\
&=-\frac{1}{2}\cdot\frac{1}{2}\cdot\frac{-1}{2}\cdot\frac{-3}{2}\cdots\frac{-(2n-3)}{2}\cdot(-4)^n/n!\\
&=\frac{(2n-2)!}{(n-1)!n!}\\
&=\frac{1}{n}{2n-2\choose n-1}.
\end{align}
</math>
So we prove the following closed form for Catalan number.
{{Theorem|Theorem|
:<math>C_n=\frac{1}{n}{2n-2\choose n-1}</math>.
}}

Combinatorics (Fall 2010)/Generating functions

2010-09-12T02:52:32Z

172.21.1.108: /* Example: multisets */

== Generating Functions ==
In Stanley's magnificent book ''Enumerative Combinatorics'', he comments the generating function as "the most useful but most difficult to understand method (for counting)".

The solution to a counting problem is usually represented as some <math>a_n</math> depending a parameter <math>n</math>. Sometimes this <math>a_n</math> is called a ''counting function'' as it is a function of the parameter <math>n</math>. <math>a_n</math> can also be treated as a infinite series:
:<math>a_0,a_1,a_2,\ldots</math>

The '''ordinary generating function (OGF)''' defined by <math>a_n</math> is
:<math>
G(x)=\sum_{n\ge 0} a_nx^n.
</math>

So <math>G(x)=a_0+a_1x+a_2x^2+\cdots</math>. An expression in this form is called a [http://en.wikipedia.org/wiki/Formal_power_series '''formal power series'''], and <math>a_0,a_1,a_2,\ldots</math> is the sequence of '''coefficients'''.

Furthermore, the generating function can be expanded as
:G(x)=<math>(\underbrace{1+\cdots+1}_{a_0})+(\underbrace{x+\cdots+x}_{a_1})+(\underbrace{x^2+\cdots+x^2}_{a_2})+\cdots+(\underbrace{x^n+\cdots+x^n}_{a_n})+\cdots</math>
so it indeed "generates" all the possible instances of the objects we want to count.

Usually, we do not evaluate the generating function <math>GF(x)</math> on any particular value. <math>x</math> remains as a '''formal variable''' without assuming any value. The numbers that we want to count are the coefficients carried by the terms in the formal power series. So far the generating function is just another way to represent the sequence
:<math>(a_0,a_1,a_2,\ldots\ldots)</math>.

The true power of generating functions comes from the various algebraic operations that we can perform on these generating functions. We use an example to demonstrate this.

=== Combinations ===

=== Fibonacci numbers ===
Consider the following counting problems.
* Count the number of ways that the nonnegative integer <math>n</math> can be written as a sum of ones and twos (in order).
: The problem asks for the number of compositions of <math>n</math> with summands from <math>\{1,2\}</math>. Formally, we are counting the number of tuples <math>(x_1,x_2,\ldots,x_k)</math> for some <math>k\le n</math> such that <math>x_i\in\{1,2\}</math> and <math>x_1+x_2+\cdots+x_k=n</math>.
: Let <math>F_n</math> be the solution. We observe that a composition either starts with a 1, in which case the rest is a composition of <math>n-1</math>; or starts with a 2, in which case the rest is a composition of <math>n-2</math>. So we have the recursion for <math>F_n</math> that
::<math>F_n=F_{n-1}+F_{n-2}</math>.
* Count the ways to completely cover a <math>2\times n</math> rectangle with <math>2\times 1</math> dominos without any overlaps.
: Dominos are identical <math>2\times 1</math> rectangles, so that only their orientations --- vertical or horizontal matter.
: Let <math>F_n</math> be the solution. It also holds that <math>F_n=F_{n-1}+F_{n-2}</math>. The proof is left as an exercise.

In both problems, the solution is given by <math>F_n</math> which satisfies the following recursion.
:<math>F_n=\begin{cases}
F_{n-1}+F_{n-2} & \mbox{if }n\ge 2,\\
1 & \mbox{if }n=1\\
0 & \mbox{if }n=0.
\end{cases}</math>

<math>F_n</math> is called the [http://en.wikipedia.org/wiki/Fibonacci_number Fibonacci number].

{{Theorem|Theorem|
::<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)</math>,
:where <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>.
}}
The quantity <math>\phi=\frac{1+\sqrt{5}}{2}</math> is the so-called [http://en.wikipedia.org/wiki/Golden_ratio golden ratio], a constant with some significance in mathematics and aesthetics.

We now prove this theorem by using generating functions.
The ordinary generating function for the Fibonacci number <math>F_{n}</math> is
:<math>G(x)=\sum_{n\ge 0}F_n x^n</math>.
We have that <math>F_{n}=F_{n-1}+F_{n-2}</math> for <math>n\ge 2</math>, thus
:<math>\begin{align}
G(x)
&=
\sum_{n\ge 0}F_n x^n
&=
x+\sum_{n\ge 2}(F_{n-1}+F_{n-2})x^n.
\end{align}
</math>
For generating functions, there are general ways to generate <math>F_{n-1}</math> and <math>F_{n-2}</math>, or the coefficients with any smaller indices.
:<math>
\begin{align}
xG(x)
&=\sum_{n\ge 0}F_n x^{n+1}=\sum_{n\ge 1}F_{n-1} x^n=\sum_{n\ge 2}F_{n-1} x^n\\
x^2G(x)
&=\sum_{n\ge 0}F_n x^{n+2}=\sum_{n\ge 2}F_{n-2} x^n.
\end{align}
</math>
So we have
:<math>G(x)=x+(x+x^2)G(x)\,</math>,
hence
:<math>G(x)=\frac{x}{1-x-x^2}</math>.
The value of <math>F_n</math> is the coefficient of <math>x^n</math> in the Taylor series for this formular, which is <math>\frac{G^{(n)}(0)}{n!}=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>. Although this expansion works in principle, the detailed calculus is rather painful.

----
It is easier to expand the generating function by breaking it into two geometric series.
{{Theorem|Proposition|
:Let <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>. It holds that
::<math>\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
}}

It is easy to verify the above equation, but to deduce it, we need some (high school) calculation.

{|border="2" width="100%" cellspacing="4" cellpadding="3" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse;empty-cells:show;"
|
:{|
|
<math>1-x-x^2</math> has two roots <math>\frac{-1\pm\sqrt{5}}{2}</math>.

Denote that <math>\phi=\frac{2}{-1+\sqrt{5}}=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{2}{-1-\sqrt{5}}=\frac{1-\sqrt{5}}{2}</math>.

Then <math>(1-x-x^2)=(1-\phi x)(1-\hat{\phi}x)</math>, so we can write
:<math>
\begin{align}
\frac{x}{1-x-x^2}
&=\frac{x}{(1-\phi x)(1-\hat{\phi} x)}\\
&=\frac{\alpha}{(1-\phi x)}+\frac{\beta}{(1-\hat{\phi} x)},
\end{align}
</math>
where <math>\alpha</math> and <math>\beta</math> satisfying that
:<math>\begin{cases}
\alpha+\beta=0\\
\alpha\phi+\beta\hat{\phi}= -1.
\end{cases}</math>
Solving this we have that <math>\alpha=\frac{1}{\sqrt{5}}</math> and <math>\beta=-\frac{1}{\sqrt{5}}</math>. Thus,
:<math>G(x)=\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
|}
:<math>\square</math>
|}

Note that the expression <math>\frac{1}{1-z}</math> has a well known geometric expansion:
:<math>\frac{1}{1-z}=\sum_{n\ge 0}z^n</math>.

Therefore, <math>G(x)</math> can be expanded as
:<math>
\begin{align}
G(x)
&=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}\\
&=\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\phi x)^n-\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\hat{\phi} x)^n\\
&=\sum_{n\ge 0}\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)x^n.
\end{align}</math>
So the <math>n</math>th Fibonacci number is given by
:<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>.

== Solving recurrences ==
The following steps describe a general methodology of solving recurrences by generating functions.
:1. Give a recursion that computes <math>a_n</math>. In the case of Fibonacci sequence
::<math>a_n=a_{n-1}+a_{n-2}</math>.
:2. Multiply both sides of the equation by <math>x^n</math> and sum over all <math>n</math>. This gives the generating function
::<math>G(x)=\sum_{n\ge 0}a_nx^n=\sum_{n\ge 0}(a_{n-1}+a_{n-2})x^n</math>.
:: And manipulate the right hand side of the equation so that it becomes some other expression involving <math>G(x)</math>.
::<math>G(x)=x+(x+x^2)G(x)\,</math>.
:3. Solve the resulting equation to derive an explicit formula for <math>G(x)</math>.
::<math>G(x)=\frac{x}{1-x-x^2}</math>.
:4. Expand <math>G(x)</math> into a power series and read off the coefficient of <math>x^n</math>, which is a closed form for <math>a_n</math>.

The first step is usually established by combinatorial observations, or explicitly given by the problem. The third step is trivial.

The second and the forth steps need some non-trivial analytic techniques.

=== Algebraic operations on generating functions ===
The second step in the above methodology is somehow tricky. It involves first applying the recurrence to the coefficients of <math>G(x)</math>, which is easy; and then manipulating the resulting formal power series to express it in terms of <math>G(x)</math>, which is more difficult (because it works backwards).

We can apply several natural algebraic operations on the formal power series.

{{Theorem|Generating function manipulation|
:Let <math>G(x)=\sum_{n\ge 0}g_nx^n</math> and <math>F(x)=\sum_{n\ge 0}f_nx^n</math>.
----

::<math>
\begin{align}
x^k G(x)
&= \sum_{n\ge k}g_{n-k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\frac{G(x)-\sum_{i=0}^{k-1}g_iz^i}{x^k}
&=\sum_{n\ge 0}g_{n+k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\alpha F(x)+\beta G(x)
&= \sum_{n\ge 0} (\alpha f_n+\beta g_n)x^n\\
F(x)G(x)
&= \sum_{n\ge 0}\sum_{k=0}^nf_kg_{n-k}x^n\\
G(cx)
&= \sum_{n\ge 0} c^ng_n x^n\\
G'(x)
&=
\sum_{n\ge 0}(n+1)g_{n+1}x^n
\end{align}
</math>
}}

When manipulating generating functions, these rules are applied backwards; that is, from the right-hand-side to the left-hand-side.

=== Expanding generating functions ===
The last step of solving recurrences by generating function is expanding the closed form generating function <math>G(x)</math> to evaluate its <math>n</math>-th coefficient. In principle, we can always use the [http://en.wikipedia.org/wiki/Taylor_series Taylor series]
:<math>G(x)=\sum_{n\ge 0}\frac{G^{(n)}(0)}{n!}x^n</math>,
where <math>G^{(n)}(0)</math> is the value of the <math>n</math>-th derivative of <math>G(x)</math> evaluated at <math>x=0</math>.

Some interesting special cases are very useful.

====Geometric sequence====
In the example of Fibonacci numbers, we use the well known geometric series:
:<math>\frac{1}{1-x}=\sum_{n\ge 0}x^n</math>.
It is useful when we can express the generating function in the form of <math>G(x)=\frac{a_1}{1-b_1x}+\frac{a_2}{1-b_2x}+\cdots+\frac{a_k}{1-b_kx}</math>. The coefficient of <math>x^n</math> in such <math>G(x)</math> is <math>a_1b_1^n+a_2b_2^n+\cdots+a_kb_k^n</math>.

====Binomial theorem====
The <math>n</math>-th derivative of <math>(1+x)^\alpha</math> for some real <math>\alpha</math> is
:<math>\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)(1+x)^{\alpha-n}</math>.
By Taylor series, we get a generalized version of the binomial theorem known as [http://en.wikipedia.org/wiki/Binomial_coefficient#Newton.27s_binomial_series '''Newton's formula''']:
{{Theorem|Newton's formular (generalized binomial theorem)|
If <math>|x|<1</math>, then
:<math>(1+x)^\alpha=\sum_{n\ge 0}{\alpha\choose n}x^{n}</math>,
where <math>{\alpha\choose n}</math> is the '''generalized binomial coefficient''' defined by
:<math>{\alpha\choose n}=\frac{\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)}{n!}</math>.
}}

=== Example: multisets ===
In the last lecture we gave a combinatorial proof of the number of <math>k</math>-multisets on an <math>n</math>-set. Now we give a generating function approach to the problem.

Let <math>S=\{x_1,x_2,\ldots,x_n\}</math> be an <math>n</math>-element set. We have
:<math>(1+x_1+x_1^2+\cdots)(1+x_2+x_2^2+\cdots)\cdots(1+x_n+x_n^2+\cdots)=\sum_{m:S\rightarrow\mathbb{N}} \prod_{x_i\in S}x_i^{m(x_i)}</math>,
where each <math>m:S\rightarrow\mathbb{N}</math> species a possible multiset on <math>S</math> with multiplicity function <math>m</math>.

Let all <math>x_i=x</math>. Then
:<math>
\begin{align}
(1+x+x^2+\cdots)^n
&=
\sum_{m:S\rightarrow\mathbb{N}}x^{m(x_1)+\cdots+m(x_n)}\\
&=
\sum_{\text{multiset }M\text{ on }S}x^{|M|}\\
&=
\sum_{k\ge 0}\left({n\choose k}\right)x^k.
\end{align}
</math>
The last equation is due to the the definition of <math>\left({n\choose k}\right)</math>. Our task is to evaluate <math>\left({n\choose k}\right)</math>.

Due to the geometric sequence and the Newton's formula
:<math>
(1+x+x^2+\cdots)^n=(1-x)^{-n}=\sum_{k\ge 0}{-n\choose k}(-x)^k.
</math>
So
:<math>
\left({n\choose k}\right)=(-1)^k{-n\choose k}={n+k-1\choose k}.
</math>
The last equation is due to the definition of the generalized binomial coefficient. We use an analytic (generating function) proof to get the same result of <math>\left({n\choose k}\right)</math> as the combinatorial proof.

=== Example: Quicksort ===

== Catalan Number ==
We now introduce a class of counting problems, all with the same solution, called [http://en.wikipedia.org/wiki/Catalan_number '''Catalan number'''].

The <math>n</math>th Catalan number is denoted as <math>C_n</math>.
In Volume 2 of Stanley's ''Enumerative Combinatorics'', a set of exercises describe 66 different interpretations of the Catalan numbers. We give a few examples, cited from Wikipedia.
* ''C''<sub>''n''</sub> is the number of '''Dyck words''' of length 2''n''. A Dyck word is a string consisting of ''n'' X's and ''n'' Y's such that no initial segment of the string has more Y's than X's (see also [http://en.wikipedia.org/wiki/Dyck_language Dyck language]). For example, the following are the Dyck words of length 6:
<div class="center"><big> XXXYYY     XYXXYY     XYXYXY     XXYYXY     XXYXYY.</big></div>

* Re-interpreting the symbol X as an open parenthesis and Y as a close parenthesis, ''C''<sub>''n''</sub> counts the number of expressions containing ''n'' pairs of parentheses which are correctly matched:
<div class="center"><big> ((()))     ()(())     ()()()     (())()     (()()) </big></div>

* ''C''<sub>''n''</sub> is the number of different ways ''n'' + 1 factors can be completely parenthesized (or the number of ways of associating ''n'' applications of a '''binary operator'''). For ''n'' = 3, for example, we have the following five different parenthesizations of four factors:
<div class="center"><math>((ab)c)d \quad (a(bc))d \quad(ab)(cd) \quad a((bc)d) \quad a(b(cd))</math></div>

* Successive applications of a binary operator can be represented in terms of a '''full binary tree'''. (A rooted binary tree is ''full'' if every vertex has either two children or no children.) It follows that ''C''<sub>''n''</sub> is the number of full binary trees with ''n'' + 1 leaves:
[[Image:Catalan number binary tree example.png|center]]

* ''C''<sub>''n''</sub> is the number of '''monotonic paths''' along the edges of a grid with ''n'' × ''n'' square cells, which do not pass above the diagonal. A monotonic path is one which starts in the lower left corner, finishes in the upper right corner, and consists entirely of edges pointing rightwards or upwards. Counting such paths is equivalent to counting Dyck words: X stands for "move right" and Y stands for "move up". The following diagrams show the case ''n'' = 4:
[[Image:Catalan number 4x4 grid example.svg.png|450px|center]]

* ''C''<sub>''n''</sub> is the number of different ways a [http://en.wikipedia.org/wiki/Convex_polygon '''convex polygon'''] with ''n'' + 2 sides can be cut into '''triangles''' by connecting vertices with straight lines. The following hexagons illustrate the case ''n'' = 4:
[[Image:Catalan-Hexagons-example.png|400px|center]]

* ''C''<sub>''n''</sub> is the number of [http://en.wikipedia.org/wiki/Stack_(data_structure) '''stack''']-sortable permutations of {1, ..., ''n''}. A permutation ''w'' is called '''stack-sortable''' if ''S''(''w'') = (1, ..., ''n''), where ''S''(''w'') is defined recursively as follows: write ''w'' = ''unv'' where ''n'' is the largest element in ''w'' and ''u'' and ''v'' are shorter sequences, and set ''S''(''w'') = ''S''(''u'')''S''(''v'')''n'', with ''S'' being the identity for one-element sequences.

* ''C''<sub>''n''</sub> is the number of ways to tile a stairstep shape of height ''n'' with ''n'' rectangles. The following figure illustrates the case ''n'' = 4:
[[Image:Catalan stairsteps 4.png|400px|center]]

{{Theorem|Recurrence relation for Catalan numbers|
:<math>C_0=0</math>, <math>C_1=1</math>, and for <math>n>1</math>,
::<math>
C_n=\sum_{i=1}^{n-1}C_iC_{n-i}.
</math>
}}

Let <math>G(x)=\sum_{n\ge 0}C_nx^n</math> be the generating function. Apply the product rule,
:<math>G(x)^2=\sum_{n\ge 0}\sum_{k=0}^{n}C_kC_{n-k}x^n=\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n</math>.
Due to the recurrence,
:<math>G(x)=\sum_{n\ge 0}C_nx^n=x+\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n=x+G(x)^2</math>.
Solving this, we obtain
:<math>G(x)=\frac{1\pm(1-4x)^{1/2}}{2}</math>.
Because <math>C_0=0</math>, it must hold that <math>G(x)=\frac{1-(1-4x)^{1/2}}{2}</math>, or otherwise the constant term is not zero. Expanding <math>(1-4x)^{1/2}</math> by Newton's formula, we have
:<math>
\begin{align}
G(x)
&=
\frac{1-(1-4x)^{1/2}}{2}\\
&=
1-\frac{1}{2}\sum_{n\ge 0}{1/2\choose n}(-4x)^n
\end{align}
</math>
Thus,
:<math>
\begin{align}
C_n
&=-\frac{1}{2}{1/2\choose n}(-4)^n\\
&=-\frac{1}{2}\cdot\frac{1}{2}\cdot\frac{-1}{2}\cdot\frac{-3}{2}\cdots\frac{-(2n-3)}{2}\cdot(-4)^n/n!\\
&=\frac{(2n-2)!}{(n-1)!n!}\\
&=\frac{1}{n}{2n-2\choose n-1}.
\end{align}
</math>
So we prove the following closed form for Catalan number.
{{Theorem|Theorem|
:<math>C_n=\frac{1}{n}{2n-2\choose n-1}</math>.
}}

Combinatorics (Fall 2010)/Generating functions

2010-09-12T02:52:17Z

172.21.1.108: /* Example: multisets */

== Generating Functions ==
In Stanley's magnificent book ''Enumerative Combinatorics'', he comments the generating function as "the most useful but most difficult to understand method (for counting)".

The solution to a counting problem is usually represented as some <math>a_n</math> depending a parameter <math>n</math>. Sometimes this <math>a_n</math> is called a ''counting function'' as it is a function of the parameter <math>n</math>. <math>a_n</math> can also be treated as a infinite series:
:<math>a_0,a_1,a_2,\ldots</math>

The '''ordinary generating function (OGF)''' defined by <math>a_n</math> is
:<math>
G(x)=\sum_{n\ge 0} a_nx^n.
</math>

So <math>G(x)=a_0+a_1x+a_2x^2+\cdots</math>. An expression in this form is called a [http://en.wikipedia.org/wiki/Formal_power_series '''formal power series'''], and <math>a_0,a_1,a_2,\ldots</math> is the sequence of '''coefficients'''.

Furthermore, the generating function can be expanded as
:G(x)=<math>(\underbrace{1+\cdots+1}_{a_0})+(\underbrace{x+\cdots+x}_{a_1})+(\underbrace{x^2+\cdots+x^2}_{a_2})+\cdots+(\underbrace{x^n+\cdots+x^n}_{a_n})+\cdots</math>
so it indeed "generates" all the possible instances of the objects we want to count.

Usually, we do not evaluate the generating function <math>GF(x)</math> on any particular value. <math>x</math> remains as a '''formal variable''' without assuming any value. The numbers that we want to count are the coefficients carried by the terms in the formal power series. So far the generating function is just another way to represent the sequence
:<math>(a_0,a_1,a_2,\ldots\ldots)</math>.

The true power of generating functions comes from the various algebraic operations that we can perform on these generating functions. We use an example to demonstrate this.

=== Combinations ===

=== Fibonacci numbers ===
Consider the following counting problems.
* Count the number of ways that the nonnegative integer <math>n</math> can be written as a sum of ones and twos (in order).
: The problem asks for the number of compositions of <math>n</math> with summands from <math>\{1,2\}</math>. Formally, we are counting the number of tuples <math>(x_1,x_2,\ldots,x_k)</math> for some <math>k\le n</math> such that <math>x_i\in\{1,2\}</math> and <math>x_1+x_2+\cdots+x_k=n</math>.
: Let <math>F_n</math> be the solution. We observe that a composition either starts with a 1, in which case the rest is a composition of <math>n-1</math>; or starts with a 2, in which case the rest is a composition of <math>n-2</math>. So we have the recursion for <math>F_n</math> that
::<math>F_n=F_{n-1}+F_{n-2}</math>.
* Count the ways to completely cover a <math>2\times n</math> rectangle with <math>2\times 1</math> dominos without any overlaps.
: Dominos are identical <math>2\times 1</math> rectangles, so that only their orientations --- vertical or horizontal matter.
: Let <math>F_n</math> be the solution. It also holds that <math>F_n=F_{n-1}+F_{n-2}</math>. The proof is left as an exercise.

In both problems, the solution is given by <math>F_n</math> which satisfies the following recursion.
:<math>F_n=\begin{cases}
F_{n-1}+F_{n-2} & \mbox{if }n\ge 2,\\
1 & \mbox{if }n=1\\
0 & \mbox{if }n=0.
\end{cases}</math>

<math>F_n</math> is called the [http://en.wikipedia.org/wiki/Fibonacci_number Fibonacci number].

{{Theorem|Theorem|
::<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)</math>,
:where <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>.
}}
The quantity <math>\phi=\frac{1+\sqrt{5}}{2}</math> is the so-called [http://en.wikipedia.org/wiki/Golden_ratio golden ratio], a constant with some significance in mathematics and aesthetics.

We now prove this theorem by using generating functions.
The ordinary generating function for the Fibonacci number <math>F_{n}</math> is
:<math>G(x)=\sum_{n\ge 0}F_n x^n</math>.
We have that <math>F_{n}=F_{n-1}+F_{n-2}</math> for <math>n\ge 2</math>, thus
:<math>\begin{align}
G(x)
&=
\sum_{n\ge 0}F_n x^n
&=
x+\sum_{n\ge 2}(F_{n-1}+F_{n-2})x^n.
\end{align}
</math>
For generating functions, there are general ways to generate <math>F_{n-1}</math> and <math>F_{n-2}</math>, or the coefficients with any smaller indices.
:<math>
\begin{align}
xG(x)
&=\sum_{n\ge 0}F_n x^{n+1}=\sum_{n\ge 1}F_{n-1} x^n=\sum_{n\ge 2}F_{n-1} x^n\\
x^2G(x)
&=\sum_{n\ge 0}F_n x^{n+2}=\sum_{n\ge 2}F_{n-2} x^n.
\end{align}
</math>
So we have
:<math>G(x)=x+(x+x^2)G(x)\,</math>,
hence
:<math>G(x)=\frac{x}{1-x-x^2}</math>.
The value of <math>F_n</math> is the coefficient of <math>x^n</math> in the Taylor series for this formular, which is <math>\frac{G^{(n)}(0)}{n!}=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>. Although this expansion works in principle, the detailed calculus is rather painful.

----
It is easier to expand the generating function by breaking it into two geometric series.
{{Theorem|Proposition|
:Let <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>. It holds that
::<math>\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
}}

It is easy to verify the above equation, but to deduce it, we need some (high school) calculation.

{|border="2" width="100%" cellspacing="4" cellpadding="3" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse;empty-cells:show;"
|
:{|
|
<math>1-x-x^2</math> has two roots <math>\frac{-1\pm\sqrt{5}}{2}</math>.

Denote that <math>\phi=\frac{2}{-1+\sqrt{5}}=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{2}{-1-\sqrt{5}}=\frac{1-\sqrt{5}}{2}</math>.

Then <math>(1-x-x^2)=(1-\phi x)(1-\hat{\phi}x)</math>, so we can write
:<math>
\begin{align}
\frac{x}{1-x-x^2}
&=\frac{x}{(1-\phi x)(1-\hat{\phi} x)}\\
&=\frac{\alpha}{(1-\phi x)}+\frac{\beta}{(1-\hat{\phi} x)},
\end{align}
</math>
where <math>\alpha</math> and <math>\beta</math> satisfying that
:<math>\begin{cases}
\alpha+\beta=0\\
\alpha\phi+\beta\hat{\phi}= -1.
\end{cases}</math>
Solving this we have that <math>\alpha=\frac{1}{\sqrt{5}}</math> and <math>\beta=-\frac{1}{\sqrt{5}}</math>. Thus,
:<math>G(x)=\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
|}
:<math>\square</math>
|}

Note that the expression <math>\frac{1}{1-z}</math> has a well known geometric expansion:
:<math>\frac{1}{1-z}=\sum_{n\ge 0}z^n</math>.

Therefore, <math>G(x)</math> can be expanded as
:<math>
\begin{align}
G(x)
&=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}\\
&=\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\phi x)^n-\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\hat{\phi} x)^n\\
&=\sum_{n\ge 0}\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)x^n.
\end{align}</math>
So the <math>n</math>th Fibonacci number is given by
:<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>.

== Solving recurrences ==
The following steps describe a general methodology of solving recurrences by generating functions.
:1. Give a recursion that computes <math>a_n</math>. In the case of Fibonacci sequence
::<math>a_n=a_{n-1}+a_{n-2}</math>.
:2. Multiply both sides of the equation by <math>x^n</math> and sum over all <math>n</math>. This gives the generating function
::<math>G(x)=\sum_{n\ge 0}a_nx^n=\sum_{n\ge 0}(a_{n-1}+a_{n-2})x^n</math>.
:: And manipulate the right hand side of the equation so that it becomes some other expression involving <math>G(x)</math>.
::<math>G(x)=x+(x+x^2)G(x)\,</math>.
:3. Solve the resulting equation to derive an explicit formula for <math>G(x)</math>.
::<math>G(x)=\frac{x}{1-x-x^2}</math>.
:4. Expand <math>G(x)</math> into a power series and read off the coefficient of <math>x^n</math>, which is a closed form for <math>a_n</math>.

The first step is usually established by combinatorial observations, or explicitly given by the problem. The third step is trivial.

The second and the forth steps need some non-trivial analytic techniques.

=== Algebraic operations on generating functions ===
The second step in the above methodology is somehow tricky. It involves first applying the recurrence to the coefficients of <math>G(x)</math>, which is easy; and then manipulating the resulting formal power series to express it in terms of <math>G(x)</math>, which is more difficult (because it works backwards).

We can apply several natural algebraic operations on the formal power series.

{{Theorem|Generating function manipulation|
:Let <math>G(x)=\sum_{n\ge 0}g_nx^n</math> and <math>F(x)=\sum_{n\ge 0}f_nx^n</math>.
----

::<math>
\begin{align}
x^k G(x)
&= \sum_{n\ge k}g_{n-k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\frac{G(x)-\sum_{i=0}^{k-1}g_iz^i}{x^k}
&=\sum_{n\ge 0}g_{n+k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\alpha F(x)+\beta G(x)
&= \sum_{n\ge 0} (\alpha f_n+\beta g_n)x^n\\
F(x)G(x)
&= \sum_{n\ge 0}\sum_{k=0}^nf_kg_{n-k}x^n\\
G(cx)
&= \sum_{n\ge 0} c^ng_n x^n\\
G'(x)
&=
\sum_{n\ge 0}(n+1)g_{n+1}x^n
\end{align}
</math>
}}

When manipulating generating functions, these rules are applied backwards; that is, from the right-hand-side to the left-hand-side.

=== Expanding generating functions ===
The last step of solving recurrences by generating function is expanding the closed form generating function <math>G(x)</math> to evaluate its <math>n</math>-th coefficient. In principle, we can always use the [http://en.wikipedia.org/wiki/Taylor_series Taylor series]
:<math>G(x)=\sum_{n\ge 0}\frac{G^{(n)}(0)}{n!}x^n</math>,
where <math>G^{(n)}(0)</math> is the value of the <math>n</math>-th derivative of <math>G(x)</math> evaluated at <math>x=0</math>.

Some interesting special cases are very useful.

====Geometric sequence====
In the example of Fibonacci numbers, we use the well known geometric series:
:<math>\frac{1}{1-x}=\sum_{n\ge 0}x^n</math>.
It is useful when we can express the generating function in the form of <math>G(x)=\frac{a_1}{1-b_1x}+\frac{a_2}{1-b_2x}+\cdots+\frac{a_k}{1-b_kx}</math>. The coefficient of <math>x^n</math> in such <math>G(x)</math> is <math>a_1b_1^n+a_2b_2^n+\cdots+a_kb_k^n</math>.

====Binomial theorem====
The <math>n</math>-th derivative of <math>(1+x)^\alpha</math> for some real <math>\alpha</math> is
:<math>\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)(1+x)^{\alpha-n}</math>.
By Taylor series, we get a generalized version of the binomial theorem known as [http://en.wikipedia.org/wiki/Binomial_coefficient#Newton.27s_binomial_series '''Newton's formula''']:
{{Theorem|Newton's formular (generalized binomial theorem)|
If <math>|x|<1</math>, then
:<math>(1+x)^\alpha=\sum_{n\ge 0}{\alpha\choose n}x^{n}</math>,
where <math>{\alpha\choose n}</math> is the '''generalized binomial coefficient''' defined by
:<math>{\alpha\choose n}=\frac{\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)}{n!}</math>.
}}

=== Example: multisets ===
In the last lecture we gave a combinatorial proof of the number of <math>k</math>-multisets on an <math>n</math>-set. Now we give a generating function approach to the problem.

Let <math>S=\{x_1,x_2,\ldots,x_n\}</math> be an <math>n</math>-element set. We have
:<math>(1+x_1+x_1^2+\cdots)(1+x_2+x_2^2+\cdots)\cdots(1+x_n+x_n^2+\cdots)=\sum_{m:S\rightarrow\mathbb{N}} \prod_{x_i\in S}x_i^{m(x_i)}</math>,
where each <math>m:S\rightarrow\mathbb{N}</math> species a possible multiset on <math>S</math> with multiplicity function <math>m</math>.

Let all <math>x_i=x</math>. Then
:<math>
\begin{align}
(1+x+x^2+\cdots)^n
&=
\sum_{m:S\rightarrow\mathbb{N}}x^{m(x_1)+\cdots+m(x_n)}\\
&=
\sum_{\text{multiset }M\text{ on }S}x^{|M|}\\
&=
\sum_{k\ge 0}\left({n\choose k}\right)x^k.
\end{align}
</math>
The last equation is due to the the definition of <math>\left({n\choose k}\right)</math>. Our task is to evaluate <math>\left({n\choose k}\right)</math>.

Due to the geometric sequence and the Newton's formula
:<math>
(1+x+x^2+\cdots)^n=(1-x)^{-n}=\sum_{k\ge 0}{-n\choose k}(-x)^k.
</math>
So
:<math>
\left({n\choose k}\right)=(-1)^k{-n\choose k}={n+k-1\choose k}.
</math>
The last equation is due to the definition of the generalized binomial coefficient. We use an analytic (generating function) proof to get the same result of <math>\left({n\choose k}\right)</math> as the combinatorial proof.

=== Example: Quicksort ===

== Catalan Number ==
We now introduce a class of counting problems, all with the same solution, called [http://en.wikipedia.org/wiki/Catalan_number '''Catalan number'''].

The <math>n</math>th Catalan number is denoted as <math>C_n</math>.
In Volume 2 of Stanley's ''Enumerative Combinatorics'', a set of exercises describe 66 different interpretations of the Catalan numbers. We give a few examples, cited from Wikipedia.
* ''C''<sub>''n''</sub> is the number of '''Dyck words''' of length 2''n''. A Dyck word is a string consisting of ''n'' X's and ''n'' Y's such that no initial segment of the string has more Y's than X's (see also [http://en.wikipedia.org/wiki/Dyck_language Dyck language]). For example, the following are the Dyck words of length 6:
<div class="center"><big> XXXYYY     XYXXYY     XYXYXY     XXYYXY     XXYXYY.</big></div>

* Re-interpreting the symbol X as an open parenthesis and Y as a close parenthesis, ''C''<sub>''n''</sub> counts the number of expressions containing ''n'' pairs of parentheses which are correctly matched:
<div class="center"><big> ((()))     ()(())     ()()()     (())()     (()()) </big></div>

* ''C''<sub>''n''</sub> is the number of different ways ''n'' + 1 factors can be completely parenthesized (or the number of ways of associating ''n'' applications of a '''binary operator'''). For ''n'' = 3, for example, we have the following five different parenthesizations of four factors:
<div class="center"><math>((ab)c)d \quad (a(bc))d \quad(ab)(cd) \quad a((bc)d) \quad a(b(cd))</math></div>

* Successive applications of a binary operator can be represented in terms of a '''full binary tree'''. (A rooted binary tree is ''full'' if every vertex has either two children or no children.) It follows that ''C''<sub>''n''</sub> is the number of full binary trees with ''n'' + 1 leaves:
[[Image:Catalan number binary tree example.png|center]]

* ''C''<sub>''n''</sub> is the number of '''monotonic paths''' along the edges of a grid with ''n'' × ''n'' square cells, which do not pass above the diagonal. A monotonic path is one which starts in the lower left corner, finishes in the upper right corner, and consists entirely of edges pointing rightwards or upwards. Counting such paths is equivalent to counting Dyck words: X stands for "move right" and Y stands for "move up". The following diagrams show the case ''n'' = 4:
[[Image:Catalan number 4x4 grid example.svg.png|450px|center]]

* ''C''<sub>''n''</sub> is the number of different ways a [http://en.wikipedia.org/wiki/Convex_polygon '''convex polygon'''] with ''n'' + 2 sides can be cut into '''triangles''' by connecting vertices with straight lines. The following hexagons illustrate the case ''n'' = 4:
[[Image:Catalan-Hexagons-example.png|400px|center]]

* ''C''<sub>''n''</sub> is the number of [http://en.wikipedia.org/wiki/Stack_(data_structure) '''stack''']-sortable permutations of {1, ..., ''n''}. A permutation ''w'' is called '''stack-sortable''' if ''S''(''w'') = (1, ..., ''n''), where ''S''(''w'') is defined recursively as follows: write ''w'' = ''unv'' where ''n'' is the largest element in ''w'' and ''u'' and ''v'' are shorter sequences, and set ''S''(''w'') = ''S''(''u'')''S''(''v'')''n'', with ''S'' being the identity for one-element sequences.

* ''C''<sub>''n''</sub> is the number of ways to tile a stairstep shape of height ''n'' with ''n'' rectangles. The following figure illustrates the case ''n'' = 4:
[[Image:Catalan stairsteps 4.png|400px|center]]

{{Theorem|Recurrence relation for Catalan numbers|
:<math>C_0=0</math>, <math>C_1=1</math>, and for <math>n>1</math>,
::<math>
C_n=\sum_{i=1}^{n-1}C_iC_{n-i}.
</math>
}}

Let <math>G(x)=\sum_{n\ge 0}C_nx^n</math> be the generating function. Apply the product rule,
:<math>G(x)^2=\sum_{n\ge 0}\sum_{k=0}^{n}C_kC_{n-k}x^n=\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n</math>.
Due to the recurrence,
:<math>G(x)=\sum_{n\ge 0}C_nx^n=x+\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n=x+G(x)^2</math>.
Solving this, we obtain
:<math>G(x)=\frac{1\pm(1-4x)^{1/2}}{2}</math>.
Because <math>C_0=0</math>, it must hold that <math>G(x)=\frac{1-(1-4x)^{1/2}}{2}</math>, or otherwise the constant term is not zero. Expanding <math>(1-4x)^{1/2}</math> by Newton's formula, we have
:<math>
\begin{align}
G(x)
&=
\frac{1-(1-4x)^{1/2}}{2}\\
&=
1-\frac{1}{2}\sum_{n\ge 0}{1/2\choose n}(-4x)^n
\end{align}
</math>
Thus,
:<math>
\begin{align}
C_n
&=-\frac{1}{2}{1/2\choose n}(-4)^n\\
&=-\frac{1}{2}\cdot\frac{1}{2}\cdot\frac{-1}{2}\cdot\frac{-3}{2}\cdots\frac{-(2n-3)}{2}\cdot(-4)^n/n!\\
&=\frac{(2n-2)!}{(n-1)!n!}\\
&=\frac{1}{n}{2n-2\choose n-1}.
\end{align}
</math>
So we prove the following closed form for Catalan number.
{{Theorem|Theorem|
:<math>C_n=\frac{1}{n}{2n-2\choose n-1}</math>.
}}

Combinatorics (Fall 2010)/Generating functions

2010-09-12T02:46:22Z

172.21.1.108: /* Pólya's problem of changing money */

== Generating Functions ==
In Stanley's magnificent book ''Enumerative Combinatorics'', he comments the generating function as "the most useful but most difficult to understand method (for counting)".

The solution to a counting problem is usually represented as some <math>a_n</math> depending a parameter <math>n</math>. Sometimes this <math>a_n</math> is called a ''counting function'' as it is a function of the parameter <math>n</math>. <math>a_n</math> can also be treated as a infinite series:
:<math>a_0,a_1,a_2,\ldots</math>

The '''ordinary generating function (OGF)''' defined by <math>a_n</math> is
:<math>
G(x)=\sum_{n\ge 0} a_nx^n.
</math>

So <math>G(x)=a_0+a_1x+a_2x^2+\cdots</math>. An expression in this form is called a [http://en.wikipedia.org/wiki/Formal_power_series '''formal power series'''], and <math>a_0,a_1,a_2,\ldots</math> is the sequence of '''coefficients'''.

Furthermore, the generating function can be expanded as
:G(x)=<math>(\underbrace{1+\cdots+1}_{a_0})+(\underbrace{x+\cdots+x}_{a_1})+(\underbrace{x^2+\cdots+x^2}_{a_2})+\cdots+(\underbrace{x^n+\cdots+x^n}_{a_n})+\cdots</math>
so it indeed "generates" all the possible instances of the objects we want to count.

Usually, we do not evaluate the generating function <math>GF(x)</math> on any particular value. <math>x</math> remains as a '''formal variable''' without assuming any value. The numbers that we want to count are the coefficients carried by the terms in the formal power series. So far the generating function is just another way to represent the sequence
:<math>(a_0,a_1,a_2,\ldots\ldots)</math>.

The true power of generating functions comes from the various algebraic operations that we can perform on these generating functions. We use an example to demonstrate this.

=== Combinations ===

=== Fibonacci numbers ===
Consider the following counting problems.
* Count the number of ways that the nonnegative integer <math>n</math> can be written as a sum of ones and twos (in order).
: The problem asks for the number of compositions of <math>n</math> with summands from <math>\{1,2\}</math>. Formally, we are counting the number of tuples <math>(x_1,x_2,\ldots,x_k)</math> for some <math>k\le n</math> such that <math>x_i\in\{1,2\}</math> and <math>x_1+x_2+\cdots+x_k=n</math>.
: Let <math>F_n</math> be the solution. We observe that a composition either starts with a 1, in which case the rest is a composition of <math>n-1</math>; or starts with a 2, in which case the rest is a composition of <math>n-2</math>. So we have the recursion for <math>F_n</math> that
::<math>F_n=F_{n-1}+F_{n-2}</math>.
* Count the ways to completely cover a <math>2\times n</math> rectangle with <math>2\times 1</math> dominos without any overlaps.
: Dominos are identical <math>2\times 1</math> rectangles, so that only their orientations --- vertical or horizontal matter.
: Let <math>F_n</math> be the solution. It also holds that <math>F_n=F_{n-1}+F_{n-2}</math>. The proof is left as an exercise.

In both problems, the solution is given by <math>F_n</math> which satisfies the following recursion.
:<math>F_n=\begin{cases}
F_{n-1}+F_{n-2} & \mbox{if }n\ge 2,\\
1 & \mbox{if }n=1\\
0 & \mbox{if }n=0.
\end{cases}</math>

<math>F_n</math> is called the [http://en.wikipedia.org/wiki/Fibonacci_number Fibonacci number].

{{Theorem|Theorem|
::<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)</math>,
:where <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>.
}}
The quantity <math>\phi=\frac{1+\sqrt{5}}{2}</math> is the so-called [http://en.wikipedia.org/wiki/Golden_ratio golden ratio], a constant with some significance in mathematics and aesthetics.

We now prove this theorem by using generating functions.
The ordinary generating function for the Fibonacci number <math>F_{n}</math> is
:<math>G(x)=\sum_{n\ge 0}F_n x^n</math>.
We have that <math>F_{n}=F_{n-1}+F_{n-2}</math> for <math>n\ge 2</math>, thus
:<math>\begin{align}
G(x)
&=
\sum_{n\ge 0}F_n x^n
&=
x+\sum_{n\ge 2}(F_{n-1}+F_{n-2})x^n.
\end{align}
</math>
For generating functions, there are general ways to generate <math>F_{n-1}</math> and <math>F_{n-2}</math>, or the coefficients with any smaller indices.
:<math>
\begin{align}
xG(x)
&=\sum_{n\ge 0}F_n x^{n+1}=\sum_{n\ge 1}F_{n-1} x^n=\sum_{n\ge 2}F_{n-1} x^n\\
x^2G(x)
&=\sum_{n\ge 0}F_n x^{n+2}=\sum_{n\ge 2}F_{n-2} x^n.
\end{align}
</math>
So we have
:<math>G(x)=x+(x+x^2)G(x)\,</math>,
hence
:<math>G(x)=\frac{x}{1-x-x^2}</math>.
The value of <math>F_n</math> is the coefficient of <math>x^n</math> in the Taylor series for this formular, which is <math>\frac{G^{(n)}(0)}{n!}=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>. Although this expansion works in principle, the detailed calculus is rather painful.

----
It is easier to expand the generating function by breaking it into two geometric series.
{{Theorem|Proposition|
:Let <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>. It holds that
::<math>\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
}}

It is easy to verify the above equation, but to deduce it, we need some (high school) calculation.

{|border="2" width="100%" cellspacing="4" cellpadding="3" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse;empty-cells:show;"
|
:{|
|
<math>1-x-x^2</math> has two roots <math>\frac{-1\pm\sqrt{5}}{2}</math>.

Denote that <math>\phi=\frac{2}{-1+\sqrt{5}}=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{2}{-1-\sqrt{5}}=\frac{1-\sqrt{5}}{2}</math>.

Then <math>(1-x-x^2)=(1-\phi x)(1-\hat{\phi}x)</math>, so we can write
:<math>
\begin{align}
\frac{x}{1-x-x^2}
&=\frac{x}{(1-\phi x)(1-\hat{\phi} x)}\\
&=\frac{\alpha}{(1-\phi x)}+\frac{\beta}{(1-\hat{\phi} x)},
\end{align}
</math>
where <math>\alpha</math> and <math>\beta</math> satisfying that
:<math>\begin{cases}
\alpha+\beta=0\\
\alpha\phi+\beta\hat{\phi}= -1.
\end{cases}</math>
Solving this we have that <math>\alpha=\frac{1}{\sqrt{5}}</math> and <math>\beta=-\frac{1}{\sqrt{5}}</math>. Thus,
:<math>G(x)=\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
|}
:<math>\square</math>
|}

Note that the expression <math>\frac{1}{1-z}</math> has a well known geometric expansion:
:<math>\frac{1}{1-z}=\sum_{n\ge 0}z^n</math>.

Therefore, <math>G(x)</math> can be expanded as
:<math>
\begin{align}
G(x)
&=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}\\
&=\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\phi x)^n-\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\hat{\phi} x)^n\\
&=\sum_{n\ge 0}\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)x^n.
\end{align}</math>
So the <math>n</math>th Fibonacci number is given by
:<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>.

== Solving recurrences ==
The following steps describe a general methodology of solving recurrences by generating functions.
:1. Give a recursion that computes <math>a_n</math>. In the case of Fibonacci sequence
::<math>a_n=a_{n-1}+a_{n-2}</math>.
:2. Multiply both sides of the equation by <math>x^n</math> and sum over all <math>n</math>. This gives the generating function
::<math>G(x)=\sum_{n\ge 0}a_nx^n=\sum_{n\ge 0}(a_{n-1}+a_{n-2})x^n</math>.
:: And manipulate the right hand side of the equation so that it becomes some other expression involving <math>G(x)</math>.
::<math>G(x)=x+(x+x^2)G(x)\,</math>.
:3. Solve the resulting equation to derive an explicit formula for <math>G(x)</math>.
::<math>G(x)=\frac{x}{1-x-x^2}</math>.
:4. Expand <math>G(x)</math> into a power series and read off the coefficient of <math>x^n</math>, which is a closed form for <math>a_n</math>.

The first step is usually established by combinatorial observations, or explicitly given by the problem. The third step is trivial.

The second and the forth steps need some non-trivial analytic techniques.

=== Algebraic operations on generating functions ===
The second step in the above methodology is somehow tricky. It involves first applying the recurrence to the coefficients of <math>G(x)</math>, which is easy; and then manipulating the resulting formal power series to express it in terms of <math>G(x)</math>, which is more difficult (because it works backwards).

We can apply several natural algebraic operations on the formal power series.

{{Theorem|Generating function manipulation|
:Let <math>G(x)=\sum_{n\ge 0}g_nx^n</math> and <math>F(x)=\sum_{n\ge 0}f_nx^n</math>.
----

::<math>
\begin{align}
x^k G(x)
&= \sum_{n\ge k}g_{n-k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\frac{G(x)-\sum_{i=0}^{k-1}g_iz^i}{x^k}
&=\sum_{n\ge 0}g_{n+k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\alpha F(x)+\beta G(x)
&= \sum_{n\ge 0} (\alpha f_n+\beta g_n)x^n\\
F(x)G(x)
&= \sum_{n\ge 0}\sum_{k=0}^nf_kg_{n-k}x^n\\
G(cx)
&= \sum_{n\ge 0} c^ng_n x^n\\
G'(x)
&=
\sum_{n\ge 0}(n+1)g_{n+1}x^n
\end{align}
</math>
}}

When manipulating generating functions, these rules are applied backwards; that is, from the right-hand-side to the left-hand-side.

=== Expanding generating functions ===
The last step of solving recurrences by generating function is expanding the closed form generating function <math>G(x)</math> to evaluate its <math>n</math>-th coefficient. In principle, we can always use the [http://en.wikipedia.org/wiki/Taylor_series Taylor series]
:<math>G(x)=\sum_{n\ge 0}\frac{G^{(n)}(0)}{n!}x^n</math>,
where <math>G^{(n)}(0)</math> is the value of the <math>n</math>-th derivative of <math>G(x)</math> evaluated at <math>x=0</math>.

Some interesting special cases are very useful.

====Geometric sequence====
In the example of Fibonacci numbers, we use the well known geometric series:
:<math>\frac{1}{1-x}=\sum_{n\ge 0}x^n</math>.
It is useful when we can express the generating function in the form of <math>G(x)=\frac{a_1}{1-b_1x}+\frac{a_2}{1-b_2x}+\cdots+\frac{a_k}{1-b_kx}</math>. The coefficient of <math>x^n</math> in such <math>G(x)</math> is <math>a_1b_1^n+a_2b_2^n+\cdots+a_kb_k^n</math>.

====Binomial theorem====
The <math>n</math>-th derivative of <math>(1+x)^\alpha</math> for some real <math>\alpha</math> is
:<math>\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)(1+x)^{\alpha-n}</math>.
By Taylor series, we get a generalized version of the binomial theorem known as [http://en.wikipedia.org/wiki/Binomial_coefficient#Newton.27s_binomial_series '''Newton's formula''']:
{{Theorem|Newton's formular (generalized binomial theorem)|
If <math>|x|<1</math>, then
:<math>(1+x)^\alpha=\sum_{n\ge 0}{\alpha\choose n}x^{n}</math>,
where <math>{\alpha\choose n}</math> is the '''generalized binomial coefficient''' defined by
:<math>{\alpha\choose n}=\frac{\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)}{n!}</math>.
}}

=== Example: multisets ===
In the last lecture we gave a combinatorial proof of the number of <math>k</math>-multisets on an <math>n</math>-set. Now we give a generating function approach to the problem.

Let <math>S=\{x_1,x_2,\ldots,x_n\}</math> be an <math>n</math>-element set. We have
:<math>(1+x_1+x_1^2+\cdots)(1+x_2+x_2^2+\cdots)\cdots(1+x_n+x_n^2+\cdots)=\sum_{m:S\rightarrow\mathbb{N}} \prod_{x_i\in S}x_i^{m(x_i)}</math>,
where each <math>m:S\rightarrow\mathbb{N}</math> species a possible multiset on <math>S</math> with multiplicity function <math>m</math>.

Let all <math>x_i=x</math>. Then
:<math>
\begin{align}
(1+x+x^2+\cdots)^n
&=
\sum_{m:S\rightarrow\mathbb{N}}x^{m(x_1)+\cdots+m(x_n)}\\
&=
\sum_{\text{multiset }M\text{ on }S}x^{|M|}\\
&=
\sum_{k\ge 0}\left({n\choose k}\right)x^k.
\end{align}
</math>
The last equation is due to the the definition of <math>\left({n\choose k}\right)</math>. Our task is to evaluate <math>\left({n\choose k}\right)</math>.

Due to the geometric sequence and the Newton's formula
:<math>
(1+x+x^2+\cdots)^n=(1-x)^{-n}=\sum_{k\ge 0}{-n\choose k}(-x)^k.
</math>
So
:<math>
\left({n\choose k}\right)=(-1)^k{-n\choose k}={n+k-1\choose k}.
</math>
The last equation is due to the definition of the generalized binomial coefficient. We use an analytic (generating function) proof to get the same result of <math>\left({n\choose k}\right)</math> as the combinatorial proof.

== Catalan Number ==
We now introduce a class of counting problems, all with the same solution, called [http://en.wikipedia.org/wiki/Catalan_number '''Catalan number'''].

The <math>n</math>th Catalan number is denoted as <math>C_n</math>.
In Volume 2 of Stanley's ''Enumerative Combinatorics'', a set of exercises describe 66 different interpretations of the Catalan numbers. We give a few examples, cited from Wikipedia.
* ''C''<sub>''n''</sub> is the number of '''Dyck words''' of length 2''n''. A Dyck word is a string consisting of ''n'' X's and ''n'' Y's such that no initial segment of the string has more Y's than X's (see also [http://en.wikipedia.org/wiki/Dyck_language Dyck language]). For example, the following are the Dyck words of length 6:
<div class="center"><big> XXXYYY     XYXXYY     XYXYXY     XXYYXY     XXYXYY.</big></div>

* Re-interpreting the symbol X as an open parenthesis and Y as a close parenthesis, ''C''<sub>''n''</sub> counts the number of expressions containing ''n'' pairs of parentheses which are correctly matched:
<div class="center"><big> ((()))     ()(())     ()()()     (())()     (()()) </big></div>

* ''C''<sub>''n''</sub> is the number of different ways ''n'' + 1 factors can be completely parenthesized (or the number of ways of associating ''n'' applications of a '''binary operator'''). For ''n'' = 3, for example, we have the following five different parenthesizations of four factors:
<div class="center"><math>((ab)c)d \quad (a(bc))d \quad(ab)(cd) \quad a((bc)d) \quad a(b(cd))</math></div>

* Successive applications of a binary operator can be represented in terms of a '''full binary tree'''. (A rooted binary tree is ''full'' if every vertex has either two children or no children.) It follows that ''C''<sub>''n''</sub> is the number of full binary trees with ''n'' + 1 leaves:
[[Image:Catalan number binary tree example.png|center]]

* ''C''<sub>''n''</sub> is the number of '''monotonic paths''' along the edges of a grid with ''n'' × ''n'' square cells, which do not pass above the diagonal. A monotonic path is one which starts in the lower left corner, finishes in the upper right corner, and consists entirely of edges pointing rightwards or upwards. Counting such paths is equivalent to counting Dyck words: X stands for "move right" and Y stands for "move up". The following diagrams show the case ''n'' = 4:
[[Image:Catalan number 4x4 grid example.svg.png|450px|center]]

* ''C''<sub>''n''</sub> is the number of different ways a [http://en.wikipedia.org/wiki/Convex_polygon '''convex polygon'''] with ''n'' + 2 sides can be cut into '''triangles''' by connecting vertices with straight lines. The following hexagons illustrate the case ''n'' = 4:
[[Image:Catalan-Hexagons-example.png|400px|center]]

* ''C''<sub>''n''</sub> is the number of [http://en.wikipedia.org/wiki/Stack_(data_structure) '''stack''']-sortable permutations of {1, ..., ''n''}. A permutation ''w'' is called '''stack-sortable''' if ''S''(''w'') = (1, ..., ''n''), where ''S''(''w'') is defined recursively as follows: write ''w'' = ''unv'' where ''n'' is the largest element in ''w'' and ''u'' and ''v'' are shorter sequences, and set ''S''(''w'') = ''S''(''u'')''S''(''v'')''n'', with ''S'' being the identity for one-element sequences.

* ''C''<sub>''n''</sub> is the number of ways to tile a stairstep shape of height ''n'' with ''n'' rectangles. The following figure illustrates the case ''n'' = 4:
[[Image:Catalan stairsteps 4.png|400px|center]]

{{Theorem|Recurrence relation for Catalan numbers|
:<math>C_0=0</math>, <math>C_1=1</math>, and for <math>n>1</math>,
::<math>
C_n=\sum_{i=1}^{n-1}C_iC_{n-i}.
</math>
}}

Let <math>G(x)=\sum_{n\ge 0}C_nx^n</math> be the generating function. Apply the product rule,
:<math>G(x)^2=\sum_{n\ge 0}\sum_{k=0}^{n}C_kC_{n-k}x^n=\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n</math>.
Due to the recurrence,
:<math>G(x)=\sum_{n\ge 0}C_nx^n=x+\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n=x+G(x)^2</math>.
Solving this, we obtain
:<math>G(x)=\frac{1\pm(1-4x)^{1/2}}{2}</math>.
Because <math>C_0=0</math>, it must hold that <math>G(x)=\frac{1-(1-4x)^{1/2}}{2}</math>, or otherwise the constant term is not zero. Expanding <math>(1-4x)^{1/2}</math> by Newton's formula, we have
:<math>
\begin{align}
G(x)
&=
\frac{1-(1-4x)^{1/2}}{2}\\
&=
1-\frac{1}{2}\sum_{n\ge 0}{1/2\choose n}(-4x)^n
\end{align}
</math>
Thus,
:<math>
\begin{align}
C_n
&=-\frac{1}{2}{1/2\choose n}(-4)^n\\
&=-\frac{1}{2}\cdot\frac{1}{2}\cdot\frac{-1}{2}\cdot\frac{-3}{2}\cdots\frac{-(2n-3)}{2}\cdot(-4)^n/n!\\
&=\frac{(2n-2)!}{(n-1)!n!}\\
&=\frac{1}{n}{2n-2\choose n-1}.
\end{align}
</math>
So we prove the following closed form for Catalan number.
{{Theorem|Theorem|
:<math>C_n=\frac{1}{n}{2n-2\choose n-1}</math>.
}}

Combinatorics (Fall 2010)/Generating functions

2010-09-12T02:19:06Z

172.21.3.5: /* Combinations */

== Generating Functions ==
In Stanley's magnificent book ''Enumerative Combinatorics'', he comments the generating function as "the most useful but most difficult to understand method (for counting)".

The solution to a counting problem is usually represented as some <math>a_n</math> depending a parameter <math>n</math>. Sometimes this <math>a_n</math> is called a ''counting function'' as it is a function of the parameter <math>n</math>. <math>a_n</math> can also be treated as a infinite series:
:<math>a_0,a_1,a_2,\ldots</math>

The '''ordinary generating function (OGF)''' defined by <math>a_n</math> is
:<math>
G(x)=\sum_{n\ge 0} a_nx^n.
</math>

So <math>G(x)=a_0+a_1x+a_2x^2+\cdots</math>. An expression in this form is called a [http://en.wikipedia.org/wiki/Formal_power_series '''formal power series'''], and <math>a_0,a_1,a_2,\ldots</math> is the sequence of '''coefficients'''.

Furthermore, the generating function can be expanded as
:G(x)=<math>(\underbrace{1+\cdots+1}_{a_0})+(\underbrace{x+\cdots+x}_{a_1})+(\underbrace{x^2+\cdots+x^2}_{a_2})+\cdots+(\underbrace{x^n+\cdots+x^n}_{a_n})+\cdots</math>
so it indeed "generates" all the possible instances of the objects we want to count.

Usually, we do not evaluate the generating function <math>GF(x)</math> on any particular value. <math>x</math> remains as a '''formal variable''' without assuming any value. The numbers that we want to count are the coefficients carried by the terms in the formal power series. So far the generating function is just another way to represent the sequence
:<math>(a_0,a_1,a_2,\ldots\ldots)</math>.

The true power of generating functions comes from the various algebraic operations that we can perform on these generating functions. We use an example to demonstrate this.

=== Combinations ===

=== Pólya's problem of changing money ===

=== Fibonacci numbers ===
Consider the following counting problems.
* Count the number of ways that the nonnegative integer <math>n</math> can be written as a sum of ones and twos (in order).
: The problem asks for the number of compositions of <math>n</math> with summands from <math>\{1,2\}</math>. Formally, we are counting the number of tuples <math>(x_1,x_2,\ldots,x_k)</math> for some <math>k\le n</math> such that <math>x_i\in\{1,2\}</math> and <math>x_1+x_2+\cdots+x_k=n</math>.
: Let <math>F_n</math> be the solution. We observe that a composition either starts with a 1, in which case the rest is a composition of <math>n-1</math>; or starts with a 2, in which case the rest is a composition of <math>n-2</math>. So we have the recursion for <math>F_n</math> that
::<math>F_n=F_{n-1}+F_{n-2}</math>.
* Count the ways to completely cover a <math>2\times n</math> rectangle with <math>2\times 1</math> dominos without any overlaps.
: Dominos are identical <math>2\times 1</math> rectangles, so that only their orientations --- vertical or horizontal matter.
: Let <math>F_n</math> be the solution. It also holds that <math>F_n=F_{n-1}+F_{n-2}</math>. The proof is left as an exercise.

In both problems, the solution is given by <math>F_n</math> which satisfies the following recursion.
:<math>F_n=\begin{cases}
F_{n-1}+F_{n-2} & \mbox{if }n\ge 2,\\
1 & \mbox{if }n=1\\
0 & \mbox{if }n=0.
\end{cases}</math>

<math>F_n</math> is called the [http://en.wikipedia.org/wiki/Fibonacci_number Fibonacci number].

{{Theorem|Theorem|
::<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)</math>,
:where <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>.
}}
The quantity <math>\phi=\frac{1+\sqrt{5}}{2}</math> is the so-called [http://en.wikipedia.org/wiki/Golden_ratio golden ratio], a constant with some significance in mathematics and aesthetics.

We now prove this theorem by using generating functions.
The ordinary generating function for the Fibonacci number <math>F_{n}</math> is
:<math>G(x)=\sum_{n\ge 0}F_n x^n</math>.
We have that <math>F_{n}=F_{n-1}+F_{n-2}</math> for <math>n\ge 2</math>, thus
:<math>\begin{align}
G(x)
&=
\sum_{n\ge 0}F_n x^n
&=
x+\sum_{n\ge 2}(F_{n-1}+F_{n-2})x^n.
\end{align}
</math>
For generating functions, there are general ways to generate <math>F_{n-1}</math> and <math>F_{n-2}</math>, or the coefficients with any smaller indices.
:<math>
\begin{align}
xG(x)
&=\sum_{n\ge 0}F_n x^{n+1}=\sum_{n\ge 1}F_{n-1} x^n=\sum_{n\ge 2}F_{n-1} x^n\\
x^2G(x)
&=\sum_{n\ge 0}F_n x^{n+2}=\sum_{n\ge 2}F_{n-2} x^n.
\end{align}
</math>
So we have
:<math>G(x)=x+(x+x^2)G(x)\,</math>,
hence
:<math>G(x)=\frac{x}{1-x-x^2}</math>.
The value of <math>F_n</math> is the coefficient of <math>x^n</math> in the Taylor series for this formular, which is <math>\frac{G^{(n)}(0)}{n!}=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>. Although this expansion works in principle, the detailed calculus is rather painful.

----
It is easier to expand the generating function by breaking it into two geometric series.
{{Theorem|Proposition|
:Let <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>. It holds that
::<math>\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
}}

It is easy to verify the above equation, but to deduce it, we need some (high school) calculation.

{|border="2" width="100%" cellspacing="4" cellpadding="3" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse;empty-cells:show;"
|
:{|
|
<math>1-x-x^2</math> has two roots <math>\frac{-1\pm\sqrt{5}}{2}</math>.

Denote that <math>\phi=\frac{2}{-1+\sqrt{5}}=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{2}{-1-\sqrt{5}}=\frac{1-\sqrt{5}}{2}</math>.

Then <math>(1-x-x^2)=(1-\phi x)(1-\hat{\phi}x)</math>, so we can write
:<math>
\begin{align}
\frac{x}{1-x-x^2}
&=\frac{x}{(1-\phi x)(1-\hat{\phi} x)}\\
&=\frac{\alpha}{(1-\phi x)}+\frac{\beta}{(1-\hat{\phi} x)},
\end{align}
</math>
where <math>\alpha</math> and <math>\beta</math> satisfying that
:<math>\begin{cases}
\alpha+\beta=0\\
\alpha\phi+\beta\hat{\phi}= -1.
\end{cases}</math>
Solving this we have that <math>\alpha=\frac{1}{\sqrt{5}}</math> and <math>\beta=-\frac{1}{\sqrt{5}}</math>. Thus,
:<math>G(x)=\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
|}
:<math>\square</math>
|}

Note that the expression <math>\frac{1}{1-z}</math> has a well known geometric expansion:
:<math>\frac{1}{1-z}=\sum_{n\ge 0}z^n</math>.

Therefore, <math>G(x)</math> can be expanded as
:<math>
\begin{align}
G(x)
&=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}\\
&=\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\phi x)^n-\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\hat{\phi} x)^n\\
&=\sum_{n\ge 0}\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)x^n.
\end{align}</math>
So the <math>n</math>th Fibonacci number is given by
:<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>.

== Solving recurrences ==
The following steps describe a general methodology of solving recurrences by generating functions.
:1. Give a recursion that computes <math>a_n</math>. In the case of Fibonacci sequence
::<math>a_n=a_{n-1}+a_{n-2}</math>.
:2. Multiply both sides of the equation by <math>x^n</math> and sum over all <math>n</math>. This gives the generating function
::<math>G(x)=\sum_{n\ge 0}a_nx^n=\sum_{n\ge 0}(a_{n-1}+a_{n-2})x^n</math>.
:: And manipulate the right hand side of the equation so that it becomes some other expression involving <math>G(x)</math>.
::<math>G(x)=x+(x+x^2)G(x)\,</math>.
:3. Solve the resulting equation to derive an explicit formula for <math>G(x)</math>.
::<math>G(x)=\frac{x}{1-x-x^2}</math>.
:4. Expand <math>G(x)</math> into a power series and read off the coefficient of <math>x^n</math>, which is a closed form for <math>a_n</math>.

The first step is usually established by combinatorial observations, or explicitly given by the problem. The third step is trivial.

The second and the forth steps need some non-trivial analytic techniques.

=== Algebraic operations on generating functions ===
The second step in the above methodology is somehow tricky. It involves first applying the recurrence to the coefficients of <math>G(x)</math>, which is easy; and then manipulating the resulting formal power series to express it in terms of <math>G(x)</math>, which is more difficult (because it works backwards).

We can apply several natural algebraic operations on the formal power series.

{{Theorem|Generating function manipulation|
:Let <math>G(x)=\sum_{n\ge 0}g_nx^n</math> and <math>F(x)=\sum_{n\ge 0}f_nx^n</math>.
----

::<math>
\begin{align}
x^k G(x)
&= \sum_{n\ge k}g_{n-k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\frac{G(x)-\sum_{i=0}^{k-1}g_iz^i}{x^k}
&=\sum_{n\ge 0}g_{n+k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\alpha F(x)+\beta G(x)
&= \sum_{n\ge 0} (\alpha f_n+\beta g_n)x^n\\
F(x)G(x)
&= \sum_{n\ge 0}\sum_{k=0}^nf_kg_{n-k}x^n\\
G(cx)
&= \sum_{n\ge 0} c^ng_n x^n\\
G'(x)
&=
\sum_{n\ge 0}(n+1)g_{n+1}x^n
\end{align}
</math>
}}

When manipulating generating functions, these rules are applied backwards; that is, from the right-hand-side to the left-hand-side.

=== Expanding generating functions ===
The last step of solving recurrences by generating function is expanding the closed form generating function <math>G(x)</math> to evaluate its <math>n</math>-th coefficient. In principle, we can always use the [http://en.wikipedia.org/wiki/Taylor_series Taylor series]
:<math>G(x)=\sum_{n\ge 0}\frac{G^{(n)}(0)}{n!}x^n</math>,
where <math>G^{(n)}(0)</math> is the value of the <math>n</math>-th derivative of <math>G(x)</math> evaluated at <math>x=0</math>.

Some interesting special cases are very useful.

====Geometric sequence====
In the example of Fibonacci numbers, we use the well known geometric series:
:<math>\frac{1}{1-x}=\sum_{n\ge 0}x^n</math>.
It is useful when we can express the generating function in the form of <math>G(x)=\frac{a_1}{1-b_1x}+\frac{a_2}{1-b_2x}+\cdots+\frac{a_k}{1-b_kx}</math>. The coefficient of <math>x^n</math> in such <math>G(x)</math> is <math>a_1b_1^n+a_2b_2^n+\cdots+a_kb_k^n</math>.

====Binomial theorem====
The <math>n</math>-th derivative of <math>(1+x)^\alpha</math> for some real <math>\alpha</math> is
:<math>\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)(1+x)^{\alpha-n}</math>.
By Taylor series, we get a generalized version of the binomial theorem known as [http://en.wikipedia.org/wiki/Binomial_coefficient#Newton.27s_binomial_series '''Newton's formula''']:
{{Theorem|Newton's formular (generalized binomial theorem)|
If <math>|x|<1</math>, then
:<math>(1+x)^\alpha=\sum_{n\ge 0}{\alpha\choose n}x^{n}</math>,
where <math>{\alpha\choose n}</math> is the '''generalized binomial coefficient''' defined by
:<math>{\alpha\choose n}=\frac{\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)}{n!}</math>.
}}

=== Example: multisets ===
In the last lecture we gave a combinatorial proof of the number of <math>k</math>-multisets on an <math>n</math>-set. Now we give a generating function approach to the problem.

Let <math>S=\{x_1,x_2,\ldots,x_n\}</math> be an <math>n</math>-element set. We have
:<math>(1+x_1+x_1^2+\cdots)(1+x_2+x_2^2+\cdots)\cdots(1+x_n+x_n^2+\cdots)=\sum_{m:S\rightarrow\mathbb{N}} \prod_{x_i\in S}x_i^{m(x_i)}</math>,
where each <math>m:S\rightarrow\mathbb{N}</math> species a possible multiset on <math>S</math> with multiplicity function <math>m</math>.

Let all <math>x_i=x</math>. Then
:<math>
\begin{align}
(1+x+x^2+\cdots)^n
&=
\sum_{m:S\rightarrow\mathbb{N}}x^{m(x_1)+\cdots+m(x_n)}\\
&=
\sum_{\text{multiset }M\text{ on }S}x^{|M|}\\
&=
\sum_{k\ge 0}\left({n\choose k}\right)x^k.
\end{align}
</math>
The last equation is due to the the definition of <math>\left({n\choose k}\right)</math>. Our task is to evaluate <math>\left({n\choose k}\right)</math>.

Due to the geometric sequence and the Newton's formula
:<math>
(1+x+x^2+\cdots)^n=(1-x)^{-n}=\sum_{k\ge 0}{-n\choose k}(-x)^k.
</math>
So
:<math>
\left({n\choose k}\right)=(-1)^k{-n\choose k}={n+k-1\choose k}.
</math>
The last equation is due to the definition of the generalized binomial coefficient. We use an analytic (generating function) proof to get the same result of <math>\left({n\choose k}\right)</math> as the combinatorial proof.

== Catalan Number ==
We now introduce a class of counting problems, all with the same solution, called [http://en.wikipedia.org/wiki/Catalan_number '''Catalan number'''].

The <math>n</math>th Catalan number is denoted as <math>C_n</math>.
In Volume 2 of Stanley's ''Enumerative Combinatorics'', a set of exercises describe 66 different interpretations of the Catalan numbers. We give a few examples, cited from Wikipedia.
* ''C''<sub>''n''</sub> is the number of '''Dyck words''' of length 2''n''. A Dyck word is a string consisting of ''n'' X's and ''n'' Y's such that no initial segment of the string has more Y's than X's (see also [http://en.wikipedia.org/wiki/Dyck_language Dyck language]). For example, the following are the Dyck words of length 6:
<div class="center"><big> XXXYYY     XYXXYY     XYXYXY     XXYYXY     XXYXYY.</big></div>

* Re-interpreting the symbol X as an open parenthesis and Y as a close parenthesis, ''C''<sub>''n''</sub> counts the number of expressions containing ''n'' pairs of parentheses which are correctly matched:
<div class="center"><big> ((()))     ()(())     ()()()     (())()     (()()) </big></div>

* ''C''<sub>''n''</sub> is the number of different ways ''n'' + 1 factors can be completely parenthesized (or the number of ways of associating ''n'' applications of a '''binary operator'''). For ''n'' = 3, for example, we have the following five different parenthesizations of four factors:
<div class="center"><math>((ab)c)d \quad (a(bc))d \quad(ab)(cd) \quad a((bc)d) \quad a(b(cd))</math></div>

* Successive applications of a binary operator can be represented in terms of a '''full binary tree'''. (A rooted binary tree is ''full'' if every vertex has either two children or no children.) It follows that ''C''<sub>''n''</sub> is the number of full binary trees with ''n'' + 1 leaves:
[[Image:Catalan number binary tree example.png|center]]

* ''C''<sub>''n''</sub> is the number of '''monotonic paths''' along the edges of a grid with ''n'' × ''n'' square cells, which do not pass above the diagonal. A monotonic path is one which starts in the lower left corner, finishes in the upper right corner, and consists entirely of edges pointing rightwards or upwards. Counting such paths is equivalent to counting Dyck words: X stands for "move right" and Y stands for "move up". The following diagrams show the case ''n'' = 4:
[[Image:Catalan number 4x4 grid example.svg.png|450px|center]]

* ''C''<sub>''n''</sub> is the number of different ways a [http://en.wikipedia.org/wiki/Convex_polygon '''convex polygon'''] with ''n'' + 2 sides can be cut into '''triangles''' by connecting vertices with straight lines. The following hexagons illustrate the case ''n'' = 4:
[[Image:Catalan-Hexagons-example.png|400px|center]]

* ''C''<sub>''n''</sub> is the number of [http://en.wikipedia.org/wiki/Stack_(data_structure) '''stack''']-sortable permutations of {1, ..., ''n''}. A permutation ''w'' is called '''stack-sortable''' if ''S''(''w'') = (1, ..., ''n''), where ''S''(''w'') is defined recursively as follows: write ''w'' = ''unv'' where ''n'' is the largest element in ''w'' and ''u'' and ''v'' are shorter sequences, and set ''S''(''w'') = ''S''(''u'')''S''(''v'')''n'', with ''S'' being the identity for one-element sequences.

* ''C''<sub>''n''</sub> is the number of ways to tile a stairstep shape of height ''n'' with ''n'' rectangles. The following figure illustrates the case ''n'' = 4:
[[Image:Catalan stairsteps 4.png|400px|center]]

{{Theorem|Recurrence relation for Catalan numbers|
:<math>C_0=0</math>, <math>C_1=1</math>, and for <math>n>1</math>,
::<math>
C_n=\sum_{i=1}^{n-1}C_iC_{n-i}.
</math>
}}

Let <math>G(x)=\sum_{n\ge 0}C_nx^n</math> be the generating function. Apply the product rule,
:<math>G(x)^2=\sum_{n\ge 0}\sum_{k=0}^{n}C_kC_{n-k}x^n=\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n</math>.
Due to the recurrence,
:<math>G(x)=\sum_{n\ge 0}C_nx^n=x+\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n=x+G(x)^2</math>.
Solving this, we obtain
:<math>G(x)=\frac{1\pm(1-4x)^{1/2}}{2}</math>.
Because <math>C_0=0</math>, it must hold that <math>G(x)=\frac{1-(1-4x)^{1/2}}{2}</math>, or otherwise the constant term is not zero. Expanding <math>(1-4x)^{1/2}</math> by Newton's formula, we have
:<math>
\begin{align}
G(x)
&=
\frac{1-(1-4x)^{1/2}}{2}\\
&=
1-\frac{1}{2}\sum_{n\ge 0}{1/2\choose n}(-4x)^n
\end{align}
</math>
Thus,
:<math>
\begin{align}
C_n
&=-\frac{1}{2}{1/2\choose n}(-4)^n\\
&=-\frac{1}{2}\cdot\frac{1}{2}\cdot\frac{-1}{2}\cdot\frac{-3}{2}\cdots\frac{-(2n-3)}{2}\cdot(-4)^n/n!\\
&=\frac{(2n-2)!}{(n-1)!n!}\\
&=\frac{1}{n}{2n-2\choose n-1}.
\end{align}
</math>
So we prove the following closed form for Catalan number.
{{Theorem|Theorem|
:<math>C_n=\frac{1}{n}{2n-2\choose n-1}</math>.
}}

Combinatorics (Fall 2010)/Generating functions

2010-09-12T02:18:48Z

172.21.3.5: /* Pólya's problem of changing money */

== Generating Functions ==
In Stanley's magnificent book ''Enumerative Combinatorics'', he comments the generating function as "the most useful but most difficult to understand method (for counting)".

The solution to a counting problem is usually represented as some <math>a_n</math> depending a parameter <math>n</math>. Sometimes this <math>a_n</math> is called a ''counting function'' as it is a function of the parameter <math>n</math>. <math>a_n</math> can also be treated as a infinite series:
:<math>a_0,a_1,a_2,\ldots</math>

The '''ordinary generating function (OGF)''' defined by <math>a_n</math> is
:<math>
G(x)=\sum_{n\ge 0} a_nx^n.
</math>

So <math>G(x)=a_0+a_1x+a_2x^2+\cdots</math>. An expression in this form is called a [http://en.wikipedia.org/wiki/Formal_power_series '''formal power series'''], and <math>a_0,a_1,a_2,\ldots</math> is the sequence of '''coefficients'''.

Furthermore, the generating function can be expanded as
:G(x)=<math>(\underbrace{1+\cdots+1}_{a_0})+(\underbrace{x+\cdots+x}_{a_1})+(\underbrace{x^2+\cdots+x^2}_{a_2})+\cdots+(\underbrace{x^n+\cdots+x^n}_{a_n})+\cdots</math>
so it indeed "generates" all the possible instances of the objects we want to count.

Usually, we do not evaluate the generating function <math>GF(x)</math> on any particular value. <math>x</math> remains as a '''formal variable''' without assuming any value. The numbers that we want to count are the coefficients carried by the terms in the formal power series. So far the generating function is just another way to represent the sequence
:<math>(a_0,a_1,a_2,\ldots\ldots)</math>.

The true power of generating functions comes from the various algebraic operations that we can perform on these generating functions. We use an example to demonstrate this.

=== Combinations ===

=== Fibonacci numbers ===
Consider the following counting problems.
* Count the number of ways that the nonnegative integer <math>n</math> can be written as a sum of ones and twos (in order).
: The problem asks for the number of compositions of <math>n</math> with summands from <math>\{1,2\}</math>. Formally, we are counting the number of tuples <math>(x_1,x_2,\ldots,x_k)</math> for some <math>k\le n</math> such that <math>x_i\in\{1,2\}</math> and <math>x_1+x_2+\cdots+x_k=n</math>.
: Let <math>F_n</math> be the solution. We observe that a composition either starts with a 1, in which case the rest is a composition of <math>n-1</math>; or starts with a 2, in which case the rest is a composition of <math>n-2</math>. So we have the recursion for <math>F_n</math> that
::<math>F_n=F_{n-1}+F_{n-2}</math>.
* Count the ways to completely cover a <math>2\times n</math> rectangle with <math>2\times 1</math> dominos without any overlaps.
: Dominos are identical <math>2\times 1</math> rectangles, so that only their orientations --- vertical or horizontal matter.
: Let <math>F_n</math> be the solution. It also holds that <math>F_n=F_{n-1}+F_{n-2}</math>. The proof is left as an exercise.

In both problems, the solution is given by <math>F_n</math> which satisfies the following recursion.
:<math>F_n=\begin{cases}
F_{n-1}+F_{n-2} & \mbox{if }n\ge 2,\\
1 & \mbox{if }n=1\\
0 & \mbox{if }n=0.
\end{cases}</math>

<math>F_n</math> is called the [http://en.wikipedia.org/wiki/Fibonacci_number Fibonacci number].

{{Theorem|Theorem|
::<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)</math>,
:where <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>.
}}
The quantity <math>\phi=\frac{1+\sqrt{5}}{2}</math> is the so-called [http://en.wikipedia.org/wiki/Golden_ratio golden ratio], a constant with some significance in mathematics and aesthetics.

We now prove this theorem by using generating functions.
The ordinary generating function for the Fibonacci number <math>F_{n}</math> is
:<math>G(x)=\sum_{n\ge 0}F_n x^n</math>.
We have that <math>F_{n}=F_{n-1}+F_{n-2}</math> for <math>n\ge 2</math>, thus
:<math>\begin{align}
G(x)
&=
\sum_{n\ge 0}F_n x^n
&=
x+\sum_{n\ge 2}(F_{n-1}+F_{n-2})x^n.
\end{align}
</math>
For generating functions, there are general ways to generate <math>F_{n-1}</math> and <math>F_{n-2}</math>, or the coefficients with any smaller indices.
:<math>
\begin{align}
xG(x)
&=\sum_{n\ge 0}F_n x^{n+1}=\sum_{n\ge 1}F_{n-1} x^n=\sum_{n\ge 2}F_{n-1} x^n\\
x^2G(x)
&=\sum_{n\ge 0}F_n x^{n+2}=\sum_{n\ge 2}F_{n-2} x^n.
\end{align}
</math>
So we have
:<math>G(x)=x+(x+x^2)G(x)\,</math>,
hence
:<math>G(x)=\frac{x}{1-x-x^2}</math>.
The value of <math>F_n</math> is the coefficient of <math>x^n</math> in the Taylor series for this formular, which is <math>\frac{G^{(n)}(0)}{n!}=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>. Although this expansion works in principle, the detailed calculus is rather painful.

----
It is easier to expand the generating function by breaking it into two geometric series.
{{Theorem|Proposition|
:Let <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>. It holds that
::<math>\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
}}

It is easy to verify the above equation, but to deduce it, we need some (high school) calculation.

{|border="2" width="100%" cellspacing="4" cellpadding="3" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse;empty-cells:show;"
|
:{|
|
<math>1-x-x^2</math> has two roots <math>\frac{-1\pm\sqrt{5}}{2}</math>.

Denote that <math>\phi=\frac{2}{-1+\sqrt{5}}=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{2}{-1-\sqrt{5}}=\frac{1-\sqrt{5}}{2}</math>.

Then <math>(1-x-x^2)=(1-\phi x)(1-\hat{\phi}x)</math>, so we can write
:<math>
\begin{align}
\frac{x}{1-x-x^2}
&=\frac{x}{(1-\phi x)(1-\hat{\phi} x)}\\
&=\frac{\alpha}{(1-\phi x)}+\frac{\beta}{(1-\hat{\phi} x)},
\end{align}
</math>
where <math>\alpha</math> and <math>\beta</math> satisfying that
:<math>\begin{cases}
\alpha+\beta=0\\
\alpha\phi+\beta\hat{\phi}= -1.
\end{cases}</math>
Solving this we have that <math>\alpha=\frac{1}{\sqrt{5}}</math> and <math>\beta=-\frac{1}{\sqrt{5}}</math>. Thus,
:<math>G(x)=\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
|}
:<math>\square</math>
|}

Note that the expression <math>\frac{1}{1-z}</math> has a well known geometric expansion:
:<math>\frac{1}{1-z}=\sum_{n\ge 0}z^n</math>.

Therefore, <math>G(x)</math> can be expanded as
:<math>
\begin{align}
G(x)
&=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}\\
&=\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\phi x)^n-\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\hat{\phi} x)^n\\
&=\sum_{n\ge 0}\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)x^n.
\end{align}</math>
So the <math>n</math>th Fibonacci number is given by
:<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>.

== Solving recurrences ==
The following steps describe a general methodology of solving recurrences by generating functions.
:1. Give a recursion that computes <math>a_n</math>. In the case of Fibonacci sequence
::<math>a_n=a_{n-1}+a_{n-2}</math>.
:2. Multiply both sides of the equation by <math>x^n</math> and sum over all <math>n</math>. This gives the generating function
::<math>G(x)=\sum_{n\ge 0}a_nx^n=\sum_{n\ge 0}(a_{n-1}+a_{n-2})x^n</math>.
:: And manipulate the right hand side of the equation so that it becomes some other expression involving <math>G(x)</math>.
::<math>G(x)=x+(x+x^2)G(x)\,</math>.
:3. Solve the resulting equation to derive an explicit formula for <math>G(x)</math>.
::<math>G(x)=\frac{x}{1-x-x^2}</math>.
:4. Expand <math>G(x)</math> into a power series and read off the coefficient of <math>x^n</math>, which is a closed form for <math>a_n</math>.

The first step is usually established by combinatorial observations, or explicitly given by the problem. The third step is trivial.

The second and the forth steps need some non-trivial analytic techniques.

=== Algebraic operations on generating functions ===
The second step in the above methodology is somehow tricky. It involves first applying the recurrence to the coefficients of <math>G(x)</math>, which is easy; and then manipulating the resulting formal power series to express it in terms of <math>G(x)</math>, which is more difficult (because it works backwards).

We can apply several natural algebraic operations on the formal power series.

{{Theorem|Generating function manipulation|
:Let <math>G(x)=\sum_{n\ge 0}g_nx^n</math> and <math>F(x)=\sum_{n\ge 0}f_nx^n</math>.
----

::<math>
\begin{align}
x^k G(x)
&= \sum_{n\ge k}g_{n-k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\frac{G(x)-\sum_{i=0}^{k-1}g_iz^i}{x^k}
&=\sum_{n\ge 0}g_{n+k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\alpha F(x)+\beta G(x)
&= \sum_{n\ge 0} (\alpha f_n+\beta g_n)x^n\\
F(x)G(x)
&= \sum_{n\ge 0}\sum_{k=0}^nf_kg_{n-k}x^n\\
G(cx)
&= \sum_{n\ge 0} c^ng_n x^n\\
G'(x)
&=
\sum_{n\ge 0}(n+1)g_{n+1}x^n
\end{align}
</math>
}}

When manipulating generating functions, these rules are applied backwards; that is, from the right-hand-side to the left-hand-side.

=== Expanding generating functions ===
The last step of solving recurrences by generating function is expanding the closed form generating function <math>G(x)</math> to evaluate its <math>n</math>-th coefficient. In principle, we can always use the [http://en.wikipedia.org/wiki/Taylor_series Taylor series]
:<math>G(x)=\sum_{n\ge 0}\frac{G^{(n)}(0)}{n!}x^n</math>,
where <math>G^{(n)}(0)</math> is the value of the <math>n</math>-th derivative of <math>G(x)</math> evaluated at <math>x=0</math>.

Some interesting special cases are very useful.

====Geometric sequence====
In the example of Fibonacci numbers, we use the well known geometric series:
:<math>\frac{1}{1-x}=\sum_{n\ge 0}x^n</math>.
It is useful when we can express the generating function in the form of <math>G(x)=\frac{a_1}{1-b_1x}+\frac{a_2}{1-b_2x}+\cdots+\frac{a_k}{1-b_kx}</math>. The coefficient of <math>x^n</math> in such <math>G(x)</math> is <math>a_1b_1^n+a_2b_2^n+\cdots+a_kb_k^n</math>.

====Binomial theorem====
The <math>n</math>-th derivative of <math>(1+x)^\alpha</math> for some real <math>\alpha</math> is
:<math>\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)(1+x)^{\alpha-n}</math>.
By Taylor series, we get a generalized version of the binomial theorem known as [http://en.wikipedia.org/wiki/Binomial_coefficient#Newton.27s_binomial_series '''Newton's formula''']:
{{Theorem|Newton's formular (generalized binomial theorem)|
If <math>|x|<1</math>, then
:<math>(1+x)^\alpha=\sum_{n\ge 0}{\alpha\choose n}x^{n}</math>,
where <math>{\alpha\choose n}</math> is the '''generalized binomial coefficient''' defined by
:<math>{\alpha\choose n}=\frac{\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)}{n!}</math>.
}}

=== Example: multisets ===
In the last lecture we gave a combinatorial proof of the number of <math>k</math>-multisets on an <math>n</math>-set. Now we give a generating function approach to the problem.

Let <math>S=\{x_1,x_2,\ldots,x_n\}</math> be an <math>n</math>-element set. We have
:<math>(1+x_1+x_1^2+\cdots)(1+x_2+x_2^2+\cdots)\cdots(1+x_n+x_n^2+\cdots)=\sum_{m:S\rightarrow\mathbb{N}} \prod_{x_i\in S}x_i^{m(x_i)}</math>,
where each <math>m:S\rightarrow\mathbb{N}</math> species a possible multiset on <math>S</math> with multiplicity function <math>m</math>.

Let all <math>x_i=x</math>. Then
:<math>
\begin{align}
(1+x+x^2+\cdots)^n
&=
\sum_{m:S\rightarrow\mathbb{N}}x^{m(x_1)+\cdots+m(x_n)}\\
&=
\sum_{\text{multiset }M\text{ on }S}x^{|M|}\\
&=
\sum_{k\ge 0}\left({n\choose k}\right)x^k.
\end{align}
</math>
The last equation is due to the the definition of <math>\left({n\choose k}\right)</math>. Our task is to evaluate <math>\left({n\choose k}\right)</math>.

Due to the geometric sequence and the Newton's formula
:<math>
(1+x+x^2+\cdots)^n=(1-x)^{-n}=\sum_{k\ge 0}{-n\choose k}(-x)^k.
</math>
So
:<math>
\left({n\choose k}\right)=(-1)^k{-n\choose k}={n+k-1\choose k}.
</math>
The last equation is due to the definition of the generalized binomial coefficient. We use an analytic (generating function) proof to get the same result of <math>\left({n\choose k}\right)</math> as the combinatorial proof.

== Catalan Number ==
We now introduce a class of counting problems, all with the same solution, called [http://en.wikipedia.org/wiki/Catalan_number '''Catalan number'''].

The <math>n</math>th Catalan number is denoted as <math>C_n</math>.
In Volume 2 of Stanley's ''Enumerative Combinatorics'', a set of exercises describe 66 different interpretations of the Catalan numbers. We give a few examples, cited from Wikipedia.
* ''C''<sub>''n''</sub> is the number of '''Dyck words''' of length 2''n''. A Dyck word is a string consisting of ''n'' X's and ''n'' Y's such that no initial segment of the string has more Y's than X's (see also [http://en.wikipedia.org/wiki/Dyck_language Dyck language]). For example, the following are the Dyck words of length 6:
<div class="center"><big> XXXYYY     XYXXYY     XYXYXY     XXYYXY     XXYXYY.</big></div>

* Re-interpreting the symbol X as an open parenthesis and Y as a close parenthesis, ''C''<sub>''n''</sub> counts the number of expressions containing ''n'' pairs of parentheses which are correctly matched:
<div class="center"><big> ((()))     ()(())     ()()()     (())()     (()()) </big></div>

* ''C''<sub>''n''</sub> is the number of different ways ''n'' + 1 factors can be completely parenthesized (or the number of ways of associating ''n'' applications of a '''binary operator'''). For ''n'' = 3, for example, we have the following five different parenthesizations of four factors:
<div class="center"><math>((ab)c)d \quad (a(bc))d \quad(ab)(cd) \quad a((bc)d) \quad a(b(cd))</math></div>

* Successive applications of a binary operator can be represented in terms of a '''full binary tree'''. (A rooted binary tree is ''full'' if every vertex has either two children or no children.) It follows that ''C''<sub>''n''</sub> is the number of full binary trees with ''n'' + 1 leaves:
[[Image:Catalan number binary tree example.png|center]]

* ''C''<sub>''n''</sub> is the number of '''monotonic paths''' along the edges of a grid with ''n'' × ''n'' square cells, which do not pass above the diagonal. A monotonic path is one which starts in the lower left corner, finishes in the upper right corner, and consists entirely of edges pointing rightwards or upwards. Counting such paths is equivalent to counting Dyck words: X stands for "move right" and Y stands for "move up". The following diagrams show the case ''n'' = 4:
[[Image:Catalan number 4x4 grid example.svg.png|450px|center]]

* ''C''<sub>''n''</sub> is the number of different ways a [http://en.wikipedia.org/wiki/Convex_polygon '''convex polygon'''] with ''n'' + 2 sides can be cut into '''triangles''' by connecting vertices with straight lines. The following hexagons illustrate the case ''n'' = 4:
[[Image:Catalan-Hexagons-example.png|400px|center]]

* ''C''<sub>''n''</sub> is the number of [http://en.wikipedia.org/wiki/Stack_(data_structure) '''stack''']-sortable permutations of {1, ..., ''n''}. A permutation ''w'' is called '''stack-sortable''' if ''S''(''w'') = (1, ..., ''n''), where ''S''(''w'') is defined recursively as follows: write ''w'' = ''unv'' where ''n'' is the largest element in ''w'' and ''u'' and ''v'' are shorter sequences, and set ''S''(''w'') = ''S''(''u'')''S''(''v'')''n'', with ''S'' being the identity for one-element sequences.

* ''C''<sub>''n''</sub> is the number of ways to tile a stairstep shape of height ''n'' with ''n'' rectangles. The following figure illustrates the case ''n'' = 4:
[[Image:Catalan stairsteps 4.png|400px|center]]

{{Theorem|Recurrence relation for Catalan numbers|
:<math>C_0=0</math>, <math>C_1=1</math>, and for <math>n>1</math>,
::<math>
C_n=\sum_{i=1}^{n-1}C_iC_{n-i}.
</math>
}}

Let <math>G(x)=\sum_{n\ge 0}C_nx^n</math> be the generating function. Apply the product rule,
:<math>G(x)^2=\sum_{n\ge 0}\sum_{k=0}^{n}C_kC_{n-k}x^n=\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n</math>.
Due to the recurrence,
:<math>G(x)=\sum_{n\ge 0}C_nx^n=x+\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n=x+G(x)^2</math>.
Solving this, we obtain
:<math>G(x)=\frac{1\pm(1-4x)^{1/2}}{2}</math>.
Because <math>C_0=0</math>, it must hold that <math>G(x)=\frac{1-(1-4x)^{1/2}}{2}</math>, or otherwise the constant term is not zero. Expanding <math>(1-4x)^{1/2}</math> by Newton's formula, we have
:<math>
\begin{align}
G(x)
&=
\frac{1-(1-4x)^{1/2}}{2}\\
&=
1-\frac{1}{2}\sum_{n\ge 0}{1/2\choose n}(-4x)^n
\end{align}
</math>
Thus,
:<math>
\begin{align}
C_n
&=-\frac{1}{2}{1/2\choose n}(-4)^n\\
&=-\frac{1}{2}\cdot\frac{1}{2}\cdot\frac{-1}{2}\cdot\frac{-3}{2}\cdots\frac{-(2n-3)}{2}\cdot(-4)^n/n!\\
&=\frac{(2n-2)!}{(n-1)!n!}\\
&=\frac{1}{n}{2n-2\choose n-1}.
\end{align}
</math>
So we prove the following closed form for Catalan number.
{{Theorem|Theorem|
:<math>C_n=\frac{1}{n}{2n-2\choose n-1}</math>.
}}

Combinatorics (Fall 2010)/Generating functions

2010-09-12T01:42:03Z

172.21.3.5: /* Fibonacci numbers */

== Generating Functions ==
In Stanley's magnificent book ''Enumerative Combinatorics'', he comments the generating function as "the most useful but most difficult to understand method (for counting)".

The solution to a counting problem is usually represented as some <math>a_n</math> depending a parameter <math>n</math>. Sometimes this <math>a_n</math> is called a ''counting function'' as it is a function of the parameter <math>n</math>. <math>a_n</math> can also be treated as a infinite series:
:<math>a_0,a_1,a_2,\ldots</math>

The '''ordinary generating function (OGF)''' defined by <math>a_n</math> is
:<math>
G(x)=\sum_{n\ge 0} a_nx^n.
</math>

So <math>G(x)=a_0+a_1x+a_2x^2+\cdots</math>. An expression in this form is called a [http://en.wikipedia.org/wiki/Formal_power_series '''formal power series'''], and <math>a_0,a_1,a_2,\ldots</math> is the sequence of '''coefficients'''.

Furthermore, the generating function can be expanded as
:G(x)=<math>(\underbrace{1+\cdots+1}_{a_0})+(\underbrace{x+\cdots+x}_{a_1})+(\underbrace{x^2+\cdots+x^2}_{a_2})+\cdots+(\underbrace{x^n+\cdots+x^n}_{a_n})+\cdots</math>
so it indeed "generates" all the possible instances of the objects we want to count.

Usually, we do not evaluate the generating function <math>GF(x)</math> on any particular value. <math>x</math> remains as a '''formal variable''' without assuming any value. The numbers that we want to count are the coefficients carried by the terms in the formal power series. So far the generating function is just another way to represent the sequence
:<math>(a_0,a_1,a_2,\ldots\ldots)</math>.

The true power of generating functions comes from the various algebraic operations that we can perform on these generating functions. We use an example to demonstrate this.

=== Combinations ===

=== Fibonacci numbers ===
Consider the following counting problems.
* Count the number of ways that the nonnegative integer <math>n</math> can be written as a sum of ones and twos (in order).
: The problem asks for the number of compositions of <math>n</math> with summands from <math>\{1,2\}</math>. Formally, we are counting the number of tuples <math>(x_1,x_2,\ldots,x_k)</math> for some <math>k\le n</math> such that <math>x_i\in\{1,2\}</math> and <math>x_1+x_2+\cdots+x_k=n</math>.
: Let <math>F_n</math> be the solution. We observe that a composition either starts with a 1, in which case the rest is a composition of <math>n-1</math>; or starts with a 2, in which case the rest is a composition of <math>n-2</math>. So we have the recursion for <math>F_n</math> that
::<math>F_n=F_{n-1}+F_{n-2}</math>.
* Count the ways to completely cover a <math>2\times n</math> rectangle with <math>2\times 1</math> dominos without any overlaps.
: Dominos are identical <math>2\times 1</math> rectangles, so that only their orientations --- vertical or horizontal matter.
: Let <math>F_n</math> be the solution. It also holds that <math>F_n=F_{n-1}+F_{n-2}</math>. The proof is left as an exercise.

In both problems, the solution is given by <math>F_n</math> which satisfies the following recursion.
:<math>F_n=\begin{cases}
F_{n-1}+F_{n-2} & \mbox{if }n\ge 2,\\
1 & \mbox{if }n=1\\
0 & \mbox{if }n=0.
\end{cases}</math>

<math>F_n</math> is called the [http://en.wikipedia.org/wiki/Fibonacci_number Fibonacci number].

{{Theorem|Theorem|
::<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)</math>,
:where <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>.
}}
The quantity <math>\phi=\frac{1+\sqrt{5}}{2}</math> is the so-called [http://en.wikipedia.org/wiki/Golden_ratio golden ratio], a constant with some significance in mathematics and aesthetics.

We now prove this theorem by using generating functions.
The ordinary generating function for the Fibonacci number <math>F_{n}</math> is
:<math>G(x)=\sum_{n\ge 0}F_n x^n</math>.
We have that <math>F_{n}=F_{n-1}+F_{n-2}</math> for <math>n\ge 2</math>, thus
:<math>\begin{align}
G(x)
&=
\sum_{n\ge 0}F_n x^n
&=
x+\sum_{n\ge 2}(F_{n-1}+F_{n-2})x^n.
\end{align}
</math>
For generating functions, there are general ways to generate <math>F_{n-1}</math> and <math>F_{n-2}</math>, or the coefficients with any smaller indices.
:<math>
\begin{align}
xG(x)
&=\sum_{n\ge 0}F_n x^{n+1}=\sum_{n\ge 1}F_{n-1} x^n=\sum_{n\ge 2}F_{n-1} x^n\\
x^2G(x)
&=\sum_{n\ge 0}F_n x^{n+2}=\sum_{n\ge 2}F_{n-2} x^n.
\end{align}
</math>
So we have
:<math>G(x)=x+(x+x^2)G(x)\,</math>,
hence
:<math>G(x)=\frac{x}{1-x-x^2}</math>.
The value of <math>F_n</math> is the coefficient of <math>x^n</math> in the Taylor series for this formular, which is <math>\frac{G^{(n)}(0)}{n!}=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>. Although this expansion works in principle, the detailed calculus is rather painful.

----
It is easier to expand the generating function by breaking it into two geometric series.
{{Theorem|Proposition|
:Let <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>. It holds that
::<math>\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
}}

It is easy to verify the above equation, but to deduce it, we need some (high school) calculation.

{|border="2" width="100%" cellspacing="4" cellpadding="3" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse;empty-cells:show;"
|
:{|
|
<math>1-x-x^2</math> has two roots <math>\frac{-1\pm\sqrt{5}}{2}</math>.

Denote that <math>\phi=\frac{2}{-1+\sqrt{5}}=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{2}{-1-\sqrt{5}}=\frac{1-\sqrt{5}}{2}</math>.

Then <math>(1-x-x^2)=(1-\phi x)(1-\hat{\phi}x)</math>, so we can write
:<math>
\begin{align}
\frac{x}{1-x-x^2}
&=\frac{x}{(1-\phi x)(1-\hat{\phi} x)}\\
&=\frac{\alpha}{(1-\phi x)}+\frac{\beta}{(1-\hat{\phi} x)},
\end{align}
</math>
where <math>\alpha</math> and <math>\beta</math> satisfying that
:<math>\begin{cases}
\alpha+\beta=0\\
\alpha\phi+\beta\hat{\phi}= -1.
\end{cases}</math>
Solving this we have that <math>\alpha=\frac{1}{\sqrt{5}}</math> and <math>\beta=-\frac{1}{\sqrt{5}}</math>. Thus,
:<math>G(x)=\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
|}
:<math>\square</math>
|}

Note that the expression <math>\frac{1}{1-z}</math> has a well known geometric expansion:
:<math>\frac{1}{1-z}=\sum_{n\ge 0}z^n</math>.

Therefore, <math>G(x)</math> can be expanded as
:<math>
\begin{align}
G(x)
&=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}\\
&=\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\phi x)^n-\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\hat{\phi} x)^n\\
&=\sum_{n\ge 0}\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)x^n.
\end{align}</math>
So the <math>n</math>th Fibonacci number is given by
:<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>.

== Solving recurrences ==
The following steps describe a general methodology of solving recurrences by generating functions.
:1. Give a recursion that computes <math>a_n</math>. In the case of Fibonacci sequence
::<math>a_n=a_{n-1}+a_{n-2}</math>.
:2. Multiply both sides of the equation by <math>x^n</math> and sum over all <math>n</math>. This gives the generating function
::<math>G(x)=\sum_{n\ge 0}a_nx^n=\sum_{n\ge 0}(a_{n-1}+a_{n-2})x^n</math>.
:: And manipulate the right hand side of the equation so that it becomes some other expression involving <math>G(x)</math>.
::<math>G(x)=x+(x+x^2)G(x)\,</math>.
:3. Solve the resulting equation to derive an explicit formula for <math>G(x)</math>.
::<math>G(x)=\frac{x}{1-x-x^2}</math>.
:4. Expand <math>G(x)</math> into a power series and read off the coefficient of <math>x^n</math>, which is a closed form for <math>a_n</math>.

The first step is usually established by combinatorial observations, or explicitly given by the problem. The third step is trivial.

The second and the forth steps need some non-trivial analytic techniques.

=== Algebraic operations on generating functions ===
The second step in the above methodology is somehow tricky. It involves first applying the recurrence to the coefficients of <math>G(x)</math>, which is easy; and then manipulating the resulting formal power series to express it in terms of <math>G(x)</math>, which is more difficult (because it works backwards).

We can apply several natural algebraic operations on the formal power series.

{{Theorem|Generating function manipulation|
:Let <math>G(x)=\sum_{n\ge 0}g_nx^n</math> and <math>F(x)=\sum_{n\ge 0}f_nx^n</math>.
----

::<math>
\begin{align}
x^k G(x)
&= \sum_{n\ge k}g_{n-k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\frac{G(x)-\sum_{i=0}^{k-1}g_iz^i}{x^k}
&=\sum_{n\ge 0}g_{n+k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\alpha F(x)+\beta G(x)
&= \sum_{n\ge 0} (\alpha f_n+\beta g_n)x^n\\
F(x)G(x)
&= \sum_{n\ge 0}\sum_{k=0}^nf_kg_{n-k}x^n\\
G(cx)
&= \sum_{n\ge 0} c^ng_n x^n\\
G'(x)
&=
\sum_{n\ge 0}(n+1)g_{n+1}x^n
\end{align}
</math>
}}

When manipulating generating functions, these rules are applied backwards; that is, from the right-hand-side to the left-hand-side.

=== Expanding generating functions ===
The last step of solving recurrences by generating function is expanding the closed form generating function <math>G(x)</math> to evaluate its <math>n</math>-th coefficient. In principle, we can always use the [http://en.wikipedia.org/wiki/Taylor_series Taylor series]
:<math>G(x)=\sum_{n\ge 0}\frac{G^{(n)}(0)}{n!}x^n</math>,
where <math>G^{(n)}(0)</math> is the value of the <math>n</math>-th derivative of <math>G(x)</math> evaluated at <math>x=0</math>.

Some interesting special cases are very useful.

====Geometric sequence====
In the example of Fibonacci numbers, we use the well known geometric series:
:<math>\frac{1}{1-x}=\sum_{n\ge 0}x^n</math>.
It is useful when we can express the generating function in the form of <math>G(x)=\frac{a_1}{1-b_1x}+\frac{a_2}{1-b_2x}+\cdots+\frac{a_k}{1-b_kx}</math>. The coefficient of <math>x^n</math> in such <math>G(x)</math> is <math>a_1b_1^n+a_2b_2^n+\cdots+a_kb_k^n</math>.

====Binomial theorem====
The <math>n</math>-th derivative of <math>(1+x)^\alpha</math> for some real <math>\alpha</math> is
:<math>\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)(1+x)^{\alpha-n}</math>.
By Taylor series, we get a generalized version of the binomial theorem known as [http://en.wikipedia.org/wiki/Binomial_coefficient#Newton.27s_binomial_series '''Newton's formula''']:
{{Theorem|Newton's formular (generalized binomial theorem)|
If <math>|x|<1</math>, then
:<math>(1+x)^\alpha=\sum_{n\ge 0}{\alpha\choose n}x^{n}</math>,
where <math>{\alpha\choose n}</math> is the '''generalized binomial coefficient''' defined by
:<math>{\alpha\choose n}=\frac{\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)}{n!}</math>.
}}

=== Example: multisets ===
In the last lecture we gave a combinatorial proof of the number of <math>k</math>-multisets on an <math>n</math>-set. Now we give a generating function approach to the problem.

Let <math>S=\{x_1,x_2,\ldots,x_n\}</math> be an <math>n</math>-element set. We have
:<math>(1+x_1+x_1^2+\cdots)(1+x_2+x_2^2+\cdots)\cdots(1+x_n+x_n^2+\cdots)=\sum_{m:S\rightarrow\mathbb{N}} \prod_{x_i\in S}x_i^{m(x_i)}</math>,
where each <math>m:S\rightarrow\mathbb{N}</math> species a possible multiset on <math>S</math> with multiplicity function <math>m</math>.

Let all <math>x_i=x</math>. Then
:<math>
\begin{align}
(1+x+x^2+\cdots)^n
&=
\sum_{m:S\rightarrow\mathbb{N}}x^{m(x_1)+\cdots+m(x_n)}\\
&=
\sum_{\text{multiset }M\text{ on }S}x^{|M|}\\
&=
\sum_{k\ge 0}\left({n\choose k}\right)x^k.
\end{align}
</math>
The last equation is due to the the definition of <math>\left({n\choose k}\right)</math>. Our task is to evaluate <math>\left({n\choose k}\right)</math>.

Due to the geometric sequence and the Newton's formula
:<math>
(1+x+x^2+\cdots)^n=(1-x)^{-n}=\sum_{k\ge 0}{-n\choose k}(-x)^k.
</math>
So
:<math>
\left({n\choose k}\right)=(-1)^k{-n\choose k}={n+k-1\choose k}.
</math>
The last equation is due to the definition of the generalized binomial coefficient. We use an analytic (generating function) proof to get the same result of <math>\left({n\choose k}\right)</math> as the combinatorial proof.

== Pólya's problem of changing money ==

== Catalan Number ==
We now introduce a class of counting problems, all with the same solution, called [http://en.wikipedia.org/wiki/Catalan_number '''Catalan number'''].

The <math>n</math>th Catalan number is denoted as <math>C_n</math>.
In Volume 2 of Stanley's ''Enumerative Combinatorics'', a set of exercises describe 66 different interpretations of the Catalan numbers. We give a few examples, cited from Wikipedia.
* ''C''<sub>''n''</sub> is the number of '''Dyck words''' of length 2''n''. A Dyck word is a string consisting of ''n'' X's and ''n'' Y's such that no initial segment of the string has more Y's than X's (see also [http://en.wikipedia.org/wiki/Dyck_language Dyck language]). For example, the following are the Dyck words of length 6:
<div class="center"><big> XXXYYY     XYXXYY     XYXYXY     XXYYXY     XXYXYY.</big></div>

* Re-interpreting the symbol X as an open parenthesis and Y as a close parenthesis, ''C''<sub>''n''</sub> counts the number of expressions containing ''n'' pairs of parentheses which are correctly matched:
<div class="center"><big> ((()))     ()(())     ()()()     (())()     (()()) </big></div>

* ''C''<sub>''n''</sub> is the number of different ways ''n'' + 1 factors can be completely parenthesized (or the number of ways of associating ''n'' applications of a '''binary operator'''). For ''n'' = 3, for example, we have the following five different parenthesizations of four factors:
<div class="center"><math>((ab)c)d \quad (a(bc))d \quad(ab)(cd) \quad a((bc)d) \quad a(b(cd))</math></div>

* Successive applications of a binary operator can be represented in terms of a '''full binary tree'''. (A rooted binary tree is ''full'' if every vertex has either two children or no children.) It follows that ''C''<sub>''n''</sub> is the number of full binary trees with ''n'' + 1 leaves:
[[Image:Catalan number binary tree example.png|center]]

* ''C''<sub>''n''</sub> is the number of '''monotonic paths''' along the edges of a grid with ''n'' × ''n'' square cells, which do not pass above the diagonal. A monotonic path is one which starts in the lower left corner, finishes in the upper right corner, and consists entirely of edges pointing rightwards or upwards. Counting such paths is equivalent to counting Dyck words: X stands for "move right" and Y stands for "move up". The following diagrams show the case ''n'' = 4:
[[Image:Catalan number 4x4 grid example.svg.png|450px|center]]

* ''C''<sub>''n''</sub> is the number of different ways a [http://en.wikipedia.org/wiki/Convex_polygon '''convex polygon'''] with ''n'' + 2 sides can be cut into '''triangles''' by connecting vertices with straight lines. The following hexagons illustrate the case ''n'' = 4:
[[Image:Catalan-Hexagons-example.png|400px|center]]

* ''C''<sub>''n''</sub> is the number of [http://en.wikipedia.org/wiki/Stack_(data_structure) '''stack''']-sortable permutations of {1, ..., ''n''}. A permutation ''w'' is called '''stack-sortable''' if ''S''(''w'') = (1, ..., ''n''), where ''S''(''w'') is defined recursively as follows: write ''w'' = ''unv'' where ''n'' is the largest element in ''w'' and ''u'' and ''v'' are shorter sequences, and set ''S''(''w'') = ''S''(''u'')''S''(''v'')''n'', with ''S'' being the identity for one-element sequences.

* ''C''<sub>''n''</sub> is the number of ways to tile a stairstep shape of height ''n'' with ''n'' rectangles. The following figure illustrates the case ''n'' = 4:
[[Image:Catalan stairsteps 4.png|400px|center]]

{{Theorem|Recurrence relation for Catalan numbers|
:<math>C_0=0</math>, <math>C_1=1</math>, and for <math>n>1</math>,
::<math>
C_n=\sum_{i=1}^{n-1}C_iC_{n-i}.
</math>
}}

Let <math>G(x)=\sum_{n\ge 0}C_nx^n</math> be the generating function. Apply the product rule,
:<math>G(x)^2=\sum_{n\ge 0}\sum_{k=0}^{n}C_kC_{n-k}x^n=\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n</math>.
Due to the recurrence,
:<math>G(x)=\sum_{n\ge 0}C_nx^n=x+\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n=x+G(x)^2</math>.
Solving this, we obtain
:<math>G(x)=\frac{1\pm(1-4x)^{1/2}}{2}</math>.
Because <math>C_0=0</math>, it must hold that <math>G(x)=\frac{1-(1-4x)^{1/2}}{2}</math>, or otherwise the constant term is not zero. Expanding <math>(1-4x)^{1/2}</math> by Newton's formula, we have
:<math>
\begin{align}
G(x)
&=
\frac{1-(1-4x)^{1/2}}{2}\\
&=
1-\frac{1}{2}\sum_{n\ge 0}{1/2\choose n}(-4x)^n
\end{align}
</math>
Thus,
:<math>
\begin{align}
C_n
&=-\frac{1}{2}{1/2\choose n}(-4)^n\\
&=-\frac{1}{2}\cdot\frac{1}{2}\cdot\frac{-1}{2}\cdot\frac{-3}{2}\cdots\frac{-(2n-3)}{2}\cdot(-4)^n/n!\\
&=\frac{(2n-2)!}{(n-1)!n!}\\
&=\frac{1}{n}{2n-2\choose n-1}.
\end{align}
</math>
So we prove the following closed form for Catalan number.
{{Theorem|Theorem|
:<math>C_n=\frac{1}{n}{2n-2\choose n-1}</math>.
}}

Combinatorics (Fall 2010)/Generating functions

2010-09-12T01:37:02Z

172.21.3.5: /* Solving recurrences */

== Generating Functions ==
In Stanley's magnificent book ''Enumerative Combinatorics'', he comments the generating function as "the most useful but most difficult to understand method (for counting)".

The solution to a counting problem is usually represented as some <math>a_n</math> depending a parameter <math>n</math>. Sometimes this <math>a_n</math> is called a ''counting function'' as it is a function of the parameter <math>n</math>. <math>a_n</math> can also be treated as a infinite series:
:<math>a_0,a_1,a_2,\ldots</math>

The '''ordinary generating function (OGF)''' defined by <math>a_n</math> is
:<math>
G(x)=\sum_{n\ge 0} a_nx^n.
</math>

So <math>G(x)=a_0+a_1x+a_2x^2+\cdots</math>. An expression in this form is called a [http://en.wikipedia.org/wiki/Formal_power_series '''formal power series'''], and <math>a_0,a_1,a_2,\ldots</math> is the sequence of '''coefficients'''.

Furthermore, the generating function can be expanded as
:G(x)=<math>(\underbrace{1+\cdots+1}_{a_0})+(\underbrace{x+\cdots+x}_{a_1})+(\underbrace{x^2+\cdots+x^2}_{a_2})+\cdots+(\underbrace{x^n+\cdots+x^n}_{a_n})+\cdots</math>
so it indeed "generates" all the possible instances of the objects we want to count.

Usually, we do not evaluate the generating function <math>GF(x)</math> on any particular value. <math>x</math> remains as a '''formal variable''' without assuming any value. The numbers that we want to count are the coefficients carried by the terms in the formal power series. So far the generating function is just another way to represent the sequence
:<math>(a_0,a_1,a_2,\ldots\ldots)</math>.

The true power of generating functions comes from the various algebraic operations that we can perform on these generating functions. We use an example to demonstrate this.

=== Fibonacci numbers ===
Consider the following counting problems.
* Count the number of ways that the nonnegative integer <math>n</math> can be written as a sum of ones and twos (in order).
: The problem asks for the number of compositions of <math>n</math> with summands from <math>\{1,2\}</math>. Formally, we are counting the number of tuples <math>(x_1,x_2,\ldots,x_k)</math> for some <math>k\le n</math> such that <math>x_i\in\{1,2\}</math> and <math>x_1+x_2+\cdots+x_k=n</math>.
: Let <math>F_n</math> be the solution. We observe that a composition either starts with a 1, in which case the rest is a composition of <math>n-1</math>; or starts with a 2, in which case the rest is a composition of <math>n-2</math>. So we have the recursion for <math>F_n</math> that
::<math>F_n=F_{n-1}+F_{n-2}</math>.
* Count the ways to completely cover a <math>2\times n</math> rectangle with <math>2\times 1</math> dominos without any overlaps.
: Dominos are identical <math>2\times 1</math> rectangles, so that only their orientations --- vertical or horizontal matter.
: Let <math>F_n</math> be the solution. It also holds that <math>F_n=F_{n-1}+F_{n-2}</math>. The proof is left as an exercise.

In both problems, the solution is given by <math>F_n</math> which satisfies the following recursion.
:<math>F_n=\begin{cases}
F_{n-1}+F_{n-2} & \mbox{if }n\ge 2,\\
1 & \mbox{if }n=1\\
0 & \mbox{if }n=0.
\end{cases}</math>

<math>F_n</math> is called the [http://en.wikipedia.org/wiki/Fibonacci_number Fibonacci number].

{{Theorem|Theorem|
::<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)</math>,
:where <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>.
}}
The quantity <math>\phi=\frac{1+\sqrt{5}}{2}</math> is the so-called [http://en.wikipedia.org/wiki/Golden_ratio golden ratio], a constant with some significance in mathematics and aesthetics.

We now prove this theorem by using generating functions.
The ordinary generating function for the Fibonacci number <math>F_{n}</math> is
:<math>G(x)=\sum_{n\ge 0}F_n x^n</math>.
We have that <math>F_{n}=F_{n-1}+F_{n-2}</math> for <math>n\ge 2</math>, thus
:<math>\begin{align}
G(x)
&=
\sum_{n\ge 0}F_n x^n
&=
x+\sum_{n\ge 2}(F_{n-1}+F_{n-2})x^n.
\end{align}
</math>
For generating functions, there are general ways to generate <math>F_{n-1}</math> and <math>F_{n-2}</math>, or the coefficients with any smaller indices.
:<math>
\begin{align}
xG(x)
&=\sum_{n\ge 0}F_n x^{n+1}=\sum_{n\ge 1}F_{n-1} x^n=\sum_{n\ge 2}F_{n-1} x^n\\
x^2G(x)
&=\sum_{n\ge 0}F_n x^{n+2}=\sum_{n\ge 2}F_{n-2} x^n.
\end{align}
</math>
So we have
:<math>G(x)=x+(x+x^2)G(x)\,</math>,
hence
:<math>G(x)=\frac{x}{1-x-x^2}</math>.
The value of <math>F_n</math> is the coefficient of <math>x^n</math> in the Taylor series for this formular, which is <math>\frac{G^{(n)}(0)}{n!}=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>. Although this expansion works in principle, the detailed calculus is rather painful.

----
It is easier to expand the generating function by breaking it into two geometric series.
{{Theorem|Proposition|
:Let <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>. It holds that
::<math>\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
}}

It is easy to verify the above equation, but to deduce it, we need some (high school) calculation.

{|border="2" width="100%" cellspacing="4" cellpadding="3" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse;empty-cells:show;"
|
:{|
|
<math>1-x-x^2</math> has two roots <math>\frac{-1\pm\sqrt{5}}{2}</math>.

Denote that <math>\phi=\frac{2}{-1+\sqrt{5}}=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{2}{-1-\sqrt{5}}=\frac{1-\sqrt{5}}{2}</math>.

Then <math>(1-x-x^2)=(1-\phi x)(1-\hat{\phi}x)</math>, so we can write
:<math>
\begin{align}
\frac{x}{1-x-x^2}
&=\frac{x}{(1-\phi x)(1-\hat{\phi} x)}\\
&=\frac{\alpha}{(1-\phi x)}+\frac{\beta}{(1-\hat{\phi} x)},
\end{align}
</math>
where <math>\alpha</math> and <math>\beta</math> satisfying that
:<math>\begin{cases}
\alpha+\beta=0\\
\alpha\phi+\beta\hat{\phi}= -1.
\end{cases}</math>
Solving this we have that <math>\alpha=\frac{1}{\sqrt{5}}</math> and <math>\beta=-\frac{1}{\sqrt{5}}</math>. Thus,
:<math>G(x)=\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
|}
:<math>\square</math>
|}

Note that the expression <math>\frac{1}{1-z}</math> has a well known geometric expansion:
:<math>\frac{1}{1-z}=\sum_{n\ge 0}z^n</math>.

Therefore, <math>G(x)</math> can be expanded as
:<math>
\begin{align}
G(x)
&=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}\\
&=\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\phi x)^n-\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\hat{\phi} x)^n\\
&=\sum_{n\ge 0}\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)x^n.
\end{align}</math>
So the <math>n</math>th Fibonacci number is given by
:<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>.

== Solving recurrences ==
The following steps describe a general methodology of solving recurrences by generating functions.
:1. Give a recursion that computes <math>a_n</math>. In the case of Fibonacci sequence
::<math>a_n=a_{n-1}+a_{n-2}</math>.
:2. Multiply both sides of the equation by <math>x^n</math> and sum over all <math>n</math>. This gives the generating function
::<math>G(x)=\sum_{n\ge 0}a_nx^n=\sum_{n\ge 0}(a_{n-1}+a_{n-2})x^n</math>.
:: And manipulate the right hand side of the equation so that it becomes some other expression involving <math>G(x)</math>.
::<math>G(x)=x+(x+x^2)G(x)\,</math>.
:3. Solve the resulting equation to derive an explicit formula for <math>G(x)</math>.
::<math>G(x)=\frac{x}{1-x-x^2}</math>.
:4. Expand <math>G(x)</math> into a power series and read off the coefficient of <math>x^n</math>, which is a closed form for <math>a_n</math>.

The first step is usually established by combinatorial observations, or explicitly given by the problem. The third step is trivial.

The second and the forth steps need some non-trivial analytic techniques.

=== Algebraic operations on generating functions ===
The second step in the above methodology is somehow tricky. It involves first applying the recurrence to the coefficients of <math>G(x)</math>, which is easy; and then manipulating the resulting formal power series to express it in terms of <math>G(x)</math>, which is more difficult (because it works backwards).

We can apply several natural algebraic operations on the formal power series.

{{Theorem|Generating function manipulation|
:Let <math>G(x)=\sum_{n\ge 0}g_nx^n</math> and <math>F(x)=\sum_{n\ge 0}f_nx^n</math>.
----

::<math>
\begin{align}
x^k G(x)
&= \sum_{n\ge k}g_{n-k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\frac{G(x)-\sum_{i=0}^{k-1}g_iz^i}{x^k}
&=\sum_{n\ge 0}g_{n+k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\alpha F(x)+\beta G(x)
&= \sum_{n\ge 0} (\alpha f_n+\beta g_n)x^n\\
F(x)G(x)
&= \sum_{n\ge 0}\sum_{k=0}^nf_kg_{n-k}x^n\\
G(cx)
&= \sum_{n\ge 0} c^ng_n x^n\\
G'(x)
&=
\sum_{n\ge 0}(n+1)g_{n+1}x^n
\end{align}
</math>
}}

When manipulating generating functions, these rules are applied backwards; that is, from the right-hand-side to the left-hand-side.

=== Expanding generating functions ===
The last step of solving recurrences by generating function is expanding the closed form generating function <math>G(x)</math> to evaluate its <math>n</math>-th coefficient. In principle, we can always use the [http://en.wikipedia.org/wiki/Taylor_series Taylor series]
:<math>G(x)=\sum_{n\ge 0}\frac{G^{(n)}(0)}{n!}x^n</math>,
where <math>G^{(n)}(0)</math> is the value of the <math>n</math>-th derivative of <math>G(x)</math> evaluated at <math>x=0</math>.

Some interesting special cases are very useful.

====Geometric sequence====
In the example of Fibonacci numbers, we use the well known geometric series:
:<math>\frac{1}{1-x}=\sum_{n\ge 0}x^n</math>.
It is useful when we can express the generating function in the form of <math>G(x)=\frac{a_1}{1-b_1x}+\frac{a_2}{1-b_2x}+\cdots+\frac{a_k}{1-b_kx}</math>. The coefficient of <math>x^n</math> in such <math>G(x)</math> is <math>a_1b_1^n+a_2b_2^n+\cdots+a_kb_k^n</math>.

====Binomial theorem====
The <math>n</math>-th derivative of <math>(1+x)^\alpha</math> for some real <math>\alpha</math> is
:<math>\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)(1+x)^{\alpha-n}</math>.
By Taylor series, we get a generalized version of the binomial theorem known as [http://en.wikipedia.org/wiki/Binomial_coefficient#Newton.27s_binomial_series '''Newton's formula''']:
{{Theorem|Newton's formular (generalized binomial theorem)|
If <math>|x|<1</math>, then
:<math>(1+x)^\alpha=\sum_{n\ge 0}{\alpha\choose n}x^{n}</math>,
where <math>{\alpha\choose n}</math> is the '''generalized binomial coefficient''' defined by
:<math>{\alpha\choose n}=\frac{\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)}{n!}</math>.
}}

=== Example: multisets ===
In the last lecture we gave a combinatorial proof of the number of <math>k</math>-multisets on an <math>n</math>-set. Now we give a generating function approach to the problem.

Let <math>S=\{x_1,x_2,\ldots,x_n\}</math> be an <math>n</math>-element set. We have
:<math>(1+x_1+x_1^2+\cdots)(1+x_2+x_2^2+\cdots)\cdots(1+x_n+x_n^2+\cdots)=\sum_{m:S\rightarrow\mathbb{N}} \prod_{x_i\in S}x_i^{m(x_i)}</math>,
where each <math>m:S\rightarrow\mathbb{N}</math> species a possible multiset on <math>S</math> with multiplicity function <math>m</math>.

Let all <math>x_i=x</math>. Then
:<math>
\begin{align}
(1+x+x^2+\cdots)^n
&=
\sum_{m:S\rightarrow\mathbb{N}}x^{m(x_1)+\cdots+m(x_n)}\\
&=
\sum_{\text{multiset }M\text{ on }S}x^{|M|}\\
&=
\sum_{k\ge 0}\left({n\choose k}\right)x^k.
\end{align}
</math>
The last equation is due to the the definition of <math>\left({n\choose k}\right)</math>. Our task is to evaluate <math>\left({n\choose k}\right)</math>.

Due to the geometric sequence and the Newton's formula
:<math>
(1+x+x^2+\cdots)^n=(1-x)^{-n}=\sum_{k\ge 0}{-n\choose k}(-x)^k.
</math>
So
:<math>
\left({n\choose k}\right)=(-1)^k{-n\choose k}={n+k-1\choose k}.
</math>
The last equation is due to the definition of the generalized binomial coefficient. We use an analytic (generating function) proof to get the same result of <math>\left({n\choose k}\right)</math> as the combinatorial proof.

== Pólya's problem of changing money ==

== Catalan Number ==
We now introduce a class of counting problems, all with the same solution, called [http://en.wikipedia.org/wiki/Catalan_number '''Catalan number'''].

The <math>n</math>th Catalan number is denoted as <math>C_n</math>.
In Volume 2 of Stanley's ''Enumerative Combinatorics'', a set of exercises describe 66 different interpretations of the Catalan numbers. We give a few examples, cited from Wikipedia.
* ''C''<sub>''n''</sub> is the number of '''Dyck words''' of length 2''n''. A Dyck word is a string consisting of ''n'' X's and ''n'' Y's such that no initial segment of the string has more Y's than X's (see also [http://en.wikipedia.org/wiki/Dyck_language Dyck language]). For example, the following are the Dyck words of length 6:
<div class="center"><big> XXXYYY     XYXXYY     XYXYXY     XXYYXY     XXYXYY.</big></div>

* Re-interpreting the symbol X as an open parenthesis and Y as a close parenthesis, ''C''<sub>''n''</sub> counts the number of expressions containing ''n'' pairs of parentheses which are correctly matched:
<div class="center"><big> ((()))     ()(())     ()()()     (())()     (()()) </big></div>

* ''C''<sub>''n''</sub> is the number of different ways ''n'' + 1 factors can be completely parenthesized (or the number of ways of associating ''n'' applications of a '''binary operator'''). For ''n'' = 3, for example, we have the following five different parenthesizations of four factors:
<div class="center"><math>((ab)c)d \quad (a(bc))d \quad(ab)(cd) \quad a((bc)d) \quad a(b(cd))</math></div>

* Successive applications of a binary operator can be represented in terms of a '''full binary tree'''. (A rooted binary tree is ''full'' if every vertex has either two children or no children.) It follows that ''C''<sub>''n''</sub> is the number of full binary trees with ''n'' + 1 leaves:
[[Image:Catalan number binary tree example.png|center]]

* ''C''<sub>''n''</sub> is the number of '''monotonic paths''' along the edges of a grid with ''n'' × ''n'' square cells, which do not pass above the diagonal. A monotonic path is one which starts in the lower left corner, finishes in the upper right corner, and consists entirely of edges pointing rightwards or upwards. Counting such paths is equivalent to counting Dyck words: X stands for "move right" and Y stands for "move up". The following diagrams show the case ''n'' = 4:
[[Image:Catalan number 4x4 grid example.svg.png|450px|center]]

* ''C''<sub>''n''</sub> is the number of different ways a [http://en.wikipedia.org/wiki/Convex_polygon '''convex polygon'''] with ''n'' + 2 sides can be cut into '''triangles''' by connecting vertices with straight lines. The following hexagons illustrate the case ''n'' = 4:
[[Image:Catalan-Hexagons-example.png|400px|center]]

* ''C''<sub>''n''</sub> is the number of [http://en.wikipedia.org/wiki/Stack_(data_structure) '''stack''']-sortable permutations of {1, ..., ''n''}. A permutation ''w'' is called '''stack-sortable''' if ''S''(''w'') = (1, ..., ''n''), where ''S''(''w'') is defined recursively as follows: write ''w'' = ''unv'' where ''n'' is the largest element in ''w'' and ''u'' and ''v'' are shorter sequences, and set ''S''(''w'') = ''S''(''u'')''S''(''v'')''n'', with ''S'' being the identity for one-element sequences.

* ''C''<sub>''n''</sub> is the number of ways to tile a stairstep shape of height ''n'' with ''n'' rectangles. The following figure illustrates the case ''n'' = 4:
[[Image:Catalan stairsteps 4.png|400px|center]]

{{Theorem|Recurrence relation for Catalan numbers|
:<math>C_0=0</math>, <math>C_1=1</math>, and for <math>n>1</math>,
::<math>
C_n=\sum_{i=1}^{n-1}C_iC_{n-i}.
</math>
}}

Let <math>G(x)=\sum_{n\ge 0}C_nx^n</math> be the generating function. Apply the product rule,
:<math>G(x)^2=\sum_{n\ge 0}\sum_{k=0}^{n}C_kC_{n-k}x^n=\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n</math>.
Due to the recurrence,
:<math>G(x)=\sum_{n\ge 0}C_nx^n=x+\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n=x+G(x)^2</math>.
Solving this, we obtain
:<math>G(x)=\frac{1\pm(1-4x)^{1/2}}{2}</math>.
Because <math>C_0=0</math>, it must hold that <math>G(x)=\frac{1-(1-4x)^{1/2}}{2}</math>, or otherwise the constant term is not zero. Expanding <math>(1-4x)^{1/2}</math> by Newton's formula, we have
:<math>
\begin{align}
G(x)
&=
\frac{1-(1-4x)^{1/2}}{2}\\
&=
1-\frac{1}{2}\sum_{n\ge 0}{1/2\choose n}(-4x)^n
\end{align}
</math>
Thus,
:<math>
\begin{align}
C_n
&=-\frac{1}{2}{1/2\choose n}(-4)^n\\
&=-\frac{1}{2}\cdot\frac{1}{2}\cdot\frac{-1}{2}\cdot\frac{-3}{2}\cdots\frac{-(2n-3)}{2}\cdot(-4)^n/n!\\
&=\frac{(2n-2)!}{(n-1)!n!}\\
&=\frac{1}{n}{2n-2\choose n-1}.
\end{align}
</math>
So we prove the following closed form for Catalan number.
{{Theorem|Theorem|
:<math>C_n=\frac{1}{n}{2n-2\choose n-1}</math>.
}}

Combinatorics (Fall 2010)/Generating functions

2010-09-12T01:32:14Z

172.21.3.5: /* Solving recurrences */

== Generating Functions ==
In Stanley's magnificent book ''Enumerative Combinatorics'', he comments the generating function as "the most useful but most difficult to understand method (for counting)".

The solution to a counting problem is usually represented as some <math>a_n</math> depending a parameter <math>n</math>. Sometimes this <math>a_n</math> is called a ''counting function'' as it is a function of the parameter <math>n</math>. <math>a_n</math> can also be treated as a infinite series:
:<math>a_0,a_1,a_2,\ldots</math>

The '''ordinary generating function (OGF)''' defined by <math>a_n</math> is
:<math>
G(x)=\sum_{n\ge 0} a_nx^n.
</math>

So <math>G(x)=a_0+a_1x+a_2x^2+\cdots</math>. An expression in this form is called a [http://en.wikipedia.org/wiki/Formal_power_series '''formal power series'''], and <math>a_0,a_1,a_2,\ldots</math> is the sequence of '''coefficients'''.

Furthermore, the generating function can be expanded as
:G(x)=<math>(\underbrace{1+\cdots+1}_{a_0})+(\underbrace{x+\cdots+x}_{a_1})+(\underbrace{x^2+\cdots+x^2}_{a_2})+\cdots+(\underbrace{x^n+\cdots+x^n}_{a_n})+\cdots</math>
so it indeed "generates" all the possible instances of the objects we want to count.

Usually, we do not evaluate the generating function <math>GF(x)</math> on any particular value. <math>x</math> remains as a '''formal variable''' without assuming any value. The numbers that we want to count are the coefficients carried by the terms in the formal power series. So far the generating function is just another way to represent the sequence
:<math>(a_0,a_1,a_2,\ldots\ldots)</math>.

The true power of generating functions comes from the various algebraic operations that we can perform on these generating functions. We use an example to demonstrate this.

=== Fibonacci numbers ===
Consider the following counting problems.
* Count the number of ways that the nonnegative integer <math>n</math> can be written as a sum of ones and twos (in order).
: The problem asks for the number of compositions of <math>n</math> with summands from <math>\{1,2\}</math>. Formally, we are counting the number of tuples <math>(x_1,x_2,\ldots,x_k)</math> for some <math>k\le n</math> such that <math>x_i\in\{1,2\}</math> and <math>x_1+x_2+\cdots+x_k=n</math>.
: Let <math>F_n</math> be the solution. We observe that a composition either starts with a 1, in which case the rest is a composition of <math>n-1</math>; or starts with a 2, in which case the rest is a composition of <math>n-2</math>. So we have the recursion for <math>F_n</math> that
::<math>F_n=F_{n-1}+F_{n-2}</math>.
* Count the ways to completely cover a <math>2\times n</math> rectangle with <math>2\times 1</math> dominos without any overlaps.
: Dominos are identical <math>2\times 1</math> rectangles, so that only their orientations --- vertical or horizontal matter.
: Let <math>F_n</math> be the solution. It also holds that <math>F_n=F_{n-1}+F_{n-2}</math>. The proof is left as an exercise.

In both problems, the solution is given by <math>F_n</math> which satisfies the following recursion.
:<math>F_n=\begin{cases}
F_{n-1}+F_{n-2} & \mbox{if }n\ge 2,\\
1 & \mbox{if }n=1\\
0 & \mbox{if }n=0.
\end{cases}</math>

<math>F_n</math> is called the [http://en.wikipedia.org/wiki/Fibonacci_number Fibonacci number].

{{Theorem|Theorem|
::<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)</math>,
:where <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>.
}}
The quantity <math>\phi=\frac{1+\sqrt{5}}{2}</math> is the so-called [http://en.wikipedia.org/wiki/Golden_ratio golden ratio], a constant with some significance in mathematics and aesthetics.

We now prove this theorem by using generating functions.
The ordinary generating function for the Fibonacci number <math>F_{n}</math> is
:<math>G(x)=\sum_{n\ge 0}F_n x^n</math>.
We have that <math>F_{n}=F_{n-1}+F_{n-2}</math> for <math>n\ge 2</math>, thus
:<math>\begin{align}
G(x)
&=
\sum_{n\ge 0}F_n x^n
&=
x+\sum_{n\ge 2}(F_{n-1}+F_{n-2})x^n.
\end{align}
</math>
For generating functions, there are general ways to generate <math>F_{n-1}</math> and <math>F_{n-2}</math>, or the coefficients with any smaller indices.
:<math>
\begin{align}
xG(x)
&=\sum_{n\ge 0}F_n x^{n+1}=\sum_{n\ge 1}F_{n-1} x^n=\sum_{n\ge 2}F_{n-1} x^n\\
x^2G(x)
&=\sum_{n\ge 0}F_n x^{n+2}=\sum_{n\ge 2}F_{n-2} x^n.
\end{align}
</math>
So we have
:<math>G(x)=x+(x+x^2)G(x)\,</math>,
hence
:<math>G(x)=\frac{x}{1-x-x^2}</math>.
The value of <math>F_n</math> is the coefficient of <math>x^n</math> in the Taylor series for this formular, which is <math>\frac{G^{(n)}(0)}{n!}=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>. Although this expansion works in principle, the detailed calculus is rather painful.

----
It is easier to expand the generating function by breaking it into two geometric series.
{{Theorem|Proposition|
:Let <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>. It holds that
::<math>\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
}}

It is easy to verify the above equation, but to deduce it, we need some (high school) calculation.

{|border="2" width="100%" cellspacing="4" cellpadding="3" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse;empty-cells:show;"
|
:{|
|
<math>1-x-x^2</math> has two roots <math>\frac{-1\pm\sqrt{5}}{2}</math>.

Denote that <math>\phi=\frac{2}{-1+\sqrt{5}}=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{2}{-1-\sqrt{5}}=\frac{1-\sqrt{5}}{2}</math>.

Then <math>(1-x-x^2)=(1-\phi x)(1-\hat{\phi}x)</math>, so we can write
:<math>
\begin{align}
\frac{x}{1-x-x^2}
&=\frac{x}{(1-\phi x)(1-\hat{\phi} x)}\\
&=\frac{\alpha}{(1-\phi x)}+\frac{\beta}{(1-\hat{\phi} x)},
\end{align}
</math>
where <math>\alpha</math> and <math>\beta</math> satisfying that
:<math>\begin{cases}
\alpha+\beta=0\\
\alpha\phi+\beta\hat{\phi}= -1.
\end{cases}</math>
Solving this we have that <math>\alpha=\frac{1}{\sqrt{5}}</math> and <math>\beta=-\frac{1}{\sqrt{5}}</math>. Thus,
:<math>G(x)=\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
|}
:<math>\square</math>
|}

Note that the expression <math>\frac{1}{1-z}</math> has a well known geometric expansion:
:<math>\frac{1}{1-z}=\sum_{n\ge 0}z^n</math>.

Therefore, <math>G(x)</math> can be expanded as
:<math>
\begin{align}
G(x)
&=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}\\
&=\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\phi x)^n-\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\hat{\phi} x)^n\\
&=\sum_{n\ge 0}\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)x^n.
\end{align}</math>
So the <math>n</math>th Fibonacci number is given by
:<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>.

== Solving recurrences ==
The following steps describe a general methodology of solving recurrences by generating functions.
:1. Give a recursion that computes <math>a_n</math>. In the case of Fibonacci sequence
::<math>a_n=a_{n-1}+a_{n-2}</math>.
:2. Multiply both sides of the equation by <math>x^n</math> and sum over all <math>n</math>. This gives the generating function
::<math>G(x)=\sum_{n\ge 0}a_nx^n=\sum_{n\ge 0}(a_{n-1}+a_{n-2})x^n</math>.
:: And manipulate the right hand side of the equation so that it becomes some other expression involving <math>G(x)</math>.
::<math>G(x)=x+(x+x^2)G(x)\,</math>.
:3. Solve the resulting equation to derive an explicit formula for <math>G(x)</math>.
::<math>G(x)=\frac{x}{1-x-x^2}</math>.
:4. Expand <math>G(x)</math> into a power series and read off the coefficient of <math>x^n</math>, which is a closed form for <math>a_n</math>.

=== Algebraic operations on generating functions ===
The second step in the above methodology is somehow tricky. It involves first applying the recurrence to the coefficients of <math>G(x)</math>, which is easy; and then manipulating the resulting formal power series to express it in terms of <math>G(x)</math>, which is more difficult (because it works backwards).

We can apply several natural algebraic operations on the formal power series.

{{Theorem|Generating function manipulation|
:Let <math>G(x)=\sum_{n\ge 0}g_nx^n</math> and <math>F(x)=\sum_{n\ge 0}f_nx^n</math>.
----

::<math>
\begin{align}
x^k G(x)
&= \sum_{n\ge k}g_{n-k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\frac{G(x)-\sum_{i=0}^{k-1}g_iz^i}{x^k}
&=\sum_{n\ge 0}g_{n+k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\alpha F(x)+\beta G(x)
&= \sum_{n\ge 0} (\alpha f_n+\beta g_n)x^n\\
F(x)G(x)
&= \sum_{n\ge 0}\sum_{k=0}^nf_kg_{n-k}x^n\\
G(cx)
&= \sum_{n\ge 0} c^ng_n x^n\\
G'(x)
&=
\sum_{n\ge 0}(n+1)g_{n+1}x^n
\end{align}
</math>
}}

When manipulating generating functions, these rules are applied backwards; that is, from the right-hand-side to the left-hand-side.

=== Expanding generating functions ===
The last step of solving recurrences by generating function is expanding the closed form generating function <math>G(x)</math> to evaluate its <math>n</math>-th coefficient. In principle, we can always use the [http://en.wikipedia.org/wiki/Taylor_series Taylor series]
:<math>G(x)=\sum_{n\ge 0}\frac{G^{(n)}(0)}{n!}x^n</math>,
where <math>G^{(n)}(0)</math> is the value of the <math>n</math>-th derivative of <math>G(x)</math> evaluated at <math>x=0</math>.

Some interesting special cases are very useful.

====Geometric sequence====
In the example of Fibonacci numbers, we use the well known geometric series:
:<math>\frac{1}{1-x}=\sum_{n\ge 0}x^n</math>.
It is useful when we can express the generating function in the form of <math>G(x)=\frac{a_1}{1-b_1x}+\frac{a_2}{1-b_2x}+\cdots+\frac{a_k}{1-b_kx}</math>. The coefficient of <math>x^n</math> in such <math>G(x)</math> is <math>a_1b_1^n+a_2b_2^n+\cdots+a_kb_k^n</math>.

====Binomial theorem====
The <math>n</math>-th derivative of <math>(1+x)^\alpha</math> for some real <math>\alpha</math> is
:<math>\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)(1+x)^{\alpha-n}</math>.
By Taylor series, we get a generalized version of the binomial theorem known as [http://en.wikipedia.org/wiki/Binomial_coefficient#Newton.27s_binomial_series '''Newton's formula''']:
{{Theorem|Newton's formular (generalized binomial theorem)|
If <math>|x|<1</math>, then
:<math>(1+x)^\alpha=\sum_{n\ge 0}{\alpha\choose n}x^{n}</math>,
where <math>{\alpha\choose n}</math> is the '''generalized binomial coefficient''' defined by
:<math>{\alpha\choose n}=\frac{\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)}{n!}</math>.
}}

=== Example: multisets ===
In the last lecture we gave a combinatorial proof of the number of <math>k</math>-multisets on an <math>n</math>-set. Now we give a generating function approach to the problem.

Let <math>S=\{x_1,x_2,\ldots,x_n\}</math> be an <math>n</math>-element set. We have
:<math>(1+x_1+x_1^2+\cdots)(1+x_2+x_2^2+\cdots)\cdots(1+x_n+x_n^2+\cdots)=\sum_{m:S\rightarrow\mathbb{N}} \prod_{x_i\in S}x_i^{m(x_i)}</math>,
where each <math>m:S\rightarrow\mathbb{N}</math> species a possible multiset on <math>S</math> with multiplicity function <math>m</math>.

Let all <math>x_i=x</math>. Then
:<math>
\begin{align}
(1+x+x^2+\cdots)^n
&=
\sum_{m:S\rightarrow\mathbb{N}}x^{m(x_1)+\cdots+m(x_n)}\\
&=
\sum_{\text{multiset }M\text{ on }S}x^{|M|}\\
&=
\sum_{k\ge 0}\left({n\choose k}\right)x^k.
\end{align}
</math>
The last equation is due to the the definition of <math>\left({n\choose k}\right)</math>. Our task is to evaluate <math>\left({n\choose k}\right)</math>.

Due to the geometric sequence and the Newton's formula
:<math>
(1+x+x^2+\cdots)^n=(1-x)^{-n}=\sum_{k\ge 0}{-n\choose k}(-x)^k.
</math>
So
:<math>
\left({n\choose k}\right)=(-1)^k{-n\choose k}={n+k-1\choose k}.
</math>
The last equation is due to the definition of the generalized binomial coefficient. We use an analytic (generating function) proof to get the same result of <math>\left({n\choose k}\right)</math> as the combinatorial proof.

== Pólya's problem of changing money ==

== Catalan Number ==
We now introduce a class of counting problems, all with the same solution, called [http://en.wikipedia.org/wiki/Catalan_number '''Catalan number'''].

The <math>n</math>th Catalan number is denoted as <math>C_n</math>.
In Volume 2 of Stanley's ''Enumerative Combinatorics'', a set of exercises describe 66 different interpretations of the Catalan numbers. We give a few examples, cited from Wikipedia.
* ''C''<sub>''n''</sub> is the number of '''Dyck words''' of length 2''n''. A Dyck word is a string consisting of ''n'' X's and ''n'' Y's such that no initial segment of the string has more Y's than X's (see also [http://en.wikipedia.org/wiki/Dyck_language Dyck language]). For example, the following are the Dyck words of length 6:
<div class="center"><big> XXXYYY     XYXXYY     XYXYXY     XXYYXY     XXYXYY.</big></div>

* Re-interpreting the symbol X as an open parenthesis and Y as a close parenthesis, ''C''<sub>''n''</sub> counts the number of expressions containing ''n'' pairs of parentheses which are correctly matched:
<div class="center"><big> ((()))     ()(())     ()()()     (())()     (()()) </big></div>

* ''C''<sub>''n''</sub> is the number of different ways ''n'' + 1 factors can be completely parenthesized (or the number of ways of associating ''n'' applications of a '''binary operator'''). For ''n'' = 3, for example, we have the following five different parenthesizations of four factors:
<div class="center"><math>((ab)c)d \quad (a(bc))d \quad(ab)(cd) \quad a((bc)d) \quad a(b(cd))</math></div>

* Successive applications of a binary operator can be represented in terms of a '''full binary tree'''. (A rooted binary tree is ''full'' if every vertex has either two children or no children.) It follows that ''C''<sub>''n''</sub> is the number of full binary trees with ''n'' + 1 leaves:
[[Image:Catalan number binary tree example.png|center]]

* ''C''<sub>''n''</sub> is the number of '''monotonic paths''' along the edges of a grid with ''n'' × ''n'' square cells, which do not pass above the diagonal. A monotonic path is one which starts in the lower left corner, finishes in the upper right corner, and consists entirely of edges pointing rightwards or upwards. Counting such paths is equivalent to counting Dyck words: X stands for "move right" and Y stands for "move up". The following diagrams show the case ''n'' = 4:
[[Image:Catalan number 4x4 grid example.svg.png|450px|center]]

* ''C''<sub>''n''</sub> is the number of different ways a [http://en.wikipedia.org/wiki/Convex_polygon '''convex polygon'''] with ''n'' + 2 sides can be cut into '''triangles''' by connecting vertices with straight lines. The following hexagons illustrate the case ''n'' = 4:
[[Image:Catalan-Hexagons-example.png|400px|center]]

* ''C''<sub>''n''</sub> is the number of [http://en.wikipedia.org/wiki/Stack_(data_structure) '''stack''']-sortable permutations of {1, ..., ''n''}. A permutation ''w'' is called '''stack-sortable''' if ''S''(''w'') = (1, ..., ''n''), where ''S''(''w'') is defined recursively as follows: write ''w'' = ''unv'' where ''n'' is the largest element in ''w'' and ''u'' and ''v'' are shorter sequences, and set ''S''(''w'') = ''S''(''u'')''S''(''v'')''n'', with ''S'' being the identity for one-element sequences.

* ''C''<sub>''n''</sub> is the number of ways to tile a stairstep shape of height ''n'' with ''n'' rectangles. The following figure illustrates the case ''n'' = 4:
[[Image:Catalan stairsteps 4.png|400px|center]]

{{Theorem|Recurrence relation for Catalan numbers|
:<math>C_0=0</math>, <math>C_1=1</math>, and for <math>n>1</math>,
::<math>
C_n=\sum_{i=1}^{n-1}C_iC_{n-i}.
</math>
}}

Let <math>G(x)=\sum_{n\ge 0}C_nx^n</math> be the generating function. Apply the product rule,
:<math>G(x)^2=\sum_{n\ge 0}\sum_{k=0}^{n}C_kC_{n-k}x^n=\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n</math>.
Due to the recurrence,
:<math>G(x)=\sum_{n\ge 0}C_nx^n=x+\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n=x+G(x)^2</math>.
Solving this, we obtain
:<math>G(x)=\frac{1\pm(1-4x)^{1/2}}{2}</math>.
Because <math>C_0=0</math>, it must hold that <math>G(x)=\frac{1-(1-4x)^{1/2}}{2}</math>, or otherwise the constant term is not zero. Expanding <math>(1-4x)^{1/2}</math> by Newton's formula, we have
:<math>
\begin{align}
G(x)
&=
\frac{1-(1-4x)^{1/2}}{2}\\
&=
1-\frac{1}{2}\sum_{n\ge 0}{1/2\choose n}(-4x)^n
\end{align}
</math>
Thus,
:<math>
\begin{align}
C_n
&=-\frac{1}{2}{1/2\choose n}(-4)^n\\
&=-\frac{1}{2}\cdot\frac{1}{2}\cdot\frac{-1}{2}\cdot\frac{-3}{2}\cdots\frac{-(2n-3)}{2}\cdot(-4)^n/n!\\
&=\frac{(2n-2)!}{(n-1)!n!}\\
&=\frac{1}{n}{2n-2\choose n-1}.
\end{align}
</math>
So we prove the following closed form for Catalan number.
{{Theorem|Theorem|
:<math>C_n=\frac{1}{n}{2n-2\choose n-1}</math>.
}}

Combinatorics (Fall 2010)/Generating functions

2010-09-12T01:24:04Z

172.21.3.5: /* Solving recurrences */

== Generating Functions ==
In Stanley's magnificent book ''Enumerative Combinatorics'', he comments the generating function as "the most useful but most difficult to understand method (for counting)".

The solution to a counting problem is usually represented as some <math>a_n</math> depending a parameter <math>n</math>. Sometimes this <math>a_n</math> is called a ''counting function'' as it is a function of the parameter <math>n</math>. <math>a_n</math> can also be treated as a infinite series:
:<math>a_0,a_1,a_2,\ldots</math>

The '''ordinary generating function (OGF)''' defined by <math>a_n</math> is
:<math>
G(x)=\sum_{n\ge 0} a_nx^n.
</math>

So <math>G(x)=a_0+a_1x+a_2x^2+\cdots</math>. An expression in this form is called a [http://en.wikipedia.org/wiki/Formal_power_series '''formal power series'''], and <math>a_0,a_1,a_2,\ldots</math> is the sequence of '''coefficients'''.

Furthermore, the generating function can be expanded as
:G(x)=<math>(\underbrace{1+\cdots+1}_{a_0})+(\underbrace{x+\cdots+x}_{a_1})+(\underbrace{x^2+\cdots+x^2}_{a_2})+\cdots+(\underbrace{x^n+\cdots+x^n}_{a_n})+\cdots</math>
so it indeed "generates" all the possible instances of the objects we want to count.

Usually, we do not evaluate the generating function <math>GF(x)</math> on any particular value. <math>x</math> remains as a '''formal variable''' without assuming any value. The numbers that we want to count are the coefficients carried by the terms in the formal power series. So far the generating function is just another way to represent the sequence
:<math>(a_0,a_1,a_2,\ldots\ldots)</math>.

The true power of generating functions comes from the various algebraic operations that we can perform on these generating functions. We use an example to demonstrate this.

=== Fibonacci numbers ===
Consider the following counting problems.
* Count the number of ways that the nonnegative integer <math>n</math> can be written as a sum of ones and twos (in order).
: The problem asks for the number of compositions of <math>n</math> with summands from <math>\{1,2\}</math>. Formally, we are counting the number of tuples <math>(x_1,x_2,\ldots,x_k)</math> for some <math>k\le n</math> such that <math>x_i\in\{1,2\}</math> and <math>x_1+x_2+\cdots+x_k=n</math>.
: Let <math>F_n</math> be the solution. We observe that a composition either starts with a 1, in which case the rest is a composition of <math>n-1</math>; or starts with a 2, in which case the rest is a composition of <math>n-2</math>. So we have the recursion for <math>F_n</math> that
::<math>F_n=F_{n-1}+F_{n-2}</math>.
* Count the ways to completely cover a <math>2\times n</math> rectangle with <math>2\times 1</math> dominos without any overlaps.
: Dominos are identical <math>2\times 1</math> rectangles, so that only their orientations --- vertical or horizontal matter.
: Let <math>F_n</math> be the solution. It also holds that <math>F_n=F_{n-1}+F_{n-2}</math>. The proof is left as an exercise.

In both problems, the solution is given by <math>F_n</math> which satisfies the following recursion.
:<math>F_n=\begin{cases}
F_{n-1}+F_{n-2} & \mbox{if }n\ge 2,\\
1 & \mbox{if }n=1\\
0 & \mbox{if }n=0.
\end{cases}</math>

<math>F_n</math> is called the [http://en.wikipedia.org/wiki/Fibonacci_number Fibonacci number].

{{Theorem|Theorem|
::<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)</math>,
:where <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>.
}}
The quantity <math>\phi=\frac{1+\sqrt{5}}{2}</math> is the so-called [http://en.wikipedia.org/wiki/Golden_ratio golden ratio], a constant with some significance in mathematics and aesthetics.

We now prove this theorem by using generating functions.
The ordinary generating function for the Fibonacci number <math>F_{n}</math> is
:<math>G(x)=\sum_{n\ge 0}F_n x^n</math>.
We have that <math>F_{n}=F_{n-1}+F_{n-2}</math> for <math>n\ge 2</math>, thus
:<math>\begin{align}
G(x)
&=
\sum_{n\ge 0}F_n x^n
&=
x+\sum_{n\ge 2}(F_{n-1}+F_{n-2})x^n.
\end{align}
</math>
For generating functions, there are general ways to generate <math>F_{n-1}</math> and <math>F_{n-2}</math>, or the coefficients with any smaller indices.
:<math>
\begin{align}
xG(x)
&=\sum_{n\ge 0}F_n x^{n+1}=\sum_{n\ge 1}F_{n-1} x^n=\sum_{n\ge 2}F_{n-1} x^n\\
x^2G(x)
&=\sum_{n\ge 0}F_n x^{n+2}=\sum_{n\ge 2}F_{n-2} x^n.
\end{align}
</math>
So we have
:<math>G(x)=x+(x+x^2)G(x)\,</math>,
hence
:<math>G(x)=\frac{x}{1-x-x^2}</math>.
The value of <math>F_n</math> is the coefficient of <math>x^n</math> in the Taylor series for this formular, which is <math>\frac{G^{(n)}(0)}{n!}=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>. Although this expansion works in principle, the detailed calculus is rather painful.

----
It is easier to expand the generating function by breaking it into two geometric series.
{{Theorem|Proposition|
:Let <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>. It holds that
::<math>\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
}}

It is easy to verify the above equation, but to deduce it, we need some (high school) calculation.

{|border="2" width="100%" cellspacing="4" cellpadding="3" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse;empty-cells:show;"
|
:{|
|
<math>1-x-x^2</math> has two roots <math>\frac{-1\pm\sqrt{5}}{2}</math>.

Denote that <math>\phi=\frac{2}{-1+\sqrt{5}}=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{2}{-1-\sqrt{5}}=\frac{1-\sqrt{5}}{2}</math>.

Then <math>(1-x-x^2)=(1-\phi x)(1-\hat{\phi}x)</math>, so we can write
:<math>
\begin{align}
\frac{x}{1-x-x^2}
&=\frac{x}{(1-\phi x)(1-\hat{\phi} x)}\\
&=\frac{\alpha}{(1-\phi x)}+\frac{\beta}{(1-\hat{\phi} x)},
\end{align}
</math>
where <math>\alpha</math> and <math>\beta</math> satisfying that
:<math>\begin{cases}
\alpha+\beta=0\\
\alpha\phi+\beta\hat{\phi}= -1.
\end{cases}</math>
Solving this we have that <math>\alpha=\frac{1}{\sqrt{5}}</math> and <math>\beta=-\frac{1}{\sqrt{5}}</math>. Thus,
:<math>G(x)=\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
|}
:<math>\square</math>
|}

Note that the expression <math>\frac{1}{1-z}</math> has a well known geometric expansion:
:<math>\frac{1}{1-z}=\sum_{n\ge 0}z^n</math>.

Therefore, <math>G(x)</math> can be expanded as
:<math>
\begin{align}
G(x)
&=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}\\
&=\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\phi x)^n-\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\hat{\phi} x)^n\\
&=\sum_{n\ge 0}\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)x^n.
\end{align}</math>
So the <math>n</math>th Fibonacci number is given by
:<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>.

== Solving recurrences ==
In the above analysis of Fibonacci numbers, we apply the following general methodology of solving recurrences by generating functions.
:1. Give a recursion that computes <math>a_n</math>, such as
::<math>a_n=f(a_0,a_1,\ldots,a_{n-1})</math> for some function <math>f</math>.
:2. Multiply both sides of the equation by <math>x^n</math> and sum over all <math>n</math>. This gives the generating function
::<math>G(x)=\sum_{n\ge 0}a_nx^n=\sum_{n\ge 0}f(a_0,a_1,\ldots,a_{n-1})x^n</math>.
:: And manipulate the right hand side of the equation so that it becomes some other expression involving <math>G(x)</math>.
:3. Solve the resulting equation to derive an explicit formula for <math>G(x)</math>.
:4. Expand <math>G(x)</math> into a power series and read off the coefficient of <math>x^n</math>, which is a closed form for <math>a_n</math>.

=== Algebraic operations on generating functions ===
The second step in the above methodology is somehow tricky. It involves first applying the recurrence to the coefficients of <math>G(x)</math>, which is easy; and then manipulating the resulting formal power series to express it in terms of <math>G(x)</math>, which is more difficult (because it works backwards).

We can apply several natural algebraic operations on the formal power series.

{{Theorem|Generating function manipulation|
:Let <math>G(x)=\sum_{n\ge 0}g_nx^n</math> and <math>F(x)=\sum_{n\ge 0}f_nx^n</math>.
----

::<math>
\begin{align}
x^k G(x)
&= \sum_{n\ge k}g_{n-k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\frac{G(x)-\sum_{i=0}^{k-1}g_iz^i}{x^k}
&=\sum_{n\ge 0}g_{n+k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\alpha F(x)+\beta G(x)
&= \sum_{n\ge 0} (\alpha f_n+\beta g_n)x^n\\
F(x)G(x)
&= \sum_{n\ge 0}\sum_{k=0}^nf_kg_{n-k}x^n\\
G(cx)
&= \sum_{n\ge 0} c^ng_n x^n\\
G'(x)
&=
\sum_{n\ge 0}(n+1)g_{n+1}x^n
\end{align}
</math>
}}

When manipulating generating functions, these rules are applied backwards; that is, from the right-hand-side to the left-hand-side.

=== Expanding generating functions ===
The last step of solving recurrences by generating function is expanding the closed form generating function <math>G(x)</math> to evaluate its <math>n</math>-th coefficient. In principle, we can always use the [http://en.wikipedia.org/wiki/Taylor_series Taylor series]
:<math>G(x)=\sum_{n\ge 0}\frac{G^{(n)}(0)}{n!}x^n</math>,
where <math>G^{(n)}(0)</math> is the value of the <math>n</math>-th derivative of <math>G(x)</math> evaluated at <math>x=0</math>.

Some interesting special cases are very useful.

====Geometric sequence====
In the example of Fibonacci numbers, we use the well known geometric series:
:<math>\frac{1}{1-x}=\sum_{n\ge 0}x^n</math>.
It is useful when we can express the generating function in the form of <math>G(x)=\frac{a_1}{1-b_1x}+\frac{a_2}{1-b_2x}+\cdots+\frac{a_k}{1-b_kx}</math>. The coefficient of <math>x^n</math> in such <math>G(x)</math> is <math>a_1b_1^n+a_2b_2^n+\cdots+a_kb_k^n</math>.

====Binomial theorem====
The <math>n</math>-th derivative of <math>(1+x)^\alpha</math> for some real <math>\alpha</math> is
:<math>\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)(1+x)^{\alpha-n}</math>.
By Taylor series, we get a generalized version of the binomial theorem known as [http://en.wikipedia.org/wiki/Binomial_coefficient#Newton.27s_binomial_series '''Newton's formula''']:
{{Theorem|Newton's formular (generalized binomial theorem)|
If <math>|x|<1</math>, then
:<math>(1+x)^\alpha=\sum_{n\ge 0}{\alpha\choose n}x^{n}</math>,
where <math>{\alpha\choose n}</math> is the '''generalized binomial coefficient''' defined by
:<math>{\alpha\choose n}=\frac{\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)}{n!}</math>.
}}

=== Example: multisets ===
In the last lecture we gave a combinatorial proof of the number of <math>k</math>-multisets on an <math>n</math>-set. Now we give a generating function approach to the problem.

Let <math>S=\{x_1,x_2,\ldots,x_n\}</math> be an <math>n</math>-element set. We have
:<math>(1+x_1+x_1^2+\cdots)(1+x_2+x_2^2+\cdots)\cdots(1+x_n+x_n^2+\cdots)=\sum_{m:S\rightarrow\mathbb{N}} \prod_{x_i\in S}x_i^{m(x_i)}</math>,
where each <math>m:S\rightarrow\mathbb{N}</math> species a possible multiset on <math>S</math> with multiplicity function <math>m</math>.

Let all <math>x_i=x</math>. Then
:<math>
\begin{align}
(1+x+x^2+\cdots)^n
&=
\sum_{m:S\rightarrow\mathbb{N}}x^{m(x_1)+\cdots+m(x_n)}\\
&=
\sum_{\text{multiset }M\text{ on }S}x^{|M|}\\
&=
\sum_{k\ge 0}\left({n\choose k}\right)x^k.
\end{align}
</math>
The last equation is due to the the definition of <math>\left({n\choose k}\right)</math>. Our task is to evaluate <math>\left({n\choose k}\right)</math>.

Due to the geometric sequence and the Newton's formula
:<math>
(1+x+x^2+\cdots)^n=(1-x)^{-n}=\sum_{k\ge 0}{-n\choose k}(-x)^k.
</math>
So
:<math>
\left({n\choose k}\right)=(-1)^k{-n\choose k}={n+k-1\choose k}.
</math>
The last equation is due to the definition of the generalized binomial coefficient. We use an analytic (generating function) proof to get the same result of <math>\left({n\choose k}\right)</math> as the combinatorial proof.

== Pólya's problem of changing money ==

== Catalan Number ==
We now introduce a class of counting problems, all with the same solution, called [http://en.wikipedia.org/wiki/Catalan_number '''Catalan number'''].

The <math>n</math>th Catalan number is denoted as <math>C_n</math>.
In Volume 2 of Stanley's ''Enumerative Combinatorics'', a set of exercises describe 66 different interpretations of the Catalan numbers. We give a few examples, cited from Wikipedia.
* ''C''<sub>''n''</sub> is the number of '''Dyck words''' of length 2''n''. A Dyck word is a string consisting of ''n'' X's and ''n'' Y's such that no initial segment of the string has more Y's than X's (see also [http://en.wikipedia.org/wiki/Dyck_language Dyck language]). For example, the following are the Dyck words of length 6:
<div class="center"><big> XXXYYY     XYXXYY     XYXYXY     XXYYXY     XXYXYY.</big></div>

* Re-interpreting the symbol X as an open parenthesis and Y as a close parenthesis, ''C''<sub>''n''</sub> counts the number of expressions containing ''n'' pairs of parentheses which are correctly matched:
<div class="center"><big> ((()))     ()(())     ()()()     (())()     (()()) </big></div>

* ''C''<sub>''n''</sub> is the number of different ways ''n'' + 1 factors can be completely parenthesized (or the number of ways of associating ''n'' applications of a '''binary operator'''). For ''n'' = 3, for example, we have the following five different parenthesizations of four factors:
<div class="center"><math>((ab)c)d \quad (a(bc))d \quad(ab)(cd) \quad a((bc)d) \quad a(b(cd))</math></div>

* Successive applications of a binary operator can be represented in terms of a '''full binary tree'''. (A rooted binary tree is ''full'' if every vertex has either two children or no children.) It follows that ''C''<sub>''n''</sub> is the number of full binary trees with ''n'' + 1 leaves:
[[Image:Catalan number binary tree example.png|center]]

* ''C''<sub>''n''</sub> is the number of '''monotonic paths''' along the edges of a grid with ''n'' × ''n'' square cells, which do not pass above the diagonal. A monotonic path is one which starts in the lower left corner, finishes in the upper right corner, and consists entirely of edges pointing rightwards or upwards. Counting such paths is equivalent to counting Dyck words: X stands for "move right" and Y stands for "move up". The following diagrams show the case ''n'' = 4:
[[Image:Catalan number 4x4 grid example.svg.png|450px|center]]

* ''C''<sub>''n''</sub> is the number of different ways a [http://en.wikipedia.org/wiki/Convex_polygon '''convex polygon'''] with ''n'' + 2 sides can be cut into '''triangles''' by connecting vertices with straight lines. The following hexagons illustrate the case ''n'' = 4:
[[Image:Catalan-Hexagons-example.png|400px|center]]

* ''C''<sub>''n''</sub> is the number of [http://en.wikipedia.org/wiki/Stack_(data_structure) '''stack''']-sortable permutations of {1, ..., ''n''}. A permutation ''w'' is called '''stack-sortable''' if ''S''(''w'') = (1, ..., ''n''), where ''S''(''w'') is defined recursively as follows: write ''w'' = ''unv'' where ''n'' is the largest element in ''w'' and ''u'' and ''v'' are shorter sequences, and set ''S''(''w'') = ''S''(''u'')''S''(''v'')''n'', with ''S'' being the identity for one-element sequences.

* ''C''<sub>''n''</sub> is the number of ways to tile a stairstep shape of height ''n'' with ''n'' rectangles. The following figure illustrates the case ''n'' = 4:
[[Image:Catalan stairsteps 4.png|400px|center]]

{{Theorem|Recurrence relation for Catalan numbers|
:<math>C_0=0</math>, <math>C_1=1</math>, and for <math>n>1</math>,
::<math>
C_n=\sum_{i=1}^{n-1}C_iC_{n-i}.
</math>
}}

Let <math>G(x)=\sum_{n\ge 0}C_nx^n</math> be the generating function. Apply the product rule,
:<math>G(x)^2=\sum_{n\ge 0}\sum_{k=0}^{n}C_kC_{n-k}x^n=\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n</math>.
Due to the recurrence,
:<math>G(x)=\sum_{n\ge 0}C_nx^n=x+\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n=x+G(x)^2</math>.
Solving this, we obtain
:<math>G(x)=\frac{1\pm(1-4x)^{1/2}}{2}</math>.
Because <math>C_0=0</math>, it must hold that <math>G(x)=\frac{1-(1-4x)^{1/2}}{2}</math>, or otherwise the constant term is not zero. Expanding <math>(1-4x)^{1/2}</math> by Newton's formula, we have
:<math>
\begin{align}
G(x)
&=
\frac{1-(1-4x)^{1/2}}{2}\\
&=
1-\frac{1}{2}\sum_{n\ge 0}{1/2\choose n}(-4x)^n
\end{align}
</math>
Thus,
:<math>
\begin{align}
C_n
&=-\frac{1}{2}{1/2\choose n}(-4)^n\\
&=-\frac{1}{2}\cdot\frac{1}{2}\cdot\frac{-1}{2}\cdot\frac{-3}{2}\cdots\frac{-(2n-3)}{2}\cdot(-4)^n/n!\\
&=\frac{(2n-2)!}{(n-1)!n!}\\
&=\frac{1}{n}{2n-2\choose n-1}.
\end{align}
</math>
So we prove the following closed form for Catalan number.
{{Theorem|Theorem|
:<math>C_n=\frac{1}{n}{2n-2\choose n-1}</math>.
}}

Combinatorics (Fall 2010)/Generating functions

2010-09-12T01:21:08Z

172.21.3.5: /* Fibonacci numbers */

== Generating Functions ==
In Stanley's magnificent book ''Enumerative Combinatorics'', he comments the generating function as "the most useful but most difficult to understand method (for counting)".

The solution to a counting problem is usually represented as some <math>a_n</math> depending a parameter <math>n</math>. Sometimes this <math>a_n</math> is called a ''counting function'' as it is a function of the parameter <math>n</math>. <math>a_n</math> can also be treated as a infinite series:
:<math>a_0,a_1,a_2,\ldots</math>

The '''ordinary generating function (OGF)''' defined by <math>a_n</math> is
:<math>
G(x)=\sum_{n\ge 0} a_nx^n.
</math>

So <math>G(x)=a_0+a_1x+a_2x^2+\cdots</math>. An expression in this form is called a [http://en.wikipedia.org/wiki/Formal_power_series '''formal power series'''], and <math>a_0,a_1,a_2,\ldots</math> is the sequence of '''coefficients'''.

Furthermore, the generating function can be expanded as
:G(x)=<math>(\underbrace{1+\cdots+1}_{a_0})+(\underbrace{x+\cdots+x}_{a_1})+(\underbrace{x^2+\cdots+x^2}_{a_2})+\cdots+(\underbrace{x^n+\cdots+x^n}_{a_n})+\cdots</math>
so it indeed "generates" all the possible instances of the objects we want to count.

Usually, we do not evaluate the generating function <math>GF(x)</math> on any particular value. <math>x</math> remains as a '''formal variable''' without assuming any value. The numbers that we want to count are the coefficients carried by the terms in the formal power series. So far the generating function is just another way to represent the sequence
:<math>(a_0,a_1,a_2,\ldots\ldots)</math>.

The true power of generating functions comes from the various algebraic operations that we can perform on these generating functions. We use an example to demonstrate this.

=== Fibonacci numbers ===
Consider the following counting problems.
* Count the number of ways that the nonnegative integer <math>n</math> can be written as a sum of ones and twos (in order).
: The problem asks for the number of compositions of <math>n</math> with summands from <math>\{1,2\}</math>. Formally, we are counting the number of tuples <math>(x_1,x_2,\ldots,x_k)</math> for some <math>k\le n</math> such that <math>x_i\in\{1,2\}</math> and <math>x_1+x_2+\cdots+x_k=n</math>.
: Let <math>F_n</math> be the solution. We observe that a composition either starts with a 1, in which case the rest is a composition of <math>n-1</math>; or starts with a 2, in which case the rest is a composition of <math>n-2</math>. So we have the recursion for <math>F_n</math> that
::<math>F_n=F_{n-1}+F_{n-2}</math>.
* Count the ways to completely cover a <math>2\times n</math> rectangle with <math>2\times 1</math> dominos without any overlaps.
: Dominos are identical <math>2\times 1</math> rectangles, so that only their orientations --- vertical or horizontal matter.
: Let <math>F_n</math> be the solution. It also holds that <math>F_n=F_{n-1}+F_{n-2}</math>. The proof is left as an exercise.

In both problems, the solution is given by <math>F_n</math> which satisfies the following recursion.
:<math>F_n=\begin{cases}
F_{n-1}+F_{n-2} & \mbox{if }n\ge 2,\\
1 & \mbox{if }n=1\\
0 & \mbox{if }n=0.
\end{cases}</math>

<math>F_n</math> is called the [http://en.wikipedia.org/wiki/Fibonacci_number Fibonacci number].

{{Theorem|Theorem|
::<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)</math>,
:where <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>.
}}
The quantity <math>\phi=\frac{1+\sqrt{5}}{2}</math> is the so-called [http://en.wikipedia.org/wiki/Golden_ratio golden ratio], a constant with some significance in mathematics and aesthetics.

We now prove this theorem by using generating functions.
The ordinary generating function for the Fibonacci number <math>F_{n}</math> is
:<math>G(x)=\sum_{n\ge 0}F_n x^n</math>.
We have that <math>F_{n}=F_{n-1}+F_{n-2}</math> for <math>n\ge 2</math>, thus
:<math>\begin{align}
G(x)
&=
\sum_{n\ge 0}F_n x^n
&=
x+\sum_{n\ge 2}(F_{n-1}+F_{n-2})x^n.
\end{align}
</math>
For generating functions, there are general ways to generate <math>F_{n-1}</math> and <math>F_{n-2}</math>, or the coefficients with any smaller indices.
:<math>
\begin{align}
xG(x)
&=\sum_{n\ge 0}F_n x^{n+1}=\sum_{n\ge 1}F_{n-1} x^n=\sum_{n\ge 2}F_{n-1} x^n\\
x^2G(x)
&=\sum_{n\ge 0}F_n x^{n+2}=\sum_{n\ge 2}F_{n-2} x^n.
\end{align}
</math>
So we have
:<math>G(x)=x+(x+x^2)G(x)\,</math>,
hence
:<math>G(x)=\frac{x}{1-x-x^2}</math>.
The value of <math>F_n</math> is the coefficient of <math>x^n</math> in the Taylor series for this formular, which is <math>\frac{G^{(n)}(0)}{n!}=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>. Although this expansion works in principle, the detailed calculus is rather painful.

----
It is easier to expand the generating function by breaking it into two geometric series.
{{Theorem|Proposition|
:Let <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>. It holds that
::<math>\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
}}

It is easy to verify the above equation, but to deduce it, we need some (high school) calculation.

{|border="2" width="100%" cellspacing="4" cellpadding="3" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse;empty-cells:show;"
|
:{|
|
<math>1-x-x^2</math> has two roots <math>\frac{-1\pm\sqrt{5}}{2}</math>.

Denote that <math>\phi=\frac{2}{-1+\sqrt{5}}=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{2}{-1-\sqrt{5}}=\frac{1-\sqrt{5}}{2}</math>.

Then <math>(1-x-x^2)=(1-\phi x)(1-\hat{\phi}x)</math>, so we can write
:<math>
\begin{align}
\frac{x}{1-x-x^2}
&=\frac{x}{(1-\phi x)(1-\hat{\phi} x)}\\
&=\frac{\alpha}{(1-\phi x)}+\frac{\beta}{(1-\hat{\phi} x)},
\end{align}
</math>
where <math>\alpha</math> and <math>\beta</math> satisfying that
:<math>\begin{cases}
\alpha+\beta=0\\
\alpha\phi+\beta\hat{\phi}= -1.
\end{cases}</math>
Solving this we have that <math>\alpha=\frac{1}{\sqrt{5}}</math> and <math>\beta=-\frac{1}{\sqrt{5}}</math>. Thus,
:<math>G(x)=\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
|}
:<math>\square</math>
|}

Note that the expression <math>\frac{1}{1-z}</math> has a well known geometric expansion:
:<math>\frac{1}{1-z}=\sum_{n\ge 0}z^n</math>.

Therefore, <math>G(x)</math> can be expanded as
:<math>
\begin{align}
G(x)
&=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}\\
&=\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\phi x)^n-\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\hat{\phi} x)^n\\
&=\sum_{n\ge 0}\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)x^n.
\end{align}</math>
So the <math>n</math>th Fibonacci number is given by
:<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>.

== Solving recurrences ==
In the above analysis of Fibonacci numbers, we apply the following general methodology of solving recurrences by generating functions.
:1. Give a recursion that computes <math>a_n</math>; that is, an equation expressing <math>a_n</math> in terms of other elements of the sequence, such as
::<math>a_n=f(a_0,a_1,\ldots,a_{n-1})</math> for some function <math>f</math>.
:2. Multiply both sides of the equation by <math>x^n</math> and sum over all <math>n</math>. This gives the generating function
::<math>G(x)=\sum_{n\ge 0}a_nx^n=\sum_{n\ge 0}f(a_0,a_1,\ldots,a_{n-1})x^n</math>.
:: And manipulate the right hand side of the equation so that it becomes some other expression involving <math>G(x)</math>.
:3. Solve the resulting equation to derive an explicit formula for <math>G(x)</math>.
:4. Expand <math>G(x)</math> into a power series and read off the coefficient of <math>x^n</math>, which is a closed form for <math>a_n</math>.

=== Algebraic operations on generating functions ===
The second step in the above methodology is somehow tricky. It involves first applying the recurrence to the coefficients of <math>G(x)</math>, which is easy; and then manipulating the resulting formal power series to express it in terms of <math>G(x)</math>, which is more difficult (because it works backwards).

We can apply several natural algebraic operations on the formal power series.

{{Theorem|Generating function manipulation|
:Let <math>G(x)=\sum_{n\ge 0}g_nx^n</math> and <math>F(x)=\sum_{n\ge 0}f_nx^n</math>.
----

::<math>
\begin{align}
x^k G(x)
&= \sum_{n\ge k}g_{n-k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\frac{G(x)-\sum_{i=0}^{k-1}g_iz^i}{x^k}
&=\sum_{n\ge 0}g_{n+k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\alpha F(x)+\beta G(x)
&= \sum_{n\ge 0} (\alpha f_n+\beta g_n)x^n\\
F(x)G(x)
&= \sum_{n\ge 0}\sum_{k=0}^nf_kg_{n-k}x^n\\
G(cx)
&= \sum_{n\ge 0} c^ng_n x^n\\
G'(x)
&=
\sum_{n\ge 0}(n+1)g_{n+1}x^n
\end{align}
</math>
}}

When manipulating generating functions, these rules are applied backwards; that is, from the right-hand-side to the left-hand-side.

=== Expanding generating functions ===
The last step of solving recurrences by generating function is expanding the closed form generating function <math>G(x)</math> to evaluate its <math>n</math>-th coefficient. In principle, we can always use the [http://en.wikipedia.org/wiki/Taylor_series Taylor series]
:<math>G(x)=\sum_{n\ge 0}\frac{G^{(n)}(0)}{n!}x^n</math>,
where <math>G^{(n)}(0)</math> is the value of the <math>n</math>-th derivative of <math>G(x)</math> evaluated at <math>x=0</math>.

Some interesting special cases are very useful.

====Geometric sequence====
In the example of Fibonacci numbers, we use the well known geometric series:
:<math>\frac{1}{1-x}=\sum_{n\ge 0}x^n</math>.
It is useful when we can express the generating function in the form of <math>G(x)=\frac{a_1}{1-b_1x}+\frac{a_2}{1-b_2x}+\cdots+\frac{a_k}{1-b_kx}</math>. The coefficient of <math>x^n</math> in such <math>G(x)</math> is <math>a_1b_1^n+a_2b_2^n+\cdots+a_kb_k^n</math>.

====Binomial theorem====
The <math>n</math>-th derivative of <math>(1+x)^\alpha</math> for some real <math>\alpha</math> is
:<math>\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)(1+x)^{\alpha-n}</math>.
By Taylor series, we get a generalized version of the binomial theorem known as [http://en.wikipedia.org/wiki/Binomial_coefficient#Newton.27s_binomial_series '''Newton's formula''']:
{{Theorem|Newton's formular (generalized binomial theorem)|
If <math>|x|<1</math>, then
:<math>(1+x)^\alpha=\sum_{n\ge 0}{\alpha\choose n}x^{n}</math>,
where <math>{\alpha\choose n}</math> is the '''generalized binomial coefficient''' defined by
:<math>{\alpha\choose n}=\frac{\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)}{n!}</math>.
}}

=== Example: multisets ===
In the last lecture we gave a combinatorial proof of the number of <math>k</math>-multisets on an <math>n</math>-set. Now we give a generating function approach to the problem.

Let <math>S=\{x_1,x_2,\ldots,x_n\}</math> be an <math>n</math>-element set. We have
:<math>(1+x_1+x_1^2+\cdots)(1+x_2+x_2^2+\cdots)\cdots(1+x_n+x_n^2+\cdots)=\sum_{m:S\rightarrow\mathbb{N}} \prod_{x_i\in S}x_i^{m(x_i)}</math>,
where each <math>m:S\rightarrow\mathbb{N}</math> species a possible multiset on <math>S</math> with multiplicity function <math>m</math>.

Let all <math>x_i=x</math>. Then
:<math>
\begin{align}
(1+x+x^2+\cdots)^n
&=
\sum_{m:S\rightarrow\mathbb{N}}x^{m(x_1)+\cdots+m(x_n)}\\
&=
\sum_{\text{multiset }M\text{ on }S}x^{|M|}\\
&=
\sum_{k\ge 0}\left({n\choose k}\right)x^k.
\end{align}
</math>
The last equation is due to the the definition of <math>\left({n\choose k}\right)</math>. Our task is to evaluate <math>\left({n\choose k}\right)</math>.

Due to the geometric sequence and the Newton's formula
:<math>
(1+x+x^2+\cdots)^n=(1-x)^{-n}=\sum_{k\ge 0}{-n\choose k}(-x)^k.
</math>
So
:<math>
\left({n\choose k}\right)=(-1)^k{-n\choose k}={n+k-1\choose k}.
</math>
The last equation is due to the definition of the generalized binomial coefficient. We use an analytic (generating function) proof to get the same result of <math>\left({n\choose k}\right)</math> as the combinatorial proof.

== Pólya's problem of changing money ==

== Catalan Number ==
We now introduce a class of counting problems, all with the same solution, called [http://en.wikipedia.org/wiki/Catalan_number '''Catalan number'''].

The <math>n</math>th Catalan number is denoted as <math>C_n</math>.
In Volume 2 of Stanley's ''Enumerative Combinatorics'', a set of exercises describe 66 different interpretations of the Catalan numbers. We give a few examples, cited from Wikipedia.
* ''C''<sub>''n''</sub> is the number of '''Dyck words''' of length 2''n''. A Dyck word is a string consisting of ''n'' X's and ''n'' Y's such that no initial segment of the string has more Y's than X's (see also [http://en.wikipedia.org/wiki/Dyck_language Dyck language]). For example, the following are the Dyck words of length 6:
<div class="center"><big> XXXYYY     XYXXYY     XYXYXY     XXYYXY     XXYXYY.</big></div>

* Re-interpreting the symbol X as an open parenthesis and Y as a close parenthesis, ''C''<sub>''n''</sub> counts the number of expressions containing ''n'' pairs of parentheses which are correctly matched:
<div class="center"><big> ((()))     ()(())     ()()()     (())()     (()()) </big></div>

* ''C''<sub>''n''</sub> is the number of different ways ''n'' + 1 factors can be completely parenthesized (or the number of ways of associating ''n'' applications of a '''binary operator'''). For ''n'' = 3, for example, we have the following five different parenthesizations of four factors:
<div class="center"><math>((ab)c)d \quad (a(bc))d \quad(ab)(cd) \quad a((bc)d) \quad a(b(cd))</math></div>

* Successive applications of a binary operator can be represented in terms of a '''full binary tree'''. (A rooted binary tree is ''full'' if every vertex has either two children or no children.) It follows that ''C''<sub>''n''</sub> is the number of full binary trees with ''n'' + 1 leaves:
[[Image:Catalan number binary tree example.png|center]]

* ''C''<sub>''n''</sub> is the number of '''monotonic paths''' along the edges of a grid with ''n'' × ''n'' square cells, which do not pass above the diagonal. A monotonic path is one which starts in the lower left corner, finishes in the upper right corner, and consists entirely of edges pointing rightwards or upwards. Counting such paths is equivalent to counting Dyck words: X stands for "move right" and Y stands for "move up". The following diagrams show the case ''n'' = 4:
[[Image:Catalan number 4x4 grid example.svg.png|450px|center]]

* ''C''<sub>''n''</sub> is the number of different ways a [http://en.wikipedia.org/wiki/Convex_polygon '''convex polygon'''] with ''n'' + 2 sides can be cut into '''triangles''' by connecting vertices with straight lines. The following hexagons illustrate the case ''n'' = 4:
[[Image:Catalan-Hexagons-example.png|400px|center]]

* ''C''<sub>''n''</sub> is the number of [http://en.wikipedia.org/wiki/Stack_(data_structure) '''stack''']-sortable permutations of {1, ..., ''n''}. A permutation ''w'' is called '''stack-sortable''' if ''S''(''w'') = (1, ..., ''n''), where ''S''(''w'') is defined recursively as follows: write ''w'' = ''unv'' where ''n'' is the largest element in ''w'' and ''u'' and ''v'' are shorter sequences, and set ''S''(''w'') = ''S''(''u'')''S''(''v'')''n'', with ''S'' being the identity for one-element sequences.

* ''C''<sub>''n''</sub> is the number of ways to tile a stairstep shape of height ''n'' with ''n'' rectangles. The following figure illustrates the case ''n'' = 4:
[[Image:Catalan stairsteps 4.png|400px|center]]

{{Theorem|Recurrence relation for Catalan numbers|
:<math>C_0=0</math>, <math>C_1=1</math>, and for <math>n>1</math>,
::<math>
C_n=\sum_{i=1}^{n-1}C_iC_{n-i}.
</math>
}}

Let <math>G(x)=\sum_{n\ge 0}C_nx^n</math> be the generating function. Apply the product rule,
:<math>G(x)^2=\sum_{n\ge 0}\sum_{k=0}^{n}C_kC_{n-k}x^n=\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n</math>.
Due to the recurrence,
:<math>G(x)=\sum_{n\ge 0}C_nx^n=x+\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n=x+G(x)^2</math>.
Solving this, we obtain
:<math>G(x)=\frac{1\pm(1-4x)^{1/2}}{2}</math>.
Because <math>C_0=0</math>, it must hold that <math>G(x)=\frac{1-(1-4x)^{1/2}}{2}</math>, or otherwise the constant term is not zero. Expanding <math>(1-4x)^{1/2}</math> by Newton's formula, we have
:<math>
\begin{align}
G(x)
&=
\frac{1-(1-4x)^{1/2}}{2}\\
&=
1-\frac{1}{2}\sum_{n\ge 0}{1/2\choose n}(-4x)^n
\end{align}
</math>
Thus,
:<math>
\begin{align}
C_n
&=-\frac{1}{2}{1/2\choose n}(-4)^n\\
&=-\frac{1}{2}\cdot\frac{1}{2}\cdot\frac{-1}{2}\cdot\frac{-3}{2}\cdots\frac{-(2n-3)}{2}\cdot(-4)^n/n!\\
&=\frac{(2n-2)!}{(n-1)!n!}\\
&=\frac{1}{n}{2n-2\choose n-1}.
\end{align}
</math>
So we prove the following closed form for Catalan number.
{{Theorem|Theorem|
:<math>C_n=\frac{1}{n}{2n-2\choose n-1}</math>.
}}

Combinatorics (Fall 2010)/Generating functions

2010-09-12T01:20:45Z

172.21.3.5: /* Fibonacci numbers */

== Generating Functions ==
In Stanley's magnificent book ''Enumerative Combinatorics'', he comments the generating function as "the most useful but most difficult to understand method (for counting)".

The solution to a counting problem is usually represented as some <math>a_n</math> depending a parameter <math>n</math>. Sometimes this <math>a_n</math> is called a ''counting function'' as it is a function of the parameter <math>n</math>. <math>a_n</math> can also be treated as a infinite series:
:<math>a_0,a_1,a_2,\ldots</math>

The '''ordinary generating function (OGF)''' defined by <math>a_n</math> is
:<math>
G(x)=\sum_{n\ge 0} a_nx^n.
</math>

So <math>G(x)=a_0+a_1x+a_2x^2+\cdots</math>. An expression in this form is called a [http://en.wikipedia.org/wiki/Formal_power_series '''formal power series'''], and <math>a_0,a_1,a_2,\ldots</math> is the sequence of '''coefficients'''.

Furthermore, the generating function can be expanded as
:G(x)=<math>(\underbrace{1+\cdots+1}_{a_0})+(\underbrace{x+\cdots+x}_{a_1})+(\underbrace{x^2+\cdots+x^2}_{a_2})+\cdots+(\underbrace{x^n+\cdots+x^n}_{a_n})+\cdots</math>
so it indeed "generates" all the possible instances of the objects we want to count.

Usually, we do not evaluate the generating function <math>GF(x)</math> on any particular value. <math>x</math> remains as a '''formal variable''' without assuming any value. The numbers that we want to count are the coefficients carried by the terms in the formal power series. So far the generating function is just another way to represent the sequence
:<math>(a_0,a_1,a_2,\ldots\ldots)</math>.

The true power of generating functions comes from the various algebraic operations that we can perform on these generating functions. We use an example to demonstrate this.

=== Fibonacci numbers ===
Consider the following counting problems.
* Count the number of ways that the nonnegative integer <math>n</math> can be written as a sum of ones and twos (in order).
: The problem asks for the number of compositions of <math>n</math> with summands from <math>\{1,2\}</math>. Formally, we are counting the number of tuples <math>(x_1,x_2,\ldots,x_k)</math> for some <math>k\le n</math> such that <math>x_i\in\{1,2\}</math> and <math>x_1+x_2+\cdots+x_k=n</math>.
: Let <math>F_n</math> be the solution. We observe that a composition either starts with a 1, in which case the rest is a composition of <math>n-1</math>; or starts with a 2, in which case the rest is a composition of <math>n-2</math>. So we have the recursion for <math>F_n</math> that
::<math>F_n=F_{n-1}+F_{n-2}</math>.
* Count the ways to completely cover a <math>2\times n</math> rectangle with <math>2\times 1</math> dominos without any overlaps.
: Dominos are identical <math>2\times 1</math> rectangles, so that only their orientations --- vertical or horizontal matter.
: Let <math>F_n</math> be the solution. It also holds that <math>F_n=F_{n-1}+F_{n-2}</math>. The proof is left as an exercise.

In both problems, the solution is given by <math>F_n</math> which satisfies the following recursion.
:<math>F_n=\begin{cases}
F_{n-1}+F_{n-2} & \mbox{if }n\ge 2,\\
1 & \mbox{if }n=1\\
0 & \mbox{if }n=0.
\end{cases}</math>

<math>F_n</math> is called the [http://en.wikipedia.org/wiki/Fibonacci_number Fibonacci number].

{{Theorem|Theorem|
::<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)</math>,
:where <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>.
}}
The quantity <math>\phi=\frac{1+\sqrt{5}}{2}</math> is the so-called [http://en.wikipedia.org/wiki/Golden_ratio golden ratio], a constant with some significance in mathematics and aesthetics.

We now prove this theorem by using generating functions.
The ordinary generating function for the Fibonacci number <math>F_{n}</math> is
:<math>G(x)=\sum_{n\ge 0}F_n x^n</math>.
We have that <math>F_{n}=F_{n-1}+F_{n-2}</math> for <math>n\ge 2</math>, thus
:<math>\begin{align}
G(x)
&=
\sum_{n\ge 0}F_n x^n
&=
x+\sum_{n\ge 2}(F_{n-1}+F_{n-2})x^n.
\end{align}
</math>
For generating functions, there are general ways to generate <math>F_{n-1}</math> and <math>F_{n-2}</math>, or the coefficients with any smaller indices.
:<math>
\begin{align}
xG(x)
&=\sum_{n\ge 0}F_n x^{n+1}=\sum_{n\ge 1}F_{n-1} x^n=\sum_{n\ge 2}F_{n-1} x^n\\
x^2G(x)
&=\sum_{n\ge 0}F_n x^{n+2}=\sum_{n\ge 2}F_{n-2} x^n.
\end{align}
</math>
So we have
:<math>G(x)=x+(x+x^2)G(x)\,</math>,
hence
:<math>G(x)=\frac{x}{1-x-x^2}</math>.
The value of <math>F_n</math> is the coefficient of <math>x^n</math> in the Taylor series for this formular, which is <math>\frac{G^{(n)}(0)}{n!}=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>. Although this expansion works in principle, the detailed calculus is rather painful.

----
It is easier to expand the generating function by breaking it into several geometric series.
{{Theorem|Proposition|
:Let <math>\phi=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{1-\sqrt{5}}{2}</math>. It holds that
::<math>\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
}}

It is easy to verify the above equation, but to deduce it, we need some (high school) calculation.

{|border="2" width="100%" cellspacing="4" cellpadding="3" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse;empty-cells:show;"
|
:{|
|
<math>1-x-x^2</math> has two roots <math>\frac{-1\pm\sqrt{5}}{2}</math>.

Denote that <math>\phi=\frac{2}{-1+\sqrt{5}}=\frac{1+\sqrt{5}}{2}</math> and <math>\hat{\phi}=\frac{2}{-1-\sqrt{5}}=\frac{1-\sqrt{5}}{2}</math>.

Then <math>(1-x-x^2)=(1-\phi x)(1-\hat{\phi}x)</math>, so we can write
:<math>
\begin{align}
\frac{x}{1-x-x^2}
&=\frac{x}{(1-\phi x)(1-\hat{\phi} x)}\\
&=\frac{\alpha}{(1-\phi x)}+\frac{\beta}{(1-\hat{\phi} x)},
\end{align}
</math>
where <math>\alpha</math> and <math>\beta</math> satisfying that
:<math>\begin{cases}
\alpha+\beta=0\\
\alpha\phi+\beta\hat{\phi}= -1.
\end{cases}</math>
Solving this we have that <math>\alpha=\frac{1}{\sqrt{5}}</math> and <math>\beta=-\frac{1}{\sqrt{5}}</math>. Thus,
:<math>G(x)=\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}</math>.
|}
:<math>\square</math>
|}

Note that the expression <math>\frac{1}{1-z}</math> has a well known geometric expansion:
:<math>\frac{1}{1-z}=\sum_{n\ge 0}z^n</math>.

Therefore, <math>G(x)</math> can be expanded as
:<math>
\begin{align}
G(x)
&=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}\\
&=\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\phi x)^n-\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\hat{\phi} x)^n\\
&=\sum_{n\ge 0}\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)x^n.
\end{align}</math>
So the <math>n</math>th Fibonacci number is given by
:<math>F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n</math>.

== Solving recurrences ==
In the above analysis of Fibonacci numbers, we apply the following general methodology of solving recurrences by generating functions.
:1. Give a recursion that computes <math>a_n</math>; that is, an equation expressing <math>a_n</math> in terms of other elements of the sequence, such as
::<math>a_n=f(a_0,a_1,\ldots,a_{n-1})</math> for some function <math>f</math>.
:2. Multiply both sides of the equation by <math>x^n</math> and sum over all <math>n</math>. This gives the generating function
::<math>G(x)=\sum_{n\ge 0}a_nx^n=\sum_{n\ge 0}f(a_0,a_1,\ldots,a_{n-1})x^n</math>.
:: And manipulate the right hand side of the equation so that it becomes some other expression involving <math>G(x)</math>.
:3. Solve the resulting equation to derive an explicit formula for <math>G(x)</math>.
:4. Expand <math>G(x)</math> into a power series and read off the coefficient of <math>x^n</math>, which is a closed form for <math>a_n</math>.

=== Algebraic operations on generating functions ===
The second step in the above methodology is somehow tricky. It involves first applying the recurrence to the coefficients of <math>G(x)</math>, which is easy; and then manipulating the resulting formal power series to express it in terms of <math>G(x)</math>, which is more difficult (because it works backwards).

We can apply several natural algebraic operations on the formal power series.

{{Theorem|Generating function manipulation|
:Let <math>G(x)=\sum_{n\ge 0}g_nx^n</math> and <math>F(x)=\sum_{n\ge 0}f_nx^n</math>.
----

::<math>
\begin{align}
x^k G(x)
&= \sum_{n\ge k}g_{n-k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\frac{G(x)-\sum_{i=0}^{k-1}g_iz^i}{x^k}
&=\sum_{n\ge 0}g_{n+k}x^n, &\qquad (\mbox{integer }k\ge 0)\\
\alpha F(x)+\beta G(x)
&= \sum_{n\ge 0} (\alpha f_n+\beta g_n)x^n\\
F(x)G(x)
&= \sum_{n\ge 0}\sum_{k=0}^nf_kg_{n-k}x^n\\
G(cx)
&= \sum_{n\ge 0} c^ng_n x^n\\
G'(x)
&=
\sum_{n\ge 0}(n+1)g_{n+1}x^n
\end{align}
</math>
}}

When manipulating generating functions, these rules are applied backwards; that is, from the right-hand-side to the left-hand-side.

=== Expanding generating functions ===
The last step of solving recurrences by generating function is expanding the closed form generating function <math>G(x)</math> to evaluate its <math>n</math>-th coefficient. In principle, we can always use the [http://en.wikipedia.org/wiki/Taylor_series Taylor series]
:<math>G(x)=\sum_{n\ge 0}\frac{G^{(n)}(0)}{n!}x^n</math>,
where <math>G^{(n)}(0)</math> is the value of the <math>n</math>-th derivative of <math>G(x)</math> evaluated at <math>x=0</math>.

Some interesting special cases are very useful.

====Geometric sequence====
In the example of Fibonacci numbers, we use the well known geometric series:
:<math>\frac{1}{1-x}=\sum_{n\ge 0}x^n</math>.
It is useful when we can express the generating function in the form of <math>G(x)=\frac{a_1}{1-b_1x}+\frac{a_2}{1-b_2x}+\cdots+\frac{a_k}{1-b_kx}</math>. The coefficient of <math>x^n</math> in such <math>G(x)</math> is <math>a_1b_1^n+a_2b_2^n+\cdots+a_kb_k^n</math>.

====Binomial theorem====
The <math>n</math>-th derivative of <math>(1+x)^\alpha</math> for some real <math>\alpha</math> is
:<math>\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)(1+x)^{\alpha-n}</math>.
By Taylor series, we get a generalized version of the binomial theorem known as [http://en.wikipedia.org/wiki/Binomial_coefficient#Newton.27s_binomial_series '''Newton's formula''']:
{{Theorem|Newton's formular (generalized binomial theorem)|
If <math>|x|<1</math>, then
:<math>(1+x)^\alpha=\sum_{n\ge 0}{\alpha\choose n}x^{n}</math>,
where <math>{\alpha\choose n}</math> is the '''generalized binomial coefficient''' defined by
:<math>{\alpha\choose n}=\frac{\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)}{n!}</math>.
}}

=== Example: multisets ===
In the last lecture we gave a combinatorial proof of the number of <math>k</math>-multisets on an <math>n</math>-set. Now we give a generating function approach to the problem.

Let <math>S=\{x_1,x_2,\ldots,x_n\}</math> be an <math>n</math>-element set. We have
:<math>(1+x_1+x_1^2+\cdots)(1+x_2+x_2^2+\cdots)\cdots(1+x_n+x_n^2+\cdots)=\sum_{m:S\rightarrow\mathbb{N}} \prod_{x_i\in S}x_i^{m(x_i)}</math>,
where each <math>m:S\rightarrow\mathbb{N}</math> species a possible multiset on <math>S</math> with multiplicity function <math>m</math>.

Let all <math>x_i=x</math>. Then
:<math>
\begin{align}
(1+x+x^2+\cdots)^n
&=
\sum_{m:S\rightarrow\mathbb{N}}x^{m(x_1)+\cdots+m(x_n)}\\
&=
\sum_{\text{multiset }M\text{ on }S}x^{|M|}\\
&=
\sum_{k\ge 0}\left({n\choose k}\right)x^k.
\end{align}
</math>
The last equation is due to the the definition of <math>\left({n\choose k}\right)</math>. Our task is to evaluate <math>\left({n\choose k}\right)</math>.

Due to the geometric sequence and the Newton's formula
:<math>
(1+x+x^2+\cdots)^n=(1-x)^{-n}=\sum_{k\ge 0}{-n\choose k}(-x)^k.
</math>
So
:<math>
\left({n\choose k}\right)=(-1)^k{-n\choose k}={n+k-1\choose k}.
</math>
The last equation is due to the definition of the generalized binomial coefficient. We use an analytic (generating function) proof to get the same result of <math>\left({n\choose k}\right)</math> as the combinatorial proof.

== Pólya's problem of changing money ==

== Catalan Number ==
We now introduce a class of counting problems, all with the same solution, called [http://en.wikipedia.org/wiki/Catalan_number '''Catalan number'''].

The <math>n</math>th Catalan number is denoted as <math>C_n</math>.
In Volume 2 of Stanley's ''Enumerative Combinatorics'', a set of exercises describe 66 different interpretations of the Catalan numbers. We give a few examples, cited from Wikipedia.
* ''C''<sub>''n''</sub> is the number of '''Dyck words''' of length 2''n''. A Dyck word is a string consisting of ''n'' X's and ''n'' Y's such that no initial segment of the string has more Y's than X's (see also [http://en.wikipedia.org/wiki/Dyck_language Dyck language]). For example, the following are the Dyck words of length 6:
<div class="center"><big> XXXYYY     XYXXYY     XYXYXY     XXYYXY     XXYXYY.</big></div>

* Re-interpreting the symbol X as an open parenthesis and Y as a close parenthesis, ''C''<sub>''n''</sub> counts the number of expressions containing ''n'' pairs of parentheses which are correctly matched:
<div class="center"><big> ((()))     ()(())     ()()()     (())()     (()()) </big></div>

* ''C''<sub>''n''</sub> is the number of different ways ''n'' + 1 factors can be completely parenthesized (or the number of ways of associating ''n'' applications of a '''binary operator'''). For ''n'' = 3, for example, we have the following five different parenthesizations of four factors:
<div class="center"><math>((ab)c)d \quad (a(bc))d \quad(ab)(cd) \quad a((bc)d) \quad a(b(cd))</math></div>

* Successive applications of a binary operator can be represented in terms of a '''full binary tree'''. (A rooted binary tree is ''full'' if every vertex has either two children or no children.) It follows that ''C''<sub>''n''</sub> is the number of full binary trees with ''n'' + 1 leaves:
[[Image:Catalan number binary tree example.png|center]]

* ''C''<sub>''n''</sub> is the number of '''monotonic paths''' along the edges of a grid with ''n'' × ''n'' square cells, which do not pass above the diagonal. A monotonic path is one which starts in the lower left corner, finishes in the upper right corner, and consists entirely of edges pointing rightwards or upwards. Counting such paths is equivalent to counting Dyck words: X stands for "move right" and Y stands for "move up". The following diagrams show the case ''n'' = 4:
[[Image:Catalan number 4x4 grid example.svg.png|450px|center]]

* ''C''<sub>''n''</sub> is the number of different ways a [http://en.wikipedia.org/wiki/Convex_polygon '''convex polygon'''] with ''n'' + 2 sides can be cut into '''triangles''' by connecting vertices with straight lines. The following hexagons illustrate the case ''n'' = 4:
[[Image:Catalan-Hexagons-example.png|400px|center]]

* ''C''<sub>''n''</sub> is the number of [http://en.wikipedia.org/wiki/Stack_(data_structure) '''stack''']-sortable permutations of {1, ..., ''n''}. A permutation ''w'' is called '''stack-sortable''' if ''S''(''w'') = (1, ..., ''n''), where ''S''(''w'') is defined recursively as follows: write ''w'' = ''unv'' where ''n'' is the largest element in ''w'' and ''u'' and ''v'' are shorter sequences, and set ''S''(''w'') = ''S''(''u'')''S''(''v'')''n'', with ''S'' being the identity for one-element sequences.

* ''C''<sub>''n''</sub> is the number of ways to tile a stairstep shape of height ''n'' with ''n'' rectangles. The following figure illustrates the case ''n'' = 4:
[[Image:Catalan stairsteps 4.png|400px|center]]

{{Theorem|Recurrence relation for Catalan numbers|
:<math>C_0=0</math>, <math>C_1=1</math>, and for <math>n>1</math>,
::<math>
C_n=\sum_{i=1}^{n-1}C_iC_{n-i}.
</math>
}}

Let <math>G(x)=\sum_{n\ge 0}C_nx^n</math> be the generating function. Apply the product rule,
:<math>G(x)^2=\sum_{n\ge 0}\sum_{k=0}^{n}C_kC_{n-k}x^n=\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n</math>.
Due to the recurrence,
:<math>G(x)=\sum_{n\ge 0}C_nx^n=x+\sum_{n\ge 2}\sum_{k=1}^{n-1}C_kC_{n-k}x^n=x+G(x)^2</math>.
Solving this, we obtain
:<math>G(x)=\frac{1\pm(1-4x)^{1/2}}{2}</math>.
Because <math>C_0=0</math>, it must hold that <math>G(x)=\frac{1-(1-4x)^{1/2}}{2}</math>, or otherwise the constant term is not zero. Expanding <math>(1-4x)^{1/2}</math> by Newton's formula, we have
:<math>
\begin{align}
G(x)
&=
\frac{1-(1-4x)^{1/2}}{2}\\
&=
1-\frac{1}{2}\sum_{n\ge 0}{1/2\choose n}(-4x)^n
\end{align}
</math>
Thus,
:<math>
\begin{align}
C_n
&=-\frac{1}{2}{1/2\choose n}(-4)^n\\
&=-\frac{1}{2}\cdot\frac{1}{2}\cdot\frac{-1}{2}\cdot\frac{-3}{2}\cdots\frac{-(2n-3)}{2}\cdot(-4)^n/n!\\
&=\frac{(2n-2)!}{(n-1)!n!}\\
&=\frac{1}{n}{2n-2\choose n-1}.
\end{align}
</math>
So we prove the following closed form for Catalan number.
{{Theorem|Theorem|
:<math>C_n=\frac{1}{n}{2n-2\choose n-1}</math>.
}}

大气环流 (Fall 2010)

2010-09-07T06:50:02Z

172.21.0.219: /* Assignments */

{{Infobox
|name = Infobox
|bodystyle =
|title = 大气环流 <br>
General Circulation of the Atmosphere
|titlestyle =

|image = [[File:zoo_2.jpg|border|150px]]
|imagestyle =
|caption =
|captionstyle =
|headerstyle = background:#ccf;
|labelstyle = background:#ddf;
|datastyle =

|header1 =Instructor
|label1 =
|data1 =
|header2 =
|label2 =
|data2 = 张洋
|header3 =
|label3 = Email
|data3 = yangzhang@nju.edu.cn yang.zhang.nju@gmail.com
|header4 =
|label4= office
|data4= 东南楼 334
|header5 = Class
|label5 =
|data5 =
|header6 =
|label6 = Class meetings
|data6 = 2 pm-4 pm, Wednesday, <br>馆I-103
|header7 =
|label7 = Place
|data7 =
|header8 =
|label8 = Office hours
|data8 = 4pm-5pm, Friday, <br>东南楼 334
|header9 = Reference book
|label9 =
|data9 =
|header10 =
|label10 =
|data10 = {{Infobox
|name =
|bodystyle =
|title =
|titlestyle =
|image = [[File:James-Circulating.jpg|border|100px]]
|imagestyle =
|caption = Introduction to Circulating Atmospheres, <br>''I. James'', Cambridge Press, 1995
|captionstyle = }}
|header11 =
|label11 =
|data11 = {{Infobox
|name =
|bodystyle =
|title =
|titlestyle =
|image = [[File:Oort.jpg|border|100px]]
|imagestyle =
|caption =Physics of Climate, ''Peixoto, J. P.'' and ''A. H. Oort'', Springer-Verlag New York, 1992
|captionstyle =
}}
|header12 =
|label12 =
|data12 = {{Infobox
|name =
|bodystyle =
|title =
|titlestyle =
|image = [[File:geoff.jpg|border|100px]]
|imagestyle =
|caption =Atmospheric and Oceanic Fluid Dynamics: Fundamentals and Large-scale Circulation, ''Vallis, G. K.'', Cambridge University Press, 2006
|captionstyle =
}}
|belowstyle = background:#ddf;
|below =
}}

This is the page for the class ''General Circulation of the Atmosphere (大气环流)'' for the Fall 2010 semester. Students who take this class should check this page periodically for content updates and new announcements.

= Announcement =
* 本学期大气环流课的上课时间已改为（1-18周）周三下午2-4pm，馆1-103. 【2010.9.4】

= Course info =
* '''Instructor ''': 张洋，
:*office: 东南楼 334
:*email: yangzhang@nju.edu.cn, yang.zhang.nju@gmail.com
* '''Class meeting''': 2 pm-4 pm, Wednesday, 馆I-103
* '''Office hour''': 4-5pm, Friday; 东南楼 334
* '''Prerequisites''': 动力气象，天气学，气候学
* '''Grading''': 平时作业（60％）＋期末考试（40％）
本课程将大致布置5次作业，每次作业一二道题目左右。题目将选择每个课题最具有代表性、需要一定思维强度和动手能力的训练用题目,意在使学生通过顺利完成作业来建立环流系统的物理模型、以对课程内容得到深刻全面地理解和掌握。期末考试题目数量将会比平时作业多,覆盖面更广,但会比作业题目简单,只涉及对基本内容的掌握和对环流理论的直接应用。平时认真完成作业的学生,将不会在考试中遇到困难。

= Course intro =
“大气环流”常指地球大气较大空间范围、较长时间尺度上的空气流动，及其对地球大气热量、动量、能量和水汽的全球输送。虽然从十七世纪起人们就开始研究大尺度的大气运动(如Hadley在1735年提出的信风理论)，但大气环流真正发展成为一门较完备的学科方向却是近半个世纪的事情。随着四五十年代探空资料等高空气象要素的取得，以及六十年代卫星等覆盖全球的观测资料的加入，大气环流的空间结构和时间变化开始被系统、全面地揭示。与此同时，大气环流的数值模拟，也开始成为研究大气环流的一个主要方法，并发展至今成为了解和预估未来气候变化的主要手段。随着观测和模拟手段的进步，大气环流的理论研究也在近三十年开始快速地发展，人们对各种环流系统的维持和变化有了更全面、更深刻、也更为现代的理解。

现代的大气环流是大气动力学、天气学和气候学相结合的产物。大气环流，既是各种天气现象产生的背景流场，又是各种气候状态形成的动力机制。大气环流在低频、季节、年际、年代际等时间尺度的变化，不但会引起天气现象的变化，也影响着气候状态的形成。而在大气科学领域面临着诸如全球暖化、气候变化、环流异常等重大科学问题的今天，大气环流研究的重要性被推到了前所未有的高度,大气环流也成为活跃发展又充满挑战的学科方向。

本课程将讲述在过去几十年里大气环流在观测、理论和模拟上取得的进展。希望学生借此课程能熟悉大气环流的基本分布和形态，掌握各主要环流系统的维持和变化机制，建立各环流系统形成的物理模型，了解现阶段的大气环流模式，知道大气环流方向有待解决的科学问题。

作为一门课程，大气环流内容的讲述常可以有两条线索。一条是全球尺度上大气热量、动量、能量和水汽的分布与输送，Lorenz（1967）和 Peixoto and Oort（1992）是按此线索介绍大气环流的优秀教材；另一条线索，是各纬度、各区域内大气环流系统的形成、维持和变化机制，James（1995）和 Vallis（2006）是按此线索介绍大气环流的经典讲义。根据现阶段大气环流方向的研究特点，本课程的讲述将主要按照后一种方式来展开，并辅以介绍各环流系统对大气各要素场的输送。在介绍各环流系统时，本课程将以观测、理论和模拟为顺序，从各大气环流系统的观测事实入手，介绍大气环流系统的分布特征和时空变化特征；着重介绍关于环流系统的各种动力学模型和现阶段对环流系统的理解；辅以对环流系统模拟研究的介绍；最后通过三者的对比，讨论各环流系统有待研究的问题。

= Syllabus =
本课程具体的内容安排如下：第一章为大气环流的概述，介绍大气环流发展的历史、包含的内容以及大气环流研究的常用观测资料和分析方法。第二章介绍大气环流产生的外部强迫：辐射强迫和下界面过程。第三至六章介绍大气环流中的各个环流系统及它们的动力机制。第七章详细介绍各复杂度的大气环流模式。第八章介绍大气环流领域现阶段最大的一个开放课题：全球暖化背景下的大气环流。这一章既是对前几章所介绍的大气环流理论的应用与检验，又是对未来大气环流研究方向的探讨。借此让学生熟悉并理解大气环流领域亟需解决的课题。具体课程安排和参考书目如下。
== Course schedule ==
*大气环流概述 (Introduction) （4课时）
*大气环流的外部强迫（3课时）
**辐射强迫 (Radiative forcing)
**下界面过程 (Surface boundaries)
*经向环流系统 (Zonally-averaged circulations)
**Hadley 环流（4课时）
**Ferrel 环流，急流，中纬度的波流相互作用（8课时）
*角动量、能量和水汽循环 (Angular momentum, energy and water vapor)（3课时）
*纬向环流系统（Non-zonal circulations）（6课时）
**Storm tracks
**Monsoon
**ENSO and Walker circulation
*不同复杂度的大气环流模式 (General circulation in a hierarchy of models)（3课时）
*全球暖化背景下的大气环流 (General circulation in the global warming scenario)（4课时）

[[详细课程安排|点击此处看详细课程安排 (click for more)]]

== References ==
*观测部分：Peixoto, J. P. and A. H. Oort, 1992: Physics of Climate. Springer-Verlag New York, Inc., 520 pp. 中文译本：气候物理学，1995，吴国雄、刘辉等译校，气象出版社。
*综合介绍：James, I., 1995: Introduction to circulating atmospheres. Cambridge University Press, 448 pp.
*理论部分：Vallis, G. K., 2006: Atmospheric and Oceanic Fluid Dynamics: Fundamentals and Large-scale Circulation. Cambridge University Press. 745 pp.
* [[其它参考书目|其它参考书目（点击看详情）]]

= Course Slides =
*[2010.09.01] Course introduction ([[file:Course_intro.pdf]]) and Chapter1_1

= Assignments =
# [[Reanalysis data and the earth's climatology]] [2010.9.8--2010.9.26]

Randomized Algorithms (Spring 2010)/Approximate counting, linear programming

2010-05-24T19:54:53Z

172.21.4.18: /* FPRAS */

== Counting Problems ==

=== Complexity model ===
Recall the class '''NP''' of decision problems (the problems with "yes" or "no" answers). Formally, denoting by <math>\{0,1\}^*</math> the set of all boolean strings of any lengths, a decision problem is a function <math>f:\{0,1\}^*\rightarrow\{0,1\}</math>.
{|border="1"
|'''Definition (NP)'''
:A function <math>f:\{0,1\}^*\rightarrow\{0,1\}</math> is in '''NP''' if there exist a polynomial <math>p</math> and a poly-time algorithm <math>V</math> with boolean output such that for every <math>x\in\{0,1\}^*</math>,
::<math>f(x)=1 \Leftrightarrow \exists y\in\{0,1\}^{p(|x|)}, \mbox{such that } V(x,y)=1\,</math>.
|}
Intuitively, the '''NP''' class contains all the decision problems such that the answer is "yes" if and only if there ''exists'' a '''certificate''' which can be verified in poly-time.

In many contexts, we are interested not just in the ''existence'' of certificate but actually in counting the ''number'' of certificates. This leads to the definition of the class '''#P''' (pronounced "sharp p"). Only now the output of the function is a natural number.

{|border="1"
|'''Definition (#P)'''
:A function <math>f:\{0,1\}^*\rightarrow\mathbb{N}</math> is in '''#P''' if there exist a polynomial <math>p</math> and a poly-time algorithm <math>V</math> with boolean output such that for every <math>x\in\{0,1\}^*</math>,
::<math>f(x)=\left|\left\{y\in\{0,1\}^{p(|x|)}\mid A(x,y)=1\right\}\right|</math>.
|}
You may notice the similarity between the two definitions. The difference is that now <math>f(x)</math> does not just indicates the existence of a certificate, but gives the actual number of certificates.

Why should we care about the number of "certificates" at all?
In combinatorics, a counting problem is usually formulated as counting the number of combinatorial objects with some particular structure, such as the number of trees, or Latin squares. The '''#P''' class contains the counting problems that the "structure" requirements are easy to check.

;Examples of problems in #P
:<nowiki>#</nowiki>cycles: given as input a graph <math>G</math>, count the number of cycles in <math>G</math>.
:<nowiki>#</nowiki>SAT: given as input a boolean formula <math>\phi</math>, count the number of satisfying assignments for <math>\phi</math>.

The second example shows that the '''<nowiki>#</nowiki>P''' may be harder than '''NP''', because we know that SAT (given a boolean formular, decide the existence of satisfying assignments) is '''NP-complete''', and its counting version <nowiki>#</nowiki>SAT now ask for the exact number of satisfying assignments.

The class '''FP''' contains all the functions <math>f:\{0,1\}^*\rightarrow\mathbb{N}</math> computable by poly-time algorithms. The classes '''FP''' and '''#P''' are the analogs of '''P''' and '''NP''' for counting problems.

A '''reduction''' from a problem <math>f</math> to a problem <math>g</math> is a mapping <math>\phi</math> which maps instances of <math>f</math> to instances of <math>g</math> such that for any instance <math>x</math> of <math>f</math>,
:<math>f(x)=g(\phi(x))</math>.
In other words, <math>\phi</math> "reduces" the task of solving <math>f</math> to the task of solving <math>g</math>. A problem <math>f</math> is said to be '''poly-time reducible''' to the problem <math>g</math>, if there exists a reduction <math>\phi</math> from <math>f</math> to <math>g</math> and <math>\phi</math> is poly-time computable.

A problem <math>f</math> is '''#P-hard''' if every problem in '''#P''' is poly-time reducible to <math>f</math>. A problem <math>f</math> is '''#P-complete''' if <math>f\in</math>'''#P''' and <math>f</math> is '''#P-hard'''. That is to say, '''#P-complete''' problems are the "hardest" problems in '''#P'''.

Generally, '''#P-complete''' problems are very hard, because if we can solve any of these problems in poly-time, then '''#P'''='''FP''', which implies that '''NP'''='''P'''. On the other hand, we do not know whether '''NP'''='''P''' could imply that '''#P'''='''FP'''.

=== FPRAS ===

{|border="1"
|'''Definition (FPRAS)'''
:A '''polynomial randomized approximation scheme (PRAS)''' for a problem <math>f:\{0,1\}^*\rightarrow\mathbb{R}</math> is a randomized algorithm <math>A</math> that takes an input instance <math>x</math> and a real number <math>\epsilon>0</math>, and in time polynomial in <math>n=|x|</math> returns <math>A(x)</math> such that
:::<math>\Pr[(1-\epsilon)f(x)\le A(x)\le (1+\epsilon)f(x)]\ge\frac{3}{4}.</math>
:A '''fully polynomial randomized approximation scheme (FPRAS)''' is a PRAS whose running time is polynomially in both <math>n</math> and <math>1/\epsilon</math>.
|}

The constant <math>\frac{3}{4}</math> in the definition can be replaced by any constant in the range <math>(1/2,1)</math> without changing the definition. In fact, it is usually more convenient to parameterize the FPRAS by the approximation error and the probability error.

{|border="1"
|'''Definition (<math>(\epsilon,\delta)</math>-FPRAS)'''
:An '''<math>(\epsilon,\delta)</math>-FPRAS''' for a problem <math>f:\{0,1\}^*\rightarrow\mathbb{R}</math> is an FPRAS with
::<math>\Pr[(1-\epsilon)f(x)\le A(x)\le (1+\epsilon)f(x)]\ge 1-\delta,</math>
:running in time polynomial of <math>n</math>, <math>\frac{1}{\epsilon}</math>, and <math>\log\frac{1}{\delta}</math>.
|}

== Approximate Counting ==
Let us consider the following abstract problem.

Let <math>U</math> be a finite set of known size, and let <math>G\subseteq U</math>. We want to compute the size of <math>G</math>, namely <math>|G|</math>.

We assume two devices:
* A '''uniform sampler''' <math>\mathcal{U}</math>, which uniformly and independently samples a member of <math>U</math> upon each calling.
* A '''membership oracle''' of <math>G</math>, denoted <math>\mathcal{O}</math>. Given as the input an <math>x\in U</math>, <math>\mathcal{O}(x)</math> indicates whether or not <math>x</math> is a member of <math>G</math>.

Equipped by <math>\mathcal{U}</math> and <math>\mathcal{O}</math>, we can have the following Monte Carlo algorithm:
*Choose <math>N</math> independent samples from <math>U</math> by the uniform sampler <math>\mathcal{U}</math>, represented by the random variables <math>X_1,X_2,\ldots, X_N</math>.
* Let <math>Y_i</math> be the indicator random variable defined as <math>Y_i=\mathcal{O}(X_i)</math>, namely, <math>Y_i</math> indicates whether <math>X_i\in G</math>.
* Define the estimator random variable
::<math>Z=\frac{|U|}{N}\sum_{i=1}^N Y_i.</math>

It is easy to see that <math>\mathbf{E}[Z]=|G|</math> and we might hope that with high probability the value of <math>Z</math> is close to <math>|G|</math>. Formally, <math>Z</math> is called an <math>\epsilon</math>-approximation of <math>|G|</math> if
:<math>
(1-\epsilon)|G|\le Z\le (1+\epsilon)|G|.
</math>

The following theorem states that the probabilistic accuracy of the estimation depends on the number of samples and the ratio between <math>|G|</math> and <math>|U|</math>

{|border="1"
|'''Theorem (estimator theorem)'''
:Let <math>\alpha=\frac{|G|}{|U|}</math>. Then the Monte Carlo method yields an <math>\epsilon</math>-approximation to <math>|G|</math> with probability at least <math>1-\delta</math> provided
::<math>N\ge\frac{4}{\epsilon \alpha}\ln\frac{2}{\delta}</math>.
|}
'''Proof''': Use the Chernoff bound.

<math>\square</math>

A counting algorithm for the set <math>G</math> has to deal with the following three complications:
* Implement the membership oracle <math>\mathcal{O}</math>. This is usually straightforward, or assumed by the model.
* Implement the uniform sampler <math>\mathcal{U}</math>. As we have seen, this is usually approximated by random walks. How to design the random walk and bound its mixing rate is usually technical challenging, if possible at all.
* Deal with exponentially small <math>\alpha=\frac{|G|}{|U|}</math>. This requires us to cleverly choose the universe <math>U</math>. Sometimes this needs some nontrivial ideas.

=== Counting DNFs ===
A disjunctive normal form (DNF) formular is a disjunction (OR) of clauses, where each clause is a conjunction (AND) of literals. For example:
:<math>(x_1\wedge \overline{x_2}\wedge x_3)\vee(x_2\wedge x_4)\vee(\overline{x_1}\wedge x_3\wedge x_4)</math>.
Note the difference from the conjunctive normal forms (CNF).

Given a DNF formular <math>\phi</math> as the input, the problem is to count the number of satisfying assignments of <math>\phi</math>. This problem is '''#P-complete'''.

Naively applying the Monte Carlo method will not give a good answer. Suppose that there are <math>n</math> variables. Let <math>U=\{\mathrm{true},\mathrm{false}\}^n</math> be the set of all truth assignments of the <math>n</math> variables. Let <math>G=\{x\in U\mid \phi(x)=\mathrm{true}\}</math> be the set of satisfying assignments for <math>\phi</math>. The straightforward use of Monte Carlo method samples <math>N</math> assignments from <math>U</math> and check how many of them satisfy <math>\phi</math>. This algorithm fails when <math>|G|/|U|</math> is exponentially small, namely, when exponentially small fraction of the assignments satisfy the input DNF formula.

;The union of sets problem
We reformulate the DNF counting problem in a more abstract framework, called the '''union of sets''' problem.

Let <math>V</math> be a finite universe. We are given <math>m</math> subsets <math>H_1,H_2,\ldots,H_m\subseteq V</math>. The following assumptions hold:
*For all <math>i</math>, <math>|H_i|</math> is computable in poly-time.
*It is possible to sample uniformly from each individual <math>H_i</math>.
*For any <math>x\in V</math>, it can be determined in poly-time whether <math>x\in H_i</math>.

The goal is to compute the size of <math>H=\bigcup_{i=1}^m H_i</math>.

DNF counting can be interpreted in this general framework as follows. Suppose that the DNF formula <math>\phi</math> is defined on <math>n</math> variables, and <math>\phi</math> contains <math>m</math> clauses <math>C_1,C_2,\ldots,C_m</math>, where clause <math>C_i</math> has <math>k_i</math> literals. Without loss of generality, we assume that in each clause, each variable appears at most once.
* <math>V</math> is the set of all assignments.
*Each <math>H_i</math> is the set of satisfying assignments for the <math>i</math>-th clause <math>C_i</math> of the DNF formular <math>\phi</math>. Then the union of sets <math>H=\bigcup_i H_i</math> gives the set of satisfying assignments for <math>\phi</math>.
* Each clause <math>C_i</math> is a conjunction (AND) of literals. It is not hard to see that <math>|H_i|=2^{n-k_i}</math>, which is efficiently computable.
* Sampling from an <math>H_i</math> is simple: we just fix the assignments of the <math>k_i</math> literals of that clause, and sample uniformly and independently the rest <math>(n-k_i)</math> variable assignments.
* For each assignment <math>x</math>, it is easy to check whether it satisfies a clause <math>C_i</math>, thus it is easy to determine whether <math>x\in H_i</math>.

;The coverage algorithm
We now introduce the coverage algorithm for the union of sets problem.

Consider the multiset <math>U</math> defined by
:<math>U=H_1\uplus H_2\uplus\cdots \uplus H_m</math>,
where <math>\uplus</math> denotes the multiset union. It is more convenient to define <math>U</math> as the set
:<math>U=\{(x,i)\mid x\in H_i\}</math>.
For each <math>x\in H</math>, there may be more than one instances of <math>(x,i)\in U</math>. We can choose a unique representative among the multiple instances <math>(x,i)\in U</math> for the same <math>x\in H</math>, by choosing the <math>(x,i)</math> with the minimum <math>i</math>, and form a set <math>G</math>.

Formally, <math>G=\{(x,i)\in U\mid \forall (x,j)\in U, j\le i\}</math>. Every <math>x\in H</math> corresponds to a unique <math>(x,i)\in G</math> where <math>i</math> is the smallest among <math>x\in H_i</math>.

It is obvious that <math>G\subseteq U</math> and
:<math>|G|=|H|</math>.

Therefore, estimation of <math>|H|</math> is reduced to estimation of <math>|G|</math> with <math>G\subseteq U</math>. Then <math>|G|</math> can have an <math>\epsilon</math>-approximation with probability <math>(1-\delta)</math> in poly-time, if we can uniformly sample from <math>U</math> and <math>|G|/|U|</math> is suitably small.

An uniform sample from <math>U</math> can be implemented as follows:
* generate an <math>i\in\{1,2,\ldots,m\}</math> with probability <math>\frac{|H_i|}{\sum_{i=1}^m|H_i|}</math>;
* uniformly sample an <math>x\in H_i</math>, and return <math>(x,i)</math>.

It is easy to see that this gives a uniform member of <math>U</math>. The above sampling procedure is poly-time because each <math>|H_i|</math> can be computed in poly-time, and sampling uniformly from each <math>H_i</math> is poly-time.

We now only need to lower bound the ratio
:<math>\alpha=\frac{|G|}{|U|}</math>.

We claim that
:<math>\alpha\ge\frac{1}{m}</math>.
It is easy to see this, because each <math>x\in H</math> has at most <math>m</math> instances of <math>(x,i)</math> in <math>U</math>, and we already know that <math>|G|=|H|</math>.

Due to the estimator theorem, this needs <math>\frac{4m}{\epsilon}\ln\frac{2}{\delta}</math> uniform random samples from <math>U</math>.

This gives the coverage algorithm for the abstract problem of the union of sets. The DNF counting is a special case of it.

=== Permanents and perfect matchings ===

Let <math>U=\{u_1,u_2,\ldots,u_n\}</math> and <math>V=\{v_1,v_2,\ldots,v_n\}</math>. Consider a bipartite graph <math>G(U,V,E)</math>. An <math>M\subseteq E</math> is a '''perfect matching''' of <math>G</math> if every vertex of <math>G</math> has exactly one edge in <math>M</math> adjacent to it.

Given a bipartite graph <math>G(U,V,E)</math> with <math>|U|=|V|=n</math>, we want to count the number of perfect matchings of <math>G</math>. This problem can be reduced to computing the '''permanent''' of a square matrix.

{|border="1"
|'''Definition (permanent)'''
:Let <math>Q</math> be an <math>n\times n</math> matrix. The '''permanent''' of the matrix is defined as
::<math>\mathrm{per}(Q)=\sum_{\pi\in\mathbb{S}_n}\prod_{i=1}^n Q_{i,\pi(i)}</math>,
:where <math>\mathbb{S}_n</math> is the symmetric group of permutation of size <math>n</math>.
|}

If we multiply each term of the sum the sign of the permutation, then it gives us the determinant of the matrix,
:<math>\det(Q)=\sum_{\pi\in\mathbb{S}_n}\sgn(\pi)\prod_{i=1}^n Q_{i,\pi(i)}</math>,
where <math>\sgn(\pi)</math>, the sign of a permutation, is either <math>-1</math> or <math>+1</math>, according to whether the minimum number of pair-wise interchanges to achieve <math>\pi</math> from <math>(1,2,\ldots,n)</math> is odd or even.

Unlike the determinants, which are computable in poly-time, permanents are hard to compute, as permanents can be used to count the number of perfect matchings in a bipartite graph, which is '''#P-complete'''.

A bipartite graph <math>G(U,V,E)</math> with <math>|U|=|V|=n</math> can be represented by an <math>n\times n</math> matrix <math>Q</math> with 0-1 entries as follows:
* Each row of <math>Q</math> corresponds to a vertex in <math>U</math> and each column of <math>Q</math> corresponds to a vertex in <math>V</math>.
::<math>Q_{ij}=\begin{cases}
1 & \mbox{if }i\sim j,\\
0 & \mbox{otherwise}.
\end{cases}</math>
Note the subtle difference between the definition of <math>Q</math> and the adjacency matrix.

Each perfect matching corresponds to a permutation <math>\pi</math> of <math>U</math> with <math>(u,\pi(u))\in E</math> for every <math>u\in U</math>, which corresponds to a permutation <math>\pi\in\mathbb{S}_n</math> with <math>\prod_{i=1}^n Q_{i,\pi(i)}=1</math>. It is than easy to see that <math>\mathrm{per}(Q)</math> gives the number of perfect matchings in the bipartite graph <math>G</math>.

It is known that counting the number of perfect matchings in a bipartite graph is '''#P-hard'''. Since this problem can be reduced to computing the permanent, thus the problem of computing the permanents is also '''#P-hard'''.

Now we show that with randomization, we can approximate the number of perfect matchings in a bipartite graph. In particular, we will give an FPRAS for counting the perfect matchings in a dense bipartite graph.

==== The Jerrum-Sinclair algorithm ====
Fix a bipartite graph <math>G(U,V,E)</math> with <math>|U|=|V|=n</math>. Let <math>\mathcal{M}_k</math> be the set of matchings of size <math>k</math> in <math>G</math>, and let <math>m_k=|\mathcal{M}_k|</math>. Thus, <math>\mathcal{M}_k</math> is the set of perfect matchings in <math>G</math>, and our goal is to compute <math>m_n</math>.

Let <math>r_k=\frac{m_k}{m_{k-1}}</math> for <math>1<k\le n</math>. Then
:<math>m_k=m_{k-1}r_k</math>,
which gives us a recursion to compute the <math>m_n</math>, as
:<math>m_n=m_{1}\frac{m_2}{m_1}\cdot\frac{m_3}{m_2}\cdots\frac{m_n}{m_{n-1}}=m_1\prod_{k=2}^n r_k</math>,
where <math>m_1=|\mathcal{M}_1|</math> is the number of matchings of size 1 in the bipartite graph <math>G</math>, which is just the number of edges in <math>G</math>. Therefore, <math>m_n</math> can be computed once we know <math>r_k</math> for <math>1<k\le n</math>.

Each <math>r_k</math> can be estimated by sampling uniformly from the set <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>. The algorithm for estimating <math>m_n</math> is outlined as:
# For each <math>1<k\le n</math>, have an FPRAS for computing the <math>r_k</math> by uniform sampling sufficiently many members from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math> as
::*uniformly sample <math>N</math> matching from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>, for some polynomially large <math>N</math>;
::* assuming that there are <math>X</math> sampled matchings of size <math>k</math>, return <math>r_k</math> as <math>r_k=\frac{X}{N-X}</math>.
:2. Compute <math>m_n</math> as <math>m_n=m_1\prod_{k=2}^n r_k</math>, where <math>m_1=|E|</math>.

There are several issues that we have to deal with in order to have a fully functional FPRAS for counting perfect matchings.
* By taking the product of <math>r_k</math>'s, the errors for individual <math>r_k</math>'s add up.
* In order to accurately estimate <math>r_k</math> by sampling from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>, the ratio <math>r_k</math> should be within the range <math>\left[\frac{1}{\alpha},\alpha\right]</math> for some <math>\alpha</math> within polynomial of <math>n</math>.
* Implement the uniform sampling from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>.

;Estimator for each <math>r_k</math>

; Accumulation of errors

; Near-uniform sampling from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>
In the last lecture, we have shown that by random walk, we can sample a near-uniform member of <math>\mathcal{M}_n\cup\mathcal{M}_{n-1}</math> in poly-time.

=== Volume estimation ===
We consider the problem of computing the volume of a given [http://en.wikipedia.org/wiki/Convex_body convex body] <math>K</math> in <math>n</math> dimensions.

We use <math>\Upsilon(K)\,</math> to denote the volume of the convex body <math>K</math>. Abstractly, the problem is that given as input a convex body <math>K</math> in <math>n</math> dimensions, return the <math>\Upsilon(K)\,</math>.

We should be more specific about the input model. Since we allow an arbitrary convex body as input, it is not even clear how to describe the body. We assume that <math>K</math> is described by means of a '''membership oracle''' <math>\mathcal{O}</math>, such that for a '''query''' of an <math>n</math>-dimensional point <math>x</math>, <math>\mathcal{O}(x)</math> indicates whether <math>x\in K</math>.

For example, the <math>n</math>-dimensional convex body defined by the intersection of <math>m</math> half-spaces, which is the set of feasible solutions to a system of <math>m</math> linear constraints, can be described as
:<math>A x\le \boldsymbol{b}</math>,
for some <math>m\times n</math> matrix <math>A</math>, and <math>m</math>-dimensional vector <math>b</math>. For a query of an <math>x</math>, the membership oracle <math>\mathcal{O}</math> just check whether <math>Ax\le b</math>.

For deterministic algorithms, there are negative news for this problem.

{|border="1"
|'''Theorem (Bárány-Füredi 1987)'''
:Suppose that a deterministic poly-time algorithm uses the membership oracle for a convex body <math>K</math> in <math>n</math> dimensions, and generates an upper bound <math>\Upsilon_u\,</math> and a lower bound <math>\Upsilon_\ell\,</math> on the volume <math>\Upsilon(K)\,</math> of <math>K</math>. Then, there is a convex body <math>K</math> and a constant <math>c</math> such that
::<math>\frac{\Upsilon_u}{\Upsilon_\ell}\ge c\left(\frac{n}{\log n}\right)^n</math>.
|}
That is said, with deterministic algorithms, we cannot even approximate the volume within a wildly loose range.

Dyer-Frieze-Kannan come up with an idea of estimating the volume of <math>K</math> by sampling near-uniformly from convex sets. They reduce the problem of computing approximately the volume of convex bodies to this sampling problem and thus give the first FPRAS for the volume of convex bodies.

For any convex body <math>K</math>, it encloses some <math>n</math>-dimensional ball and is also enclosed by another ball with larger radius. We assume that the convex body <math>K</math> encloses the unit ball around the origin, and is enclosed by a larger ball round the origin with polynomially large radius. Formally, we assume that
:<math>B(0,1)\subseteq K\subseteq B(0,n^c)</math> for some constant <math>c</math>,
where <math>B(p,r)</math> denotes a ball of radius <math>r</math> with <math>p</math> as center, i.e. <math>B(p,r)=\{x\mid \|x-p\|\le r \}</math>. We can make this assumption because it is known that for any convex body <math>K</math>, there exists a linear transformation <math>\tau</math> which can be found within poly-time such that <math>\tau K</math> transforms the <math>K</math> to a convex body that satisfies the assumption, and preserves the ratio of the volumes of the balls to the convex body.

The volumes of the balls are easy to compute. If only the ratio between the convex body and the ball which encloses it, is sufficiently large, then we can apply the Monte Carlo method to estimate <math>\Upsilon(K)\,</math> by uniformly sampling from the ball.

However, in high-dimension, the ratio between the volume of a convex body and the volume of ball which encloses it, can be exponentially small. This is caused by the so called the "[http://en.wikipedia.org/wiki/Curse_of_dimensionality curse of dimensionality]".

Instead of having an outer ball and an inner ball, we define a sequence of balls:
:<math>B_0=B(0,\lambda^0), B_1=B(0,\lambda^1), B_2=B(0,\lambda^2),\ldots, B_m=B(0,\lambda^m)</math>,
where <math>\lambda=(1+\frac{1}{n})</math> and <math>m</math> is the smallest positive integer such that <math>\lambda^m\ge n^c</math>. Therefore, the inner ball <math>B(0,1)=B_0</math>, the outer ball <math>B(0,n^c)\subseteq B_m</math>, and <math>m</math> is within polynomial of <math>n</math>. In fact, <math>m\approx cn\ln n</math>.

This sequence of balls naturally defines a sequence of convex bodies by intersections as <math>K_i=B_i\cap K</math>. It is obvious that
:<math>B(0,1)=K_0\subseteq K_1\subseteq K_2\subseteq\cdots\subseteq K_m=K</math>.
Balls are convex, and since the intersection of convex bodies is still convex, the sequence of <math>K_i</math> is a sequence of convex bodies.

We have the telescopic product:
:<math>
\frac{\Upsilon(K_0)}{\Upsilon(K_1)}\cdot\frac{\Upsilon(K_1)}{\Upsilon(K_2)}\cdots\frac{\Upsilon(K_{m-1})}{\Upsilon(K_m)}=\frac{\Upsilon(K_0)}{\Upsilon(K_m)}=\frac{\Upsilon(B(0,1))}{\Upsilon(K)}.
</math>
Therefore, the volume <math>\Upsilon(K)\,</math> can be computed as
:<math>\Upsilon(K)=\Upsilon(B(0,1))\cdot\prod_{i=1}^{m}\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math>.

The volume of unite ball <math>\Upsilon(B(0,1))\,</math> can be precisely computed in poly-time. Each <math>\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math> is computed by near-uniform sampling from <math>K_{i}</math>, which encloses <math>K_{i-1}</math>.

Another observation is that the ratio <math>\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math> is well-bounded. Recall that <math>K_i=B(0,\lambda^i)\cap K</math> where <math>\lambda=(1+\frac{1}{n})</math>. It can be proved that in <math>n</math> dimensions, the volume of <math>K_i</math> is at most <math>\lambda^n</math> times the <math>K_{i-1}</math>, thus the ratio <math>\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math> is at most <math>\lambda^n=O(1)</math>. By the estimator theorem, we can have an FPRAS for the ratio <math>\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math> if we can uniformly sample from <math>K_i</math>.

Uniformly sampling from an arbitrary convex body is replaced by near-uniform sampling achieved by random walks. In the original walk of Dyer-Frieze-Kannan, they consider the random walk over <math>n</math>-dimensional discrete grid points enclosed by <math>K</math>, and prove that the walk is rapid mixing. This gives us the first FPRAS for volume estimation which runs in <math>\tilde{O}(n^{23})</math>, where <math>\tilde{O}(\cdot)</math> ignores the polylogarithmic factors.

The time bound was later improved by a series of works, each introducing some new ideas.

{|border="1"
| || complexity || new ingredient(s)
|-
| Dyer-Frieze-Kannan 1991 || align="center" | <math>n^{23}</math> || everything
|-
| Lovász-Simonovits 1990 || align="center" | <math>n^{16}</math> || localization lemma
|-
| Applegate-Kannan 1990 || align="center" | <math>n^{10}</math> || logconcave sampling
|-
| Lovász 1990 || align="center" | <math>n^{10}</math> || ball walk
|-
| Dyer-Frieze 1991 || align="center" | <math>n^8</math> || better error analysis
|-
| Lovász-Simonovits 1993 || align="center" | <math>n^7</math> || many improvements
|-
| Kannan-Lovász-Simonovits 1997 || align="center" | <math>n^5</math> || isotropy, speedy walk
|-
| Lovász-Vempala 2003 || align="center" | <math>n^4</math> || simulated annealing, hit-and-run
|-
|}
(cited from "Geometric Random Walks: A Survey" by Santosh Vempala.)

The current best upper bound is <math>\tilde{O}(n^4)</math> due to Lovász and Vempala in 2003. It is conjectured that the optimal bound is <math>\Theta(n^3)</math>.

== Linear Programming ==
Given a function <math>f:\mathbb{R}^n\rightarrow\mathbb{R}</math> and a set <math>\mathcal{F}\subseteq\mathbb{R}^n</math>, an '''optimization problem''' is the problem of finding an <math>x\in\mathcal{F}</math> with the optimal (minimum) <math>f(x)</math>. Formally, the problem can be expressed as:
::<math>
\begin{align}
\mbox{minimize} & \quad f(x)\\
\mbox{subject to} &\quad x\in\mathcal{F}
\end{align}
</math>
where <math>x\in\mathcal{R}^n</math> is a vector of <math>n</math> variables. For the problem of maximizing a function <math>f</math>, we just minimizes <math>-f</math> instead.

We call the function <math>f</math> the '''objective function''' and call any <math>x\in\mathcal{F}</math> a '''feasible solution''' of the problem. A feasible solution that minimizes <math>f(x)</math> is called an '''optimal solution'''. Our task is to find an optimal solution.

The feasible set <math>\mathcal{F}</math> is usually given by a number of '''constraints''' <math>P_1,P_2,\ldots,P_m</math>, which are predicates defined on <math>x</math>. An <math>x</math> is a feasible solution if it satisfies all the constraints.

A '''linear programming (LP)''' problem is an optimization problem with a linear objective function subject to a number of linear constraints. Formally, an LP is a problem that can be expressed in the following form:
::<math>
\begin{align}
\mbox{minimize} & \quad c_1x_1+c_2x_2+\cdots+c_nx_n\\
\mbox{subject to} & \quad a_{11}x_1+a_{12}x_2+\cdots+a_{1n}x_n\le b_1\\
& \quad a_{21}x_1+a_{22}x_2+\cdots+a_{2n}x_n\le b_2\\
& \qquad\qquad \vdots\\
& \quad a_{m1}x_1+a_{m2}x_2+\cdots+a_{mn}x_n\le b_m\\
\end{align}
</math>
where <math>a_{ij}</math>, <math>b_i</math> and <math>c_j</math> are constants and <math>x_j</math> are variables. For maximization problem, or constraints given by "<math>\ge</math>", we can multiply the coefficients by <math>-1</math> and still write the LP in the above form.

We can describe the programming in the vector form. Let <math>x</math> be a vector of <math>n</math> variables. Let <math>A=(a_{ij})</math> be an <math>m\times n</math> matrix of constant entries, <math>b=(b_1,\ldots,b_m)</math> be a vector of <math>m</math> constant entries, and <math>c=(c_1,\ldots,c_n)</math> be a vector of <math>n</math> constant entries. Then an LP is expressed as:
::<math>
\begin{align}
\mbox{minimize} & \quad c^T x\\
\mbox{subject to} & \quad Ax\le b
\end{align}
</math>
We call it the '''canonical form''' of linear programming.

=== The geometry of LPs ===
For <math>n</math>-dimensional space <math>\mathbb{R}^n</math>, a linear constraint <math>ax\le b</math> specifies a [http://en.wikipedia.org/wiki/Half-space halfspace]. The feasible set specified by <math>m</math> constraints together is an intersection of <math>m</math> halfspaces, thus, a [http://en.wikipedia.org/wiki/Convex_polytope convex polytope].

The convex polytope is defined by <math>Ax\le b</math>.

The LP can be thought as given an <math>n</math>-dimensional polytope and a linear (affine) function <math>f:\mathbb{R}^n\rightarrow \mathbb{R}</math>, looking for a point <math>x</math> in the polytope with the smallest function value <math>f(x)</math>.

We can choose a number of constraints in the system <math>Ax\le b</math>, and make them hold with equality. This would define a subspace of <math>\mathbb{R}^n</math>. In particular, if we choose <math>n</math> linearly independent constraints to form an <math>n\times n</math> submatrix <math>A'</math> and the corresponding <math>n</math>-dimensional vector <math>b'</math>, solving <math>A'x=b'</math> would give us exactly one point <math>x</math>. If this <math>x</math> is in the polytope, i.e. <math>x</math> satisfies that <math>Ax\le b</math>, we call such <math>x</math> a '''vertex''' of the polytope <math>Ax\le b</math>. In the context of LP, it is also called a '''basic feasible solution (''bfs'')''' of the LP.

A key observation for LP is that there is a basic feasible solution which is optimal, i.e. the optimal solution is a vertex of the polytope.

=== The simplex algorithms ===
The [http://en.wikipedia.org/wiki/Simplex_algorithm simplex algorithm] by George Dantzig solves the linear programming by moving from vertex to vertex of the convex polytope, each time making some progress towards optimizing the objective function. Eventually the algorithm reaches a vertex which is a [http://en.wikipedia.org/wiki/Local_optimum local optimum]. In a convex set, a locally optimal point is also global optimal, thus the simplex algorithm returns an optimal solution in finite steps.

The problem with the simplex algorithm is that in some bad polytopes, it takes the simplex algorithm exponentially many steps to reach the optimum. The simplex algorithm is actually a class of algorithms defined by various '''pivoting rules''', which describe how to move locally from one vertex to another. The original pivoting rule proposed by Dantzig has an exponentially large worst-case time complexity (though works very good in practice). To-date, it is still unknown whether there exists any deterministic simplex algorithm with sub-exponential worst-case complexity.

People have tried randomized pivoting rules, and there are some amazing progresses have been made:
* Kalai 1992: there is a randomized simplex algorithm with sub-exponential time complexity.
* Kelner-Spielman 2006: there is a polynomial time randomized simplex algorithm.

Although we do not know whether there exists deterministic poly-time simplex algorithm, we do know that LP can be solved by deterministic poly-time algorithms, i.e. LP is in '''P'''. The following two algorithms use different ideas than the simplex algorithm, and are both in poly-time:
* The ellipsoid algorithm.
* Interior point methods.
Although these algorithms guarantee polynomial time complexity in the worst-case, their performances are worse than the simplex algorithms. However, the ideas of these algorithms can be used to solve more general mathematical programmings, such as [http://en.wikipedia.org/wiki/Convex_programming convex programmings].

=== An LP solver via random walks ===
We discuss a new algorithm for solving LP which is based on random sampling, introduced by Bertsimas and Vempala in 2004.

The problem of solving an LP can be reduced to that given a convex polytope <math>Ax\le b</math>, test whether the polytope is empty. We call this problem the "''feasibility test''".

Suppose we have an LP:
::<math>
\begin{align}
\mbox{minimize} & \quad c^T x\\
\mbox{subject to} & \quad Ax\le b
\end{align}
</math>
To see this LP can be solved by feasibility testing, we treat the <math>c^Tx\le d</math> for some parameter <math>d</math> as an additional constraint, and define a new polytope <math>A'x\le b'</math>. The smallest <math>d</math> that the polytope <math>A'x\le b'</math> is not empty is the optimal solution to the original LP. We can find this smallest <math>d</math> by binary search if we can efficiently test the emptiness of a convex polytope.

We assume that the convex polytope <math>P</math> is contained in the axis-aligned cube of width <math>R</math> centered at the origin; further if <math>P</math> is non-empty then it contains a cube of width <math>r</math>. The parameter <math>L</math> is equal to <math>\log\frac{R}{r}</math>.

{|border="1"
|'''Algorithm''': feasibility test
|-
|'''Input''': an <math>n</math>-dimensional convex polytope <math>P</math> defined by the system <math>Ax\le b</math>.
'''Output''': a point <math>x\in P</math> in the polytope, or a guarantee that <math>P</math> is empty.
|-
|
:Let <math>K</math> be the axis-aligned cube of side length <math>R</math> and center <math>z=\boldsymbol{0}</math>.
:Repeat for <math>2nL</math> times do:
:: If <math>Az\le b</math>, return <math>z</math>.
:: Pick a constraint violated by <math>z</math>, say that <math>A_i z>b_i</math>, and define the halfspace
:::<math>H=\{x\mid A_i x\le A_i z\}</math>.
::Set <math>K=K\cap H</math>. Uniformly sample <math>N</math> random points <math>y^{(1)},y^{(2)},\ldots,y^{(N)}</math> from <math>K</math> and let
:::<math>z=\frac{1}{N}\sum_{i=1}^Ny^{(i)}</math>.
:Report <math>P</math> is empty.
|}

The number of samples required in each iteration, <math>N=O((\log m)^2)</math>, where <math>m</math> is the number of linear constraints.

It is easy to see that at any iteration, <math>K</math> is a convex set, and <math>P\subseteq K</math> for the current convex set <math>K</math>.

The idea of the algorithm is based on computing the [http://en.wikipedia.org/wiki/Centroid centroid] of the convex set. It is know that if we could compute the exact centroid in each iteration, then the volume of <math>K</math> drops by a constant factor <math>(1-1/e)</math> in each iteration, thus reaches the smallest possible volume of a cube of width <math>r</math> within polynomial number of iterations. But, finding the centroid of a convex polytope, is '''#P-hard'''.

The idea behind the algorithm is that an approximate centroid can be computed using <math>N=O((\log m)^2)</math> random points and the volume of <math>K</math> is drops by a constant factor with high probability in each iteration with this choice of <math>z</math>.

The uniform sampling in <math>K</math> is not easy. But since <math>K</math> is also a convex polytope, we can approximate the uniform sampling by near-uniform sampling from <math>K</math> by rapid mixing random walks. The random walks used here are similar to the ones used in the volume estimation of convex bodies. We could use the basic grid walk, or more advanced ball walk, or hit-and-run walk.

With rapid mixing random walks, the resulting LP solver runs in poly-time.

The details are omitted. You may check the following paper if interested:
* Bertsimas and Vempala, “''Solving convex programs by random walks''.” JACM 2004.

Randomized Algorithms (Spring 2010)/Approximate counting, linear programming

2010-05-24T19:47:55Z

172.21.4.18: /* FPRAS */

== Counting Problems ==

=== Complexity model ===
Recall the class '''NP''' of decision problems (the problems with "yes" or "no" answers). Formally, denoting by <math>\{0,1\}^*</math> the set of all boolean strings of any lengths, a decision problem is a function <math>f:\{0,1\}^*\rightarrow\{0,1\}</math>.
{|border="1"
|'''Definition (NP)'''
:A function <math>f:\{0,1\}^*\rightarrow\{0,1\}</math> is in '''NP''' if there exist a polynomial <math>p</math> and a poly-time algorithm <math>V</math> with boolean output such that for every <math>x\in\{0,1\}^*</math>,
::<math>f(x)=1 \Leftrightarrow \exists y\in\{0,1\}^{p(|x|)}, \mbox{such that } V(x,y)=1\,</math>.
|}
Intuitively, the '''NP''' class contains all the decision problems such that the answer is "yes" if and only if there ''exists'' a '''certificate''' which can be verified in poly-time.

In many contexts, we are interested not just in the ''existence'' of certificate but actually in counting the ''number'' of certificates. This leads to the definition of the class '''#P''' (pronounced "sharp p"). Only now the output of the function is a natural number.

{|border="1"
|'''Definition (#P)'''
:A function <math>f:\{0,1\}^*\rightarrow\mathbb{N}</math> is in '''#P''' if there exist a polynomial <math>p</math> and a poly-time algorithm <math>V</math> with boolean output such that for every <math>x\in\{0,1\}^*</math>,
::<math>f(x)=\left|\left\{y\in\{0,1\}^{p(|x|)}\mid A(x,y)=1\right\}\right|</math>.
|}
You may notice the similarity between the two definitions. The difference is that now <math>f(x)</math> does not just indicates the existence of a certificate, but gives the actual number of certificates.

Why should we care about the number of "certificates" at all?
In combinatorics, a counting problem is usually formulated as counting the number of combinatorial objects with some particular structure, such as the number of trees, or Latin squares. The '''#P''' class contains the counting problems that the "structure" requirements are easy to check.

;Examples of problems in #P
:<nowiki>#</nowiki>cycles: given as input a graph <math>G</math>, count the number of cycles in <math>G</math>.
:<nowiki>#</nowiki>SAT: given as input a boolean formula <math>\phi</math>, count the number of satisfying assignments for <math>\phi</math>.

The second example shows that the '''<nowiki>#</nowiki>P''' may be harder than '''NP''', because we know that SAT (given a boolean formular, decide the existence of satisfying assignments) is '''NP-complete''', and its counting version <nowiki>#</nowiki>SAT now ask for the exact number of satisfying assignments.

The class '''FP''' contains all the functions <math>f:\{0,1\}^*\rightarrow\mathbb{N}</math> computable by poly-time algorithms. The classes '''FP''' and '''#P''' are the analogs of '''P''' and '''NP''' for counting problems.

A '''reduction''' from a problem <math>f</math> to a problem <math>g</math> is a mapping <math>\phi</math> which maps instances of <math>f</math> to instances of <math>g</math> such that for any instance <math>x</math> of <math>f</math>,
:<math>f(x)=g(\phi(x))</math>.
In other words, <math>\phi</math> "reduces" the task of solving <math>f</math> to the task of solving <math>g</math>. A problem <math>f</math> is said to be '''poly-time reducible''' to the problem <math>g</math>, if there exists a reduction <math>\phi</math> from <math>f</math> to <math>g</math> and <math>\phi</math> is poly-time computable.

A problem <math>f</math> is '''#P-hard''' if every problem in '''#P''' is poly-time reducible to <math>f</math>. A problem <math>f</math> is '''#P-complete''' if <math>f\in</math>'''#P''' and <math>f</math> is '''#P-hard'''. That is to say, '''#P-complete''' problems are the "hardest" problems in '''#P'''.

Generally, '''#P-complete''' problems are very hard, because if we can solve any of these problems in poly-time, then '''#P'''='''FP''', which implies that '''NP'''='''P'''. On the other hand, we do not know whether '''NP'''='''P''' could imply that '''#P'''='''FP'''.

=== FPRAS ===

{|border="1"
|'''Definition (FPRAS)'''
:A '''polynomial randomized approximation scheme (PRAS)''' for a problem <math>f:\{0,1\}^*\rightarrow\mathbb{R}</math> is a randomized algorithm <math>A</math> that takes an input instance <math>x</math> and a real number <math>\epsilon>0</math>, and in time polynomial in <math>n=|x|</math> returns <math>A(x)</math> such that
::<math>\Pr[(1-\epsilon)f(x)\le A(x)\le (1+\epsilon)f(x)]\ge\frac{3}{4}.</math>
:A '''fully polynomial randomized approximation scheme (FPRAS)''' is a PRAS whose running time is polynomially in both <math>n</math> and <math>1/\epsilon</math>.
|}

== Approximate Counting ==
Let us consider the following abstract problem.

Let <math>U</math> be a finite set of known size, and let <math>G\subseteq U</math>. We want to compute the size of <math>G</math>, namely <math>|G|</math>.

We assume two devices:
* A '''uniform sampler''' <math>\mathcal{U}</math>, which uniformly and independently samples a member of <math>U</math> upon each calling.
* A '''membership oracle''' of <math>G</math>, denoted <math>\mathcal{O}</math>. Given as the input an <math>x\in U</math>, <math>\mathcal{O}(x)</math> indicates whether or not <math>x</math> is a member of <math>G</math>.

Equipped by <math>\mathcal{U}</math> and <math>\mathcal{O}</math>, we can have the following Monte Carlo algorithm:
*Choose <math>N</math> independent samples from <math>U</math> by the uniform sampler <math>\mathcal{U}</math>, represented by the random variables <math>X_1,X_2,\ldots, X_N</math>.
* Let <math>Y_i</math> be the indicator random variable defined as <math>Y_i=\mathcal{O}(X_i)</math>, namely, <math>Y_i</math> indicates whether <math>X_i\in G</math>.
* Define the estimator random variable
::<math>Z=\frac{|U|}{N}\sum_{i=1}^N Y_i.</math>

It is easy to see that <math>\mathbf{E}[Z]=|G|</math> and we might hope that with high probability the value of <math>Z</math> is close to <math>|G|</math>. Formally, <math>Z</math> is called an <math>\epsilon</math>-approximation of <math>|G|</math> if
:<math>
(1-\epsilon)|G|\le Z\le (1+\epsilon)|G|.
</math>

The following theorem states that the probabilistic accuracy of the estimation depends on the number of samples and the ratio between <math>|G|</math> and <math>|U|</math>

{|border="1"
|'''Theorem (estimator theorem)'''
:Let <math>\alpha=\frac{|G|}{|U|}</math>. Then the Monte Carlo method yields an <math>\epsilon</math>-approximation to <math>|G|</math> with probability at least <math>1-\delta</math> provided
::<math>N\ge\frac{4}{\epsilon \alpha}\ln\frac{2}{\delta}</math>.
|}
'''Proof''': Use the Chernoff bound.

<math>\square</math>

A counting algorithm for the set <math>G</math> has to deal with the following three complications:
* Implement the membership oracle <math>\mathcal{O}</math>. This is usually straightforward, or assumed by the model.
* Implement the uniform sampler <math>\mathcal{U}</math>. As we have seen, this is usually approximated by random walks. How to design the random walk and bound its mixing rate is usually technical challenging, if possible at all.
* Deal with exponentially small <math>\alpha=\frac{|G|}{|U|}</math>. This requires us to cleverly choose the universe <math>U</math>. Sometimes this needs some nontrivial ideas.

=== Counting DNFs ===
A disjunctive normal form (DNF) formular is a disjunction (OR) of clauses, where each clause is a conjunction (AND) of literals. For example:
:<math>(x_1\wedge \overline{x_2}\wedge x_3)\vee(x_2\wedge x_4)\vee(\overline{x_1}\wedge x_3\wedge x_4)</math>.
Note the difference from the conjunctive normal forms (CNF).

Given a DNF formular <math>\phi</math> as the input, the problem is to count the number of satisfying assignments of <math>\phi</math>. This problem is '''#P-complete'''.

Naively applying the Monte Carlo method will not give a good answer. Suppose that there are <math>n</math> variables. Let <math>U=\{\mathrm{true},\mathrm{false}\}^n</math> be the set of all truth assignments of the <math>n</math> variables. Let <math>G=\{x\in U\mid \phi(x)=\mathrm{true}\}</math> be the set of satisfying assignments for <math>\phi</math>. The straightforward use of Monte Carlo method samples <math>N</math> assignments from <math>U</math> and check how many of them satisfy <math>\phi</math>. This algorithm fails when <math>|G|/|U|</math> is exponentially small, namely, when exponentially small fraction of the assignments satisfy the input DNF formula.

;The union of sets problem
We reformulate the DNF counting problem in a more abstract framework, called the '''union of sets''' problem.

Let <math>V</math> be a finite universe. We are given <math>m</math> subsets <math>H_1,H_2,\ldots,H_m\subseteq V</math>. The following assumptions hold:
*For all <math>i</math>, <math>|H_i|</math> is computable in poly-time.
*It is possible to sample uniformly from each individual <math>H_i</math>.
*For any <math>x\in V</math>, it can be determined in poly-time whether <math>x\in H_i</math>.

The goal is to compute the size of <math>H=\bigcup_{i=1}^m H_i</math>.

DNF counting can be interpreted in this general framework as follows. Suppose that the DNF formula <math>\phi</math> is defined on <math>n</math> variables, and <math>\phi</math> contains <math>m</math> clauses <math>C_1,C_2,\ldots,C_m</math>, where clause <math>C_i</math> has <math>k_i</math> literals. Without loss of generality, we assume that in each clause, each variable appears at most once.
* <math>V</math> is the set of all assignments.
*Each <math>H_i</math> is the set of satisfying assignments for the <math>i</math>-th clause <math>C_i</math> of the DNF formular <math>\phi</math>. Then the union of sets <math>H=\bigcup_i H_i</math> gives the set of satisfying assignments for <math>\phi</math>.
* Each clause <math>C_i</math> is a conjunction (AND) of literals. It is not hard to see that <math>|H_i|=2^{n-k_i}</math>, which is efficiently computable.
* Sampling from an <math>H_i</math> is simple: we just fix the assignments of the <math>k_i</math> literals of that clause, and sample uniformly and independently the rest <math>(n-k_i)</math> variable assignments.
* For each assignment <math>x</math>, it is easy to check whether it satisfies a clause <math>C_i</math>, thus it is easy to determine whether <math>x\in H_i</math>.

;The coverage algorithm
We now introduce the coverage algorithm for the union of sets problem.

Consider the multiset <math>U</math> defined by
:<math>U=H_1\uplus H_2\uplus\cdots \uplus H_m</math>,
where <math>\uplus</math> denotes the multiset union. It is more convenient to define <math>U</math> as the set
:<math>U=\{(x,i)\mid x\in H_i\}</math>.
For each <math>x\in H</math>, there may be more than one instances of <math>(x,i)\in U</math>. We can choose a unique representative among the multiple instances <math>(x,i)\in U</math> for the same <math>x\in H</math>, by choosing the <math>(x,i)</math> with the minimum <math>i</math>, and form a set <math>G</math>.

Formally, <math>G=\{(x,i)\in U\mid \forall (x,j)\in U, j\le i\}</math>. Every <math>x\in H</math> corresponds to a unique <math>(x,i)\in G</math> where <math>i</math> is the smallest among <math>x\in H_i</math>.

It is obvious that <math>G\subseteq U</math> and
:<math>|G|=|H|</math>.

Therefore, estimation of <math>|H|</math> is reduced to estimation of <math>|G|</math> with <math>G\subseteq U</math>. Then <math>|G|</math> can have an <math>\epsilon</math>-approximation with probability <math>(1-\delta)</math> in poly-time, if we can uniformly sample from <math>U</math> and <math>|G|/|U|</math> is suitably small.

An uniform sample from <math>U</math> can be implemented as follows:
* generate an <math>i\in\{1,2,\ldots,m\}</math> with probability <math>\frac{|H_i|}{\sum_{i=1}^m|H_i|}</math>;
* uniformly sample an <math>x\in H_i</math>, and return <math>(x,i)</math>.

It is easy to see that this gives a uniform member of <math>U</math>. The above sampling procedure is poly-time because each <math>|H_i|</math> can be computed in poly-time, and sampling uniformly from each <math>H_i</math> is poly-time.

We now only need to lower bound the ratio
:<math>\alpha=\frac{|G|}{|U|}</math>.

We claim that
:<math>\alpha\ge\frac{1}{m}</math>.
It is easy to see this, because each <math>x\in H</math> has at most <math>m</math> instances of <math>(x,i)</math> in <math>U</math>, and we already know that <math>|G|=|H|</math>.

Due to the estimator theorem, this needs <math>\frac{4m}{\epsilon}\ln\frac{2}{\delta}</math> uniform random samples from <math>U</math>.

This gives the coverage algorithm for the abstract problem of the union of sets. The DNF counting is a special case of it.

=== Permanents and perfect matchings ===

Let <math>U=\{u_1,u_2,\ldots,u_n\}</math> and <math>V=\{v_1,v_2,\ldots,v_n\}</math>. Consider a bipartite graph <math>G(U,V,E)</math>. An <math>M\subseteq E</math> is a '''perfect matching''' of <math>G</math> if every vertex of <math>G</math> has exactly one edge in <math>M</math> adjacent to it.

Given a bipartite graph <math>G(U,V,E)</math> with <math>|U|=|V|=n</math>, we want to count the number of perfect matchings of <math>G</math>. This problem can be reduced to computing the '''permanent''' of a square matrix.

{|border="1"
|'''Definition (permanent)'''
:Let <math>Q</math> be an <math>n\times n</math> matrix. The '''permanent''' of the matrix is defined as
::<math>\mathrm{per}(Q)=\sum_{\pi\in\mathbb{S}_n}\prod_{i=1}^n Q_{i,\pi(i)}</math>,
:where <math>\mathbb{S}_n</math> is the symmetric group of permutation of size <math>n</math>.
|}

If we multiply each term of the sum the sign of the permutation, then it gives us the determinant of the matrix,
:<math>\det(Q)=\sum_{\pi\in\mathbb{S}_n}\sgn(\pi)\prod_{i=1}^n Q_{i,\pi(i)}</math>,
where <math>\sgn(\pi)</math>, the sign of a permutation, is either <math>-1</math> or <math>+1</math>, according to whether the minimum number of pair-wise interchanges to achieve <math>\pi</math> from <math>(1,2,\ldots,n)</math> is odd or even.

Unlike the determinants, which are computable in poly-time, permanents are hard to compute, as permanents can be used to count the number of perfect matchings in a bipartite graph, which is '''#P-complete'''.

A bipartite graph <math>G(U,V,E)</math> with <math>|U|=|V|=n</math> can be represented by an <math>n\times n</math> matrix <math>Q</math> with 0-1 entries as follows:
* Each row of <math>Q</math> corresponds to a vertex in <math>U</math> and each column of <math>Q</math> corresponds to a vertex in <math>V</math>.
::<math>Q_{ij}=\begin{cases}
1 & \mbox{if }i\sim j,\\
0 & \mbox{otherwise}.
\end{cases}</math>
Note the subtle difference between the definition of <math>Q</math> and the adjacency matrix.

Each perfect matching corresponds to a permutation <math>\pi</math> of <math>U</math> with <math>(u,\pi(u))\in E</math> for every <math>u\in U</math>, which corresponds to a permutation <math>\pi\in\mathbb{S}_n</math> with <math>\prod_{i=1}^n Q_{i,\pi(i)}=1</math>. It is than easy to see that <math>\mathrm{per}(Q)</math> gives the number of perfect matchings in the bipartite graph <math>G</math>.

It is known that counting the number of perfect matchings in a bipartite graph is '''#P-hard'''. Since this problem can be reduced to computing the permanent, thus the problem of computing the permanents is also '''#P-hard'''.

Now we show that with randomization, we can approximate the number of perfect matchings in a bipartite graph. In particular, we will give an FPRAS for counting the perfect matchings in a dense bipartite graph.

==== The Jerrum-Sinclair algorithm ====
Fix a bipartite graph <math>G(U,V,E)</math> with <math>|U|=|V|=n</math>. Let <math>\mathcal{M}_k</math> be the set of matchings of size <math>k</math> in <math>G</math>, and let <math>m_k=|\mathcal{M}_k|</math>. Thus, <math>\mathcal{M}_k</math> is the set of perfect matchings in <math>G</math>, and our goal is to compute <math>m_n</math>.

Let <math>r_k=\frac{m_k}{m_{k-1}}</math> for <math>1<k\le n</math>. Then
:<math>m_k=m_{k-1}r_k</math>,
which gives us a recursion to compute the <math>m_n</math>, as
:<math>m_n=m_{1}\frac{m_2}{m_1}\cdot\frac{m_3}{m_2}\cdots\frac{m_n}{m_{n-1}}=m_1\prod_{k=2}^n r_k</math>,
where <math>m_1=|\mathcal{M}_1|</math> is the number of matchings of size 1 in the bipartite graph <math>G</math>, which is just the number of edges in <math>G</math>. Therefore, <math>m_n</math> can be computed once we know <math>r_k</math> for <math>1<k\le n</math>.

Each <math>r_k</math> can be estimated by sampling uniformly from the set <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>. The algorithm for estimating <math>m_n</math> is outlined as:
# For each <math>1<k\le n</math>, have an FPRAS for computing the <math>r_k</math> by uniform sampling sufficiently many members from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math> as
::*uniformly sample <math>N</math> matching from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>, for some polynomially large <math>N</math>;
::* assuming that there are <math>X</math> sampled matchings of size <math>k</math>, return <math>r_k</math> as <math>r_k=\frac{X}{N-X}</math>.
:2. Compute <math>m_n</math> as <math>m_n=m_1\prod_{k=2}^n r_k</math>, where <math>m_1=|E|</math>.

There are several issues that we have to deal with in order to have a fully functional FPRAS for counting perfect matchings.
* By taking the product of <math>r_k</math>'s, the errors for individual <math>r_k</math>'s add up.
* In order to accurately estimate <math>r_k</math> by sampling from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>, the ratio <math>r_k</math> should be within the range <math>\left[\frac{1}{\alpha},\alpha\right]</math> for some <math>\alpha</math> within polynomial of <math>n</math>.
* Implement the uniform sampling from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>.

;Estimator for each <math>r_k</math>

; Accumulation of errors

; Near-uniform sampling from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>
In the last lecture, we have shown that by random walk, we can sample a near-uniform member of <math>\mathcal{M}_n\cup\mathcal{M}_{n-1}</math> in poly-time.

=== Volume estimation ===
We consider the problem of computing the volume of a given [http://en.wikipedia.org/wiki/Convex_body convex body] <math>K</math> in <math>n</math> dimensions.

We use <math>\Upsilon(K)\,</math> to denote the volume of the convex body <math>K</math>. Abstractly, the problem is that given as input a convex body <math>K</math> in <math>n</math> dimensions, return the <math>\Upsilon(K)\,</math>.

We should be more specific about the input model. Since we allow an arbitrary convex body as input, it is not even clear how to describe the body. We assume that <math>K</math> is described by means of a '''membership oracle''' <math>\mathcal{O}</math>, such that for a '''query''' of an <math>n</math>-dimensional point <math>x</math>, <math>\mathcal{O}(x)</math> indicates whether <math>x\in K</math>.

For example, the <math>n</math>-dimensional convex body defined by the intersection of <math>m</math> half-spaces, which is the set of feasible solutions to a system of <math>m</math> linear constraints, can be described as
:<math>A x\le \boldsymbol{b}</math>,
for some <math>m\times n</math> matrix <math>A</math>, and <math>m</math>-dimensional vector <math>b</math>. For a query of an <math>x</math>, the membership oracle <math>\mathcal{O}</math> just check whether <math>Ax\le b</math>.

For deterministic algorithms, there are negative news for this problem.

{|border="1"
|'''Theorem (Bárány-Füredi 1987)'''
:Suppose that a deterministic poly-time algorithm uses the membership oracle for a convex body <math>K</math> in <math>n</math> dimensions, and generates an upper bound <math>\Upsilon_u\,</math> and a lower bound <math>\Upsilon_\ell\,</math> on the volume <math>\Upsilon(K)\,</math> of <math>K</math>. Then, there is a convex body <math>K</math> and a constant <math>c</math> such that
::<math>\frac{\Upsilon_u}{\Upsilon_\ell}\ge c\left(\frac{n}{\log n}\right)^n</math>.
|}
That is said, with deterministic algorithms, we cannot even approximate the volume within a wildly loose range.

Dyer-Frieze-Kannan come up with an idea of estimating the volume of <math>K</math> by sampling near-uniformly from convex sets. They reduce the problem of computing approximately the volume of convex bodies to this sampling problem and thus give the first FPRAS for the volume of convex bodies.

For any convex body <math>K</math>, it encloses some <math>n</math>-dimensional ball and is also enclosed by another ball with larger radius. We assume that the convex body <math>K</math> encloses the unit ball around the origin, and is enclosed by a larger ball round the origin with polynomially large radius. Formally, we assume that
:<math>B(0,1)\subseteq K\subseteq B(0,n^c)</math> for some constant <math>c</math>,
where <math>B(p,r)</math> denotes a ball of radius <math>r</math> with <math>p</math> as center, i.e. <math>B(p,r)=\{x\mid \|x-p\|\le r \}</math>. We can make this assumption because it is known that for any convex body <math>K</math>, there exists a linear transformation <math>\tau</math> which can be found within poly-time such that <math>\tau K</math> transforms the <math>K</math> to a convex body that satisfies the assumption, and preserves the ratio of the volumes of the balls to the convex body.

The volumes of the balls are easy to compute. If only the ratio between the convex body and the ball which encloses it, is sufficiently large, then we can apply the Monte Carlo method to estimate <math>\Upsilon(K)\,</math> by uniformly sampling from the ball.

However, in high-dimension, the ratio between the volume of a convex body and the volume of ball which encloses it, can be exponentially small. This is caused by the so called the "[http://en.wikipedia.org/wiki/Curse_of_dimensionality curse of dimensionality]".

Instead of having an outer ball and an inner ball, we define a sequence of balls:
:<math>B_0=B(0,\lambda^0), B_1=B(0,\lambda^1), B_2=B(0,\lambda^2),\ldots, B_m=B(0,\lambda^m)</math>,
where <math>\lambda=(1+\frac{1}{n})</math> and <math>m</math> is the smallest positive integer such that <math>\lambda^m\ge n^c</math>. Therefore, the inner ball <math>B(0,1)=B_0</math>, the outer ball <math>B(0,n^c)\subseteq B_m</math>, and <math>m</math> is within polynomial of <math>n</math>. In fact, <math>m\approx cn\ln n</math>.

This sequence of balls naturally defines a sequence of convex bodies by intersections as <math>K_i=B_i\cap K</math>. It is obvious that
:<math>B(0,1)=K_0\subseteq K_1\subseteq K_2\subseteq\cdots\subseteq K_m=K</math>.
Balls are convex, and since the intersection of convex bodies is still convex, the sequence of <math>K_i</math> is a sequence of convex bodies.

We have the telescopic product:
:<math>
\frac{\Upsilon(K_0)}{\Upsilon(K_1)}\cdot\frac{\Upsilon(K_1)}{\Upsilon(K_2)}\cdots\frac{\Upsilon(K_{m-1})}{\Upsilon(K_m)}=\frac{\Upsilon(K_0)}{\Upsilon(K_m)}=\frac{\Upsilon(B(0,1))}{\Upsilon(K)}.
</math>
Therefore, the volume <math>\Upsilon(K)\,</math> can be computed as
:<math>\Upsilon(K)=\Upsilon(B(0,1))\cdot\prod_{i=1}^{m}\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math>.

The volume of unite ball <math>\Upsilon(B(0,1))\,</math> can be precisely computed in poly-time. Each <math>\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math> is computed by near-uniform sampling from <math>K_{i}</math>, which encloses <math>K_{i-1}</math>.

Another observation is that the ratio <math>\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math> is well-bounded. Recall that <math>K_i=B(0,\lambda^i)\cap K</math> where <math>\lambda=(1+\frac{1}{n})</math>. It can be proved that in <math>n</math> dimensions, the volume of <math>K_i</math> is at most <math>\lambda^n</math> times the <math>K_{i-1}</math>, thus the ratio <math>\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math> is at most <math>\lambda^n=O(1)</math>. By the estimator theorem, we can have an FPRAS for the ratio <math>\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math> if we can uniformly sample from <math>K_i</math>.

Uniformly sampling from an arbitrary convex body is replaced by near-uniform sampling achieved by random walks. In the original walk of Dyer-Frieze-Kannan, they consider the random walk over <math>n</math>-dimensional discrete grid points enclosed by <math>K</math>, and prove that the walk is rapid mixing. This gives us the first FPRAS for volume estimation which runs in <math>\tilde{O}(n^{23})</math>, where <math>\tilde{O}(\cdot)</math> ignores the polylogarithmic factors.

The time bound was later improved by a series of works, each introducing some new ideas.

{|border="1"
| || complexity || new ingredient(s)
|-
| Dyer-Frieze-Kannan 1991 || align="center" | <math>n^{23}</math> || everything
|-
| Lovász-Simonovits 1990 || align="center" | <math>n^{16}</math> || localization lemma
|-
| Applegate-Kannan 1990 || align="center" | <math>n^{10}</math> || logconcave sampling
|-
| Lovász 1990 || align="center" | <math>n^{10}</math> || ball walk
|-
| Dyer-Frieze 1991 || align="center" | <math>n^8</math> || better error analysis
|-
| Lovász-Simonovits 1993 || align="center" | <math>n^7</math> || many improvements
|-
| Kannan-Lovász-Simonovits 1997 || align="center" | <math>n^5</math> || isotropy, speedy walk
|-
| Lovász-Vempala 2003 || align="center" | <math>n^4</math> || simulated annealing, hit-and-run
|-
|}
(cited from "Geometric Random Walks: A Survey" by Santosh Vempala.)

The current best upper bound is <math>\tilde{O}(n^4)</math> due to Lovász and Vempala in 2003. It is conjectured that the optimal bound is <math>\Theta(n^3)</math>.

== Linear Programming ==
Given a function <math>f:\mathbb{R}^n\rightarrow\mathbb{R}</math> and a set <math>\mathcal{F}\subseteq\mathbb{R}^n</math>, an '''optimization problem''' is the problem of finding an <math>x\in\mathcal{F}</math> with the optimal (minimum) <math>f(x)</math>. Formally, the problem can be expressed as:
::<math>
\begin{align}
\mbox{minimize} & \quad f(x)\\
\mbox{subject to} &\quad x\in\mathcal{F}
\end{align}
</math>
where <math>x\in\mathcal{R}^n</math> is a vector of <math>n</math> variables. For the problem of maximizing a function <math>f</math>, we just minimizes <math>-f</math> instead.

We call the function <math>f</math> the '''objective function''' and call any <math>x\in\mathcal{F}</math> a '''feasible solution''' of the problem. A feasible solution that minimizes <math>f(x)</math> is called an '''optimal solution'''. Our task is to find an optimal solution.

The feasible set <math>\mathcal{F}</math> is usually given by a number of '''constraints''' <math>P_1,P_2,\ldots,P_m</math>, which are predicates defined on <math>x</math>. An <math>x</math> is a feasible solution if it satisfies all the constraints.

A '''linear programming (LP)''' problem is an optimization problem with a linear objective function subject to a number of linear constraints. Formally, an LP is a problem that can be expressed in the following form:
::<math>
\begin{align}
\mbox{minimize} & \quad c_1x_1+c_2x_2+\cdots+c_nx_n\\
\mbox{subject to} & \quad a_{11}x_1+a_{12}x_2+\cdots+a_{1n}x_n\le b_1\\
& \quad a_{21}x_1+a_{22}x_2+\cdots+a_{2n}x_n\le b_2\\
& \qquad\qquad \vdots\\
& \quad a_{m1}x_1+a_{m2}x_2+\cdots+a_{mn}x_n\le b_m\\
\end{align}
</math>
where <math>a_{ij}</math>, <math>b_i</math> and <math>c_j</math> are constants and <math>x_j</math> are variables. For maximization problem, or constraints given by "<math>\ge</math>", we can multiply the coefficients by <math>-1</math> and still write the LP in the above form.

We can describe the programming in the vector form. Let <math>x</math> be a vector of <math>n</math> variables. Let <math>A=(a_{ij})</math> be an <math>m\times n</math> matrix of constant entries, <math>b=(b_1,\ldots,b_m)</math> be a vector of <math>m</math> constant entries, and <math>c=(c_1,\ldots,c_n)</math> be a vector of <math>n</math> constant entries. Then an LP is expressed as:
::<math>
\begin{align}
\mbox{minimize} & \quad c^T x\\
\mbox{subject to} & \quad Ax\le b
\end{align}
</math>
We call it the '''canonical form''' of linear programming.

=== The geometry of LPs ===
For <math>n</math>-dimensional space <math>\mathbb{R}^n</math>, a linear constraint <math>ax\le b</math> specifies a [http://en.wikipedia.org/wiki/Half-space halfspace]. The feasible set specified by <math>m</math> constraints together is an intersection of <math>m</math> halfspaces, thus, a [http://en.wikipedia.org/wiki/Convex_polytope convex polytope].

The convex polytope is defined by <math>Ax\le b</math>.

The LP can be thought as given an <math>n</math>-dimensional polytope and a linear (affine) function <math>f:\mathbb{R}^n\rightarrow \mathbb{R}</math>, looking for a point <math>x</math> in the polytope with the smallest function value <math>f(x)</math>.

We can choose a number of constraints in the system <math>Ax\le b</math>, and make them hold with equality. This would define a subspace of <math>\mathbb{R}^n</math>. In particular, if we choose <math>n</math> linearly independent constraints to form an <math>n\times n</math> submatrix <math>A'</math> and the corresponding <math>n</math>-dimensional vector <math>b'</math>, solving <math>A'x=b'</math> would give us exactly one point <math>x</math>. If this <math>x</math> is in the polytope, i.e. <math>x</math> satisfies that <math>Ax\le b</math>, we call such <math>x</math> a '''vertex''' of the polytope <math>Ax\le b</math>. In the context of LP, it is also called a '''basic feasible solution (''bfs'')''' of the LP.

A key observation for LP is that there is a basic feasible solution which is optimal, i.e. the optimal solution is a vertex of the polytope.

=== The simplex algorithms ===
The [http://en.wikipedia.org/wiki/Simplex_algorithm simplex algorithm] by George Dantzig solves the linear programming by moving from vertex to vertex of the convex polytope, each time making some progress towards optimizing the objective function. Eventually the algorithm reaches a vertex which is a [http://en.wikipedia.org/wiki/Local_optimum local optimum]. In a convex set, a locally optimal point is also global optimal, thus the simplex algorithm returns an optimal solution in finite steps.

The problem with the simplex algorithm is that in some bad polytopes, it takes the simplex algorithm exponentially many steps to reach the optimum. The simplex algorithm is actually a class of algorithms defined by various '''pivoting rules''', which describe how to move locally from one vertex to another. The original pivoting rule proposed by Dantzig has an exponentially large worst-case time complexity (though works very good in practice). To-date, it is still unknown whether there exists any deterministic simplex algorithm with sub-exponential worst-case complexity.

People have tried randomized pivoting rules, and there are some amazing progresses have been made:
* Kalai 1992: there is a randomized simplex algorithm with sub-exponential time complexity.
* Kelner-Spielman 2006: there is a polynomial time randomized simplex algorithm.

Although we do not know whether there exists deterministic poly-time simplex algorithm, we do know that LP can be solved by deterministic poly-time algorithms, i.e. LP is in '''P'''. The following two algorithms use different ideas than the simplex algorithm, and are both in poly-time:
* The ellipsoid algorithm.
* Interior point methods.
Although these algorithms guarantee polynomial time complexity in the worst-case, their performances are worse than the simplex algorithms. However, the ideas of these algorithms can be used to solve more general mathematical programmings, such as [http://en.wikipedia.org/wiki/Convex_programming convex programmings].

=== An LP solver via random walks ===
We discuss a new algorithm for solving LP which is based on random sampling, introduced by Bertsimas and Vempala in 2004.

The problem of solving an LP can be reduced to that given a convex polytope <math>Ax\le b</math>, test whether the polytope is empty. We call this problem the "''feasibility test''".

Suppose we have an LP:
::<math>
\begin{align}
\mbox{minimize} & \quad c^T x\\
\mbox{subject to} & \quad Ax\le b
\end{align}
</math>
To see this LP can be solved by feasibility testing, we treat the <math>c^Tx\le d</math> for some parameter <math>d</math> as an additional constraint, and define a new polytope <math>A'x\le b'</math>. The smallest <math>d</math> that the polytope <math>A'x\le b'</math> is not empty is the optimal solution to the original LP. We can find this smallest <math>d</math> by binary search if we can efficiently test the emptiness of a convex polytope.

We assume that the convex polytope <math>P</math> is contained in the axis-aligned cube of width <math>R</math> centered at the origin; further if <math>P</math> is non-empty then it contains a cube of width <math>r</math>. The parameter <math>L</math> is equal to <math>\log\frac{R}{r}</math>.

{|border="1"
|'''Algorithm''': feasibility test
|-
|'''Input''': an <math>n</math>-dimensional convex polytope <math>P</math> defined by the system <math>Ax\le b</math>.
'''Output''': a point <math>x\in P</math> in the polytope, or a guarantee that <math>P</math> is empty.
|-
|
:Let <math>K</math> be the axis-aligned cube of side length <math>R</math> and center <math>z=\boldsymbol{0}</math>.
:Repeat for <math>2nL</math> times do:
:: If <math>Az\le b</math>, return <math>z</math>.
:: Pick a constraint violated by <math>z</math>, say that <math>A_i z>b_i</math>, and define the halfspace
:::<math>H=\{x\mid A_i x\le A_i z\}</math>.
::Set <math>K=K\cap H</math>. Uniformly sample <math>N</math> random points <math>y^{(1)},y^{(2)},\ldots,y^{(N)}</math> from <math>K</math> and let
:::<math>z=\frac{1}{N}\sum_{i=1}^Ny^{(i)}</math>.
:Report <math>P</math> is empty.
|}

The number of samples required in each iteration, <math>N=O((\log m)^2)</math>, where <math>m</math> is the number of linear constraints.

It is easy to see that at any iteration, <math>K</math> is a convex set, and <math>P\subseteq K</math> for the current convex set <math>K</math>.

The idea of the algorithm is based on computing the [http://en.wikipedia.org/wiki/Centroid centroid] of the convex set. It is know that if we could compute the exact centroid in each iteration, then the volume of <math>K</math> drops by a constant factor <math>(1-1/e)</math> in each iteration, thus reaches the smallest possible volume of a cube of width <math>r</math> within polynomial number of iterations. But, finding the centroid of a convex polytope, is '''#P-hard'''.

The idea behind the algorithm is that an approximate centroid can be computed using <math>N=O((\log m)^2)</math> random points and the volume of <math>K</math> is drops by a constant factor with high probability in each iteration with this choice of <math>z</math>.

The uniform sampling in <math>K</math> is not easy. But since <math>K</math> is also a convex polytope, we can approximate the uniform sampling by near-uniform sampling from <math>K</math> by rapid mixing random walks. The random walks used here are similar to the ones used in the volume estimation of convex bodies. We could use the basic grid walk, or more advanced ball walk, or hit-and-run walk.

With rapid mixing random walks, the resulting LP solver runs in poly-time.

The details are omitted. You may check the following paper if interested:
* Bertsimas and Vempala, “''Solving convex programs by random walks''.” JACM 2004.

Randomized Algorithms (Spring 2010)/Approximate counting, linear programming

2010-05-24T19:24:21Z

172.21.4.18: /* Complexity model */

== Counting Problems ==

=== Complexity model ===
Recall the class '''NP''' of decision problems (the problems with "yes" or "no" answers). Formally, denoting by <math>\{0,1\}^*</math> the set of all boolean strings of any lengths, a decision problem is a function <math>f:\{0,1\}^*\rightarrow\{0,1\}</math>.
{|border="1"
|'''Definition (NP)'''
:A function <math>f:\{0,1\}^*\rightarrow\{0,1\}</math> is in '''NP''' if there exist a polynomial <math>p</math> and a poly-time algorithm <math>V</math> with boolean output such that for every <math>x\in\{0,1\}^*</math>,
::<math>f(x)=1 \Leftrightarrow \exists y\in\{0,1\}^{p(|x|)}, \mbox{such that } V(x,y)=1\,</math>.
|}
Intuitively, the '''NP''' class contains all the decision problems such that the answer is "yes" if and only if there ''exists'' a '''certificate''' which can be verified in poly-time.

In many contexts, we are interested not just in the ''existence'' of certificate but actually in counting the ''number'' of certificates. This leads to the definition of the class '''#P''' (pronounced "sharp p"). Only now the output of the function is a natural number.

{|border="1"
|'''Definition (#P)'''
:A function <math>f:\{0,1\}^*\rightarrow\mathbb{N}</math> is in '''#P''' if there exist a polynomial <math>p</math> and a poly-time algorithm <math>V</math> with boolean output such that for every <math>x\in\{0,1\}^*</math>,
::<math>f(x)=\left|\left\{y\in\{0,1\}^{p(|x|)}\mid A(x,y)=1\right\}\right|</math>.
|}
You may notice the similarity between the two definitions. The difference is that now <math>f(x)</math> does not just indicates the existence of a certificate, but gives the actual number of certificates.

Why should we care about the number of "certificates" at all?
In combinatorics, a counting problem is usually formulated as counting the number of combinatorial objects with some particular structure, such as the number of trees, or Latin squares. The '''#P''' class contains the counting problems that the "structure" requirements are easy to check.

;Examples of problems in #P
:<nowiki>#</nowiki>cycles: given as input a graph <math>G</math>, count the number of cycles in <math>G</math>.
:<nowiki>#</nowiki>SAT: given as input a boolean formula <math>\phi</math>, count the number of satisfying assignments for <math>\phi</math>.

The second example shows that the '''<nowiki>#</nowiki>P''' may be harder than '''NP''', because we know that SAT (given a boolean formular, decide the existence of satisfying assignments) is '''NP-complete''', and its counting version <nowiki>#</nowiki>SAT now ask for the exact number of satisfying assignments.

The class '''FP''' contains all the functions <math>f:\{0,1\}^*\rightarrow\mathbb{N}</math> computable by poly-time algorithms. The classes '''FP''' and '''#P''' are the analogs of '''P''' and '''NP''' for counting problems.

A '''reduction''' from a problem <math>f</math> to a problem <math>g</math> is a mapping <math>\phi</math> which maps instances of <math>f</math> to instances of <math>g</math> such that for any instance <math>x</math> of <math>f</math>,
:<math>f(x)=g(\phi(x))</math>.
In other words, <math>\phi</math> "reduces" the task of solving <math>f</math> to the task of solving <math>g</math>. A problem <math>f</math> is said to be '''poly-time reducible''' to the problem <math>g</math>, if there exists a reduction <math>\phi</math> from <math>f</math> to <math>g</math> and <math>\phi</math> is poly-time computable.

A problem <math>f</math> is '''#P-hard''' if every problem in '''#P''' is poly-time reducible to <math>f</math>. A problem <math>f</math> is '''#P-complete''' if <math>f\in</math>'''#P''' and <math>f</math> is '''#P-hard'''. That is to say, '''#P-complete''' problems are the "hardest" problems in '''#P'''.

Generally, '''#P-complete''' problems are very hard, because if we can solve any of these problems in poly-time, then '''#P'''='''FP''', which implies that '''NP'''='''P'''. On the other hand, we do not know whether '''NP'''='''P''' could imply that '''#P'''='''FP'''.

=== FPRAS ===

== Approximate Counting ==
Let us consider the following abstract problem.

Let <math>U</math> be a finite set of known size, and let <math>G\subseteq U</math>. We want to compute the size of <math>G</math>, namely <math>|G|</math>.

We assume two devices:
* A '''uniform sampler''' <math>\mathcal{U}</math>, which uniformly and independently samples a member of <math>U</math> upon each calling.
* A '''membership oracle''' of <math>G</math>, denoted <math>\mathcal{O}</math>. Given as the input an <math>x\in U</math>, <math>\mathcal{O}(x)</math> indicates whether or not <math>x</math> is a member of <math>G</math>.

Equipped by <math>\mathcal{U}</math> and <math>\mathcal{O}</math>, we can have the following Monte Carlo algorithm:
*Choose <math>N</math> independent samples from <math>U</math> by the uniform sampler <math>\mathcal{U}</math>, represented by the random variables <math>X_1,X_2,\ldots, X_N</math>.
* Let <math>Y_i</math> be the indicator random variable defined as <math>Y_i=\mathcal{O}(X_i)</math>, namely, <math>Y_i</math> indicates whether <math>X_i\in G</math>.
* Define the estimator random variable
::<math>Z=\frac{|U|}{N}\sum_{i=1}^N Y_i.</math>

It is easy to see that <math>\mathbf{E}[Z]=|G|</math> and we might hope that with high probability the value of <math>Z</math> is close to <math>|G|</math>. Formally, <math>Z</math> is called an <math>\epsilon</math>-approximation of <math>|G|</math> if
:<math>
(1-\epsilon)|G|\le Z\le (1+\epsilon)|G|.
</math>

The following theorem states that the probabilistic accuracy of the estimation depends on the number of samples and the ratio between <math>|G|</math> and <math>|U|</math>

{|border="1"
|'''Theorem (estimator theorem)'''
:Let <math>\alpha=\frac{|G|}{|U|}</math>. Then the Monte Carlo method yields an <math>\epsilon</math>-approximation to <math>|G|</math> with probability at least <math>1-\delta</math> provided
::<math>N\ge\frac{4}{\epsilon \alpha}\ln\frac{2}{\delta}</math>.
|}
'''Proof''': Use the Chernoff bound.

<math>\square</math>

A counting algorithm for the set <math>G</math> has to deal with the following three complications:
* Implement the membership oracle <math>\mathcal{O}</math>. This is usually straightforward, or assumed by the model.
* Implement the uniform sampler <math>\mathcal{U}</math>. As we have seen, this is usually approximated by random walks. How to design the random walk and bound its mixing rate is usually technical challenging, if possible at all.
* Deal with exponentially small <math>\alpha=\frac{|G|}{|U|}</math>. This requires us to cleverly choose the universe <math>U</math>. Sometimes this needs some nontrivial ideas.

=== Counting DNFs ===
A disjunctive normal form (DNF) formular is a disjunction (OR) of clauses, where each clause is a conjunction (AND) of literals. For example:
:<math>(x_1\wedge \overline{x_2}\wedge x_3)\vee(x_2\wedge x_4)\vee(\overline{x_1}\wedge x_3\wedge x_4)</math>.
Note the difference from the conjunctive normal forms (CNF).

Given a DNF formular <math>\phi</math> as the input, the problem is to count the number of satisfying assignments of <math>\phi</math>. This problem is '''#P-complete'''.

Naively applying the Monte Carlo method will not give a good answer. Suppose that there are <math>n</math> variables. Let <math>U=\{\mathrm{true},\mathrm{false}\}^n</math> be the set of all truth assignments of the <math>n</math> variables. Let <math>G=\{x\in U\mid \phi(x)=\mathrm{true}\}</math> be the set of satisfying assignments for <math>\phi</math>. The straightforward use of Monte Carlo method samples <math>N</math> assignments from <math>U</math> and check how many of them satisfy <math>\phi</math>. This algorithm fails when <math>|G|/|U|</math> is exponentially small, namely, when exponentially small fraction of the assignments satisfy the input DNF formula.

;The union of sets problem
We reformulate the DNF counting problem in a more abstract framework, called the '''union of sets''' problem.

Let <math>V</math> be a finite universe. We are given <math>m</math> subsets <math>H_1,H_2,\ldots,H_m\subseteq V</math>. The following assumptions hold:
*For all <math>i</math>, <math>|H_i|</math> is computable in poly-time.
*It is possible to sample uniformly from each individual <math>H_i</math>.
*For any <math>x\in V</math>, it can be determined in poly-time whether <math>x\in H_i</math>.

The goal is to compute the size of <math>H=\bigcup_{i=1}^m H_i</math>.

DNF counting can be interpreted in this general framework as follows. Suppose that the DNF formula <math>\phi</math> is defined on <math>n</math> variables, and <math>\phi</math> contains <math>m</math> clauses <math>C_1,C_2,\ldots,C_m</math>, where clause <math>C_i</math> has <math>k_i</math> literals. Without loss of generality, we assume that in each clause, each variable appears at most once.
* <math>V</math> is the set of all assignments.
*Each <math>H_i</math> is the set of satisfying assignments for the <math>i</math>-th clause <math>C_i</math> of the DNF formular <math>\phi</math>. Then the union of sets <math>H=\bigcup_i H_i</math> gives the set of satisfying assignments for <math>\phi</math>.
* Each clause <math>C_i</math> is a conjunction (AND) of literals. It is not hard to see that <math>|H_i|=2^{n-k_i}</math>, which is efficiently computable.
* Sampling from an <math>H_i</math> is simple: we just fix the assignments of the <math>k_i</math> literals of that clause, and sample uniformly and independently the rest <math>(n-k_i)</math> variable assignments.
* For each assignment <math>x</math>, it is easy to check whether it satisfies a clause <math>C_i</math>, thus it is easy to determine whether <math>x\in H_i</math>.

;The coverage algorithm
We now introduce the coverage algorithm for the union of sets problem.

Consider the multiset <math>U</math> defined by
:<math>U=H_1\uplus H_2\uplus\cdots \uplus H_m</math>,
where <math>\uplus</math> denotes the multiset union. It is more convenient to define <math>U</math> as the set
:<math>U=\{(x,i)\mid x\in H_i\}</math>.
For each <math>x\in H</math>, there may be more than one instances of <math>(x,i)\in U</math>. We can choose a unique representative among the multiple instances <math>(x,i)\in U</math> for the same <math>x\in H</math>, by choosing the <math>(x,i)</math> with the minimum <math>i</math>, and form a set <math>G</math>.

Formally, <math>G=\{(x,i)\in U\mid \forall (x,j)\in U, j\le i\}</math>. Every <math>x\in H</math> corresponds to a unique <math>(x,i)\in G</math> where <math>i</math> is the smallest among <math>x\in H_i</math>.

It is obvious that <math>G\subseteq U</math> and
:<math>|G|=|H|</math>.

Therefore, estimation of <math>|H|</math> is reduced to estimation of <math>|G|</math> with <math>G\subseteq U</math>. Then <math>|G|</math> can have an <math>\epsilon</math>-approximation with probability <math>(1-\delta)</math> in poly-time, if we can uniformly sample from <math>U</math> and <math>|G|/|U|</math> is suitably small.

An uniform sample from <math>U</math> can be implemented as follows:
* generate an <math>i\in\{1,2,\ldots,m\}</math> with probability <math>\frac{|H_i|}{\sum_{i=1}^m|H_i|}</math>;
* uniformly sample an <math>x\in H_i</math>, and return <math>(x,i)</math>.

It is easy to see that this gives a uniform member of <math>U</math>. The above sampling procedure is poly-time because each <math>|H_i|</math> can be computed in poly-time, and sampling uniformly from each <math>H_i</math> is poly-time.

We now only need to lower bound the ratio
:<math>\alpha=\frac{|G|}{|U|}</math>.

We claim that
:<math>\alpha\ge\frac{1}{m}</math>.
It is easy to see this, because each <math>x\in H</math> has at most <math>m</math> instances of <math>(x,i)</math> in <math>U</math>, and we already know that <math>|G|=|H|</math>.

Due to the estimator theorem, this needs <math>\frac{4m}{\epsilon}\ln\frac{2}{\delta}</math> uniform random samples from <math>U</math>.

This gives the coverage algorithm for the abstract problem of the union of sets. The DNF counting is a special case of it.

=== Permanents and perfect matchings ===

Let <math>U=\{u_1,u_2,\ldots,u_n\}</math> and <math>V=\{v_1,v_2,\ldots,v_n\}</math>. Consider a bipartite graph <math>G(U,V,E)</math>. An <math>M\subseteq E</math> is a '''perfect matching''' of <math>G</math> if every vertex of <math>G</math> has exactly one edge in <math>M</math> adjacent to it.

Given a bipartite graph <math>G(U,V,E)</math> with <math>|U|=|V|=n</math>, we want to count the number of perfect matchings of <math>G</math>. This problem can be reduced to computing the '''permanent''' of a square matrix.

{|border="1"
|'''Definition (permanent)'''
:Let <math>Q</math> be an <math>n\times n</math> matrix. The '''permanent''' of the matrix is defined as
::<math>\mathrm{per}(Q)=\sum_{\pi\in\mathbb{S}_n}\prod_{i=1}^n Q_{i,\pi(i)}</math>,
:where <math>\mathbb{S}_n</math> is the symmetric group of permutation of size <math>n</math>.
|}

If we multiply each term of the sum the sign of the permutation, then it gives us the determinant of the matrix,
:<math>\det(Q)=\sum_{\pi\in\mathbb{S}_n}\sgn(\pi)\prod_{i=1}^n Q_{i,\pi(i)}</math>,
where <math>\sgn(\pi)</math>, the sign of a permutation, is either <math>-1</math> or <math>+1</math>, according to whether the minimum number of pair-wise interchanges to achieve <math>\pi</math> from <math>(1,2,\ldots,n)</math> is odd or even.

Unlike the determinants, which are computable in poly-time, permanents are hard to compute, as permanents can be used to count the number of perfect matchings in a bipartite graph, which is '''#P-complete'''.

A bipartite graph <math>G(U,V,E)</math> with <math>|U|=|V|=n</math> can be represented by an <math>n\times n</math> matrix <math>Q</math> with 0-1 entries as follows:
* Each row of <math>Q</math> corresponds to a vertex in <math>U</math> and each column of <math>Q</math> corresponds to a vertex in <math>V</math>.
::<math>Q_{ij}=\begin{cases}
1 & \mbox{if }i\sim j,\\
0 & \mbox{otherwise}.
\end{cases}</math>
Note the subtle difference between the definition of <math>Q</math> and the adjacency matrix.

Each perfect matching corresponds to a permutation <math>\pi</math> of <math>U</math> with <math>(u,\pi(u))\in E</math> for every <math>u\in U</math>, which corresponds to a permutation <math>\pi\in\mathbb{S}_n</math> with <math>\prod_{i=1}^n Q_{i,\pi(i)}=1</math>. It is than easy to see that <math>\mathrm{per}(Q)</math> gives the number of perfect matchings in the bipartite graph <math>G</math>.

It is known that counting the number of perfect matchings in a bipartite graph is '''#P-hard'''. Since this problem can be reduced to computing the permanent, thus the problem of computing the permanents is also '''#P-hard'''.

Now we show that with randomization, we can approximate the number of perfect matchings in a bipartite graph. In particular, we will give an FPRAS for counting the perfect matchings in a dense bipartite graph.

==== The Jerrum-Sinclair algorithm ====
Fix a bipartite graph <math>G(U,V,E)</math> with <math>|U|=|V|=n</math>. Let <math>\mathcal{M}_k</math> be the set of matchings of size <math>k</math> in <math>G</math>, and let <math>m_k=|\mathcal{M}_k|</math>. Thus, <math>\mathcal{M}_k</math> is the set of perfect matchings in <math>G</math>, and our goal is to compute <math>m_n</math>.

Let <math>r_k=\frac{m_k}{m_{k-1}}</math> for <math>1<k\le n</math>. Then
:<math>m_k=m_{k-1}r_k</math>,
which gives us a recursion to compute the <math>m_n</math>, as
:<math>m_n=m_{1}\frac{m_2}{m_1}\cdot\frac{m_3}{m_2}\cdots\frac{m_n}{m_{n-1}}=m_1\prod_{k=2}^n r_k</math>,
where <math>m_1=|\mathcal{M}_1|</math> is the number of matchings of size 1 in the bipartite graph <math>G</math>, which is just the number of edges in <math>G</math>. Therefore, <math>m_n</math> can be computed once we know <math>r_k</math> for <math>1<k\le n</math>.

Each <math>r_k</math> can be estimated by sampling uniformly from the set <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>. The algorithm for estimating <math>m_n</math> is outlined as:
# For each <math>1<k\le n</math>, have an FPRAS for computing the <math>r_k</math> by uniform sampling sufficiently many members from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math> as
::*uniformly sample <math>N</math> matching from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>, for some polynomially large <math>N</math>;
::* assuming that there are <math>X</math> sampled matchings of size <math>k</math>, return <math>r_k</math> as <math>r_k=\frac{X}{N-X}</math>.
:2. Compute <math>m_n</math> as <math>m_n=m_1\prod_{k=2}^n r_k</math>, where <math>m_1=|E|</math>.

There are several issues that we have to deal with in order to have a fully functional FPRAS for counting perfect matchings.
* By taking the product of <math>r_k</math>'s, the errors for individual <math>r_k</math>'s add up.
* In order to accurately estimate <math>r_k</math> by sampling from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>, the ratio <math>r_k</math> should be within the range <math>\left[\frac{1}{\alpha},\alpha\right]</math> for some <math>\alpha</math> within polynomial of <math>n</math>.
* Implement the uniform sampling from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>.

;Estimator for each <math>r_k</math>

; Accumulation of errors

; Near-uniform sampling from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>
In the last lecture, we have shown that by random walk, we can sample a near-uniform member of <math>\mathcal{M}_n\cup\mathcal{M}_{n-1}</math> in poly-time.

=== Volume estimation ===
We consider the problem of computing the volume of a given [http://en.wikipedia.org/wiki/Convex_body convex body] <math>K</math> in <math>n</math> dimensions.

We use <math>\Upsilon(K)\,</math> to denote the volume of the convex body <math>K</math>. Abstractly, the problem is that given as input a convex body <math>K</math> in <math>n</math> dimensions, return the <math>\Upsilon(K)\,</math>.

We should be more specific about the input model. Since we allow an arbitrary convex body as input, it is not even clear how to describe the body. We assume that <math>K</math> is described by means of a '''membership oracle''' <math>\mathcal{O}</math>, such that for a '''query''' of an <math>n</math>-dimensional point <math>x</math>, <math>\mathcal{O}(x)</math> indicates whether <math>x\in K</math>.

For example, the <math>n</math>-dimensional convex body defined by the intersection of <math>m</math> half-spaces, which is the set of feasible solutions to a system of <math>m</math> linear constraints, can be described as
:<math>A x\le \boldsymbol{b}</math>,
for some <math>m\times n</math> matrix <math>A</math>, and <math>m</math>-dimensional vector <math>b</math>. For a query of an <math>x</math>, the membership oracle <math>\mathcal{O}</math> just check whether <math>Ax\le b</math>.

For deterministic algorithms, there are negative news for this problem.

{|border="1"
|'''Theorem (Bárány-Füredi 1987)'''
:Suppose that a deterministic poly-time algorithm uses the membership oracle for a convex body <math>K</math> in <math>n</math> dimensions, and generates an upper bound <math>\Upsilon_u\,</math> and a lower bound <math>\Upsilon_\ell\,</math> on the volume <math>\Upsilon(K)\,</math> of <math>K</math>. Then, there is a convex body <math>K</math> and a constant <math>c</math> such that
::<math>\frac{\Upsilon_u}{\Upsilon_\ell}\ge c\left(\frac{n}{\log n}\right)^n</math>.
|}
That is said, with deterministic algorithms, we cannot even approximate the volume within a wildly loose range.

Dyer-Frieze-Kannan come up with an idea of estimating the volume of <math>K</math> by sampling near-uniformly from convex sets. They reduce the problem of computing approximately the volume of convex bodies to this sampling problem and thus give the first FPRAS for the volume of convex bodies.

For any convex body <math>K</math>, it encloses some <math>n</math>-dimensional ball and is also enclosed by another ball with larger radius. We assume that the convex body <math>K</math> encloses the unit ball around the origin, and is enclosed by a larger ball round the origin with polynomially large radius. Formally, we assume that
:<math>B(0,1)\subseteq K\subseteq B(0,n^c)</math> for some constant <math>c</math>,
where <math>B(p,r)</math> denotes a ball of radius <math>r</math> with <math>p</math> as center, i.e. <math>B(p,r)=\{x\mid \|x-p\|\le r \}</math>. We can make this assumption because it is known that for any convex body <math>K</math>, there exists a linear transformation <math>\tau</math> which can be found within poly-time such that <math>\tau K</math> transforms the <math>K</math> to a convex body that satisfies the assumption, and preserves the ratio of the volumes of the balls to the convex body.

The volumes of the balls are easy to compute. If only the ratio between the convex body and the ball which encloses it, is sufficiently large, then we can apply the Monte Carlo method to estimate <math>\Upsilon(K)\,</math> by uniformly sampling from the ball.

However, in high-dimension, the ratio between the volume of a convex body and the volume of ball which encloses it, can be exponentially small. This is caused by the so called the "[http://en.wikipedia.org/wiki/Curse_of_dimensionality curse of dimensionality]".

Instead of having an outer ball and an inner ball, we define a sequence of balls:
:<math>B_0=B(0,\lambda^0), B_1=B(0,\lambda^1), B_2=B(0,\lambda^2),\ldots, B_m=B(0,\lambda^m)</math>,
where <math>\lambda=(1+\frac{1}{n})</math> and <math>m</math> is the smallest positive integer such that <math>\lambda^m\ge n^c</math>. Therefore, the inner ball <math>B(0,1)=B_0</math>, the outer ball <math>B(0,n^c)\subseteq B_m</math>, and <math>m</math> is within polynomial of <math>n</math>. In fact, <math>m\approx cn\ln n</math>.

This sequence of balls naturally defines a sequence of convex bodies by intersections as <math>K_i=B_i\cap K</math>. It is obvious that
:<math>B(0,1)=K_0\subseteq K_1\subseteq K_2\subseteq\cdots\subseteq K_m=K</math>.
Balls are convex, and since the intersection of convex bodies is still convex, the sequence of <math>K_i</math> is a sequence of convex bodies.

We have the telescopic product:
:<math>
\frac{\Upsilon(K_0)}{\Upsilon(K_1)}\cdot\frac{\Upsilon(K_1)}{\Upsilon(K_2)}\cdots\frac{\Upsilon(K_{m-1})}{\Upsilon(K_m)}=\frac{\Upsilon(K_0)}{\Upsilon(K_m)}=\frac{\Upsilon(B(0,1))}{\Upsilon(K)}.
</math>
Therefore, the volume <math>\Upsilon(K)\,</math> can be computed as
:<math>\Upsilon(K)=\Upsilon(B(0,1))\cdot\prod_{i=1}^{m}\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math>.

The volume of unite ball <math>\Upsilon(B(0,1))\,</math> can be precisely computed in poly-time. Each <math>\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math> is computed by near-uniform sampling from <math>K_{i}</math>, which encloses <math>K_{i-1}</math>.

Another observation is that the ratio <math>\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math> is well-bounded. Recall that <math>K_i=B(0,\lambda^i)\cap K</math> where <math>\lambda=(1+\frac{1}{n})</math>. It can be proved that in <math>n</math> dimensions, the volume of <math>K_i</math> is at most <math>\lambda^n</math> times the <math>K_{i-1}</math>, thus the ratio <math>\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math> is at most <math>\lambda^n=O(1)</math>. By the estimator theorem, we can have an FPRAS for the ratio <math>\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math> if we can uniformly sample from <math>K_i</math>.

Uniformly sampling from an arbitrary convex body is replaced by near-uniform sampling achieved by random walks. In the original walk of Dyer-Frieze-Kannan, they consider the random walk over <math>n</math>-dimensional discrete grid points enclosed by <math>K</math>, and prove that the walk is rapid mixing. This gives us the first FPRAS for volume estimation which runs in <math>\tilde{O}(n^{23})</math>, where <math>\tilde{O}(\cdot)</math> ignores the polylogarithmic factors.

The time bound was later improved by a series of works, each introducing some new ideas.

{|border="1"
| || complexity || new ingredient(s)
|-
| Dyer-Frieze-Kannan 1991 || align="center" | <math>n^{23}</math> || everything
|-
| Lovász-Simonovits 1990 || align="center" | <math>n^{16}</math> || localization lemma
|-
| Applegate-Kannan 1990 || align="center" | <math>n^{10}</math> || logconcave sampling
|-
| Lovász 1990 || align="center" | <math>n^{10}</math> || ball walk
|-
| Dyer-Frieze 1991 || align="center" | <math>n^8</math> || better error analysis
|-
| Lovász-Simonovits 1993 || align="center" | <math>n^7</math> || many improvements
|-
| Kannan-Lovász-Simonovits 1997 || align="center" | <math>n^5</math> || isotropy, speedy walk
|-
| Lovász-Vempala 2003 || align="center" | <math>n^4</math> || simulated annealing, hit-and-run
|-
|}
(cited from "Geometric Random Walks: A Survey" by Santosh Vempala.)

The current best upper bound is <math>\tilde{O}(n^4)</math> due to Lovász and Vempala in 2003. It is conjectured that the optimal bound is <math>\Theta(n^3)</math>.

== Linear Programming ==
Given a function <math>f:\mathbb{R}^n\rightarrow\mathbb{R}</math> and a set <math>\mathcal{F}\subseteq\mathbb{R}^n</math>, an '''optimization problem''' is the problem of finding an <math>x\in\mathcal{F}</math> with the optimal (minimum) <math>f(x)</math>. Formally, the problem can be expressed as:
::<math>
\begin{align}
\mbox{minimize} & \quad f(x)\\
\mbox{subject to} &\quad x\in\mathcal{F}
\end{align}
</math>
where <math>x\in\mathcal{R}^n</math> is a vector of <math>n</math> variables. For the problem of maximizing a function <math>f</math>, we just minimizes <math>-f</math> instead.

We call the function <math>f</math> the '''objective function''' and call any <math>x\in\mathcal{F}</math> a '''feasible solution''' of the problem. A feasible solution that minimizes <math>f(x)</math> is called an '''optimal solution'''. Our task is to find an optimal solution.

The feasible set <math>\mathcal{F}</math> is usually given by a number of '''constraints''' <math>P_1,P_2,\ldots,P_m</math>, which are predicates defined on <math>x</math>. An <math>x</math> is a feasible solution if it satisfies all the constraints.

A '''linear programming (LP)''' problem is an optimization problem with a linear objective function subject to a number of linear constraints. Formally, an LP is a problem that can be expressed in the following form:
::<math>
\begin{align}
\mbox{minimize} & \quad c_1x_1+c_2x_2+\cdots+c_nx_n\\
\mbox{subject to} & \quad a_{11}x_1+a_{12}x_2+\cdots+a_{1n}x_n\le b_1\\
& \quad a_{21}x_1+a_{22}x_2+\cdots+a_{2n}x_n\le b_2\\
& \qquad\qquad \vdots\\
& \quad a_{m1}x_1+a_{m2}x_2+\cdots+a_{mn}x_n\le b_m\\
\end{align}
</math>
where <math>a_{ij}</math>, <math>b_i</math> and <math>c_j</math> are constants and <math>x_j</math> are variables. For maximization problem, or constraints given by "<math>\ge</math>", we can multiply the coefficients by <math>-1</math> and still write the LP in the above form.

We can describe the programming in the vector form. Let <math>x</math> be a vector of <math>n</math> variables. Let <math>A=(a_{ij})</math> be an <math>m\times n</math> matrix of constant entries, <math>b=(b_1,\ldots,b_m)</math> be a vector of <math>m</math> constant entries, and <math>c=(c_1,\ldots,c_n)</math> be a vector of <math>n</math> constant entries. Then an LP is expressed as:
::<math>
\begin{align}
\mbox{minimize} & \quad c^T x\\
\mbox{subject to} & \quad Ax\le b
\end{align}
</math>
We call it the '''canonical form''' of linear programming.

=== The geometry of LPs ===
For <math>n</math>-dimensional space <math>\mathbb{R}^n</math>, a linear constraint <math>ax\le b</math> specifies a [http://en.wikipedia.org/wiki/Half-space halfspace]. The feasible set specified by <math>m</math> constraints together is an intersection of <math>m</math> halfspaces, thus, a [http://en.wikipedia.org/wiki/Convex_polytope convex polytope].

The convex polytope is defined by <math>Ax\le b</math>.

The LP can be thought as given an <math>n</math>-dimensional polytope and a linear (affine) function <math>f:\mathbb{R}^n\rightarrow \mathbb{R}</math>, looking for a point <math>x</math> in the polytope with the smallest function value <math>f(x)</math>.

We can choose a number of constraints in the system <math>Ax\le b</math>, and make them hold with equality. This would define a subspace of <math>\mathbb{R}^n</math>. In particular, if we choose <math>n</math> linearly independent constraints to form an <math>n\times n</math> submatrix <math>A'</math> and the corresponding <math>n</math>-dimensional vector <math>b'</math>, solving <math>A'x=b'</math> would give us exactly one point <math>x</math>. If this <math>x</math> is in the polytope, i.e. <math>x</math> satisfies that <math>Ax\le b</math>, we call such <math>x</math> a '''vertex''' of the polytope <math>Ax\le b</math>. In the context of LP, it is also called a '''basic feasible solution (''bfs'')''' of the LP.

A key observation for LP is that there is a basic feasible solution which is optimal, i.e. the optimal solution is a vertex of the polytope.

=== The simplex algorithms ===
The [http://en.wikipedia.org/wiki/Simplex_algorithm simplex algorithm] by George Dantzig solves the linear programming by moving from vertex to vertex of the convex polytope, each time making some progress towards optimizing the objective function. Eventually the algorithm reaches a vertex which is a [http://en.wikipedia.org/wiki/Local_optimum local optimum]. In a convex set, a locally optimal point is also global optimal, thus the simplex algorithm returns an optimal solution in finite steps.

The problem with the simplex algorithm is that in some bad polytopes, it takes the simplex algorithm exponentially many steps to reach the optimum. The simplex algorithm is actually a class of algorithms defined by various '''pivoting rules''', which describe how to move locally from one vertex to another. The original pivoting rule proposed by Dantzig has an exponentially large worst-case time complexity (though works very good in practice). To-date, it is still unknown whether there exists any deterministic simplex algorithm with sub-exponential worst-case complexity.

People have tried randomized pivoting rules, and there are some amazing progresses have been made:
* Kalai 1992: there is a randomized simplex algorithm with sub-exponential time complexity.
* Kelner-Spielman 2006: there is a polynomial time randomized simplex algorithm.

Although we do not know whether there exists deterministic poly-time simplex algorithm, we do know that LP can be solved by deterministic poly-time algorithms, i.e. LP is in '''P'''. The following two algorithms use different ideas than the simplex algorithm, and are both in poly-time:
* The ellipsoid algorithm.
* Interior point methods.
Although these algorithms guarantee polynomial time complexity in the worst-case, their performances are worse than the simplex algorithms. However, the ideas of these algorithms can be used to solve more general mathematical programmings, such as [http://en.wikipedia.org/wiki/Convex_programming convex programmings].

=== An LP solver via random walks ===
We discuss a new algorithm for solving LP which is based on random sampling, introduced by Bertsimas and Vempala in 2004.

The problem of solving an LP can be reduced to that given a convex polytope <math>Ax\le b</math>, test whether the polytope is empty. We call this problem the "''feasibility test''".

Suppose we have an LP:
::<math>
\begin{align}
\mbox{minimize} & \quad c^T x\\
\mbox{subject to} & \quad Ax\le b
\end{align}
</math>
To see this LP can be solved by feasibility testing, we treat the <math>c^Tx\le d</math> for some parameter <math>d</math> as an additional constraint, and define a new polytope <math>A'x\le b'</math>. The smallest <math>d</math> that the polytope <math>A'x\le b'</math> is not empty is the optimal solution to the original LP. We can find this smallest <math>d</math> by binary search if we can efficiently test the emptiness of a convex polytope.

We assume that the convex polytope <math>P</math> is contained in the axis-aligned cube of width <math>R</math> centered at the origin; further if <math>P</math> is non-empty then it contains a cube of width <math>r</math>. The parameter <math>L</math> is equal to <math>\log\frac{R}{r}</math>.

{|border="1"
|'''Algorithm''': feasibility test
|-
|'''Input''': an <math>n</math>-dimensional convex polytope <math>P</math> defined by the system <math>Ax\le b</math>.
'''Output''': a point <math>x\in P</math> in the polytope, or a guarantee that <math>P</math> is empty.
|-
|
:Let <math>K</math> be the axis-aligned cube of side length <math>R</math> and center <math>z=\boldsymbol{0}</math>.
:Repeat for <math>2nL</math> times do:
:: If <math>Az\le b</math>, return <math>z</math>.
:: Pick a constraint violated by <math>z</math>, say that <math>A_i z>b_i</math>, and define the halfspace
:::<math>H=\{x\mid A_i x\le A_i z\}</math>.
::Set <math>K=K\cap H</math>. Uniformly sample <math>N</math> random points <math>y^{(1)},y^{(2)},\ldots,y^{(N)}</math> from <math>K</math> and let
:::<math>z=\frac{1}{N}\sum_{i=1}^Ny^{(i)}</math>.
:Report <math>P</math> is empty.
|}

The number of samples required in each iteration, <math>N=O((\log m)^2)</math>, where <math>m</math> is the number of linear constraints.

It is easy to see that at any iteration, <math>K</math> is a convex set, and <math>P\subseteq K</math> for the current convex set <math>K</math>.

The idea of the algorithm is based on computing the [http://en.wikipedia.org/wiki/Centroid centroid] of the convex set. It is know that if we could compute the exact centroid in each iteration, then the volume of <math>K</math> drops by a constant factor <math>(1-1/e)</math> in each iteration, thus reaches the smallest possible volume of a cube of width <math>r</math> within polynomial number of iterations. But, finding the centroid of a convex polytope, is '''#P-hard'''.

The idea behind the algorithm is that an approximate centroid can be computed using <math>N=O((\log m)^2)</math> random points and the volume of <math>K</math> is drops by a constant factor with high probability in each iteration with this choice of <math>z</math>.

The uniform sampling in <math>K</math> is not easy. But since <math>K</math> is also a convex polytope, we can approximate the uniform sampling by near-uniform sampling from <math>K</math> by rapid mixing random walks. The random walks used here are similar to the ones used in the volume estimation of convex bodies. We could use the basic grid walk, or more advanced ball walk, or hit-and-run walk.

With rapid mixing random walks, the resulting LP solver runs in poly-time.

The details are omitted. You may check the following paper if interested:
* Bertsimas and Vempala, “''Solving convex programs by random walks''.” JACM 2004.

Randomized Algorithms (Spring 2010)/Approximate counting, linear programming

2010-05-24T19:21:31Z

172.21.4.18: /* Complexity model */

== Counting Problems ==

=== Complexity model ===
Recall the class '''NP''' of decision problems (the problems with "yes" or "no" answers). Formally, denoting by <math>\{0,1\}^*</math> the set of all boolean strings of any lengths, a decision problem is a function <math>f:\{0,1\}^*\rightarrow\{0,1\}</math>.
{|border="1"
|'''Definition (NP)'''
:A function <math>f:\{0,1\}^*\rightarrow\{0,1\}</math> is in '''NP''' if there exist a polynomial <math>p</math> and a poly-time algorithm <math>V</math> with boolean output such that for every <math>x\in\{0,1\}^*</math>,
::<math>f(x)=1 \Leftrightarrow \exists y\in\{0,1\}^{p(|x|)}, \mbox{such that } V(x,y)=1\,</math>.
|}
Intuitively, the '''NP''' class contains all the decision problems such that the answer is "yes" if and only if there ''exists'' a '''certificate''' which can be verified in poly-time.

In many contexts, we are interested not just in the ''existence'' of certificate but actually in counting the ''number'' of certificates. This leads to the definition of the class '''#P''' (pronounced "sharp p"). Only now the output of the function is a natural number.

{|border="1"
|'''Definition (#P)'''
:A function <math>f:\{0,1\}^*\rightarrow\mathbb{N}</math> is in '''#P''' if there exist a polynomial <math>p</math> and a poly-time algorithm <math>V</math> with boolean output such that for every <math>x\in\{0,1\}^*</math>,
::<math>f(x)=\left|\left\{y\in\{0,1\}^{p(|x|)}\mid A(x,y)=1\right\}\right|</math>.
|}
You may notice the similarity between the two definitions. The difference is that now <math>f(x)</math> does not just indicates the existence of a certificate, but gives the actual number of certificates.

Why should we care about the number of "certificates" at all?
In combinatorics, a counting problem is usually formulated as counting the number of combinatorial objects with some particular structure, such as the number of trees, or Latin squares. The '''#P''' class contains the counting problems that the "structure" requirements are easy to check.

;Examples of problems in #P
:<nowiki>#</nowiki>cycles: given as input a graph <math>G</math>, count the number of cycles in <math>G</math>.
:<nowiki>#</nowiki>SAT: given as input a boolean formula <math>\phi</math>, count the number of satisfying assignments for <math>\phi</math>.

The second example shows that the '''<nowiki>#</nowiki>P''' may be harder than '''NP''', because we know that SAT (given a boolean formular, decide the existence of satisfying assignments) is '''NP-complete''', and its counting version <nowiki>#</nowiki>SAT now ask for the exact number of satisfying assignments.

The class '''FP''' contains all the functions <math>f:\{0,1\}^*\rightarrow\mathbb{N}</math> computable by poly-time algorithms. The classes '''FP''' and '''#P''' are the analogs of '''P''' and '''NP''' for counting problems.

A '''reduction''' from a problem <math>f</math> to a problem <math>g</math> is a mapping <math>\phi</math> which maps instances of <math>f</math> to instances of <math>g</math> such that for any instance <math>x</math> of <math>f</math>,
:<math>f(x)=g(\phi(x))</math>.
In other words, <math>\phi</math> "reduces" the task of solving <math>f</math> to the task of solving <math>g</math>. A problem <math>f</math> is said to be '''poly-time reducible''' to the problem <math>g</math>, if there exists a reduction <math>\phi</math> from <math>f</math> to <math>g</math> and <math>\phi</math> is poly-time computable.

A problem <math>f</math> is '''#P-hard''' if every problem in '''#P''' is poly-time reducible to <math>f</math>. A problem <math>f</math> is '''#P-complete''' if <math>f\in</math>'''#P''' and <math>f</math> is '''#P-hard'''. That is to say, '''#P-complete''' problems are the "hardest" problems in '''#P'''.

=== FPRAS ===

== Approximate Counting ==
Let us consider the following abstract problem.

Let <math>U</math> be a finite set of known size, and let <math>G\subseteq U</math>. We want to compute the size of <math>G</math>, namely <math>|G|</math>.

We assume two devices:
* A '''uniform sampler''' <math>\mathcal{U}</math>, which uniformly and independently samples a member of <math>U</math> upon each calling.
* A '''membership oracle''' of <math>G</math>, denoted <math>\mathcal{O}</math>. Given as the input an <math>x\in U</math>, <math>\mathcal{O}(x)</math> indicates whether or not <math>x</math> is a member of <math>G</math>.

Equipped by <math>\mathcal{U}</math> and <math>\mathcal{O}</math>, we can have the following Monte Carlo algorithm:
*Choose <math>N</math> independent samples from <math>U</math> by the uniform sampler <math>\mathcal{U}</math>, represented by the random variables <math>X_1,X_2,\ldots, X_N</math>.
* Let <math>Y_i</math> be the indicator random variable defined as <math>Y_i=\mathcal{O}(X_i)</math>, namely, <math>Y_i</math> indicates whether <math>X_i\in G</math>.
* Define the estimator random variable
::<math>Z=\frac{|U|}{N}\sum_{i=1}^N Y_i.</math>

It is easy to see that <math>\mathbf{E}[Z]=|G|</math> and we might hope that with high probability the value of <math>Z</math> is close to <math>|G|</math>. Formally, <math>Z</math> is called an <math>\epsilon</math>-approximation of <math>|G|</math> if
:<math>
(1-\epsilon)|G|\le Z\le (1+\epsilon)|G|.
</math>

The following theorem states that the probabilistic accuracy of the estimation depends on the number of samples and the ratio between <math>|G|</math> and <math>|U|</math>

{|border="1"
|'''Theorem (estimator theorem)'''
:Let <math>\alpha=\frac{|G|}{|U|}</math>. Then the Monte Carlo method yields an <math>\epsilon</math>-approximation to <math>|G|</math> with probability at least <math>1-\delta</math> provided
::<math>N\ge\frac{4}{\epsilon \alpha}\ln\frac{2}{\delta}</math>.
|}
'''Proof''': Use the Chernoff bound.

<math>\square</math>

A counting algorithm for the set <math>G</math> has to deal with the following three complications:
* Implement the membership oracle <math>\mathcal{O}</math>. This is usually straightforward, or assumed by the model.
* Implement the uniform sampler <math>\mathcal{U}</math>. As we have seen, this is usually approximated by random walks. How to design the random walk and bound its mixing rate is usually technical challenging, if possible at all.
* Deal with exponentially small <math>\alpha=\frac{|G|}{|U|}</math>. This requires us to cleverly choose the universe <math>U</math>. Sometimes this needs some nontrivial ideas.

=== Counting DNFs ===
A disjunctive normal form (DNF) formular is a disjunction (OR) of clauses, where each clause is a conjunction (AND) of literals. For example:
:<math>(x_1\wedge \overline{x_2}\wedge x_3)\vee(x_2\wedge x_4)\vee(\overline{x_1}\wedge x_3\wedge x_4)</math>.
Note the difference from the conjunctive normal forms (CNF).

Given a DNF formular <math>\phi</math> as the input, the problem is to count the number of satisfying assignments of <math>\phi</math>. This problem is '''#P-complete'''.

Naively applying the Monte Carlo method will not give a good answer. Suppose that there are <math>n</math> variables. Let <math>U=\{\mathrm{true},\mathrm{false}\}^n</math> be the set of all truth assignments of the <math>n</math> variables. Let <math>G=\{x\in U\mid \phi(x)=\mathrm{true}\}</math> be the set of satisfying assignments for <math>\phi</math>. The straightforward use of Monte Carlo method samples <math>N</math> assignments from <math>U</math> and check how many of them satisfy <math>\phi</math>. This algorithm fails when <math>|G|/|U|</math> is exponentially small, namely, when exponentially small fraction of the assignments satisfy the input DNF formula.

;The union of sets problem
We reformulate the DNF counting problem in a more abstract framework, called the '''union of sets''' problem.

Let <math>V</math> be a finite universe. We are given <math>m</math> subsets <math>H_1,H_2,\ldots,H_m\subseteq V</math>. The following assumptions hold:
*For all <math>i</math>, <math>|H_i|</math> is computable in poly-time.
*It is possible to sample uniformly from each individual <math>H_i</math>.
*For any <math>x\in V</math>, it can be determined in poly-time whether <math>x\in H_i</math>.

The goal is to compute the size of <math>H=\bigcup_{i=1}^m H_i</math>.

DNF counting can be interpreted in this general framework as follows. Suppose that the DNF formula <math>\phi</math> is defined on <math>n</math> variables, and <math>\phi</math> contains <math>m</math> clauses <math>C_1,C_2,\ldots,C_m</math>, where clause <math>C_i</math> has <math>k_i</math> literals. Without loss of generality, we assume that in each clause, each variable appears at most once.
* <math>V</math> is the set of all assignments.
*Each <math>H_i</math> is the set of satisfying assignments for the <math>i</math>-th clause <math>C_i</math> of the DNF formular <math>\phi</math>. Then the union of sets <math>H=\bigcup_i H_i</math> gives the set of satisfying assignments for <math>\phi</math>.
* Each clause <math>C_i</math> is a conjunction (AND) of literals. It is not hard to see that <math>|H_i|=2^{n-k_i}</math>, which is efficiently computable.
* Sampling from an <math>H_i</math> is simple: we just fix the assignments of the <math>k_i</math> literals of that clause, and sample uniformly and independently the rest <math>(n-k_i)</math> variable assignments.
* For each assignment <math>x</math>, it is easy to check whether it satisfies a clause <math>C_i</math>, thus it is easy to determine whether <math>x\in H_i</math>.

;The coverage algorithm
We now introduce the coverage algorithm for the union of sets problem.

Consider the multiset <math>U</math> defined by
:<math>U=H_1\uplus H_2\uplus\cdots \uplus H_m</math>,
where <math>\uplus</math> denotes the multiset union. It is more convenient to define <math>U</math> as the set
:<math>U=\{(x,i)\mid x\in H_i\}</math>.
For each <math>x\in H</math>, there may be more than one instances of <math>(x,i)\in U</math>. We can choose a unique representative among the multiple instances <math>(x,i)\in U</math> for the same <math>x\in H</math>, by choosing the <math>(x,i)</math> with the minimum <math>i</math>, and form a set <math>G</math>.

Formally, <math>G=\{(x,i)\in U\mid \forall (x,j)\in U, j\le i\}</math>. Every <math>x\in H</math> corresponds to a unique <math>(x,i)\in G</math> where <math>i</math> is the smallest among <math>x\in H_i</math>.

It is obvious that <math>G\subseteq U</math> and
:<math>|G|=|H|</math>.

Therefore, estimation of <math>|H|</math> is reduced to estimation of <math>|G|</math> with <math>G\subseteq U</math>. Then <math>|G|</math> can have an <math>\epsilon</math>-approximation with probability <math>(1-\delta)</math> in poly-time, if we can uniformly sample from <math>U</math> and <math>|G|/|U|</math> is suitably small.

An uniform sample from <math>U</math> can be implemented as follows:
* generate an <math>i\in\{1,2,\ldots,m\}</math> with probability <math>\frac{|H_i|}{\sum_{i=1}^m|H_i|}</math>;
* uniformly sample an <math>x\in H_i</math>, and return <math>(x,i)</math>.

It is easy to see that this gives a uniform member of <math>U</math>. The above sampling procedure is poly-time because each <math>|H_i|</math> can be computed in poly-time, and sampling uniformly from each <math>H_i</math> is poly-time.

We now only need to lower bound the ratio
:<math>\alpha=\frac{|G|}{|U|}</math>.

We claim that
:<math>\alpha\ge\frac{1}{m}</math>.
It is easy to see this, because each <math>x\in H</math> has at most <math>m</math> instances of <math>(x,i)</math> in <math>U</math>, and we already know that <math>|G|=|H|</math>.

Due to the estimator theorem, this needs <math>\frac{4m}{\epsilon}\ln\frac{2}{\delta}</math> uniform random samples from <math>U</math>.

This gives the coverage algorithm for the abstract problem of the union of sets. The DNF counting is a special case of it.

=== Permanents and perfect matchings ===

Let <math>U=\{u_1,u_2,\ldots,u_n\}</math> and <math>V=\{v_1,v_2,\ldots,v_n\}</math>. Consider a bipartite graph <math>G(U,V,E)</math>. An <math>M\subseteq E</math> is a '''perfect matching''' of <math>G</math> if every vertex of <math>G</math> has exactly one edge in <math>M</math> adjacent to it.

Given a bipartite graph <math>G(U,V,E)</math> with <math>|U|=|V|=n</math>, we want to count the number of perfect matchings of <math>G</math>. This problem can be reduced to computing the '''permanent''' of a square matrix.

{|border="1"
|'''Definition (permanent)'''
:Let <math>Q</math> be an <math>n\times n</math> matrix. The '''permanent''' of the matrix is defined as
::<math>\mathrm{per}(Q)=\sum_{\pi\in\mathbb{S}_n}\prod_{i=1}^n Q_{i,\pi(i)}</math>,
:where <math>\mathbb{S}_n</math> is the symmetric group of permutation of size <math>n</math>.
|}

If we multiply each term of the sum the sign of the permutation, then it gives us the determinant of the matrix,
:<math>\det(Q)=\sum_{\pi\in\mathbb{S}_n}\sgn(\pi)\prod_{i=1}^n Q_{i,\pi(i)}</math>,
where <math>\sgn(\pi)</math>, the sign of a permutation, is either <math>-1</math> or <math>+1</math>, according to whether the minimum number of pair-wise interchanges to achieve <math>\pi</math> from <math>(1,2,\ldots,n)</math> is odd or even.

Unlike the determinants, which are computable in poly-time, permanents are hard to compute, as permanents can be used to count the number of perfect matchings in a bipartite graph, which is '''#P-complete'''.

A bipartite graph <math>G(U,V,E)</math> with <math>|U|=|V|=n</math> can be represented by an <math>n\times n</math> matrix <math>Q</math> with 0-1 entries as follows:
* Each row of <math>Q</math> corresponds to a vertex in <math>U</math> and each column of <math>Q</math> corresponds to a vertex in <math>V</math>.
::<math>Q_{ij}=\begin{cases}
1 & \mbox{if }i\sim j,\\
0 & \mbox{otherwise}.
\end{cases}</math>
Note the subtle difference between the definition of <math>Q</math> and the adjacency matrix.

Each perfect matching corresponds to a permutation <math>\pi</math> of <math>U</math> with <math>(u,\pi(u))\in E</math> for every <math>u\in U</math>, which corresponds to a permutation <math>\pi\in\mathbb{S}_n</math> with <math>\prod_{i=1}^n Q_{i,\pi(i)}=1</math>. It is than easy to see that <math>\mathrm{per}(Q)</math> gives the number of perfect matchings in the bipartite graph <math>G</math>.

It is known that counting the number of perfect matchings in a bipartite graph is '''#P-hard'''. Since this problem can be reduced to computing the permanent, thus the problem of computing the permanents is also '''#P-hard'''.

Now we show that with randomization, we can approximate the number of perfect matchings in a bipartite graph. In particular, we will give an FPRAS for counting the perfect matchings in a dense bipartite graph.

==== The Jerrum-Sinclair algorithm ====
Fix a bipartite graph <math>G(U,V,E)</math> with <math>|U|=|V|=n</math>. Let <math>\mathcal{M}_k</math> be the set of matchings of size <math>k</math> in <math>G</math>, and let <math>m_k=|\mathcal{M}_k|</math>. Thus, <math>\mathcal{M}_k</math> is the set of perfect matchings in <math>G</math>, and our goal is to compute <math>m_n</math>.

Let <math>r_k=\frac{m_k}{m_{k-1}}</math> for <math>1<k\le n</math>. Then
:<math>m_k=m_{k-1}r_k</math>,
which gives us a recursion to compute the <math>m_n</math>, as
:<math>m_n=m_{1}\frac{m_2}{m_1}\cdot\frac{m_3}{m_2}\cdots\frac{m_n}{m_{n-1}}=m_1\prod_{k=2}^n r_k</math>,
where <math>m_1=|\mathcal{M}_1|</math> is the number of matchings of size 1 in the bipartite graph <math>G</math>, which is just the number of edges in <math>G</math>. Therefore, <math>m_n</math> can be computed once we know <math>r_k</math> for <math>1<k\le n</math>.

Each <math>r_k</math> can be estimated by sampling uniformly from the set <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>. The algorithm for estimating <math>m_n</math> is outlined as:
# For each <math>1<k\le n</math>, have an FPRAS for computing the <math>r_k</math> by uniform sampling sufficiently many members from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math> as
::*uniformly sample <math>N</math> matching from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>, for some polynomially large <math>N</math>;
::* assuming that there are <math>X</math> sampled matchings of size <math>k</math>, return <math>r_k</math> as <math>r_k=\frac{X}{N-X}</math>.
:2. Compute <math>m_n</math> as <math>m_n=m_1\prod_{k=2}^n r_k</math>, where <math>m_1=|E|</math>.

There are several issues that we have to deal with in order to have a fully functional FPRAS for counting perfect matchings.
* By taking the product of <math>r_k</math>'s, the errors for individual <math>r_k</math>'s add up.
* In order to accurately estimate <math>r_k</math> by sampling from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>, the ratio <math>r_k</math> should be within the range <math>\left[\frac{1}{\alpha},\alpha\right]</math> for some <math>\alpha</math> within polynomial of <math>n</math>.
* Implement the uniform sampling from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>.

;Estimator for each <math>r_k</math>

; Accumulation of errors

; Near-uniform sampling from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>
In the last lecture, we have shown that by random walk, we can sample a near-uniform member of <math>\mathcal{M}_n\cup\mathcal{M}_{n-1}</math> in poly-time.

=== Volume estimation ===
We consider the problem of computing the volume of a given [http://en.wikipedia.org/wiki/Convex_body convex body] <math>K</math> in <math>n</math> dimensions.

We use <math>\Upsilon(K)\,</math> to denote the volume of the convex body <math>K</math>. Abstractly, the problem is that given as input a convex body <math>K</math> in <math>n</math> dimensions, return the <math>\Upsilon(K)\,</math>.

We should be more specific about the input model. Since we allow an arbitrary convex body as input, it is not even clear how to describe the body. We assume that <math>K</math> is described by means of a '''membership oracle''' <math>\mathcal{O}</math>, such that for a '''query''' of an <math>n</math>-dimensional point <math>x</math>, <math>\mathcal{O}(x)</math> indicates whether <math>x\in K</math>.

For example, the <math>n</math>-dimensional convex body defined by the intersection of <math>m</math> half-spaces, which is the set of feasible solutions to a system of <math>m</math> linear constraints, can be described as
:<math>A x\le \boldsymbol{b}</math>,
for some <math>m\times n</math> matrix <math>A</math>, and <math>m</math>-dimensional vector <math>b</math>. For a query of an <math>x</math>, the membership oracle <math>\mathcal{O}</math> just check whether <math>Ax\le b</math>.

For deterministic algorithms, there are negative news for this problem.

{|border="1"
|'''Theorem (Bárány-Füredi 1987)'''
:Suppose that a deterministic poly-time algorithm uses the membership oracle for a convex body <math>K</math> in <math>n</math> dimensions, and generates an upper bound <math>\Upsilon_u\,</math> and a lower bound <math>\Upsilon_\ell\,</math> on the volume <math>\Upsilon(K)\,</math> of <math>K</math>. Then, there is a convex body <math>K</math> and a constant <math>c</math> such that
::<math>\frac{\Upsilon_u}{\Upsilon_\ell}\ge c\left(\frac{n}{\log n}\right)^n</math>.
|}
That is said, with deterministic algorithms, we cannot even approximate the volume within a wildly loose range.

Dyer-Frieze-Kannan come up with an idea of estimating the volume of <math>K</math> by sampling near-uniformly from convex sets. They reduce the problem of computing approximately the volume of convex bodies to this sampling problem and thus give the first FPRAS for the volume of convex bodies.

For any convex body <math>K</math>, it encloses some <math>n</math>-dimensional ball and is also enclosed by another ball with larger radius. We assume that the convex body <math>K</math> encloses the unit ball around the origin, and is enclosed by a larger ball round the origin with polynomially large radius. Formally, we assume that
:<math>B(0,1)\subseteq K\subseteq B(0,n^c)</math> for some constant <math>c</math>,
where <math>B(p,r)</math> denotes a ball of radius <math>r</math> with <math>p</math> as center, i.e. <math>B(p,r)=\{x\mid \|x-p\|\le r \}</math>. We can make this assumption because it is known that for any convex body <math>K</math>, there exists a linear transformation <math>\tau</math> which can be found within poly-time such that <math>\tau K</math> transforms the <math>K</math> to a convex body that satisfies the assumption, and preserves the ratio of the volumes of the balls to the convex body.

The volumes of the balls are easy to compute. If only the ratio between the convex body and the ball which encloses it, is sufficiently large, then we can apply the Monte Carlo method to estimate <math>\Upsilon(K)\,</math> by uniformly sampling from the ball.

However, in high-dimension, the ratio between the volume of a convex body and the volume of ball which encloses it, can be exponentially small. This is caused by the so called the "[http://en.wikipedia.org/wiki/Curse_of_dimensionality curse of dimensionality]".

Instead of having an outer ball and an inner ball, we define a sequence of balls:
:<math>B_0=B(0,\lambda^0), B_1=B(0,\lambda^1), B_2=B(0,\lambda^2),\ldots, B_m=B(0,\lambda^m)</math>,
where <math>\lambda=(1+\frac{1}{n})</math> and <math>m</math> is the smallest positive integer such that <math>\lambda^m\ge n^c</math>. Therefore, the inner ball <math>B(0,1)=B_0</math>, the outer ball <math>B(0,n^c)\subseteq B_m</math>, and <math>m</math> is within polynomial of <math>n</math>. In fact, <math>m\approx cn\ln n</math>.

This sequence of balls naturally defines a sequence of convex bodies by intersections as <math>K_i=B_i\cap K</math>. It is obvious that
:<math>B(0,1)=K_0\subseteq K_1\subseteq K_2\subseteq\cdots\subseteq K_m=K</math>.
Balls are convex, and since the intersection of convex bodies is still convex, the sequence of <math>K_i</math> is a sequence of convex bodies.

We have the telescopic product:
:<math>
\frac{\Upsilon(K_0)}{\Upsilon(K_1)}\cdot\frac{\Upsilon(K_1)}{\Upsilon(K_2)}\cdots\frac{\Upsilon(K_{m-1})}{\Upsilon(K_m)}=\frac{\Upsilon(K_0)}{\Upsilon(K_m)}=\frac{\Upsilon(B(0,1))}{\Upsilon(K)}.
</math>
Therefore, the volume <math>\Upsilon(K)\,</math> can be computed as
:<math>\Upsilon(K)=\Upsilon(B(0,1))\cdot\prod_{i=1}^{m}\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math>.

The volume of unite ball <math>\Upsilon(B(0,1))\,</math> can be precisely computed in poly-time. Each <math>\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math> is computed by near-uniform sampling from <math>K_{i}</math>, which encloses <math>K_{i-1}</math>.

Another observation is that the ratio <math>\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math> is well-bounded. Recall that <math>K_i=B(0,\lambda^i)\cap K</math> where <math>\lambda=(1+\frac{1}{n})</math>. It can be proved that in <math>n</math> dimensions, the volume of <math>K_i</math> is at most <math>\lambda^n</math> times the <math>K_{i-1}</math>, thus the ratio <math>\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math> is at most <math>\lambda^n=O(1)</math>. By the estimator theorem, we can have an FPRAS for the ratio <math>\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math> if we can uniformly sample from <math>K_i</math>.

Uniformly sampling from an arbitrary convex body is replaced by near-uniform sampling achieved by random walks. In the original walk of Dyer-Frieze-Kannan, they consider the random walk over <math>n</math>-dimensional discrete grid points enclosed by <math>K</math>, and prove that the walk is rapid mixing. This gives us the first FPRAS for volume estimation which runs in <math>\tilde{O}(n^{23})</math>, where <math>\tilde{O}(\cdot)</math> ignores the polylogarithmic factors.

The time bound was later improved by a series of works, each introducing some new ideas.

{|border="1"
| || complexity || new ingredient(s)
|-
| Dyer-Frieze-Kannan 1991 || align="center" | <math>n^{23}</math> || everything
|-
| Lovász-Simonovits 1990 || align="center" | <math>n^{16}</math> || localization lemma
|-
| Applegate-Kannan 1990 || align="center" | <math>n^{10}</math> || logconcave sampling
|-
| Lovász 1990 || align="center" | <math>n^{10}</math> || ball walk
|-
| Dyer-Frieze 1991 || align="center" | <math>n^8</math> || better error analysis
|-
| Lovász-Simonovits 1993 || align="center" | <math>n^7</math> || many improvements
|-
| Kannan-Lovász-Simonovits 1997 || align="center" | <math>n^5</math> || isotropy, speedy walk
|-
| Lovász-Vempala 2003 || align="center" | <math>n^4</math> || simulated annealing, hit-and-run
|-
|}
(cited from "Geometric Random Walks: A Survey" by Santosh Vempala.)

The current best upper bound is <math>\tilde{O}(n^4)</math> due to Lovász and Vempala in 2003. It is conjectured that the optimal bound is <math>\Theta(n^3)</math>.

== Linear Programming ==
Given a function <math>f:\mathbb{R}^n\rightarrow\mathbb{R}</math> and a set <math>\mathcal{F}\subseteq\mathbb{R}^n</math>, an '''optimization problem''' is the problem of finding an <math>x\in\mathcal{F}</math> with the optimal (minimum) <math>f(x)</math>. Formally, the problem can be expressed as:
::<math>
\begin{align}
\mbox{minimize} & \quad f(x)\\
\mbox{subject to} &\quad x\in\mathcal{F}
\end{align}
</math>
where <math>x\in\mathcal{R}^n</math> is a vector of <math>n</math> variables. For the problem of maximizing a function <math>f</math>, we just minimizes <math>-f</math> instead.

We call the function <math>f</math> the '''objective function''' and call any <math>x\in\mathcal{F}</math> a '''feasible solution''' of the problem. A feasible solution that minimizes <math>f(x)</math> is called an '''optimal solution'''. Our task is to find an optimal solution.

The feasible set <math>\mathcal{F}</math> is usually given by a number of '''constraints''' <math>P_1,P_2,\ldots,P_m</math>, which are predicates defined on <math>x</math>. An <math>x</math> is a feasible solution if it satisfies all the constraints.

A '''linear programming (LP)''' problem is an optimization problem with a linear objective function subject to a number of linear constraints. Formally, an LP is a problem that can be expressed in the following form:
::<math>
\begin{align}
\mbox{minimize} & \quad c_1x_1+c_2x_2+\cdots+c_nx_n\\
\mbox{subject to} & \quad a_{11}x_1+a_{12}x_2+\cdots+a_{1n}x_n\le b_1\\
& \quad a_{21}x_1+a_{22}x_2+\cdots+a_{2n}x_n\le b_2\\
& \qquad\qquad \vdots\\
& \quad a_{m1}x_1+a_{m2}x_2+\cdots+a_{mn}x_n\le b_m\\
\end{align}
</math>
where <math>a_{ij}</math>, <math>b_i</math> and <math>c_j</math> are constants and <math>x_j</math> are variables. For maximization problem, or constraints given by "<math>\ge</math>", we can multiply the coefficients by <math>-1</math> and still write the LP in the above form.

We can describe the programming in the vector form. Let <math>x</math> be a vector of <math>n</math> variables. Let <math>A=(a_{ij})</math> be an <math>m\times n</math> matrix of constant entries, <math>b=(b_1,\ldots,b_m)</math> be a vector of <math>m</math> constant entries, and <math>c=(c_1,\ldots,c_n)</math> be a vector of <math>n</math> constant entries. Then an LP is expressed as:
::<math>
\begin{align}
\mbox{minimize} & \quad c^T x\\
\mbox{subject to} & \quad Ax\le b
\end{align}
</math>
We call it the '''canonical form''' of linear programming.

=== The geometry of LPs ===
For <math>n</math>-dimensional space <math>\mathbb{R}^n</math>, a linear constraint <math>ax\le b</math> specifies a [http://en.wikipedia.org/wiki/Half-space halfspace]. The feasible set specified by <math>m</math> constraints together is an intersection of <math>m</math> halfspaces, thus, a [http://en.wikipedia.org/wiki/Convex_polytope convex polytope].

The convex polytope is defined by <math>Ax\le b</math>.

The LP can be thought as given an <math>n</math>-dimensional polytope and a linear (affine) function <math>f:\mathbb{R}^n\rightarrow \mathbb{R}</math>, looking for a point <math>x</math> in the polytope with the smallest function value <math>f(x)</math>.

We can choose a number of constraints in the system <math>Ax\le b</math>, and make them hold with equality. This would define a subspace of <math>\mathbb{R}^n</math>. In particular, if we choose <math>n</math> linearly independent constraints to form an <math>n\times n</math> submatrix <math>A'</math> and the corresponding <math>n</math>-dimensional vector <math>b'</math>, solving <math>A'x=b'</math> would give us exactly one point <math>x</math>. If this <math>x</math> is in the polytope, i.e. <math>x</math> satisfies that <math>Ax\le b</math>, we call such <math>x</math> a '''vertex''' of the polytope <math>Ax\le b</math>. In the context of LP, it is also called a '''basic feasible solution (''bfs'')''' of the LP.

A key observation for LP is that there is a basic feasible solution which is optimal, i.e. the optimal solution is a vertex of the polytope.

=== The simplex algorithms ===
The [http://en.wikipedia.org/wiki/Simplex_algorithm simplex algorithm] by George Dantzig solves the linear programming by moving from vertex to vertex of the convex polytope, each time making some progress towards optimizing the objective function. Eventually the algorithm reaches a vertex which is a [http://en.wikipedia.org/wiki/Local_optimum local optimum]. In a convex set, a locally optimal point is also global optimal, thus the simplex algorithm returns an optimal solution in finite steps.

The problem with the simplex algorithm is that in some bad polytopes, it takes the simplex algorithm exponentially many steps to reach the optimum. The simplex algorithm is actually a class of algorithms defined by various '''pivoting rules''', which describe how to move locally from one vertex to another. The original pivoting rule proposed by Dantzig has an exponentially large worst-case time complexity (though works very good in practice). To-date, it is still unknown whether there exists any deterministic simplex algorithm with sub-exponential worst-case complexity.

People have tried randomized pivoting rules, and there are some amazing progresses have been made:
* Kalai 1992: there is a randomized simplex algorithm with sub-exponential time complexity.
* Kelner-Spielman 2006: there is a polynomial time randomized simplex algorithm.

Although we do not know whether there exists deterministic poly-time simplex algorithm, we do know that LP can be solved by deterministic poly-time algorithms, i.e. LP is in '''P'''. The following two algorithms use different ideas than the simplex algorithm, and are both in poly-time:
* The ellipsoid algorithm.
* Interior point methods.
Although these algorithms guarantee polynomial time complexity in the worst-case, their performances are worse than the simplex algorithms. However, the ideas of these algorithms can be used to solve more general mathematical programmings, such as [http://en.wikipedia.org/wiki/Convex_programming convex programmings].

=== An LP solver via random walks ===
We discuss a new algorithm for solving LP which is based on random sampling, introduced by Bertsimas and Vempala in 2004.

The problem of solving an LP can be reduced to that given a convex polytope <math>Ax\le b</math>, test whether the polytope is empty. We call this problem the "''feasibility test''".

Suppose we have an LP:
::<math>
\begin{align}
\mbox{minimize} & \quad c^T x\\
\mbox{subject to} & \quad Ax\le b
\end{align}
</math>
To see this LP can be solved by feasibility testing, we treat the <math>c^Tx\le d</math> for some parameter <math>d</math> as an additional constraint, and define a new polytope <math>A'x\le b'</math>. The smallest <math>d</math> that the polytope <math>A'x\le b'</math> is not empty is the optimal solution to the original LP. We can find this smallest <math>d</math> by binary search if we can efficiently test the emptiness of a convex polytope.

We assume that the convex polytope <math>P</math> is contained in the axis-aligned cube of width <math>R</math> centered at the origin; further if <math>P</math> is non-empty then it contains a cube of width <math>r</math>. The parameter <math>L</math> is equal to <math>\log\frac{R}{r}</math>.

{|border="1"
|'''Algorithm''': feasibility test
|-
|'''Input''': an <math>n</math>-dimensional convex polytope <math>P</math> defined by the system <math>Ax\le b</math>.
'''Output''': a point <math>x\in P</math> in the polytope, or a guarantee that <math>P</math> is empty.
|-
|
:Let <math>K</math> be the axis-aligned cube of side length <math>R</math> and center <math>z=\boldsymbol{0}</math>.
:Repeat for <math>2nL</math> times do:
:: If <math>Az\le b</math>, return <math>z</math>.
:: Pick a constraint violated by <math>z</math>, say that <math>A_i z>b_i</math>, and define the halfspace
:::<math>H=\{x\mid A_i x\le A_i z\}</math>.
::Set <math>K=K\cap H</math>. Uniformly sample <math>N</math> random points <math>y^{(1)},y^{(2)},\ldots,y^{(N)}</math> from <math>K</math> and let
:::<math>z=\frac{1}{N}\sum_{i=1}^Ny^{(i)}</math>.
:Report <math>P</math> is empty.
|}

The number of samples required in each iteration, <math>N=O((\log m)^2)</math>, where <math>m</math> is the number of linear constraints.

It is easy to see that at any iteration, <math>K</math> is a convex set, and <math>P\subseteq K</math> for the current convex set <math>K</math>.

The idea of the algorithm is based on computing the [http://en.wikipedia.org/wiki/Centroid centroid] of the convex set. It is know that if we could compute the exact centroid in each iteration, then the volume of <math>K</math> drops by a constant factor <math>(1-1/e)</math> in each iteration, thus reaches the smallest possible volume of a cube of width <math>r</math> within polynomial number of iterations. But, finding the centroid of a convex polytope, is '''#P-hard'''.

The idea behind the algorithm is that an approximate centroid can be computed using <math>N=O((\log m)^2)</math> random points and the volume of <math>K</math> is drops by a constant factor with high probability in each iteration with this choice of <math>z</math>.

The uniform sampling in <math>K</math> is not easy. But since <math>K</math> is also a convex polytope, we can approximate the uniform sampling by near-uniform sampling from <math>K</math> by rapid mixing random walks. The random walks used here are similar to the ones used in the volume estimation of convex bodies. We could use the basic grid walk, or more advanced ball walk, or hit-and-run walk.

With rapid mixing random walks, the resulting LP solver runs in poly-time.

The details are omitted. You may check the following paper if interested:
* Bertsimas and Vempala, “''Solving convex programs by random walks''.” JACM 2004.

Randomized Algorithms (Spring 2010)/Approximate counting, linear programming

2010-05-24T18:56:18Z

172.21.4.18: /* Complexity model */

== Counting Problems ==

=== Complexity model ===
Recall the class '''NP''' of decision problems (the problems with "yes" or "no" answers). Formally, denoting by <math>\{0,1\}^*</math> the set of all boolean strings of any lengths, a decision problem is a function <math>f:\{0,1\}^*\rightarrow\{0,1\}</math>.
{|border="1"
|'''Definition (NP)'''
:A function <math>f:\{0,1\}^*\rightarrow\{0,1\}</math> is in '''NP''' if there exist a polynomial <math>p</math> and a poly-time algorithm <math>V</math> with boolean output such that for every <math>x\in\{0,1\}^*</math>,
::<math>f(x)=1 \Leftrightarrow \exists y\in\{0,1\}^{p(|x|)}, \mbox{such that } V(x,y)=1\,</math>.
|}
Intuitively, the '''NP''' class contains all the decision problems such that the answer is "yes" if and only if there ''exists'' a '''certificate''' which can be verified in poly-time.

In many contexts, we are interested not just in the ''existence'' of certificate but actually in counting the ''number'' of certificates. This leads to the definition of the class '''#P''' (pronounced "sharp p"). Only now the output of the function is a natural number.

{|border="1"
|'''Definition (#P)'''
:A function <math>f:\{0,1\}^*\rightarrow\mathbb{N}</math> is in '''#P''' if there exist a polynomial <math>p</math> and a poly-time algorithm <math>V</math> with boolean output such that for every <math>x\in\{0,1\}^*</math>,
::<math>f(x)=\left|\left\{y\in\{0,1\}^{p(|x|)}\mid A(x,y)=1\right\}\right|</math>.
|}
You may notice the similarity between the two definitions. The difference is that now <math>f(x)</math> does not just indicates the existence of a certificate, but gives the actual number of certificates.

;Examples
:<nowiki>#</nowiki>cycles: given as input a graph <math>G</math>, count the number of cycles in <math>G</math>.
:<nowiki>#</nowiki>SAT: given as input a boolean formula <math>\phi</math>, count the number of satisfying assignments for <math>\phi</math>.

=== FPRAS ===

== Approximate Counting ==
Let us consider the following abstract problem.

Let <math>U</math> be a finite set of known size, and let <math>G\subseteq U</math>. We want to compute the size of <math>G</math>, namely <math>|G|</math>.

We assume two devices:
* A '''uniform sampler''' <math>\mathcal{U}</math>, which uniformly and independently samples a member of <math>U</math> upon each calling.
* A '''membership oracle''' of <math>G</math>, denoted <math>\mathcal{O}</math>. Given as the input an <math>x\in U</math>, <math>\mathcal{O}(x)</math> indicates whether or not <math>x</math> is a member of <math>G</math>.

Equipped by <math>\mathcal{U}</math> and <math>\mathcal{O}</math>, we can have the following Monte Carlo algorithm:
*Choose <math>N</math> independent samples from <math>U</math> by the uniform sampler <math>\mathcal{U}</math>, represented by the random variables <math>X_1,X_2,\ldots, X_N</math>.
* Let <math>Y_i</math> be the indicator random variable defined as <math>Y_i=\mathcal{O}(X_i)</math>, namely, <math>Y_i</math> indicates whether <math>X_i\in G</math>.
* Define the estimator random variable
::<math>Z=\frac{|U|}{N}\sum_{i=1}^N Y_i.</math>

It is easy to see that <math>\mathbf{E}[Z]=|G|</math> and we might hope that with high probability the value of <math>Z</math> is close to <math>|G|</math>. Formally, <math>Z</math> is called an <math>\epsilon</math>-approximation of <math>|G|</math> if
:<math>
(1-\epsilon)|G|\le Z\le (1+\epsilon)|G|.
</math>

The following theorem states that the probabilistic accuracy of the estimation depends on the number of samples and the ratio between <math>|G|</math> and <math>|U|</math>

{|border="1"
|'''Theorem (estimator theorem)'''
:Let <math>\alpha=\frac{|G|}{|U|}</math>. Then the Monte Carlo method yields an <math>\epsilon</math>-approximation to <math>|G|</math> with probability at least <math>1-\delta</math> provided
::<math>N\ge\frac{4}{\epsilon \alpha}\ln\frac{2}{\delta}</math>.
|}
'''Proof''': Use the Chernoff bound.

<math>\square</math>

A counting algorithm for the set <math>G</math> has to deal with the following three complications:
* Implement the membership oracle <math>\mathcal{O}</math>. This is usually straightforward, or assumed by the model.
* Implement the uniform sampler <math>\mathcal{U}</math>. As we have seen, this is usually approximated by random walks. How to design the random walk and bound its mixing rate is usually technical challenging, if possible at all.
* Deal with exponentially small <math>\alpha=\frac{|G|}{|U|}</math>. This requires us to cleverly choose the universe <math>U</math>. Sometimes this needs some nontrivial ideas.

=== Counting DNFs ===
A disjunctive normal form (DNF) formular is a disjunction (OR) of clauses, where each clause is a conjunction (AND) of literals. For example:
:<math>(x_1\wedge \overline{x_2}\wedge x_3)\vee(x_2\wedge x_4)\vee(\overline{x_1}\wedge x_3\wedge x_4)</math>.
Note the difference from the conjunctive normal forms (CNF).

Given a DNF formular <math>\phi</math> as the input, the problem is to count the number of satisfying assignments of <math>\phi</math>. This problem is '''#P-complete'''.

Naively applying the Monte Carlo method will not give a good answer. Suppose that there are <math>n</math> variables. Let <math>U=\{\mathrm{true},\mathrm{false}\}^n</math> be the set of all truth assignments of the <math>n</math> variables. Let <math>G=\{x\in U\mid \phi(x)=\mathrm{true}\}</math> be the set of satisfying assignments for <math>\phi</math>. The straightforward use of Monte Carlo method samples <math>N</math> assignments from <math>U</math> and check how many of them satisfy <math>\phi</math>. This algorithm fails when <math>|G|/|U|</math> is exponentially small, namely, when exponentially small fraction of the assignments satisfy the input DNF formula.

;The union of sets problem
We reformulate the DNF counting problem in a more abstract framework, called the '''union of sets''' problem.

Let <math>V</math> be a finite universe. We are given <math>m</math> subsets <math>H_1,H_2,\ldots,H_m\subseteq V</math>. The following assumptions hold:
*For all <math>i</math>, <math>|H_i|</math> is computable in poly-time.
*It is possible to sample uniformly from each individual <math>H_i</math>.
*For any <math>x\in V</math>, it can be determined in poly-time whether <math>x\in H_i</math>.

The goal is to compute the size of <math>H=\bigcup_{i=1}^m H_i</math>.

DNF counting can be interpreted in this general framework as follows. Suppose that the DNF formula <math>\phi</math> is defined on <math>n</math> variables, and <math>\phi</math> contains <math>m</math> clauses <math>C_1,C_2,\ldots,C_m</math>, where clause <math>C_i</math> has <math>k_i</math> literals. Without loss of generality, we assume that in each clause, each variable appears at most once.
* <math>V</math> is the set of all assignments.
*Each <math>H_i</math> is the set of satisfying assignments for the <math>i</math>-th clause <math>C_i</math> of the DNF formular <math>\phi</math>. Then the union of sets <math>H=\bigcup_i H_i</math> gives the set of satisfying assignments for <math>\phi</math>.
* Each clause <math>C_i</math> is a conjunction (AND) of literals. It is not hard to see that <math>|H_i|=2^{n-k_i}</math>, which is efficiently computable.
* Sampling from an <math>H_i</math> is simple: we just fix the assignments of the <math>k_i</math> literals of that clause, and sample uniformly and independently the rest <math>(n-k_i)</math> variable assignments.
* For each assignment <math>x</math>, it is easy to check whether it satisfies a clause <math>C_i</math>, thus it is easy to determine whether <math>x\in H_i</math>.

;The coverage algorithm
We now introduce the coverage algorithm for the union of sets problem.

Consider the multiset <math>U</math> defined by
:<math>U=H_1\uplus H_2\uplus\cdots \uplus H_m</math>,
where <math>\uplus</math> denotes the multiset union. It is more convenient to define <math>U</math> as the set
:<math>U=\{(x,i)\mid x\in H_i\}</math>.
For each <math>x\in H</math>, there may be more than one instances of <math>(x,i)\in U</math>. We can choose a unique representative among the multiple instances <math>(x,i)\in U</math> for the same <math>x\in H</math>, by choosing the <math>(x,i)</math> with the minimum <math>i</math>, and form a set <math>G</math>.

Formally, <math>G=\{(x,i)\in U\mid \forall (x,j)\in U, j\le i\}</math>. Every <math>x\in H</math> corresponds to a unique <math>(x,i)\in G</math> where <math>i</math> is the smallest among <math>x\in H_i</math>.

It is obvious that <math>G\subseteq U</math> and
:<math>|G|=|H|</math>.

Therefore, estimation of <math>|H|</math> is reduced to estimation of <math>|G|</math> with <math>G\subseteq U</math>. Then <math>|G|</math> can have an <math>\epsilon</math>-approximation with probability <math>(1-\delta)</math> in poly-time, if we can uniformly sample from <math>U</math> and <math>|G|/|U|</math> is suitably small.

An uniform sample from <math>U</math> can be implemented as follows:
* generate an <math>i\in\{1,2,\ldots,m\}</math> with probability <math>\frac{|H_i|}{\sum_{i=1}^m|H_i|}</math>;
* uniformly sample an <math>x\in H_i</math>, and return <math>(x,i)</math>.

It is easy to see that this gives a uniform member of <math>U</math>. The above sampling procedure is poly-time because each <math>|H_i|</math> can be computed in poly-time, and sampling uniformly from each <math>H_i</math> is poly-time.

We now only need to lower bound the ratio
:<math>\alpha=\frac{|G|}{|U|}</math>.

We claim that
:<math>\alpha\ge\frac{1}{m}</math>.
It is easy to see this, because each <math>x\in H</math> has at most <math>m</math> instances of <math>(x,i)</math> in <math>U</math>, and we already know that <math>|G|=|H|</math>.

Due to the estimator theorem, this needs <math>\frac{4m}{\epsilon}\ln\frac{2}{\delta}</math> uniform random samples from <math>U</math>.

This gives the coverage algorithm for the abstract problem of the union of sets. The DNF counting is a special case of it.

=== Permanents and perfect matchings ===

Let <math>U=\{u_1,u_2,\ldots,u_n\}</math> and <math>V=\{v_1,v_2,\ldots,v_n\}</math>. Consider a bipartite graph <math>G(U,V,E)</math>. An <math>M\subseteq E</math> is a '''perfect matching''' of <math>G</math> if every vertex of <math>G</math> has exactly one edge in <math>M</math> adjacent to it.

Given a bipartite graph <math>G(U,V,E)</math> with <math>|U|=|V|=n</math>, we want to count the number of perfect matchings of <math>G</math>. This problem can be reduced to computing the '''permanent''' of a square matrix.

{|border="1"
|'''Definition (permanent)'''
:Let <math>Q</math> be an <math>n\times n</math> matrix. The '''permanent''' of the matrix is defined as
::<math>\mathrm{per}(Q)=\sum_{\pi\in\mathbb{S}_n}\prod_{i=1}^n Q_{i,\pi(i)}</math>,
:where <math>\mathbb{S}_n</math> is the symmetric group of permutation of size <math>n</math>.
|}

If we multiply each term of the sum the sign of the permutation, then it gives us the determinant of the matrix,
:<math>\det(Q)=\sum_{\pi\in\mathbb{S}_n}\sgn(\pi)\prod_{i=1}^n Q_{i,\pi(i)}</math>,
where <math>\sgn(\pi)</math>, the sign of a permutation, is either <math>-1</math> or <math>+1</math>, according to whether the minimum number of pair-wise interchanges to achieve <math>\pi</math> from <math>(1,2,\ldots,n)</math> is odd or even.

Unlike the determinants, which are computable in poly-time, permanents are hard to compute, as permanents can be used to count the number of perfect matchings in a bipartite graph, which is '''#P-complete'''.

A bipartite graph <math>G(U,V,E)</math> with <math>|U|=|V|=n</math> can be represented by an <math>n\times n</math> matrix <math>Q</math> with 0-1 entries as follows:
* Each row of <math>Q</math> corresponds to a vertex in <math>U</math> and each column of <math>Q</math> corresponds to a vertex in <math>V</math>.
::<math>Q_{ij}=\begin{cases}
1 & \mbox{if }i\sim j,\\
0 & \mbox{otherwise}.
\end{cases}</math>
Note the subtle difference between the definition of <math>Q</math> and the adjacency matrix.

Each perfect matching corresponds to a permutation <math>\pi</math> of <math>U</math> with <math>(u,\pi(u))\in E</math> for every <math>u\in U</math>, which corresponds to a permutation <math>\pi\in\mathbb{S}_n</math> with <math>\prod_{i=1}^n Q_{i,\pi(i)}=1</math>. It is than easy to see that <math>\mathrm{per}(Q)</math> gives the number of perfect matchings in the bipartite graph <math>G</math>.

It is known that counting the number of perfect matchings in a bipartite graph is '''#P-hard'''. Since this problem can be reduced to computing the permanent, thus the problem of computing the permanents is also '''#P-hard'''.

Now we show that with randomization, we can approximate the number of perfect matchings in a bipartite graph. In particular, we will give an FPRAS for counting the perfect matchings in a dense bipartite graph.

==== The Jerrum-Sinclair algorithm ====
Fix a bipartite graph <math>G(U,V,E)</math> with <math>|U|=|V|=n</math>. Let <math>\mathcal{M}_k</math> be the set of matchings of size <math>k</math> in <math>G</math>, and let <math>m_k=|\mathcal{M}_k|</math>. Thus, <math>\mathcal{M}_k</math> is the set of perfect matchings in <math>G</math>, and our goal is to compute <math>m_n</math>.

Let <math>r_k=\frac{m_k}{m_{k-1}}</math> for <math>1<k\le n</math>. Then
:<math>m_k=m_{k-1}r_k</math>,
which gives us a recursion to compute the <math>m_n</math>, as
:<math>m_n=m_{1}\frac{m_2}{m_1}\cdot\frac{m_3}{m_2}\cdots\frac{m_n}{m_{n-1}}=m_1\prod_{k=2}^n r_k</math>,
where <math>m_1=|\mathcal{M}_1|</math> is the number of matchings of size 1 in the bipartite graph <math>G</math>, which is just the number of edges in <math>G</math>. Therefore, <math>m_n</math> can be computed once we know <math>r_k</math> for <math>1<k\le n</math>.

Each <math>r_k</math> can be estimated by sampling uniformly from the set <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>. The algorithm for estimating <math>m_n</math> is outlined as:
# For each <math>1<k\le n</math>, have an FPRAS for computing the <math>r_k</math> by uniform sampling sufficiently many members from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math> as
::*uniformly sample <math>N</math> matching from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>, for some polynomially large <math>N</math>;
::* assuming that there are <math>X</math> sampled matchings of size <math>k</math>, return <math>r_k</math> as <math>r_k=\frac{X}{N-X}</math>.
:2. Compute <math>m_n</math> as <math>m_n=m_1\prod_{k=2}^n r_k</math>, where <math>m_1=|E|</math>.

There are several issues that we have to deal with in order to have a fully functional FPRAS for counting perfect matchings.
* By taking the product of <math>r_k</math>'s, the errors for individual <math>r_k</math>'s add up.
* In order to accurately estimate <math>r_k</math> by sampling from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>, the ratio <math>r_k</math> should be within the range <math>\left[\frac{1}{\alpha},\alpha\right]</math> for some <math>\alpha</math> within polynomial of <math>n</math>.
* Implement the uniform sampling from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>.

;Estimator for each <math>r_k</math>

; Accumulation of errors

; Near-uniform sampling from <math>\mathcal{M}_k\cup\mathcal{M}_{k-1}</math>
In the last lecture, we have shown that by random walk, we can sample a near-uniform member of <math>\mathcal{M}_n\cup\mathcal{M}_{n-1}</math> in poly-time.

=== Volume estimation ===
We consider the problem of computing the volume of a given [http://en.wikipedia.org/wiki/Convex_body convex body] <math>K</math> in <math>n</math> dimensions.

We use <math>\Upsilon(K)\,</math> to denote the volume of the convex body <math>K</math>. Abstractly, the problem is that given as input a convex body <math>K</math> in <math>n</math> dimensions, return the <math>\Upsilon(K)\,</math>.

We should be more specific about the input model. Since we allow an arbitrary convex body as input, it is not even clear how to describe the body. We assume that <math>K</math> is described by means of a '''membership oracle''' <math>\mathcal{O}</math>, such that for a '''query''' of an <math>n</math>-dimensional point <math>x</math>, <math>\mathcal{O}(x)</math> indicates whether <math>x\in K</math>.

For example, the <math>n</math>-dimensional convex body defined by the intersection of <math>m</math> half-spaces, which is the set of feasible solutions to a system of <math>m</math> linear constraints, can be described as
:<math>A x\le \boldsymbol{b}</math>,
for some <math>m\times n</math> matrix <math>A</math>, and <math>m</math>-dimensional vector <math>b</math>. For a query of an <math>x</math>, the membership oracle <math>\mathcal{O}</math> just check whether <math>Ax\le b</math>.

For deterministic algorithms, there are negative news for this problem.

{|border="1"
|'''Theorem (Bárány-Füredi 1987)'''
:Suppose that a deterministic poly-time algorithm uses the membership oracle for a convex body <math>K</math> in <math>n</math> dimensions, and generates an upper bound <math>\Upsilon_u\,</math> and a lower bound <math>\Upsilon_\ell\,</math> on the volume <math>\Upsilon(K)\,</math> of <math>K</math>. Then, there is a convex body <math>K</math> and a constant <math>c</math> such that
::<math>\frac{\Upsilon_u}{\Upsilon_\ell}\ge c\left(\frac{n}{\log n}\right)^n</math>.
|}
That is said, with deterministic algorithms, we cannot even approximate the volume within a wildly loose range.

Dyer-Frieze-Kannan come up with an idea of estimating the volume of <math>K</math> by sampling near-uniformly from convex sets. They reduce the problem of computing approximately the volume of convex bodies to this sampling problem and thus give the first FPRAS for the volume of convex bodies.

For any convex body <math>K</math>, it encloses some <math>n</math>-dimensional ball and is also enclosed by another ball with larger radius. We assume that the convex body <math>K</math> encloses the unit ball around the origin, and is enclosed by a larger ball round the origin with polynomially large radius. Formally, we assume that
:<math>B(0,1)\subseteq K\subseteq B(0,n^c)</math> for some constant <math>c</math>,
where <math>B(p,r)</math> denotes a ball of radius <math>r</math> with <math>p</math> as center, i.e. <math>B(p,r)=\{x\mid \|x-p\|\le r \}</math>. We can make this assumption because it is known that for any convex body <math>K</math>, there exists a linear transformation <math>\tau</math> which can be found within poly-time such that <math>\tau K</math> transforms the <math>K</math> to a convex body that satisfies the assumption, and preserves the ratio of the volumes of the balls to the convex body.

The volumes of the balls are easy to compute. If only the ratio between the convex body and the ball which encloses it, is sufficiently large, then we can apply the Monte Carlo method to estimate <math>\Upsilon(K)\,</math> by uniformly sampling from the ball.

However, in high-dimension, the ratio between the volume of a convex body and the volume of ball which encloses it, can be exponentially small. This is caused by the so called the "[http://en.wikipedia.org/wiki/Curse_of_dimensionality curse of dimensionality]".

Instead of having an outer ball and an inner ball, we define a sequence of balls:
:<math>B_0=B(0,\lambda^0), B_1=B(0,\lambda^1), B_2=B(0,\lambda^2),\ldots, B_m=B(0,\lambda^m)</math>,
where <math>\lambda=(1+\frac{1}{n})</math> and <math>m</math> is the smallest positive integer such that <math>\lambda^m\ge n^c</math>. Therefore, the inner ball <math>B(0,1)=B_0</math>, the outer ball <math>B(0,n^c)\subseteq B_m</math>, and <math>m</math> is within polynomial of <math>n</math>. In fact, <math>m\approx cn\ln n</math>.

This sequence of balls naturally defines a sequence of convex bodies by intersections as <math>K_i=B_i\cap K</math>. It is obvious that
:<math>B(0,1)=K_0\subseteq K_1\subseteq K_2\subseteq\cdots\subseteq K_m=K</math>.
Balls are convex, and since the intersection of convex bodies is still convex, the sequence of <math>K_i</math> is a sequence of convex bodies.

We have the telescopic product:
:<math>
\frac{\Upsilon(K_0)}{\Upsilon(K_1)}\cdot\frac{\Upsilon(K_1)}{\Upsilon(K_2)}\cdots\frac{\Upsilon(K_{m-1})}{\Upsilon(K_m)}=\frac{\Upsilon(K_0)}{\Upsilon(K_m)}=\frac{\Upsilon(B(0,1))}{\Upsilon(K)}.
</math>
Therefore, the volume <math>\Upsilon(K)\,</math> can be computed as
:<math>\Upsilon(K)=\Upsilon(B(0,1))\cdot\prod_{i=1}^{m}\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math>.

The volume of unite ball <math>\Upsilon(B(0,1))\,</math> can be precisely computed in poly-time. Each <math>\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math> is computed by near-uniform sampling from <math>K_{i}</math>, which encloses <math>K_{i-1}</math>.

Another observation is that the ratio <math>\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math> is well-bounded. Recall that <math>K_i=B(0,\lambda^i)\cap K</math> where <math>\lambda=(1+\frac{1}{n})</math>. It can be proved that in <math>n</math> dimensions, the volume of <math>K_i</math> is at most <math>\lambda^n</math> times the <math>K_{i-1}</math>, thus the ratio <math>\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math> is at most <math>\lambda^n=O(1)</math>. By the estimator theorem, we can have an FPRAS for the ratio <math>\frac{\Upsilon(K_{i})}{\Upsilon(K_{i-1})}</math> if we can uniformly sample from <math>K_i</math>.

Uniformly sampling from an arbitrary convex body is replaced by near-uniform sampling achieved by random walks. In the original walk of Dyer-Frieze-Kannan, they consider the random walk over <math>n</math>-dimensional discrete grid points enclosed by <math>K</math>, and prove that the walk is rapid mixing. This gives us the first FPRAS for volume estimation which runs in <math>\tilde{O}(n^{23})</math>, where <math>\tilde{O}(\cdot)</math> ignores the polylogarithmic factors.

The time bound was later improved by a series of works, each introducing some new ideas.

{|border="1"
| || complexity || new ingredient(s)
|-
| Dyer-Frieze-Kannan 1991 || align="center" | <math>n^{23}</math> || everything
|-
| Lovász-Simonovits 1990 || align="center" | <math>n^{16}</math> || localization lemma
|-
| Applegate-Kannan 1990 || align="center" | <math>n^{10}</math> || logconcave sampling
|-
| Lovász 1990 || align="center" | <math>n^{10}</math> || ball walk
|-
| Dyer-Frieze 1991 || align="center" | <math>n^8</math> || better error analysis
|-
| Lovász-Simonovits 1993 || align="center" | <math>n^7</math> || many improvements
|-
| Kannan-Lovász-Simonovits 1997 || align="center" | <math>n^5</math> || isotropy, speedy walk
|-
| Lovász-Vempala 2003 || align="center" | <math>n^4</math> || simulated annealing, hit-and-run
|-
|}
(cited from "Geometric Random Walks: A Survey" by Santosh Vempala.)

The current best upper bound is <math>\tilde{O}(n^4)</math> due to Lovász and Vempala in 2003. It is conjectured that the optimal bound is <math>\Theta(n^3)</math>.

== Linear Programming ==
Given a function <math>f:\mathbb{R}^n\rightarrow\mathbb{R}</math> and a set <math>\mathcal{F}\subseteq\mathbb{R}^n</math>, an '''optimization problem''' is the problem of finding an <math>x\in\mathcal{F}</math> with the optimal (minimum) <math>f(x)</math>. Formally, the problem can be expressed as:
::<math>
\begin{align}
\mbox{minimize} & \quad f(x)\\
\mbox{subject to} &\quad x\in\mathcal{F}
\end{align}
</math>
where <math>x\in\mathcal{R}^n</math> is a vector of <math>n</math> variables. For the problem of maximizing a function <math>f</math>, we just minimizes <math>-f</math> instead.

We call the function <math>f</math> the '''objective function''' and call any <math>x\in\mathcal{F}</math> a '''feasible solution''' of the problem. A feasible solution that minimizes <math>f(x)</math> is called an '''optimal solution'''. Our task is to find an optimal solution.

The feasible set <math>\mathcal{F}</math> is usually given by a number of '''constraints''' <math>P_1,P_2,\ldots,P_m</math>, which are predicates defined on <math>x</math>. An <math>x</math> is a feasible solution if it satisfies all the constraints.

A '''linear programming (LP)''' problem is an optimization problem with a linear objective function subject to a number of linear constraints. Formally, an LP is a problem that can be expressed in the following form:
::<math>
\begin{align}
\mbox{minimize} & \quad c_1x_1+c_2x_2+\cdots+c_nx_n\\
\mbox{subject to} & \quad a_{11}x_1+a_{12}x_2+\cdots+a_{1n}x_n\le b_1\\
& \quad a_{21}x_1+a_{22}x_2+\cdots+a_{2n}x_n\le b_2\\
& \qquad\qquad \vdots\\
& \quad a_{m1}x_1+a_{m2}x_2+\cdots+a_{mn}x_n\le b_m\\
\end{align}
</math>
where <math>a_{ij}</math>, <math>b_i</math> and <math>c_j</math> are constants and <math>x_j</math> are variables. For maximization problem, or constraints given by "<math>\ge</math>", we can multiply the coefficients by <math>-1</math> and still write the LP in the above form.

We can describe the programming in the vector form. Let <math>x</math> be a vector of <math>n</math> variables. Let <math>A=(a_{ij})</math> be an <math>m\times n</math> matrix of constant entries, <math>b=(b_1,\ldots,b_m)</math> be a vector of <math>m</math> constant entries, and <math>c=(c_1,\ldots,c_n)</math> be a vector of <math>n</math> constant entries. Then an LP is expressed as:
::<math>
\begin{align}
\mbox{minimize} & \quad c^T x\\
\mbox{subject to} & \quad Ax\le b
\end{align}
</math>
We call it the '''canonical form''' of linear programming.

=== The geometry of LPs ===
For <math>n</math>-dimensional space <math>\mathbb{R}^n</math>, a linear constraint <math>ax\le b</math> specifies a [http://en.wikipedia.org/wiki/Half-space halfspace]. The feasible set specified by <math>m</math> constraints together is an intersection of <math>m</math> halfspaces, thus, a [http://en.wikipedia.org/wiki/Convex_polytope convex polytope].

The convex polytope is defined by <math>Ax\le b</math>.

The LP can be thought as given an <math>n</math>-dimensional polytope and a linear (affine) function <math>f:\mathbb{R}^n\rightarrow \mathbb{R}</math>, looking for a point <math>x</math> in the polytope with the smallest function value <math>f(x)</math>.

We can choose a number of constraints in the system <math>Ax\le b</math>, and make them hold with equality. This would define a subspace of <math>\mathbb{R}^n</math>. In particular, if we choose <math>n</math> linearly independent constraints to form an <math>n\times n</math> submatrix <math>A'</math> and the corresponding <math>n</math>-dimensional vector <math>b'</math>, solving <math>A'x=b'</math> would give us exactly one point <math>x</math>. If this <math>x</math> is in the polytope, i.e. <math>x</math> satisfies that <math>Ax\le b</math>, we call such <math>x</math> a '''vertex''' of the polytope <math>Ax\le b</math>. In the context of LP, it is also called a '''basic feasible solution (''bfs'')''' of the LP.

A key observation for LP is that there is a basic feasible solution which is optimal, i.e. the optimal solution is a vertex of the polytope.

=== The simplex algorithms ===
The [http://en.wikipedia.org/wiki/Simplex_algorithm simplex algorithm] by George Dantzig solves the linear programming by moving from vertex to vertex of the convex polytope, each time making some progress towards optimizing the objective function. Eventually the algorithm reaches a vertex which is a [http://en.wikipedia.org/wiki/Local_optimum local optimum]. In a convex set, a locally optimal point is also global optimal, thus the simplex algorithm returns an optimal solution in finite steps.

The problem with the simplex algorithm is that in some bad polytopes, it takes the simplex algorithm exponentially many steps to reach the optimum. The simplex algorithm is actually a class of algorithms defined by various '''pivoting rules''', which describe how to move locally from one vertex to another. The original pivoting rule proposed by Dantzig has an exponentially large worst-case time complexity (though works very good in practice). To-date, it is still unknown whether there exists any deterministic simplex algorithm with sub-exponential worst-case complexity.

People have tried randomized pivoting rules, and there are some amazing progresses have been made:
* Kalai 1992: there is a randomized simplex algorithm with sub-exponential time complexity.
* Kelner-Spielman 2006: there is a polynomial time randomized simplex algorithm.

Although we do not know whether there exists deterministic poly-time simplex algorithm, we do know that LP can be solved by deterministic poly-time algorithms, i.e. LP is in '''P'''. The following two algorithms use different ideas than the simplex algorithm, and are both in poly-time:
* The ellipsoid algorithm.
* Interior point methods.
Although these algorithms guarantee polynomial time complexity in the worst-case, their performances are worse than the simplex algorithms. However, the ideas of these algorithms can be used to solve more general mathematical programmings, such as [http://en.wikipedia.org/wiki/Convex_programming convex programmings].

=== An LP solver via random walks ===
We discuss a new algorithm for solving LP which is based on random sampling, introduced by Bertsimas and Vempala in 2004.

The problem of solving an LP can be reduced to that given a convex polytope <math>Ax\le b</math>, test whether the polytope is empty. We call this problem the "''feasibility test''".

Suppose we have an LP:
::<math>
\begin{align}
\mbox{minimize} & \quad c^T x\\
\mbox{subject to} & \quad Ax\le b
\end{align}
</math>
To see this LP can be solved by feasibility testing, we treat the <math>c^Tx\le d</math> for some parameter <math>d</math> as an additional constraint, and define a new polytope <math>A'x\le b'</math>. The smallest <math>d</math> that the polytope <math>A'x\le b'</math> is not empty is the optimal solution to the original LP. We can find this smallest <math>d</math> by binary search if we can efficiently test the emptiness of a convex polytope.

We assume that the convex polytope <math>P</math> is contained in the axis-aligned cube of width <math>R</math> centered at the origin; further if <math>P</math> is non-empty then it contains a cube of width <math>r</math>. The parameter <math>L</math> is equal to <math>\log\frac{R}{r}</math>.

{|border="1"
|'''Algorithm''': feasibility test
|-
|'''Input''': an <math>n</math>-dimensional convex polytope <math>P</math> defined by the system <math>Ax\le b</math>.
'''Output''': a point <math>x\in P</math> in the polytope, or a guarantee that <math>P</math> is empty.
|-
|
:Let <math>K</math> be the axis-aligned cube of side length <math>R</math> and center <math>z=\boldsymbol{0}</math>.
:Repeat for <math>2nL</math> times do:
:: If <math>Az\le b</math>, return <math>z</math>.
:: Pick a constraint violated by <math>z</math>, say that <math>A_i z>b_i</math>, and define the halfspace
:::<math>H=\{x\mid A_i x\le A_i z\}</math>.
::Set <math>K=K\cap H</math>. Uniformly sample <math>N</math> random points <math>y^{(1)},y^{(2)},\ldots,y^{(N)}</math> from <math>K</math> and let
:::<math>z=\frac{1}{N}\sum_{i=1}^Ny^{(i)}</math>.
:Report <math>P</math> is empty.
|}

The number of samples required in each iteration, <math>N=O((\log m)^2)</math>, where <math>m</math> is the number of linear constraints.

It is easy to see that at any iteration, <math>K</math> is a convex set, and <math>P\subseteq K</math> for the current convex set <math>K</math>.

The idea of the algorithm is based on computing the [http://en.wikipedia.org/wiki/Centroid centroid] of the convex set. It is know that if we could compute the exact centroid in each iteration, then the volume of <math>K</math> drops by a constant factor <math>(1-1/e)</math> in each iteration, thus reaches the smallest possible volume of a cube of width <math>r</math> within polynomial number of iterations. But, finding the centroid of a convex polytope, is '''#P-hard'''.

The idea behind the algorithm is that an approximate centroid can be computed using <math>N=O((\log m)^2)</math> random points and the volume of <math>K</math> is drops by a constant factor with high probability in each iteration with this choice of <math>z</math>.

The uniform sampling in <math>K</math> is not easy. But since <math>K</math> is also a convex polytope, we can approximate the uniform sampling by near-uniform sampling from <math>K</math> by rapid mixing random walks. The random walks used here are similar to the ones used in the volume estimation of convex bodies. We could use the basic grid walk, or more advanced ball walk, or hit-and-run walk.

With rapid mixing random walks, the resulting LP solver runs in poly-time.

The details are omitted. You may check the following paper if interested:
* Bertsimas and Vempala, “''Solving convex programs by random walks''.” JACM 2004.