Combinatorics (Fall 2010)/Generating functions

Stirling's Approximation

As we have seen, answers to many counting problems are expressed in terms of factorials. It is important for us to know roughly how large [math]\displaystyle{ n! }[/math] is, i.e. the asymptotic order of [math]\displaystyle{ n! }[/math]. This is done by the famous Stirling's approximation, also called Stirling's formula.

Theorem (Stirling's approximation)

[math]\displaystyle{ n!\sim \sqrt{2\pi n}\left(\frac{n}{e}\right)^n }[/math].

The symbol [math]\displaystyle{ \sim\, }[/math] means that [math]\displaystyle{ \lim_{n\rightarrow\infty}\frac{n!}{\sqrt{2\pi n}\left(\frac{n}{e}\right)^n}=1 }[/math].

The proof of the Stirling's approximation is complicated. We will prove the following easier proposition.

Proposition

[math]\displaystyle{ \ln (n!)=n\ln n-n+O(\ln n)\, }[/math]

Proof. The logarithm of [math]\displaystyle{ n! }[/math] is [math]\displaystyle{ \ln (n!)=\sum_{k=1}^n\ln k }[/math]. Since [math]\displaystyle{ f(x)=\ln x }[/math] is an increasing function of [math]\displaystyle{ x }[/math] for all positive [math]\displaystyle{ x }[/math], we have [math]\displaystyle{ \ln (n!)=\sum_{k=1}^n\ln k \le \int_2^{n+1}\ln x\, \mathrm{d}x =(n+1)\ln(n+1)-(n+1)-2\ln 2+2. }[/math] Note that [math]\displaystyle{ \ln(n+1)-\ln n=\int_{n}^{n+1}\frac{1}{x}\,\mathrm{d}x\lt \frac{1}{n}, }[/math] thus [math]\displaystyle{ n\ln (n+1)\lt n\ln n+1 }[/math]. Combining this with the above upper bound, [math]\displaystyle{ \ln (n!)\le n\ln n-n+O(\ln n). }[/math] [math]\displaystyle{ \square }[/math]

The full proof of Stirling's approximation involves a more precise estimation of the constant factor in [math]\displaystyle{ O(\ln n) }[/math].

Binomial coefficient

For binomial coefficient, we can approximate it by the Stirling's approximation of factorial, but we also have an easier (but loose) estimation.

Proposition

[math]\displaystyle{ \left(\frac{n}{k}\right)^k\le {n\choose k} \lt \left(\frac{\mathrm{e}n}{k}\right)^k }[/math]

Proof. The lower bound is easy: [math]\displaystyle{ {n\choose k}=\frac{n}{k}\cdot\frac{n-1}{k-1}\cdots\frac{n-k+1}{1}\ge\frac{n}{k}\cdot\frac{n}{k}\cdots\frac{n}{k}=\left(\frac{n}{k}\right)^k }[/math]. For the upper bound, [math]\displaystyle{ \mathrm{e}^{nx}\gt (1+x)^n=\sum_{i=1}^n{n\choose i}x^i\gt {n\choose k}x^k, }[/math] for any [math]\displaystyle{ 0\le k\le n }[/math]. Setting [math]\displaystyle{ x=\frac{k}{n} }[/math], we have [math]\displaystyle{ \mathrm{e}^k\gt {n\choose k}\left(\frac{k}{n}\right)^k. }[/math] Therefore, [math]\displaystyle{ {n\choose k} \lt \left(\frac{\mathrm{e}n}{k}\right)^k }[/math]. [math]\displaystyle{ \square }[/math]

Generating Functions

In Stanley's magnificent book Enumerative Combinatorics, he comments the generating function as "the most useful but most difficult to understand method (for counting)".

The solution to a counting problem is usually represented as some [math]\displaystyle{ a_n }[/math] depending a parameter [math]\displaystyle{ n }[/math]. Sometimes this [math]\displaystyle{ a_n }[/math] is called a counting function as it is a function of the parameter [math]\displaystyle{ n }[/math]. [math]\displaystyle{ a_n }[/math] can also be treated as a infinite series:

[math]\displaystyle{ a_0,a_1,a_2,\ldots }[/math]

The ordinary generating function (OGF) defined by [math]\displaystyle{ a_n }[/math] is

[math]\displaystyle{ G(x)=\sum_{n\ge 0} a_nx^n. }[/math]

So [math]\displaystyle{ G(x)=a_0+a_1x+a_2x^2+\cdots }[/math]. An expression in this form is called a formal power series, and [math]\displaystyle{ a_0,a_1,a_2,\ldots }[/math] is the sequence of coefficients.

Furthermore, the generating function can be expanded as

G(x)=[math]\displaystyle{ (\underbrace{1+\cdots+1}_{a_0})+(\underbrace{x+\cdots+x}_{a_1})+(\underbrace{x^2+\cdots+x^2}_{a_2})+\cdots+(\underbrace{x^n+\cdots+x^n}_{a_n})+\cdots }[/math]

so it indeed "generates" all the possible instances of the objects we want to count.

Usually, we do not evaluate the generating function [math]\displaystyle{ GF(x) }[/math] on any particular value. [math]\displaystyle{ x }[/math] remains as a formal variable without assuming any value. The numbers that we want to count are the coefficients carried by the terms in the formal power series. So far the generating function is just another way to represent the sequence

[math]\displaystyle{ (a_0,a_1,a_2,\ldots\ldots) }[/math].

The true power of generating functions comes from the various algebraic operations that we can perform on these generating functions. We use an example to demonstrate this.

Fibonacci numbers

Consider the following counting problems.

• Count the number of ways that the nonnegative integer [math]\displaystyle{ n }[/math] can be written as a sum of ones and twos (in order).

The problem asks for the number of compositions of [math]\displaystyle{ n }[/math] with summands from [math]\displaystyle{ \{1,2\} }[/math]. Formally, we are counting the number of tuples [math]\displaystyle{ (x_1,x_2,\ldots,x_k) }[/math] for some [math]\displaystyle{ k\le n }[/math] such that [math]\displaystyle{ x_i\in\{1,2\} }[/math] and [math]\displaystyle{ \sum_{i=1}^k x_i=n }[/math].

Let [math]\displaystyle{ F_n }[/math] be the solution. We observe that a composition either starts with a 1, in which case the rest is a composition of [math]\displaystyle{ n-1 }[/math]; or starts with a 2, in which case the rest is a composition of [math]\displaystyle{ n-2 }[/math]. So we have the recursion for [math]\displaystyle{ F_n }[/math] that

[math]\displaystyle{ F_n=F_{n-1}+F_{n-2} }[/math].

• Count the ways to completely cover a [math]\displaystyle{ 2\times n }[/math] rectangle with [math]\displaystyle{ 2\times 1 }[/math] dominos without any overlaps.

Dominos are identical [math]\displaystyle{ 2\times 1 }[/math] rectangles, so that only their orientations --- vertical or horizontal matter.

Let [math]\displaystyle{ F_n }[/math] be the solution. It also holds that [math]\displaystyle{ F_n=F_{n-1}+F_{n-2} }[/math]. The proof is left as an exercise.

In both problems, the solution is given by [math]\displaystyle{ F_n }[/math] which satisfies the following recursion.

[math]\displaystyle{ F_n=\begin{cases} 0 & \mbox{if }n=0\\ 1 & \mbox{if }n=1\\ F_{n-1}+F_{n-2} & \mbox{if}n\ge 2. \end{cases} }[/math]

[math]\displaystyle{ F_n }[/math] is called the Fibonacci number.

Theorem

[math]\displaystyle{ F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right) }[/math], where [math]\displaystyle{ \phi=\frac{1+\sqrt{5}}{2} }[/math] and [math]\displaystyle{ \hat{\phi}=\frac{1-\sqrt{5}}{2} }[/math].

The quantity [math]\displaystyle{ \phi=\frac{1+\sqrt{5}}{2} }[/math] is the so-called golden ratio, a constant with some significance in mathematics and aesthetics.

We now prove this theorem by using generating functions.

The ordinary generating function for the Fibonacci number [math]\displaystyle{ F_{n} }[/math] is

[math]\displaystyle{ G(x)=\sum_{n\ge 0}F_n x^n }[/math].

We have that [math]\displaystyle{ F_{n}=F_{n-1}+F_{n-2} }[/math] for [math]\displaystyle{ n\ge 2 }[/math], thus

[math]\displaystyle{ \begin{align} G(x) &= \sum_{n\ge 0}F_n x^n &= x+\sum_{n\ge 2}(F_{n-1}+F_{n-2})x^n. \end{align} }[/math]

For generating functions, there are general ways to generate [math]\displaystyle{ F_{n-1} }[/math] and [math]\displaystyle{ F_{n-2} }[/math], or the coefficients with any smaller indices.

[math]\displaystyle{ \begin{align} xG(x) &=\sum_{n\ge 0}F_n x^{n+1}=\sum_{n\ge 1}F_{n-1} x^n=\sum_{n\ge 2}F_{n-1} x^n\\ x^2G(x) &=\sum_{n\ge 0}F_n x^{n+2}=\sum_{n\ge 2}F_{n-2} x^n. \end{align} }[/math]

So we have

[math]\displaystyle{ G(x)=x+(x+x^2)G(x)\, }[/math],

hence

[math]\displaystyle{ G(x)=\frac{x}{1-x-x^2} }[/math].

The value of [math]\displaystyle{ F_n }[/math] is the coefficient of [math]\displaystyle{ x^n }[/math] in the Taylor series for this formular, which is [math]\displaystyle{ \frac{G^{(n)}(0)}{n!}=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n }[/math]. Although this expansion works in principle, the detailed calculus is rather painful.

---

There is an easier way to get this coefficient than directly expanding the Taylor series.

[math]\displaystyle{ 1-x-x^2 }[/math] has two roots [math]\displaystyle{ \frac{-1\pm\sqrt{5}}{2} }[/math].

Denote that [math]\displaystyle{ \phi=\frac{2}{-1+\sqrt{5}}=\frac{1+\sqrt{5}}{2} }[/math] and [math]\displaystyle{ \hat{\phi}=\frac{2}{-1-\sqrt{5}}=\frac{1-\sqrt{5}}{2} }[/math]. Then [math]\displaystyle{ (1-x-x^2)=(1-\phi x)(1-\hat{\phi}x) }[/math], so we can write

[math]\displaystyle{ \begin{align} \frac{x}{1-x-x^2} &=\frac{x}{(1-\phi x)(1-\hat{\phi} x)}\\ &=\frac{\alpha}{(1-\phi x)}+\frac{\beta}{(1-\hat{\phi} x)}, \end{align} }[/math]

where [math]\displaystyle{ \alpha }[/math] and [math]\displaystyle{ \beta }[/math] satisfying that

[math]\displaystyle{ \begin{cases} \alpha+\beta=0\\ \alpha\phi+\beta\hat{\phi}= -1. \end{cases} }[/math]

Solving this we have that [math]\displaystyle{ \alpha=\frac{1}{\sqrt{5}} }[/math] and [math]\displaystyle{ \beta=-\frac{1}{\sqrt{5}} }[/math]. And

[math]\displaystyle{ G(x)=\frac{x}{1-x-x^2}=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x} }[/math]

where [math]\displaystyle{ \phi=\frac{1+\sqrt{5}}{2} }[/math] and [math]\displaystyle{ \hat{\phi}=\frac{1-\sqrt{5}}{2} }[/math].

Note that the expression [math]\displaystyle{ \frac{1}{1-z} }[/math] has a well known expansion:

[math]\displaystyle{ \frac{1}{1-z}=\sum_{n\ge 0}z^n }[/math].

Therefore, [math]\displaystyle{ G(x) }[/math] can be expanded as

[math]\displaystyle{ \begin{align} G(x) &=\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\phi x}-\frac{1}{\sqrt{5}}\cdot\frac{1}{1-\hat{\phi} x}\\ &=\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\phi x)^n-\frac{1}{\sqrt{5}}\sum_{n\ge 0}(\hat{\phi} x)^n\\ &=\sum_{n\ge 0}\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)x^n. \end{align} }[/math]

So the [math]\displaystyle{ n }[/math]th Fibonacci number is given by

[math]\displaystyle{ F_n=\frac{1}{\sqrt{5}}\left(\phi^n-\hat{\phi}^n\right)=\frac{1}{\sqrt{5}}\left(\frac{1+\sqrt{5}}{2}\right)^n-\frac{1}{\sqrt{5}}\left(\frac{1-\sqrt{5}}{2}\right)^n }[/math].

Solving recurrences

In the above analysis of Fibonacci numbers, we apply the following general methodology of solving recurrences by generating functions.

1. Give a recursion that computes [math]\displaystyle{ a_n }[/math]; that is, an equation expressing [math]\displaystyle{ a_n }[/math] in terms of other elements of the sequence, such as

[math]\displaystyle{ a_n=f(a_0,a_1,\ldots,a_{n-1}) }[/math] for some function [math]\displaystyle{ f }[/math].

2. Multiply both sides of the equation by [math]\displaystyle{ x^n }[/math] and sum over all [math]\displaystyle{ n }[/math]. This gives the generating function

[math]\displaystyle{ G(x)=\sum_{n\ge 0}a_nx^n=\sum_{n\ge 0}f(a_0,a_1,\ldots,a_{n-1})x^n }[/math].

And manipulate the right hand side of the equation so that it becomes some other expression involving [math]\displaystyle{ G(x) }[/math].

3. Solve the resulting equation to derive an explicit formula for [math]\displaystyle{ G(x) }[/math].

4. Expand [math]\displaystyle{ G(x) }[/math] into a power series and read off the coefficient of [math]\displaystyle{ x^n }[/math], which is a closed form for [math]\displaystyle{ a_n }[/math].

Algebraic operations on generating functions

The second step in the above methodology is somehow tricky. It involves first applying the recurrence to the coefficients of [math]\displaystyle{ G(x) }[/math], which is easy; and then manipulating the resulting formal power series to express it in terms of [math]\displaystyle{ G(x) }[/math], which is more difficult (because it works backwards).

We can apply several natural algebraic operations on the formal power series.

Generating function manipulation

Let [math]\displaystyle{ G(x)=\sum_{n\ge 0}g_nx^n }[/math] and [math]\displaystyle{ F(x)=\sum_{n\ge 0}f_nx^n }[/math]. [math]\displaystyle{ \begin{align} x^k G(x) &= \sum_{n\ge k}g_{n-k}x^n, &\qquad (\mbox{integer }k\ge 0)\\ \frac{G(x)-\sum_{i=0}^{k-1}g_iz^i}{x^k} &=\sum_{n\ge 0}g_{n+k}x^n, &\qquad (\mbox{integer }k\ge 0)\\ \alpha F(x)+\beta G(x) &= \sum_{n\ge 0} (\alpha f_n+\beta g_n)x^n\\ F(x)G(x) &= \sum_{n\ge 0}\sum_{k=0}^nf_kg_{n-k}x^n\\ G(cx) &= \sum_{n\ge 0} c^ng_n x^n\\ G'(x) &= \sum_{n\ge 0}(n+1)g_{n+1}x^n\\ \frac{G(x)}{1-x} &=\sum_{n\ge 0}\left(\sum_{k=0}^ng_k\right)x^n \end{align} }[/math]

When manipulating generating functions, these rules are applied backwards; that is, from the right-hand-side to the left-hand-side.

Expanding generating functions

The last step of solving recurrences by generating function is expanding the closed form generating function [math]\displaystyle{ G(x) }[/math] to evaluate its [math]\displaystyle{ n }[/math]-th coefficient. In principle, we can always use the Taylor series

[math]\displaystyle{ G(x)=\sum_{n\ge 0}\frac{G^{(n)}(0)}{n!}x^n }[/math],

where [math]\displaystyle{ G^{(n)}(0) }[/math] is the value of the [math]\displaystyle{ n }[/math]-th derivative of [math]\displaystyle{ G(x) }[/math] evaluated at [math]\displaystyle{ x=0 }[/math].

Some interesting special cases are very useful.

Geometric sequence

In the example of Fibonacci numbers, we use the well known geometric series:

[math]\displaystyle{ \frac{1}{1-x}=\sum_{n\ge 0}x^n }[/math].

It is useful when we can express the generating function in the form of [math]\displaystyle{ G(x)=\frac{a_1}{1-b_1x}+\frac{a_2}{1-b_2x}+\cdots+\frac{a_k}{1-b_kx} }[/math]. The coefficient of [math]\displaystyle{ x^n }[/math] in such [math]\displaystyle{ G(x) }[/math] is [math]\displaystyle{ a_1b_1^n+a_2b_2^n+\cdots+a_kb_k^n }[/math].

Binomial theorem

The [math]\displaystyle{ n }[/math]-th derivative of [math]\displaystyle{ (1+x)^\alpha }[/math] for some real [math]\displaystyle{ \alpha }[/math] is

[math]\displaystyle{ \alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)(1+x)^{\alpha-n} }[/math].

By Taylor series, we get a generalized version of the binomial theorem known as Newton's formula:

[math]\displaystyle{ (1+x)^\alpha=\sum_{n\ge 0}{\alpha\choose n}x^{n} }[/math],

where [math]\displaystyle{ {\alpha\choose n} }[/math] is the generalized binomial coefficient defined by

[math]\displaystyle{ {\alpha\choose n}=\frac{\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-n+1)}{n!} }[/math].

In the last lecture we gave a combinatorial proof of the number of [math]\displaystyle{ k }[/math]-multisets on an [math]\displaystyle{ n }[/math]-set. Now we give a generating function approach to the problem.

Let [math]\displaystyle{ S=\{x_1,x_2,\ldots,x_n\} }[/math] be an [math]\displaystyle{ n }[/math]-element set. We have

[math]\displaystyle{ (1+x_1+x_1^2+\cdots)(1+x_2+x_2^2+\cdots)\cdots(1+x_n+x_n^2+\cdots)=\sum_{m:S\rightarrow\mathbb{N}} \prod_{x_i\in S}x_i^{m(x_i)} }[/math],

where each [math]\displaystyle{ m:S\rightarrow\mathbb{N} }[/math] species a possible multiset on [math]\displaystyle{ S }[/math] with multiplicity function [math]\displaystyle{ m }[/math].

Let all [math]\displaystyle{ x_i=x }[/math]. Then

[math]\displaystyle{ \begin{align} (1+x+x^2+\cdots)^n &= \sum_{m:S\rightarrow\mathbb{N}}x^{m(x_1)+\cdots+m(x_n)}\\ &= \sum_{\text{multiset }M\text{ on }S}x^{|M|}\\ &= \sum_{k\ge 0}\left({n\choose k}\right)x^k. \end{align} }[/math]

The last equation is due to the the definition of [math]\displaystyle{ \left({n\choose k}\right) }[/math]. Our task is to evaluate [math]\displaystyle{ \left({n\choose k}\right) }[/math].

Due to the geometric sequence and the Newton's formula

[math]\displaystyle{ (1+x+x^2+\cdots)^n=(1-x)^{-n}=\sum_{k\ge 0}{-n\choose k}(-x)^k. }[/math]

So

[math]\displaystyle{ \left({n\choose k}\right)=(-1)^k{-n\choose k}={n+k-1\choose k}. }[/math]

The last equation is due to the definition of the generalized binomial coefficient. We use an analytic (generating function) proof to get the same result of [math]\displaystyle{ \left({n\choose k}\right) }[/math] as the combinatorial proof.