Analytic Combinatorics – A Worked Example

https://grossack.site/2025/04/08/analytic-combinatorics-example.html

by mathgenius • 1 year ago

105 54 1 year ago

antics6• 1 year ago

This is only tangentially related, but one thing that I am still kind of hopeful about is analytic combinatorics will free us from the tyranny of asymptotic analysis.

I suppose this is kind of a hot take, and in some ways less relevant than it was (say) 10 years ago, but I believe the theoretical CS community has spent a lot of time analyzing the performance of various algorithms over the years and kind of pointedly not inventing algorithms which accomplish some result at all, even if it is only known to be computable in a pathologically inefficient way. This means a lot of breakthroughs that could have been TCS breakthroughs have been invented, usually worse, by other communities.

I am not the only one who thinks so, here[1] is an obscure talk where Michael Mitzenmacher (a well-known CS theory prof at Harvard) makes more or less this specific complaint.

One of the nice things about Flajolet/Sedgewick[2] (mentioned in this post) is that it gives a nice set of combinators for (at least it seems to me, in theory) precisely enumerating how many "steps" an algorithm would take given some input. I optimistically imagine a world where we replace the relatively non-specific asymptotic bounds of an algorithm with a very specific quantification of the work an algorithm will take. And that this in turn would free TCS to focus on things that are more germane to the field as a whole.

With that all said I'm open to the argument that TCS has fundamentally changed and this is no longer such a big issue. But perusing the various TCS conferences I sense it isn't.

[1] https://www.youtube.com/watch?v=p3anjSAnKBo

[2] https://algo.inria.fr/flajolet/Publications/book.pdf

krick• 1 year ago

I optimistically imagine a world where we replace the relatively non-specific asymptotic bounds of an algorithm with a very specific quantification of the work an algorithm will take.

My understanding is that it's pretty much how it started, but proved to be impractical. Meaning, we absolutely can quantify the work any algorithm will take, but for that we need to define the architecture first, i.e., to specify what is the unit of work, what are operations available to us. And, sure, we can do that for any given task (and we always could), but it's usually too messy to include that in general discussions about algorithms, not specifically concerned with the architecture.

antics6• 1 year ago

Yep, That's exactly why we moved to asymptotic analysis. I was just hoping analytic combinatorics would help us move back to something like this.

qsort3• 1 year ago

I'm not a researcher, so start with the default assumption that I don't know what I'm talking about, but I had the same impression when studying those topics, asymptotic analysis is kind of broken.

For example, we assume that arbitrary arithmetic takes constant time. Good. Except that if you take that seriously, then P=PSPACE. So we might say that bit operations on fixed registers are constant time. But nobody does that because then arithmetic is O(lg n) and it's too much trouble to keep track of everything (and if you do, then hashtables aren't O(1) any longer!) If we are more sophisticated we might take a transidchotomous model, but that's really weird and we have results like sorting in less than n lg n time. Not to mention that the mathematical foundations get a bit shaky with more than one variable.

I think the hard problem underneath is that it's hard to define a computational model that (a) can be reasoned about and isn't extremely weird, (b) has solid mathematical foundations, (c) is reasonably predictive of real world performance, and (d) is not linked to a specific architecture. Asymptotic analysis is kind of a compromise. It can be formulated in very elementary terms, it makes mathematical sense and if you plot the performance of a sorting algorithm on a chart it kind of looks like the graph of n log n if you squint a bit.

Knuth does something similar to what you're suggesting IIRC, albeit without the analytic combinatorics machinery. For some programs he derives a precise expression as a function of the size of input that describes how many operations of each kind the underlying machine will perform. In numerical analysis there is a similar concept, e.g. evaluating a polynomial with Horner has the same asymptotic complexity as the naive method, but we say it's better because it takes exactly the optimal amount of floating point operations (and, in practice, because it uses fused multiply-adds, but that's another story...)

A research program along those lines would be certainly fascinating, but the combination of "needs to be reasonably predictive" and "needs to be portable" is a tough nut to crack.

antics6• 1 year ago

Yep, I mentioned this elsewhere in the threads—Knuth does do this quite a bit in TAOCP. I think in a lot of ways we've ended up completely full circle, in that we are now counting FLOPs. This time though I think we have very little hope for something as nice and pretty as asymptotics. At least ones that are useful.

Tainnor2• 1 year ago

For example, we assume that arbitrary arithmetic takes constant time.

Arithmetic on fixed-width number types is constant. It's very much not constant on arbitrary precision integers, something that underpins much of cryptography. Indeed, if you try to e.g. use Gaussian elimination (which uses O(n^3) arithmetic operations) to compute the determinant of a large integer matrix, you'll run into trouble, which is why there is another algorithm for this use case: https://en.m.wikipedia.org/wiki/Bareiss_algorithm

(Disclaimer: Also not a researcher)

PollardsRho4• 1 year ago

I don't think it's controversial to say that asymptotic analysis has flaws: the conclusions you draw from it only hold in the limit of larger inputs, and sometims "larger" means "larger than anything you'd be able to run it on." Perhaps as Moore's law dies we'll be increasingly able to talk more about specific problem sizes in a way that won't become obsolete immediately.

I suppose my question is why you think TCS people would do this analysis and development better than non-TCS people. Once you leave the warm cocoon of big-O, the actual practical value of an algorithm depends hugely on specific hardware details. Similarly, once you stop dealing with worst-case or naive average-case complexity, you have to try and define a data distribution relevant for specific real-world problems. My (relatively uninformed) sense is that the skill set required to, say, implement transformer attention customizing to the specific hierarchical memory layout of NVIDIA datacenter GPUs, or evaluate evolutionary optimization algorithms on a specific real-world problem domain, isn't necessarily something you gain in TCS itself.

When you can connect theory to the real world, it's fantastic, but my sense is that such connections are often desired and rarely found. At the very least, I'd expect that to often be a response to applied CS and not coming first from TCS: it's observed empirically that the simplex algorithm works well in practice, and then that encourages people to revisit the asymptotic analysis and refine it. I'd worry that TCS work trying to project onto applications from the blackboard would lead to less rigorous presentations and a lot of work that's only good on paper.

LegionMammal9783• 1 year ago

Average-case complexity can be a fickle beast. As you say, the simplex algorithm for LP is great in practice, so it's rarely problematic to use. But meanwhile, people also say, "Modern SAT/SMT solvers are great, they can solve huge problems!" Yet when I try using one, I usually run into exponential runtimes very quickly, especially for unsatisfiable instances.

Overall, it's annoying to tell whether an NP-hard problem is always really hard, or if ~all practical instances can be solved with a clever heuristic. It doesn't help that most problems receive little attention (e.g., to find solvable special cases) after being labeled NP-hard.

antics6• 1 year ago

I just meant that there are lots of problems where I think TCS could have contributed but didn't, because the relevant analysis was not acceptable at FOCS or SODA (or whatever). A good example is the original transformers paper, which is vastly over-complicated relative to (say) a later treatment[1] by Ruslan Salakhutdinov's lab, which shows that attention is "just" plain-old Nadaraya-Watson kernel smoothing—which if you don't know is a weighted average of points covered in elementary grad stats.

I'm not saying it was TCS people would be "better" at discovering this or anything like that, I'm just saying that the opportunity for contribution from TCS people is very high, and because the fields are not well-integrated you often have to wait years to uncover what ML concepts "really" are. I think working together would benefit both ML and TCS!

[1] https://arxiv.org/pdf/1908.11775

openasocket• 1 year ago

One of the nice things about Flajolet/Sedgewick[2] (mentioned in this post) is that it gives a nice set of combinators for (at least it seems to me, in theory) precisely enumerating how many "steps" an algorithm would take given some input. I optimistically imagine a world where we replace the relatively non-specific asymptotic bounds of an algorithm with a very specific quantification of the work an algorithm will take. And that this in turn would free TCS to focus on things that are more germane to the field as a whole.

Check out the Art of Computer Programming, by Knuth. He writes all of the algorithms in a hypothetical machine code called MIX (MMIX in later editions). He does go through algorithmic analysis to the degree you describe, showing exactly how many cycles a particular routine may take to complete, based on different variables. This is actually fairly complex even for simple programs. Consider the simple case of computing the maximum in a list:

    let current_max := 0
    for x in array_of_numbers {
        if x > current_max {
            current_max = x;
        }
    }
    return current_max;

Anyone can tell you this runs in O(N) time. But exactly how many cycles would this take? Ignoring the architectural details, you would need to know how often the condition "x > current_max" is true. If the list of numbers is in ascending order, this will be true every time, if it is in descending order it will only be true once. The difference in the number of cycles between those two cases is significant. I believe Knuth works through this example, and shows that, for a randomly-ordered array, this would be true around log_2(n) times.

antics6• 1 year ago

I actually have read a chunk of the relevant TAOCP, but didn't mention it because the comment was already kind of long. I remarked to a friend the other day that it's kind of funny that we've a little bit come full circle on asymptotic analysis in that we are now counting FLOPs, and it is not a little bit like what Knuth does here. :)

pfdietz2• 1 year ago

Natural log of n, actually. That's because the expected number of times a new max is found is H_n, the nth harmonic number, where n is the size of the array. And H_n = ln n + gamma + o(1).

thomasahle3• 1 year ago

I optimistically imagine a world where we replace the relatively non-specific asymptotic bounds of an algorithm with a very specific quantification of the work an algorithm will take.

I think you're getting this backwards. The (good) point you and Mitzenmacher are making is that our analysis standards are _too strict_, and hence we avoid good algorithms that are just too hard to analyze.

If we instead require exact combinatorics, e.g., using Flajolet/Sedgewick, that will be _even harder_ than if we _just_ try to do asymptotic analysis. Remember that asymptotics are a way to _approximate_ the exact number of steps. It's supposed to make it _easier_ for you.

antics6• 1 year ago

Oh I think there is no real doubt that this is true, it's why analytic combinatorics did not succeed in replacing asymptotic analysis. I just wish it had. If it was tractable in a similar set of areas I think a lot of things would be quite a bit better.

ww520• 1 year ago

Coincidentally I just did an asymptotic complexity analysis yesterday (something I rarely do). It seemed right since it's by the book but I wasn't sure. After running the extensive benchmarks on the algorithm, I was so happy to find the results matched the asymptotic prediction. The lesson is verify, verify, and always verify.

[1] Algorithm, https://github.com/williamw520/toposort/blob/master/Algorith...

[2] Benchmarks, https://github.com/williamw520/toposort/blob/master/README.m...

DevelopingElk• 1 year ago

In cryptography it is common to get exact estimates for the number of bit operations required to break a cipher. Asymptomatic analysis is great in some circumstances. It for example explains why quicksort is faster than bubble sort for large inputs. Beyond that you want some empirical results.

svat4• 1 year ago

In fact in these slides from a 2011/2012 talk, Sedgewick talks about exactly this, even going as far as to draw a contrast between "TCS" and "analysis of algorithms" (the "scientific method"): https://sedgewick.io/talks/#algorithms-for-the-masses = https://sedgewick.io/wp-content/uploads/2022/03/2011-12AlgsM... onwards.

(Also touched upon briefly in https://sedgewick.io/talks/#lasting-legacy-of-flajolet = https://sedgewick.io/wp-content/uploads/2022/03/2013-14Flajo...)

His point is exactly that, often researchers care only about worst-case analysis and asymptotics, which is not necessarily a useful scientific analysis of how long an algorithm is expected to take on a machine as a function of the input size (which was the original motivation for Knuth etc to develop the field).

The same authors (but ordered as Sedgewick and Flajolet!) also have a book called An Introduction to the Analysis of Algorithms; there's a course website at https://aofa.cs.princeton.edu/home/ and videos on Coursera (https://www.coursera.org/learn/analysis-of-algorithms) — it is advanced mathematics but less advanced than the Analytic Combinatorics book.

PollardsRho4• 1 year ago

Very cool!

What's meant by "it’s already too much to ask for a closed form for fibonacci numbers"? Binet's formula is usually called a closed form in my experience. Is "closed form" here supposed to mean "closed form we can evaluate without needing arbitrary-precision arithmetic"?

sfpotter• 1 year ago

Author probably just unaware of Binet's formula. But I'd agree with their assessment that there's unlikely to be a closed form for the (indeed, much more complicated!) quantity that they're interested in.

PollardsRho4• 1 year ago

They then say there's an approximation for Fibonacci, which makes me think that's what they're calling Binet's formula. (I'd also expect an author with this mathematical sophistication to be aware of Binet's formula, but maybe I'm projecting.)

LegionMammal9783• 1 year ago

In fact, for that 'warmup' problem, the leading term has a base and coefficient that are roots of cubic polynomials, given in the OEIS entry. But often the coefficient will be trancendental for these sorts of problems.

aleph_minus_one4• 1 year ago

Is "closed form" here supposed to mean "closed form we can evaluate without needing arbitrary-precision arithmetic"?

You don't need arbitrary-precision arithmetic to evaluate Binet's formula - big integer arithmetic suffices (simply do your calculations in the ring Z[X]/(X^2-X-1) ).

PollardsRho4• 1 year ago

At that point, you'd be better off just using a recursive algorithm like the one in GMP. You're swapping out arbitrary-length for arbitrary-precision.

paulpauper5• 1 year ago

It is closed form .the author makes so many mistakes here. All linear recusions are closed form by simply finding the roots of the characteristic equation. This is separate from the generating function, which the author confuses with the characteristic equation. A generating function is used when it's not possible to find a closed-form expression.

1minusp• 1 year ago

I find this interesting but the nomenclature escapes me: How is T(z) = z + zT + zT^2 + ... ? I find the jump from the functional programming description of the tree to this recurrence relation not intuitive

jstrieb• 1 year ago

Generating functions are definitely strange and unintuitive, but they can be really helpful for combinatorics.

The introduction to generating functions in this well known text on the subject (at the beginning of Chapter 1) might be more helpful than the Wikipedia article linked in the original post:

https://www2.math.upenn.edu/~wilf/gfology2.pdf

From that text:

Although giving a simple formula for the members of the sequence may be out of the question, we might be able to give a simple formula for the sum of a power series, whose coefficients are the sequence that we’re looking for.

qsort3• 1 year ago

I'm afraid the author is writing for an audience that's familiar with generating functions. It's using a common trick in analytic combinatorics, the notation is actually "shorthand".

Very informally, let T(z) be the generating function where the coefficient of the z^n term is the number of trees we are searching with exactly n nodes.

We know from the functional description that such a tree can be:

  - a leaf;
  - a node with 1 child;
  - a node with 2 children;
  - a node with 3 children.

A leaf is a tree with one node and that's it, so it "contributes" as a single tree of size 1.

How many trees of size n are there that fall in the second case? Exactly as many as the total trees of size (n-1), hence this case "contributes" as zT(z).

How many trees of size n fall in the third case? That's a bit harder to see, because now you have (n-1) nodes in total, split between the two branches. So you can have 1 in the left branch and n-2 in the right branch, 2 left n-3 right and so on. With a bit of basic combinatorics you can notice that this is exactly the convolution product of the series with itself, in analytical terms T^2(z). Hence the third case "contributes" for zT^2(z).

The fourth case is similar.

At this point we have constructed the inverse of the generating function we are looking for, i.e. z as a function of T(z) rather than the reverse, but luckily there's a standard trick for this: Lagrange inversion.

wasabi991011• 1 year ago

I had the same question as OP, passing knowledge of generating functions was not enough to see the link between the description and formula.

Your explanation was exactly what I needed, thanks.

paulpauper5• 1 year ago

He doesn't explain any of the notation, like how the brackets [z] work. Also the generating function he supplies is is wrong.

paulpauper5• 1 year ago

This article gets things wrong and overcomplicates other things.

he confuses the characteristic equation for the generating function. This : T(z) = z + zT + zT^2 + zT^3 is the characteristic equation.

There is no simple generating function ,as it's not possible to easily invert a cubic closed in form.

Successive derivatives of the generating function are taken to find the coefficients of the infinite series that encodes the sought values. It's hard enough explaining this with a quadratic; a cubic so much more involved.

This so far beyond the scope of a blog post. You need to spend at least a few weeks working on this or take some college courses.

svat4• 1 year ago

The article is correct. T(z) (in the warmup problem) is the generating function that counts the number of rooted ordered ternary trees, i.e. if we write T(z) = t_0 + t_1z + t_2z^2 + …, then t_n is the number of such trees with n nodes.

Yes there is no "simple" generating function, i.e. there's no simple expression for T(z). Nevertheless, T(z) (as further explained in another comment here in this thread) satisfies the equation T(z) = z + T(z) + zT(z)^2 + zT(z)^3, and this fact can be used to find the coefficients of T (i.e. the numbers t_n) via Lagrange inversion, and the code in the post does so and gets the same numbers as https://oeis.org/A036765.

(You use the term "characteristic equation", but that is a term only used in the context of linear recurrence relations AFAIK, and not applicable here.)

(Also IMO the post does not overcomplicate anything; what it presents is in fact the easiest way of solving the problem — that of finding the asymptotics of the number of unordered rooted ternary trees, up to isomorphism. If you think you have a simpler method even for counting small cases, feel free to post an answer at https://math.stackexchange.com/questions/5050941/counting-tr....)

paulpauper5• 1 year ago

thanks for clearing it up

thomasahle3• 1 year ago

There's an interesting extension to Analytic Combinatorics by Flajolet/Sedgewick called "Analytic Combinatorics in Several Variables" by Pemantle and Wilson: https://www2.math.upenn.edu/~pemantle/papers/ACSV.pdf and https://acsvproject.com/

It extends the original framework with a lot of useful new primitives, like Hadamard products. Super useful!

mrcode007• 1 year ago

Thanks!

3abiton• 1 year ago

While this interesting, what are its main applications?

thomasahle3• 1 year ago

I'm currently using it to analyze a version of the Kaczmarz algorithm, generalizing my answer here: https://mathoverflow.net/a/490506/5429

But more generally, why do we want to analyze combinatorics and algorithms? I suppose it gives us some confidence that we are actually making progress.

globnomulous2• 1 year ago

This article is technically way beyond me, but can anybody explain to me what on earth a "worked example" is? That expression is completely incoherent.

getnormality• 1 year ago

It's an idiom in math meaning "a problem, together with its solution, that illustrates a general principle or technique".

A worked example of the quadratic formula would be solving a particular equation by plugging in values of a, b, c into the formula.

A worked example of analytic combinatorics is solving a combinatorics problem with techniques from real/complex analysis.

globnomulous2• 1 year ago

Thanks, I get it now: I'm even more ignorant than I thought.

WhitneyLand2• 1 year ago

What math lets you most easily be convinced you’ve come up with the right answer when you actually haven’t?

Seems like combinatorics comes in second only to statistics.

svat4• 1 year ago

What is the relevance of this comment to this post? Are you insinuating that the author did not come up with the right answer? That's odd here because the answers are actually verified in multiple ways (Sage code, looking up the numbers on OEIS).

WhitneyLand2• 1 year ago

Not at all, the post seems solid. Just fond memories of a combinatorics class in school. I like problems that are deceptively simple. Easy to get lured in to, then satisfying as layers of complexity are peeled back.

DadBase• 1 year ago

The key is to always square the generating function—it doesn’t matter why, it just feels more correct. Learned that from a proof in an old cereal box puzzle.

vmchale• 1 year ago

in Haskell you can read off the generating function's coefficients directly from the functional equation T(z) = z + zT + zT² + zT²

ghci> ts=0:(1+ts+ts^2+ts^3) ghci> take 12 ts [0,1,1,2,5,13,36,104,309,939,2905,9118]

Using the Num instance for power series (very cute application of laziness expounded by Karczmarczuk and then McIlroy https://dl.acm.org/doi/abs/10.1017/s0956796899003299)

svat4• 1 year ago

This is cool! What does one need to do to get this to work? I get:

    $ ghci
    GHCi, version 9.12.2: https://www.haskell.org/ghc/  :? for help
    ghci> ts=0:(1+ts+ts^2+ts^3)
    ghci> take 12 ts
    <interactive>:2:1: error: [GHC-39999]
        • No instance for ‘Num [Integer]’ arising from a use of ‘it’
        • In the first argument of ‘print’, namely ‘it’
          In a stmt of an interactive GHCi command: print it

Edit: After reading Bentley's paper that you linked, figured it out:

    $ cat > bentley.hs
    ps0:: Num a => [a]
    ps0 = 0 : ps0

    -- (.*):: Num a => a->[a]->[a]
    c .* (f:fs) = c*f : c.*fs

    instance Num a => Num [a] where
        -- negate (f:fs) = (negate f) : (negate fs)
        (f:fs) + (g:gs) = f+g : fs+gs
        (f:fs) * (g:gs) = f*g : (f.*gs + fs*(g:gs))
        fromInteger c = fromInteger c : ps0

    ^D
    $ ghci bentley.hs
    GHCi, version 9.12.2: https://www.haskell.org/ghc/  :? for help
    [1 of 2] Compiling Main             ( bentley.hs, interpreted )
    bentley.hs:7:10: warning: [GHC-06201] [-Wmissing-methods]
        • No explicit implementation for
            ‘abs’, ‘signum’, and (either ‘negate’ or ‘-’)
        • In the instance declaration for ‘Num [a]’
    |
    7 | instance Num a => Num [a] where
    |          ^^^^^^^^^^^^^^^^

    Ok, one module loaded.
    ghci> ts=0:(1+ts+ts^2+ts^3)
    ghci> take 12 ts
    [0,1,1,2,5,13,36,104,309,939,2905,9118]

Analytic Combinatorics – A Worked Example

Related Stories