The Lindy Effect

https://arxiv.org/abs/2308.09045

by merename • 3 years ago

75 78 3 years ago

arxiv.org

scoofy• 3 years ago

“Tried and true”

coke12• 3 years ago

Sayings about how old things are surprisingly effective:

"Don't knock it till you try it" "Chesterton's fence" "Reinventing the wheel"

Any others?

lovecg• 3 years ago

“if it ain’t broke, don’t fix it”

“better the devil you know than the devil you don’t”

“old is gold”

“stood the test of time”

treerock• 3 years ago

Not sure it fits, but I like The Gods of Copy Book Headings

https://en.m.wikipedia.org/wiki/The_Gods_of_the_Copybook_Hea...

kawhah4• 3 years ago

The opposite of Chesterton's fence is "Chesterton's facehugger". Sometimes, when you discover something bad, the best strategy is to immediately try to remove it as quickly as possible, before spending any energy trying to find out why it appeared.

atleastoptimal• 3 years ago

There's a subset of this I think about sometimes, not sure if it's actually a thing or been looked at/is valid.

Let's say I'm waiting for a package to arrive and it's late. As each minute passes, does the chance of it coming the very next minute increase or decrease?

One could say it increases because since it will eventually come and I have passed all the previous minutes they haven't come, then the remaining minutes until it comes comprise a smaller percentage of the all the remaining minutes, increasing the chance each minute.

However up to a certain point the chance may decrease, as in, if it is "late" in a substantial sense, whatever caused it being something that delayed it a significant amount of time increases as time elapses, as the chances of it being something small that caused the lateness decreases as each minute passes.

kqr• 3 years ago

And you can model this through two survival functions, one thin-tailed and one heavy-tailed. As time increases, whatever your initial probability of "package will be very late" it will be dwarfed by the posterior after evidence of a long waiting time.

It's the exact same technique used in https://two-wrongs.com/forecasting-covid-19-variants.html except applied to "packages that may be late" instead.

codetrotter• 3 years ago

it will eventually come

It might also never come.

Sometimes packages just disappear in transit.

Maybe it lost its label.

Maybe it got broken.

Maybe someone stole it.

Etc.

My experience with packages goes like:

- Most often things I order in my own country or from a neighbouring country arrive exactly on the expected day, or at least the next business day after that. This makes sense because shorter travel has less opportunity for things to go wrong.

- Things from far away have their estimate set for like 60 days from now, because no one knows what boat it will end up etc. Sometimes these things show up quickly. Most of the time it takes a while, but still it arrives before the long estimate date. Sometimes they show up a bit late.

- If they are very delayed then in both cases it means the package is just gone and will never arrive. So I inform the platform that the package was lost in transit and they refund the purchase.

mlsu• 3 years ago

This is a sort of transformation of the bathtub curve.

3np• 3 years ago

Replace it with waiting for a bus and you have a classic textbook example for teaching Poisson process!

https://en.wikipedia.org/wiki/Poisson_point_process

https://stats.stackexchange.com/questions/122722/please-expl...

dmurray• 3 years ago

It's specifically not a Poisson process, which assumes that the chance of a bus arriving at any given time is constant.

A bus arrival has some kind of multimodal distribution, where the bus either comes on time or very late, with some noise in each case.

DantesKite• 3 years ago

One implication is that even things which are becoming less robust over time can display the Lindy effect.

Interesting. And somewhat counterintuitive.

riffic• 3 years ago

https://www.google.com/search?q=lindy+site:https://news.ycom...

marttt5• 3 years ago

I like to use the Lindy effect as an everyday heuristic, quite often for choosing what books to read. That is: if a book has been highly relevant, say, for the past 20 years, it may have considerable relevance 20 years from today as well. Conclusion: I will think about buying that physical book for my home library, so my children would have a carefully-filtered collection of "classics" from many eras by the time they reach young adulthood.

As a result, I almost never buy (or often even read) current-day bestsellers. Usually, one is not in a hurry with reading a book; also, new books can be really well marketed these days, which always makes me cautious. It seems to make more sense to read a book 5-10 years after it was first released.

The Lindy effect does seem to hold true when I think about e.g. SICP, K&R's C book -- or Kernighan et al's 1988 AWK book for which the authors release a second edition in October this year! [1]

Not to mention classic books in fiction or philosophy. Or cultural memes and ideas -- as a rough heuristic, the Lindy effect does often seem to hold true with these as well. It is a fun and extremely useful thinking tool for longer-term everyday prognostics.

1: https://www.awk.dev/

kjellsbells• 3 years ago

There's an implicit value judgment there though: would you stick a book that exhibited the Lindy effect even if you fundamentally disapproved of it.

For example: the King James English Bible is surely very high on the Lindyness scale, massively influential, and on that basis deserves a place on every educated reader's library: but its also the source of bigotry, suffering and death for thousands of people. Do you stock it, or no?

j7ake• 3 years ago

The bible is a hugely influential book. It is probably the most referenced book in western literature, art, and history.

Reading a book is not the same as agreeing with its premises.

One must also judge a book by their historical context.

marttt5• 3 years ago

I definitely would stock the Bible, yes -- with the argument that Western societies and cultures are riddled with biblical metaphors and everyday customs, so it does make sense for the kids to be familiar with them.

Not sure about stocking e.g. Mein Kampf, though. This one I would probably pass, due to disgust towards what the Nazis did. From another angle, I might be "brainwashed" here -- by the cultural meme that this is a Dangerous Book. I don't really know. In the end, it is still just a book. Can a book be dangerous in itself? Not sure that possessing it would turn somebody into a Nazi; a youngster's appreciation of those ideas is surely more complicated a mechanism than just reading a book.

Tangentially related: in my (post-soviet) country, there has been discussion whether former Soviet military buildings are a part of cultural heritage, or should they all be demolished. As a regular person, I'm definitely siding with keeping some of those buildings -- the Soviet occupation was a sad, devastating part of our past, and younger generations should have physical examples of what it was like. Whether those buildings will be used as some kind of cult objects etc -- this is a matter of values education at school or at home.

So I fully agree with @j7ake's comment in this thread that reading (or -- my addition -- possessing) a book is not the same as agreeing with its premises. Hiding the past won't work, I think.

Great angle @kjellsbells, though. Interesting to think about. Thanks for bringing this up.

kimi• 3 years ago

Yes definitely. Currently in four languages, for the record.

lemper• 3 years ago

Yes. I'd stock Koran too. Satanic Verses by Rushdie would also be included. If I can have access to Mein Kampf, to the library it goes.

vasco• 3 years ago

What someone says is the justification for their actions, and what actually is the reason for their actions are often different things.

flohofwoe2• 3 years ago

TBF that's less because of the actual content of the bible, but what people in various eras read into it (except for some of the Old Testament "eye for an eye" stuff of course).

PS: ...and even though I'm agnostic, of course I have the bible in my book shelf and occasionally read it, if only for the entertainment value (not the King James version though, but the Luther version).

bazoom42• 3 years ago

its also the source of bigotry, suffering and death for thousands of people

But are you afraid reading it will turn you into a murderous bigot?

DontchaKnowit2• 3 years ago

Really? The bible is the source of bigotry and death for thousands? A vit dramatic no? The bible does not implore anyone to violence. But it is a convenient scapegoat for those with ill will

jrflowers• 3 years ago

As a result, I almost never buy (or often even read) current-day bestsellers.

I like this strategy because it maximizes the value you get out of old books while you minimize the value of new books. A phenomenal and important book could be published today but the optimal strategy is to die before the Lindy effect becomes apparent and let another generation enjoy its value. This makes sense because books have a finite value that is divisible by the number of readers they get.

zogrodea2• 3 years ago

Why do you think books have a finite value divisible by the number of readers they get? That sounds like a very odd claim to me.

Let's say there are two universes U1 and U2 and a book A which is identical in both. In U1, book A gets 100 readers, while the same book in U2 gets 1000 readers. I think most people would say that the book has the same value in both universes, independent of how many read it.

Maybe your thought process involves the physical condition of a book (there is more wear and tear as it passes through more readers), but I'd like to understand your point of view.

RamblingCTO7• 3 years ago

You overfit for classics and skip on maybe very entertaining contemporary reads. Feels weird to me to hipster out on the "I build a library of classics only". With this approach you'd never read Dark Matter, which is one of my favorites in the past few years. I also don't care about rankings, top sellers etc. etc., but contemporary stuff is still worth the read. Feels very weird to me tbh.

marttt5• 3 years ago

Not considering myself a hipster. I'm closely following new releases on several fields (including experimental fiction and contemporary philosophy), but I like to wait -- typically some years -- before I really read them.

Yodel0914• 3 years ago

You're not wrong, which is why for me longevity is only one of the factors that goes into my book select - but it is a big one.

The thing is, we have so much brilliant art and literature available that you need to filter it somehow. The likelihood that something released this year is better than something that has been heralded and great for 100 years is very, very low. Not impossible, mind you. I've definitely read modern books that have had a huge impact on me (including, ironically, Taleb, who introduced me to the Lindy effect), and art and music have a certain cultural context that can make modern expressions more relevant.

gizmo• 3 years ago

Also, we are too kind to some classics out of nostalgia perhaps. I've been greatly disappointed by top rated classic spy and sci-fi novels. The kind of storytelling people got away with 50 years ago really doesn't work anymore today. Think of bad pacing, characters without depth, lazy stereotyping, deus ex machinas.

So on the one hand recency bias results in us giving undue attention to new books. On the other hand we are too forgiving to classics. There is no lesson here. Finding the right books to read is hard and you can't trust reviews.

jackothy• 3 years ago

The fact that you mention Dark Matter (no hate, taste is subjective etc.) as potentially rivaling actual classics is for me just further proof that fighting recency bias and hype is important.

But yes, of course some modern books will end up standing the test of time as well. But it's hard to judge that a priori without actually reading the book yourself.

You will spend a finite amount of time reading in your life. If you read Dark Matter now, that could for example mean that you will never end up reading Dune, or The Count of Monte-Cristo. If you're gonna sit down and read, an older book is a much safer bet if you want to maximize the value of your limited time.

Still, modern books can sometimes offer other qualities that simply don't exist in older books. Sci-Fi is a great example of that, where scienctific developments affect the writing deeply. So (unfortunately) in certain cases you might have to gamble on a newer book. But I try to avoid that.

somedude895• 3 years ago

Same with films and videogames. With books and films it's probably more the current Zeitgeist that doesn't agree with me, so if something's hyped up I tend to think it's probably due to the current social activism (which I believe won't age well) permeating everything. There is more than enough media from the past hundred years that has stood the test of time.

kwhitefoot• 3 years ago

so my children would have a carefully-filtered collection of "classics" from many eras by the time they reach young adulthood.

I have that. My three adult children read none of them.

golol3• 3 years ago

It seems like the Lindy effect for a positive real number valued ramdom variable X, is the following identity

E[X | X >= a] = 2 * a

nonrandomstring• 3 years ago

It's more likely described by the non-linear portion corresponding to the first stage of the Bathtub Curve [0] we use in engineering.

[0] https://en.wikipedia.org/wiki/Bathtub_curve

Most complex but faulted things fail very early.

sieste2• 3 years ago

E[X | X >= a] = 2 * a

Is it possible to derive the maximum entropy probability distribution subject to that constraint?

golol3• 3 years ago

Let f: R -> R be the probability density function of X. It fulfills int f(x) dx = 1 and we furthermore assume that f is differentiable.

We can write the identity as

    int_a^oo x f(x) dx / int_a^oo f(x) dx = 2 a.

We rewrite this as

    int_a^oo x f(x) dx = 2 a int_a^oo f(x) dx.

Taking the derivative with respect to a on both sides yields

    - a f(a) = 2 int_a^oo f(x) dx + 2 a (- f(a)).

Taking another derivative yields

    - f(a) - a f'(a) = - 2 f(a) - 2 f(a) - 2 a f'(a).

We rewrite this as

    f'(a) = - 3 f(a) / a,

and furthermore

    f'(a) / f(a) = - 3 / a.

We can write this as

    (ln ○ f)'(a) = - 3 / a.

As far as I can tell, we can not find a solution on all of [0, oo). Let's assume that failure is impossible up to an early time t. This means that we assume f(x) = 0 for all x < t, and that possibly f(t) > 0, meaning we have a jump discontinuity in the probability density at x = t.

We integrate from t to x

    int_t^x (ln ○ f)'(a) da = int_t^x - 3 / a da.

This evaluates to

    ln(f(x)) = ln(f(t)) - int_t^x 3 / a da
             = ln(f(t)) - 3 ln(x) + 3 ln(t)
             = ln(f(t)) - ln(x^3) + ln(t^3)
             = ln(f(t) t^3 / x^3).

In particular

    f(x) = f(t) t^3 / x^3.

Right now f(t) is still a free parameter, but recall that

    1 = int f(x) dx
      = int_t^oo f(t) t^3 / x^3 dx
      = f(t) t^3 [- 1 / (2 oo^2) + 1 / (2 t^2)]
      = f(t) t / 2

has to hold, so we find that

    f(t) = 2 / t

and

    f(x) = 2 t^2 / x^3.

So these are the probability densities that fulfill the "modified" Lindy effect, where failure is impossible before time t > 0.

I haven't double checked this so maybe there is a mistake, but I'm quite convinced that this line of reasoning leads to a unique distribution for each t > 0 and impossibility when t = 0.

stakhanov2• 3 years ago

Here is one way, I use the Lindy effect as a mental model:

When picking a tech stack for a software investment and you want a reasonable expectation that it will still be actively maintained 10 years from now, you need to go back in time by 10 years. I’ve found this 10 years time horizon a useful one to work with, based on version histories of various bits of software I build upon. For example, for Python, this takes me back to version 3.4, before type decorators and many other things hit the scene that I disapprove of. So now, I test all my code with both version 3.4 and the newest version (3.11). I won’t use language features from 3.11 that weren’t already there in 3.4, and I won’t use code in 3.4 that breaks or throws deprecation warnings in 3.11. I also apply this test to dependencies, paying close attention to what happens if I try to get the newest version of some Python library running on version 3.4 of the interpreter or a 10-year-old version of the library on version 3.11 of the interpreter.

This means my code is engineered so that it could have been in continuous operation for the last 10 years while running continuous updates on what’s underneath it. And this gives me a reasonable expectation that my code will require only minimal code changes over the next 10 years to keep up with whatever might arise.

With the python interpreter itself, it’s remarkably easy to do that. I feel it doesn’t limit my coding in any meaningful way. With other bits of software, including many python libraries that one might depend on, it would be absolutely unworkable. In such a case, I take that as a clear signal not to use the dependency at all. This is a lot of work, but also a good forcing function that prevents me from becoming a dependency hog myself.

emblaegh2• 3 years ago

Sounds like there would be barely any up to date dependency that you could use, which in some cases could have known security issues, no?

stakhanov2• 3 years ago

To clarify: I still run the latest version of all the packages I depend on. For example, the newest version of SSL that you can still use with Python 3.4 has known issues. But I don't use that in production. I use Python 3.11 with an up-to-date SSL, but the code that I've written myself is code that I've also tested on Python 3.4 with outdated SSL. So the subset of features I use is a set of features considered by Python and SSL maintainers as so essential that they've maintained compatibility around them for ten years, and are therefore likely to maintain compatibility around for another ten years. -- Whether that's true no one can know, but applying the Lindy effect mental model would imply that this would be a useful way to think about it.

The packages I get to use after applying the above filter are a minority, but, surprisingly, it's not "barely any" either. If you decide to live like this, you'll have less eyecandy in your life and less capability to capitalize on all the latest hotness all the time. But you'll get the job done, pretty much regardless of what the job might be.

Your dependency hog colleagues will call you a dinosaur for your choice of tech stack. But when security vulnerabilities happen then, more often than not, there won't be any action needed on your part because, more often than not, you'll discover that you don't depend on the thing that has the vulnerability in the first place. You'll be home for dinner, while those same colleagues try to patch their dependencies, watch their software break, and try to work their way out of dependency hell (presumably by introducing more dependencies).

vishnugupta2• 3 years ago

I really wish that we, software developers, apply Lindy rule while choosing library/framework/language/SaaS tool etc., Lot of developers underestimate the value of collective brain power that has gone into making a system mature, not to talk about community and ecosystem support. I was extremely hesitant to get onboard Golang bandwagon 5-7 years ago. I reluctantly said OK to try it out in a non critical service and all my concerns turned out to be true.

I also apply this rule while buying car and choosing a bank. It's been interesting to see all the fintechs who badmouthed incumbent banks now going through exactly same challenges that banks went through and addressed.

lexoj• 3 years ago

Care to elaborate on the Go issues you encountered? Nowadays Go is very well established, unless you specifically meant that the issues happened 5-7 years ago.

vishnugupta2• 3 years ago

Of the top of my head:

1) Lack of libraries.

2) Little to non-existent tooling & observability support.

3) Lots of challenges around gRPC. That it needed a discovery service such as etcd was a big source of pain. I remember two or three massive downtimes where an adjacent team went through hell for a day just to get back a high throughput system back online. gRPC + etcd was very susceptible to thundering herd problem.

4) Lack of experienced developers who would guide others idiomatic way of doing things. And related, go routines need a different mindset and somewhat a grounds up re-learning. The code would routinely get into deadlock sort of scenarios.

I conceded that most of the above are not inherent to Go but to any new system/language. I'm sure they all have been addressed now as the language and ecosystem has matured. That was the point I was trying to make though; don't jump onto a new bandwagon. Of course you want to learn it on your own and experiment that's totally fine but to use it in production is fraught with risk IMO.

hyperthesis3• 3 years ago

java will continue for 28 years; c much longer; and when both are long gone, lisp and fortran will still be here.

bmacho• 3 years ago

So something whose lifespan is exponentially distributed (such as a radioactive particle) does not display the Lindy effect.

Huh? Is this true? If I had particles with unknown half-times, I think I would expect the older particles to stay around longer, if that makes sense. (It is possible that there is no 'I expect' in the probability theory sense, like in the two envelopes problem.)

mikewarot• 3 years ago

Clearly, my choice to program in Free Pascal/Lazarus, using the old GUI toolkits is vindicated! ;-)

In the world of machining, 1940s-1960s American cast iron machine tools (Mills, Lathes, etc) are the best available. Grind / Scrape the bearing surfaces true, update the control systems, and they'll be good to go for at least 2 more generations.

The Lindy Effect

Related Stories