About Linkrot: Proportion of Working Links on Pinboard.in, 1997-2011

http://blog.pinboard.in/2011/05/remembrance_of_links_past/

by chl • 15 years ago

118 34 15 years ago

blog.pinboard.in

patio11• 15 years ago

There is an easy way to increase your detection of redirects and parked pages: make two requests, one to the real URL and one to a URL which is intentionally broken. (example.com/i-am-a-link and example.com/fklsdfasdifo for example) Run a heuristic for difference on the resulting content. This won't catch all of them, particularly if you use a really naive heuristic that can't deal with e.g. ads changing, but it's a heck of a lot quicker than comparing manually.

hcho• 15 years ago

I suspect that's what Google does too. I see lots of example.com/fklsdfasdifo type requests from Big G in my logs.

bauchidgw• 15 years ago

if you see a lot of them go to webmaster tools, if you see them there too its not some kind of test but some other reasone, mostly their shitty js parsinf, which treats anything with an / as a relative url...

benhoyt• 15 years ago

Yeah, that's a good method to detect "soft 404s". I wrote a Python library for DecentURL to do this some time ago: http://blog.brush.co.nz/2008/01/soft404s/ (Ironically, the original paper I got the idea from seems to be a dead link: http://www.tomkinshome.com/andrew/papers/decay/final/p444-ba...)

ible• 15 years ago

I had a project/startup working on dealing with link rot for a while. It would not just tell you a link was 404ing, but recognize when the content of the page had changed significantly and let you know. The fun part was to automatically recommend a good replacement page from within the site, nearby, or the internet archive.

Based on a quick test it seemed that it would take an site owner about 5-10 minutes per link to find a good replacement once they knew it was broken . That's fine for a personal portfolio with 50 links, but for a site like Boing Boing getting all the broken links working again looked like full time work for a year.

I'm curious if you have many outbound links in your 'scalable content'? Do you spend much time maintaining them?

true_religion• 15 years ago

I just want to point out that this won't really work if the site tries to redirect you to a search engine. Naturally broken links will come up with all kinds of search results, but deliberately broken links will come up with "None".

bmm6o3• 15 years ago

"Links appear to die at a steady rate (they don't have a half life), and you can expect to lose about a quarter of them every seven years."

Is this self-contradictory, or is just poor wording of his findings?

yahelc• 15 years ago

Half life isn't a steady rate; it's exponential gradual decay. For example, if the element yahelium had a half-life of a year, it would degrade by 1/2 every year. So, the degradation during the first year would be twice the degradation of the second year. That's not linear.

pyninja2• 15 years ago

What's not clear about it? They die at a steady rate of 25% per 7 years.

bahadden• 15 years ago

So they have a quarter-life of 7 years?

bmm6o3• 15 years ago

As reemrevnivek explains, the answer to "25% of what?" is the issue here. An exponential decay can be described with the same words.

reemrevnivek• 15 years ago

"You can expect to lose about a quarter of them every seven years" can be translated in one of two ways:

1. Every 7 years, 25% of the original links are lost: in 14 years half are working, and in 28 years you have approximately 0% of your links working. This is a steady, linear rate which does not have a half life.

2. Every 7 years, 25% of the remaining links are lost: in 7 years 75% are working, in 14 years 0.75^2 = 56% of the links are still working, and in 28 years 31.6% are working. This is exponential decay, which has a half-life and is contradictory.

From looking at the data, and from the explicitness with which he states that the decay is linear, I'd conclude that the former is true. It would be hard to make a case for the later.

fhars2• 15 years ago

Given that the data for the first few years result from a small number of pages, the graph doesn't allow to reliably distinguish between both interpretations, and his (and your) interpretation is not really justified. If you look just at the complete, reliable years (links from 2000 to 2010), the graph looks convex, just like you'd expect for exponential decay but not for linear decay. So while the first interpratation is probably intended by the author, I'd say he is overstating his case and is most probably wrong.

ugh• 15 years ago

I think it's about time that some government or billionaire throws a few millions at an internet archive project. The Internet Archive is nice but more regular snapshots with a wider coverage would be something I'm certain future historians would love to get their hands on (and they will hate us if we don't do it).

tokenadult• 15 years ago

I think it's about time that some government or billionaire throws a few millions at an internet archive project.

It may be that one or two governments have already done that. You are, of course, referring to a publicly accessible Internet archive.

As for what a benevolent millionaire (it wouldn't have to be a full billionaire for this to start up) could fund, pg has suggested, "There is room to do to Wikipedia what Wikipedia did to Britannica."

http://ycombinator.com/ideas.html

It's interesting that pg thought then that Wikipedia's problem is excessive deletionism, while I (after being a registered Wikipedian and working on various articles) think that Wikipedia's problem is lack of thorough research to prepare article content.

http://strategy.wikimedia.org/wiki/Wikimedia_Movement_Strate...

Whatever one's opinion of what's wrong with Wikipedia, the best way to prompt improvement in Wikipedia (or replace it, if you prefer that) is to build another site that does some of what Wikipedia does but does it better somehow. That's not easy, not easy at all, but it's not terribly expensive. I have looked at the Wikimedia Foundation financial reports, and building a strong competitor to Wikipedia is a project that is well within the grasp of several individual millionaires, and within the grasp of quite a few nonprofit charitable organizations. A business corporation that can find out a way to monetize a Wikipedia competitor might have a great business opportunity.

tmarthal• 15 years ago

Fitting, in regards to the original pinboard.in story, is that in the CRM section of the http://ycombinator.com/ideas.html (number 7.) links to an article http://articles.latimes.com/2008/may/29/business/fiw-octopar... that has rotted and is now a link to a generic not-found(w/HTTP 200) latimes.com page.

neilk• 15 years ago

> It's interesting that pg thought then that Wikipedia's problem is excessive deletionism, while I (after being a registered Wikipedian and working on various articles) think that Wikipedia's problem is lack of thorough research to prepare article content.

Yep, that's the usual distinction. Non-Wikipedians believe that Wikipedia should be a compendium of any information that could be useful, however unverifiable or incomplete. Wikipedians want there to be higher standards, but pg makes the common mistake of thinking this is because they are all OCD.[1] The Wikipedia system relies on group verifiability. Low quality info imposes long-term costs on the administrators. The article will be flagged more frequently than others. So pruning low quality info is a matter of administrator self-defence, even if you ignore the ideals of achieving a trustworthy encyclopedia.

A Wikipedia successor would have to abandon trustworthiness (or figure out some way to indicate that certain pages were untrustworthy). Or figure out how not to impose the costs of maintaining unverifiable information on administrators. One way might be to connect the info with the community that cares about it in a more direct and intimate way. Wikipedia fails REALLY badly at the latter, to the point where the wiki-insiders sometimes have more control than the audience for a topic.

> That's not easy, not easy at all, but it's not terribly expensive.

In terms of software and services, it would be no problem at all. But you are overlooking the cost of creating a new Wikipedia in a world where Wikipedia already exists.

Wikipedia content is also famously intractable to reuse in any system other than MediaWiki. We hope to begin alleviating that this year with the big parser redesign. A side effect should be to enable competitors to try different things with our content.

[1] They are, but this is not the primary reason. ;)

idlewords4• 15 years ago

A big problem with the Internet Archive is how easy it is to opt out of it, either by robots.txt or explicit request. I understand why they do it, but it seriously damages their mission.

_delirium• 15 years ago

The fact that you can retroactively opt out of it also makes it particularly odd, compared to previous media. If you change your mind about publishing a book, you can't unpublish existing copies, and chances are that some library will still have a copy that they continue to make available to the public (a handful of people in history have succeeded in suppressing already-published books, but it's hard). But if you decide you want to unpublish a webpage, you can actually get it excised from all the archives available to the public as well, erasing it from the historical record, even if it had been available for years and read by thousands--- because, unlike with books, typically none of those thousands of readers will have a copy that they can claim ownership to.

sbierwagen• 15 years ago

And yet HN explodes with hate whenever Archive Team does something.

GoodIntentions• 15 years ago

AT telling the world they will rip any content they want regardless of owner's wishes, and then trolling about it probably soured most of us on your 'team'.

Have you considered that if you're despised by many, it might be for a reason?

sp332• 15 years ago

There's a big difference between the way the Archive Team operates and the way the Internet Archive operates. IA is professional and polite, AT is brutish.

gojomo2• 15 years ago

Users may prune their own bookmarks when they discover the links broken – especially when considering some of the pre-Pinboard systems (like in-browser bookmarking) from which the earliest data in this analysis comes. So I suspect this underestimates link-rot.

idlewords4• 15 years ago

Just like it says in the article!

gojomo2• 15 years ago

D'oh! Overlooked that in my quick-read (or seeing as how I chose the same 'prune' word, only noticed it subconsciously).

Still an important enough point to pull out for highlight here; I should have used a direct quote.

InclinedPlane• 15 years ago

Sadly I think technological advances has only accelerated this phenomenon. We've gone from an era of static pages that would require considerable effort to change the overall layout of to CMSes that we can twiddle and upgrade with nary a concern for backward link compatibility.

Personally I think it should be a principle of every professional web developer that you just don't break links, period.

About Linkrot: Proportion of Working Links on Pinboard.in, 1997-2011

Related Stories