There’s a simple alternative to the current web

http://hapgood.us/2014/08/14/the-web-is-broken-and-we-should-fix-it/

by mgunes • 12 years ago

150 143 12 years ago

Y U no archive.org?

pbhjpbhj• 12 years ago

Archive.org doesn't have everything and their archives can be cleared of specific content on request of the copyright holder.

There is a big copyright issue here as in the UK we don't have the relatively liberal Fair Use exceptions that are in the USC - we only just got permission to format shift and make personal-use backups (so MP3 players [using tracks ripped from CD] are legal as of June this year!). Copying down a website, beyond caching, is generally speaking copyright infringement for those in the UK.

CyberShadow• 12 years ago

their archives can be cleared of specific content on request of the copyright holder.

Not only that. If a domain expires and is picked up by a squatter, the squatter can instruct Archive.org to delete ALL copies of content archived from that domain. Unfortunately many do so.

dilap• 12 years ago

"The Tyranny of Print" has a nice ring to it, but mediums that give the creator more control over appearance+behavior are going to lend themselves to crafting more compelling experiences.

Sure, not disappearing in 10 years (or whenever the original server goes poof) would also be nice, but it's of little benefit if no one ever sees the thing in the first place.

And disappearing is the default, natural state of things.

If I see some people playing music on a corner and return the next night to see they've left, I may be wistful, but it would be silly to argue "playing live music is broken and we should fix it".

If you think of web sites as performances put on for a limited time by the server, it doesn't seem so terrible that they disappear after a while.

seiji2• 12 years ago

The mental model of web browsing and bookmarking is: "If I see it, I can get to it again." There's a partial feeling of ownership. "I've read it, so I should be able to refer back to it later."

Nobody (at least, no sane regular person) reads a webpage and thinks "I have a time-limited license from the originator of this content to consume the material and only use it for their expressly condoned purposes."

The vanishing content problem is like if books in your house randomly walked away just because it's the "natural state" of things to disappear.

idlewords17• 12 years ago

There should be room for a spectrum of stuff online, from evanescent to permanent. It's one thing for the musicians on that corner to be gone the next night, but you do expect the corner to still be there. Online, it's depressingly common for even large bits of infrastructure (like GeoCities) to just go poof.

mgunes• 12 years ago

And disappearing is the default, natural state of things.

Books, clay tablets, scrolls, engraved stone, to which humans owe their entire knowledge of their premodern history, seem to have put up pretty well against entropy. The same is not the case for information disseminated in a controlled manner from privately owned servers.

If I see some people playing music on a corner and return the next night to see they've left, I may be wistful, but it would be silly to argue "playing live music is broken and we should fix it".
If you think of web sites as performances put on for a limited time by the server, it doesn't seem so terrible that they disappear after a while.

Thankfully, the generations who produced and preserved knowledge on paper, clay and stone before the onset of digital technology - that is, every generation of humans that has ever lived, except ours - did not think of books and libraries as throwaway pamphlets. And it would take more than an arbitrary interchange of modes of cultural production to argue that we should be doing otherwise in the technological circumstance we find ourselves in.

The "tyrants of the server" are not thinking of server-centric aggregation and dissemination of as a performance put on for a limited time: they are betting on it as the future of all human literary activity. Google doesn't want to read you a paragraph, take your money and say goodbye; it wants to swallow all the world's books and information, chop it to tiny pieces, store and own it forever, and extract the maximum profit from each tiny piece, without having you pay a penny. And it wants you to come back for more. The persistence of the server-centric model of content dissemination is not an accident; it is dictated by the political economy of the web brought about by the Googles of the world.

al2o3cr• 12 years ago

"Books, clay tablets, scrolls, engraved stone, to which humans owe their entire knowledge of their premodern history, seem to have put up pretty well against entropy."

Only the ones that have survived. For every book or tablet we have, there are certainly tens of THOUSANDS of which every copy ever published has been lost - most of those are ephemera that wouldn't mean much to us anyways, but the lost also include things that would be nice: the majority of Livy, any of the original source material for the Gospels, etc.

Even considerably more modern material has been vanishing at a significant rate; for instance, most of the output of the silent film era has already been lost.

AnthonyMouse• 12 years ago

mediums that give the creator more control over appearance+behavior are going to lend themselves to crafting more compelling experiences.

There is a difference between the creator and the server. Most of the content you consume is created by people who don't own the servers. Separating appearance+behavior from content source would help actual creators because they wouldn't have to worry about their host deciding one day to delete all their content because the service is being discontinued or the creator is competing with some business interest of the host.

If you think of web sites as performances put on for a limited time by the server, it doesn't seem so terrible that they disappear after a while.

The problem is that the web is being used for everything, even things that can and should work like books rather than like live performances.

BerislavLopac• 12 years ago

If I see some people playing music on a corner and return the next night to see they've left, I may be wistful, but it would be silly to argue "playing live music is broken and we should fix it".

Actually, that's exactly what the inventors of (various) recording machines did. Something might have disappeared in one form, only to return in another. Just ask the Project Gutenberg people.

nb13• 12 years ago

This wouldn't work for any web page that has dynamic content stored in a database. If the database no longer exists a decade from now this doesn't solve that problem.

Also, wouldn't this break analytics and reporting for most websites too? It'll be much tougher to track user behavior to improve user experience. And debugging using log data? I get what the author is suggesting but "fixing the web" this way would break more things that large websites and companies rely on.

seiji2• 12 years ago

It'll be much tougher to track user behavior to improve user experience.

That sounds tragically Orwellian. "We do [BAD THING] to help you! [and it also, just by chance, makes us more money too by exploiting you, but never you mind that]"

break more things that large websites and companies rely on.

The historical "spy everywhere" privacy model of the web isn't a natural state anybody has a right to exploit. Breaking the current centralized curation model would be a benefit for everyone.

idlewords17• 12 years ago

Well, as the Internet once said to the music companies, it's not our fault if our new technology breaks your business model. People would find new ways to solve these problems.

corin_• 12 years ago

It's fine to be that stubborn if you can win. But against the world of business that needs analytics... I suspect their business model trumps your technology for now.

larrys• 12 years ago

"Also, wouldn't this break analytics and reporting for most websites too?"

True but that's a "YP" not a "MP". (Bogie Nights, "Your problem" not "My Problem".

I mean we're not talking about a public health issue after all.

idlewords17• 12 years ago

Clearly I'm a biased observer, but I really think people should take steps to archive stuff that is important to them. Of course it's terrible when large sites go offline and take vast swaths of the Internet with them, and we should continue to shame the ones that do it. At the same time, if something is really important to you, you shouldn't store it in the form of links to random third-party servers.

One problem we need to solve as coders is giving people better tools for saving stuff. It's really hard right now to save a webpage (or worse, series of connected pages) with any confidence that you've captured everything you need to see it again if the original server disappears.

A project that I think has struck a really good balance between permanence and retaining authors' control over their writing is the Archive of Our Own (AO3). A bunch of fanfic authors got tired of sites falling out from under them, and decided to implement their own system, along with sensible governance and a way to fund its ongoing operations. The only broken links I've ever seen to AO3 are ones where the author consciously decided to take the material offline.

rando289• 12 years ago

It always seemed like such a regression to me when browsers disabled entire page caches and only ever show the freshest content. Before, if a page served 404, I could easily see my local cached version from my previous visit. Now, I'm shit out of luck.

walterbell4• 12 years ago

What is amazing is that:

1) Never in the history of civilization has local knowledge storage (disk) and local compute been so cheap

2) Never have we had a larger free software ecosystem or more hackers to deploy free software locally

3) Never have we had more evidence about the differences in civilizational freedom between local and central storage/compute

What if backup/restore skills were taught alongside home economics? Is a home a place of both shelter & storage? Are we abandoning a set of possible futures because we want the "convenience" of "someone else" backing up our digital selves/souls?

"For want of a nail the shoe was lost.

For want of a shoe the horse was lost.

For want of a horse the rider was lost.

For want of a rider the battle was lost.

For want of a battle the kingdom was lost.

And all for the want of a horseshoe nail."

MetaCosm2• 12 years ago

Clearly I'm a biased observer, but I really think people should take steps to archive stuff that is important to them.

I recently converted all my bookmarks to saved copies of the pages in Evernote. This means I have full text search over everything I "bookmark" -- still have the links -- and never lose it regardless of what happens.

In this process, I was shocked at the number of 404s I encountered from relatively recent (last 18 months) bookmarks.

Evernote is a bit of a wreck, but so far the combination of decent copy of page, browser plugins, tags, search, accessibility (phone clients, web client, some native clients) have made it my best option.

Springtime• 12 years ago

Since 2008 or so I've been saving all pages that I would otherwise be bookmarking. Only in MHT format, but it's convenient. After reading a few quotes on linkrot years ago, and noticing many pages I'd come back to 404'ing I decided bookmarks are useless long term (or even at times short term).

walterbell4• 12 years ago

Is this a feature built into Evernote? If not, what tools did you use for the conversion?

oridecon• 12 years ago

Luckly if Stack Overflow goes down the one billion shady scrapper bots will have (at least) the most viewed questions.

jaimebuelta• 12 years ago

This is the only real solution (even if it's through an intermediate agent, which may have the same problem. Remember Google Reader). Of course, then the problem to keep that info in a good way (indexable, accessible, backed up, etc...) is another interesting problem. I'm afraid that most of the people are simply not aware of all of this (both on the content supplier or the content consumer)

This is also somehow of a problem on printed books. There are tons of books that are out of print and cannot be easily found.

DanBC• 12 years ago

Many people just don't have the clue to make usable local archives.

You're also proposing mass copyright infringement which - stupidly in this example - is not legal.

sillysaurus312• 12 years ago

Is saving a webpage to your local harddrive copyright infringement? The data is already on your local system when you view a webpage.

anon4• 12 years ago

If the system itself is federated the same way as the rest of the data, then it doesn't matter if it's legal or not.

You can't make an omelette without breaking a few eggs and you can't fix the world without breaking a few laws.

grkvlt3• 12 years ago

I don't think copyright infringement is the issue. Making a copy to view offline is always legal (well, fair use clauses sort-of make it mostly legal) and you are doing that with a web browser anyway, with cached pages, and cacheing proxies and similar devices also perform a similar function. The protections afforded by copyright all pertain to public reproduction and dissemination.

The WSJ point about restricting the number of free article views made in another reply is similarly nothing to do with copyright. This is more of a contract issue, and anyway, if I can view only N articles for free, I can only save those N, so it is really moot.

sroerick13• 12 years ago

I'm interested in artists who release things under Creative Commons, or works that exist or are released into the public domain.

These tools would work fine for me, without risk of copyright infringement. Just because you don't use libre content doesn't mean that the rest of us should have a broken internet.

sroerick13• 12 years ago

Slightly related to this, the other day I tried to download every youtube video in my watch history.

Turns out it's pretty much impossible. The Youtube API only delivers about 20 results, which is a bug that has existed for about 2 years. I tried manually loading the watch history page, and was only able to get about 1000 out of ~8000 results. When I selected those 1000 results and tried to add them to a playlist, the interface crashed.

Does anyone have a solution for this, or is my watch history just at the mercy of Google?

lutusp5• 12 years ago

The Youtube API only delivers about 20 results, which is a bug that has existed for about 2 years.

That may not be a bug. It might be a way to limit the traffic created by scrapers who submit requests and then download all the videos listed on the result page.

yuvadam• 12 years ago

You can use youtube-dl to do this:

    $ youtube-dl ':ythistory' -u USER -p PWD --write-pages

Of course, you won't get your entire history since g00gle would rather mine all your personal data itself, and never share it back with you again.

hnha• 12 years ago

It baffles me how we got from "tracking is creepy" to "please track all I do on your site" in ~10 years.

Your watch history should be stored by your browser on your machine. It is only your business and only locally it is fully in your control.

sroerick13• 12 years ago

My second reply, but, I think that this is really important.

We only need to look at early film history to know how easy it is to lose massive parts of our history.

Going back to old pages, I frequently get 404 results. For politically sensitive documents, the problem is much more widespread.

I would like something that not only archives pages I visit, but also versions them and tracks changes. If there was a bookmarking tool that did this, you could easily have an opt-in feature that shared content. This type of system would be a huge boost to something like the wayback machine.

lutusp5• 12 years ago

It clearly hasn't occurred to you that private collections evaporate over time, just like public ones.

I would like something that not only archives pages I visit, but also versions them and tracks changes.

That's a huge storage requirement, you must realize this. If you're an avid Web browser, and if every archived page had to look as it originally looked (i.e. all the linked resources) you could accumulate several terabytes per week.

This type of system would be a huge boost to something like the wayback machine.

Here's the road to madness. Someone, aware of the rapidly declining cost of storage, rebuilds the Wayback machine based on your scheme, with the intent of archiving every Web page in existence, including all required resources so the pages look just as they originally did. Then, as the project approaches completion, this genius says, "For the next phase, I need to archive the Wayback machine itself." At that point, as the implications of what he's said occur to him, a strange look crosses his face and his imagination begins writing checks his intellect can't cash.

walterbell4• 12 years ago

What do you think of this generalized architecture?

HARDWARE: differs depending on whether you want local search/analytics or just network storage.

For mobile use, either a VPN back to your personal home/cloud server, or a hackable wifi hard drive proxy, e.g. Seagate Wirless Plus + HackGFS.

For non-analytics home use, hackable router with USB3 storage and Linux software RAID, connected to a USB3 drive chassis with room for 2-4 disks.

For analytics home use, a microserver like HP N54L, Dell T20 or Lenovo TS140. Up to Xeon processor with ECC memory, plus 4-6 internal disks and up to 32GB RAM. Sold without a Windows tax, supports hardware virtualization and Linux. Possibly FreeBSD with ZFS.

SOFTWARE: generalized multi-tier cache AND compute. Camlistore and git-annex are tackling multi-device storage sync. For archives, we need a search interface that will query a series of caches, e.g. mobile > home > trusted friends private VPN (tinc overlay) > public paid cloud archive (pinboard et al) > public free cloud archive (archive.org).

It's important for usability to have a simple, local UX that will take a search string, propagate across all private/public federated tiers of storage and compute, then aggregate the metasearch results on the client.

With this approach, we can collectively pool resoures to improve on CommonCrawl.org, without locking up the 300TB index at AWS. This would turn web search engines into a secondary source, rather than a primary source. First search your archive + trusted friends, then trusted verticals (e.g. HN, StackOverflow), then a generic web search.

Let's be clear: the goal is not to archive "everything" in the world, only that which is personally important to the viewer. This attention metadata has long-term value. With this architecture, it is always optional to escalate a query to a public archive or search engine. Most importantly, there is technical autonomy and low-latency compute for local queries.

For web pages, wget of WARC formats (per HN advice on another thread) and wkhtmltopdf (available as Firefox plugin to print to PDF) will keep local archives. Recoll.org (xapian front-end with user-customizable python filters) on Linux will search full text and provide preview snippets, or lucene/solr can be adapted.

Terretta• 12 years ago

I really think people should take steps to archive stuff that is important to them

I understand why you didn't post a "self promoting" link, but I want others to know about this option:

https://pinboard.in/tour/#archive

Currently $10 for a lifetime bookmarking account, plus $25/year to archive every bookmarked page:

"Pinboard offers a bookmark archiving service for an annual fee of $25. The site will crawl and store a copy of every bookmark in your account, and display a special icon you can click to see the cached copy. If the page you bookmarked goes offline, you'll still be able to see the archived copy indefinitely."

dang• 12 years ago

We changed the title to the first sentence of the article because it is less linkbaity.

purplerails12• 12 years ago

I've been thinking about this for a while now. Please check out my web app to solve this problem: https://www.purplerails.com/

The main idea is to use a browser extension to automatically save pages that you read to the cloud (including the images, stylesheets etc) in the background. Saved pages are searchable and sharable.

lingben2• 12 years ago

sounds like evernote, if I'm mistaken, please enlighten me :)

purplerails12• 12 years ago

1) saving is automatic

2) privacy-first architecture (e.g., plaintext is never uploaded, plaintext URLs are never uploaded etc)

idlewords17• 12 years ago

An early design idea I had for Pinboard was as a browser plugin that just saved everything it saw in passing to an upstream server. But the problem that stumped me was that there's much more downstream bandwidth than upstream on a typical residential connection, so it was hard to push things to a server in anything like real time. How did you end up dealing with this issue?

purplerails12• 12 years ago

Pages are saved in the background. Nothing too fancy. I dedup when uploading: that helps a lot.

hollerith• 12 years ago

I'd prefer for the pages to be saved to the hard drive of the machine running the web browser.

But maybe browser extensions cannot obtain permission to do that?

purplerails12• 12 years ago

I see your point. I may release installable native apps that can serve as the local storage backend for the truly paranoid. Time limitations etc: the usual. :)

I anticipate that PurpleRails will be used on multiple computers and over several years. Which is why pages are sync'd to the cloud.

I've adopted the current architecture because I feel that the energy barrier that needs to be overcome to persuade somebody to install an app is much higher than the one to install an extension.

zargon• 12 years ago

This sounded really great until I went to the website and saw that I can't use my own cloud storage, only purplerails'. As soon as purplerails disappears all my saved pages are gone. I already have this functionality with diigo and it makes me very uncomfortable not to have a copy of the data.

purplerails12• 12 years ago

Excellent point. The ability to download your data in a well-documented format is coming soon. See also my reply to hollerith on a native client.

Time limitations are what is preventing me from doing this.

Thanks for your feedback! Hope you will use Purplerails. :)

asaddhamani4• 12 years ago

I tried giving purplerails a shot. I use LastPass for password management, and I am not going to type in a 25/30 character password every single time I want to log into an application. I think you're going way too hard on that part. This is the first time I've had this happen to me when using a web app, and it immediately made me close the page.

purplerails12• 12 years ago

Thanks for taking the time to write your feedback. I appreciate it.

With PurpleRails, you will rarely be typing your password once you successfully login. It works kinda like Gmail/Facebook etc: you remain logged in for months at a time.

This page explains why password saving doesn't work yet: https://www.purplerails.com/blog/saving-passwords-why-we-hav...

I'll see if I can do something fancier than what I do now to allow saved passwords to work. I suspect that using a Javascript-based AJAX authentication system instead of a plain-old HTML form, I can achieve the privacy goals as well as ease of use.

I hope you will hang in there and use the current version until I figure out a way to fix this. Thanks!

magila• 12 years ago

The fact that he thinks a federated wiki would be "simple" or "easy" leads me to believe he has not actually thought through the details of how it would work in practice.

michaelchisari2• 12 years ago

It would not be simple or easy, but crypto-currency blockchains make it more possible than ever.

grkvlt3• 12 years ago

What does this mean? I can't understand the relationship between the two ideas?

azakai• 12 years ago

Yes. 2 problems that immediately come to mind are

1. Copyright law.

2. Dynamic content.

ricardolopes• 12 years ago

It's a great idea for a real problem that needs to be solved. Still, for dynamic pages, what would be the desired behaviour? Updating it whenever possible, which could lead to the specific info we wanted to save potentially disappearing or changing? Leaving it outdated? It is really something that I cannot answer.

hackaflocka• 12 years ago

I will pay good money for a Chrome extension that does the following:

1) I can select (or do select all) Chrome bookmarks that I want to keep offline page backups/archives of (saved to google drive or dropbox or some such).

2) Whenever I want, instead of seeing the current online version of that bookmarked page, I can look up the originally bookmarked archived page.

3) It allows me to choose the level of links to the bookmarked page to also backup/archive (e.g., every single page that is linked to that page, x links deep, is also automatically archived -- think httrack or wget).

As someone on Hacker News once said to me: my bookmarks are my knowledge graph. As important to me as any library.

Detrus• 12 years ago

Pinboard archiving costs about $25 a month. Not sure it does the deep link archiving.

steele• 12 years ago

Right to be forgotten?

idlewords17• 12 years ago

I forgot about that. AS REQUIRED BY EUROPEAN LAW.

wyager2• 12 years ago

Right to be forgotten?

A ridiculous concept of a "right". I do not recognize that anyone has a right to force other people to forget things.

spindritf• 12 years ago

It’s interesting that Andreessen can’t see the solution, but perhaps expected.

What a weird dig. It's neither expected, nor established that he can't see a solution. I'm not as smart as Andreessen and I could come up with half a dozen solutions.

Author's favourite is fine but far, far from obvious. How viable is it to run your own federated wiki anyway? Are there packages for popular systems? Are there plugins for major browsers? Is there any federation actually happening? I skimmed the resources[1] and don't know. Does anyone here run one? That would be a solution, this seems more like an idea.

And it's not like no one's doing anything. There are services like Pocket or Readability to store an article until you want to read it, Evernotes, Google Keeps... Our very own 'idlewords will archive the contents of your bookmarks for a fee[2]. Finally, there's archive.org.

[1] https://github.com/WardCunningham/Smallest-Federated-Wiki#ho...

[2] https://pinboard.in/tour/#archive

can09• 12 years ago

What he seems to be getting at is a much larger, more revolutionary approach to not just "the web", but "the internet" as we know it:

https://en.wikipedia.org/wiki/Named_data_networking

MrBuddyCasino• 12 years ago

Thats a very academic and static view of content - I don't see how that would work in todays hyper-dynamic environment, where the Ads that are displayed on a site are priced by millisecond real-time auctions before they are delivered to the user, and websites are single-page apps with REST APIs in the background. How would that work?

grmarcil• 12 years ago

I find Bret Victor's comparison between the internet and the LOC a little weird. I've always thought of the internet as a publishing/sharing medium, not an archive.

There are plenty of books that go out of print within ten years, we just happen to have infrastructure beyond publishers (libraries) that preserve published copies.

idlewords17• 12 years ago

I think it's significant that the Library of Congress has funding, an official mandate, employees, a clear legal status, and stores complete copies of the works in its catalog. A similar model would work great for the Internet (and archive.org is doing its best to fill the role).

chimeracoder• 12 years ago

I find Bret Victor's comparison between the internet and the LOC a little weird. I've always thought of the internet as a publishing/sharing medium, not an archive.

The Library of Congress is archiving all tweets from the US[0], which I think is what he is referring to.

[0] http://www.businessinsider.com/library-of-congress-is-archiv...

wyager2• 12 years ago

This would not work with dynamic content.

We already have systems like this (bittorrent, freenet, etc.), and almost no one sees them as a viable replacement for the web because they can't do 99.9% of the things we want (social networks, forums, email, etc.)

unicornporn• 12 years ago

The museums, libraries and cultural heritage institutions are working hard on digital preservation. I think there's a lot to learn from there. Check out this for an instance: http://www.lockss.org/about/how-it-works/

TelmoMenezes• 12 years ago

Coming up with architectures to decentralize servers is the fun part. Convincing people outside of our bubble to use the new system is the very hard part. It has to be able to do something the regular person really wants that the previous system didn't allow. This is why Linux never caught up on the desktop.

Now excuse me while I go curate my socks collection.

twoodfin• 12 years ago

I'd love it if browsers natively supported URI's derived from cryptographic hashes of content by looking them up in a distributed store a la BitTorrent. Imagine if Chrome supported such a thing, for example. Perfectly reliable cache-ability (or archive-ability), P2P hosting, ... All the good stuff for any web content that its creator wants to so expose, albeit at the price of immutability.

ottonomy• 12 years ago

Especially under the current copyright regime, finding some solution that preserves the intent of the creator to publish in a fixed format, would be a great component of a distributed publishing system. I don't think this proposal has as good a fair use defense as the Internet Archive wayback machine does.

In the world today, we often think of publishing online as providing access to something under our control. I think a technology that aims to solve these problems should embody a different spirit, one closer to "making public". The word "mine" doesn't need to imply ownership in the sense of exclusive control. I mean, "My children" is at least as meaningful a relationship as "my property". Some kind of copyright license ability built into a distributed document publishing system would be nice.

mark_l_watson• 12 years ago

I like the idea of the federated wiki, but search engines rank copies of pages poorly, so it is not clear how visible copies would be after the original content disappears.

I used Evernote for years, but recently canceled the service because I spent too much time curating compared to reading old material.

One option that I am considering is archiving really good web content as web archive files and saving them locally in folders indicating the year of capture. Local file search would quickly find old stuff and if I stored the yearly web archive folders in Dropbox, I would have them available on different systems.

bajsejohannes• 12 years ago

I think the federated wiki is a neat idea, but in it's current incarnation, I find it to be exceedingly unlikely that a page I'm looking at there will stay around 10 years.

Even if I'm making a copy of every page I see, I'm not sure I'll still run a federated wiki on my server in 10 years.

I don't think this is a real solution to the problem posed by Bret Victor.

lutusp5• 12 years ago

There’s actually a pretty simple alternative to the current web. In federated wiki, when you find a page you like, you curate it to your own server (which may even be running on your laptop). That forms part of a named-content system, and if later that page disappears at the source, the system can find dozens of curated copies across the web.

This is a simple and very bad idea. If it were the norm, instead of one or no copies of a particular work online, you would have any number of "curated" copies of uncertain vintage, downloaded at different times in the lifetime of an original whose content might well have changed as time passed. You would have curations of curations, and curations of those, ad infinitum.

Pages that depended on remote Web content (increasingly common) and/or that linked to online references, would gradually become unreadable or incomprehensible as its links vanished into other offline "curations".

Not to mention the copyright issues. And I'm not crazy about the term "curation" either -- it's obviously meant to try to elevate the practice of downloading anything we please, without regard to copyright.

pjbrunet2• 12 years ago

I'd rather a page go offline than have it taken out of context. As if plagiarism wasn't bad enough already. (Yahoo Answers cough)

These crooks will even steal your copyright notice. It's quite possible the original content producers are offline because scraper thieves stole so much content that it's no longer possible to earn a living.

As an artist, this reminds me of the condescending attitude that gave us fake Rolexes, Facebook & North Korea's 28 state-approved haircuts. Either it's "just content" to stuff in a database somewhere or you understand the medium is the message too.

sroerick13• 12 years ago

As someone with a teensy bit of film background, I have to disagree. The number of early Hollywood films that were lost is astounding. This is a massive part of our visual history that is completely gone. It will never be restored.

With the current environment on the internet, with DRM'd video, music and text, I have to assume that we will lose far more from this time period than we ever had before.

While I don't pirate things (I'd rather just consume Creative Commons and Public Domain content), I wholehartedly support people who are trying to archive the things that are part of our collective culture. When I have kids, I'd like to be able to show them where they came from.

pjbrunet2• 12 years ago

I'm not against reproduction if care is taken to preserve what can be preserved, as close to the original as possible while giving credit, compensating creators, etc. In your example, I think reproduction/conservation technology was available but the studios couldn't justify spending the time/money it would cost to preserve their entire library. Who would have paid for that? I don't know. Supposedly, half of Van Gogh's entire lifetime output was burned because he was too poor to find a place to store it. At the same time, I'd rather see a bug-eaten Van Gogh with fugitive reds long faded away, than a flat, lifeless high-def copy. I suppose it's a complex subject and each work is unique. Sometimes I'm thrilled when I can find an old page in "Wayback Machine" but what they manage to save is typically broken and low quality.

samdroid• 12 years ago

The author is not suggesting taking others content and calling it their own - he is suggesting almost keeping a backup with a bibliography. Infact, this already exists today - do you hate archive.org?

Houshalter3• 12 years ago

Link rot is a serious problem: http://www.gwern.net/Archiving%20URLs#link-rot

In a 2003 experiment, Fetterly et al. discovered that about one link out of every 200 disappeared each week from the Internet. McCown et al. (2005) discovered that half of the URLs cited in D-Lib Magazine articles were no longer accessible 10 years after publication [the irony!], and other studies have shown link rot in academic literature to be even worse (Spinellis, 2003, Lawrence et al., 2001). Nelson and Allen (2002) examined link rot in digital libraries and found that about 3% of the objects were no longer accessible after one year.
Bruce Schneier remarks that one friend experienced 50% linkrot in one of his pages over less than 9 years (not that the situation was any better in 1998), and that his own blog posts link to news articles that go dead in days; the Internet Archive has estimated the average lifespan of a Web page at 100 days. A Science study looked at articles in prestigious journals; they didn’t use many Internet links, but when they did, 2 years later ~13% were dead. The French company Linterweb studied external links on the French Wikipedia before setting up their cache of French external links, and found - back in 2008 - already 5% were dead. (The English Wikipedia has seen a 2010-2011 spike from a few thousand dead links to ~110,000 out of ~17.5m live links.) The dismal studies just go on and on and on (and on). Even in a highly stable, funded, curated environment, link rot happens anyway. For example, about 11% of Arab Spring-related tweets were gone within a year (even though Twitter is - currently - still around).

idlewords17• 12 years ago

My own research (which I hope to publish soon) shows a slightly better link rot rate for bookmarked URLs (which are presumably ones people are most interested in keeping). The attrition rate I see so far is roughly linear and about 5% a year. Which is still shocking by any non-web standard, but a little better than the figures cited above.

sillysaurus312• 12 years ago

You do research using bookmarks on Pinboard as your dataset? May I ask how this data is used and disclosed to others?

Houshalter3• 12 years ago

That's consistent with most of those studies. The next paragraph does an interesting experiment showing a very conservative rate of 3% per year:

My specific target date is 2070, 60 years from now. As of 10 March 2011, gwern.net has around 6800 external links (with around 2200 to non-Wikipedia websites)4. Even at the lowest estimate of 3% annual linkrot, few will survive to 2070. If each link has a 97% chance of surviving each year, then the chance a link will be alive in 2070 is 0.972070−2011=0.16 (or to put it another way, an 84% chance any given link will die). The 95% confidence interval for such a binomial distribution says that of the 2200 non-Wikipedia links, ~336-394 will survive to 20705. If we try to predict using a more reasonable estimate of 50% linkrot, then an average of 0 links will survive (0.502070−2011×2200=1.735×10−16×2200≃0). It would be a good idea to simply assume that no link will survive.

ilaksh• 12 years ago

Named data networking of some kind is likely to become popular at some point. This is that kind of idea but doesn't look like a really general protocol since he mentions a specific wiki.

I wonder if there are browser extensions that do p2p caching/distribution of content. Then you could standardize a protocol used for that type of communication.

I believe there are many efforts along these lines. The trick is as usual getting everyone on the same page or at least working together more.

CCs• 12 years ago

Hosting your own server might not be a scalable solution either. There's a reason why SaaS is popular: it's not that easy.

On the downside: stuff hosted by others might go away. Web pages, web services, apps requiring server side support...

Investing a lot in a service makes it more painful to lose, like the apparently discontinued Amazon Cloud Drive (supposed to be a cheaper Dropbox): https://news.ycombinator.com/item?id=8219257

There’s a simple alternative to the current web

Related Stories