Internet Archive, decentralized

https://dweb.archive.org/

by justin_ • 8 years ago

242 84 8 years ago

dweb.archive.org

krautsourced• 8 years ago

Look, feel and content-wise this feels a lot like the internet of the 90s...

pmoriarty• 8 years ago

Unfortunately, it's not 90's enough, as it still requires fucking javascript.

theoctopus• 8 years ago

This is one of the (admittedly few) cases where javascript is actually required.

SquareWheel• 8 years ago

Would you like to add more dependencies on server-side code for this decentralized project?

ryanlol• 8 years ago

Unsupported mediatype: data

mitraardron5• 8 years ago

Yes - we don't yet support some of the more unusual types on the Archive (including "data" and "software")

CuriousSkeptic• 8 years ago

When cloudflare screwed up and published everyones secrets there was a coordinated effort involving, amongst others, archive.org to try and scrub the internet of those secrets. Are there any mechanism available to allow similar efforts with this dweb version?

comboy2• 8 years ago

There should not be a mechanism for that. China mastered such mechanisms. We should rather optimize for not having that kind of points of failure.

burkemw3• 8 years ago

In addition, optimize for dealing with those situations: secret rotation plans

0xcde4c3db• 8 years ago

There's nothing inherently wrong with an opt-in coordination mechanism. It makes sense that if I trust someone to make a blacklist, I can have my node refer to that list. The problem is when an organization threatens you for "publishing" the wrong thing. That's a legal or social problem, not a technical one, and such organizations typically don't care in the slightest about how convenient it is for you to comply with their demands. If there's no blacklist mechanism, they're more than happy to demand that you shut down your node altogether.

alkonaut4• 8 years ago

I think of the two evils, the possobility of censorship or mistakes being eternal, I think I prefer the former.

For the same reason I find it comforting that an old fashioned bank transfer can be corrected if I transfer money and make a mistake writing the account number. Mutable history is a powerful feature.

lucb1e• 8 years ago

Downvote because I think the nationalistic "us vs them" is uncalled for. Let's not make it about country politics when the topic isn't already politics.

marknadal6• 8 years ago

GUN author here (https://github.com/amark/gun) happy to answer any questions.

IA did this integration in 1 week, Mitra is awesome.

Also, decentralized Reddit (https://notabug.io) was built in 1 week on us, and pushed 0.5TB P2P traffic on 1st day.

Note: I may not be awake for several hours, and might not be able to reply until Monday.

pandasun• 8 years ago

Wow, your work (GUN) is really great. I hope you don't mind me asking how it works in layman's terms? Thank you for your work.

marknadal6• 8 years ago

Huge honor! Yes:

- http://gun.js.org/distributed/matters.html

- https://youtu.be/EHZyaupYjYo?t=55m52s

- http://gun.js.org/explainers/data/security.html (not about GUN itself, directly, but SEA)

Is at least 1 of those helpful? If not, have any ideas on how I can improve the explanation?

modernerd• 8 years ago

GUN looks great! I love the quick-start tutorial too: https://gun.eco/think.html

How suitable is GUN for live multiplayer (non turn-based) web games? (Similar to https://airma.sh/.)

I see one game example at https://github.com/amark/gun/wiki/Awesome-GUN, although it's turn-based.

marknadal6• 8 years ago

Thank you!!!

I need to fix the organization of the documentation (and update the docs, oye!). I'm impressed you found Awesome-GUN.

https://github.com/amark/gun/blob/master/examples/game/space... is probably what you were searching for.

What would be better is if I made a blog/tutorial for ^ link. Not a priority for me, sadly, but maybe it is for somebody out there, that they could help?

Thanks again!

justin_2• 8 years ago

Thank you for creating GUN! The decentralized Reddit demo loads so quickly.

To prevent spam, it looks like they require a PoW on each vote. Does every update on a GUN database from untrusted peers require something like this? Is authentication intended to stop spammers in the future?

PS. For anyone reading, there's a recording of a talk Mark gave on GUN from the DWeb Summit, available here: https://youtu.be/kW6e1GCpqpE?t=43m22s

go1dfish5• 8 years ago

Author of notabug here.

GUN uses a proof of work for account creation/login I think, but otherwise no there is no proof of work requirement for updates.

I added the proof of work requirement to votes as part of my own validation. The difficulty at https://notabug.io is set quite low, but https://dontsuemebro.com is a peer that still has it set quite a bit higher, it rejects the cheaper votes at notabug.io so the scores/sorts are different.

Spent a lot of time focusing on performance, notabug.io is running GUN with redis as a storage adapter and doing server side rendering to speed up the user experience.

Domain pages are currently all gun/clientside though with out the server doing anything special to help at all.

https://notabug.io/domain/youtube.com/

Also when using the infinite scroll feature or chat most all content is loaded directly through gun without intermediary REST calls.

e12e3• 8 years ago

Hi, thanks for stopping by :)

I'm still trying to get a handle on the security aspects of gun. Say you want to create a blog/note app - that holds both private notes, drafts - and things that are to be shared to some friends.

Would you effectively have to store data encrypted in gun, and manage access via sharing encryption keys - in order to be able to both securely store data, and share it?

In the examples, it appears things like "create user" is called in client side code - which seems to imply anyone can write any data to a gun db? (by adding themselves as admin?)

Is the use-case of gun more a public, structured wiki - where all content is fundamentally untrusted - but easily updated by anyone?

marknadal6• 8 years ago

Right back to you! :)

That is probably because I've done a poor job communicating it, since I'm still finding time to write about it. Thank you for bringing this up!

Probably most relevant: I kinda sorta had a demo of a P2P LinkedIn working https://www.youtube.com/watch?v=ZiELAFqNSLQ .

So we do have an unstable API that automates key management and key sharing, but all production apps (notabug.io , etc.) today directly use our https://gun.eco/docs/SEA shim over WebCrypto.

Unfortunately, that means you have to be aware of how to apply it - thankfully, we did make a cartoon cryptography crash course on this (in link), so it is viable to get started.

Obviously, if you have any new insights, would love to hear it!

Without SEA, gun is very much like what you say. With SEA, you can protect against just anyone randomly writing to GUN. Jump in and ask more Qs on https://gitter.im/amark/gun about it, or you'll circle back around later - hopefully that is helpful directions?

Thanks. Most important statement of mine: I bet you'll enjoy the cartoon cryptography series.

rambojazz• 8 years ago

Is that website related to https://notabug.org in any way?

go1dfish5• 8 years ago

nanna• 8 years ago

How does this work?

krylon2• 8 years ago

This is really cool! A bit sluggish, but I love the idea!

If you think back to, say, the library of Alexandria, to how much knowledge has been lost over the ages, it is so important to preserve as much as we can for future generations.

And building a decentralized foundation for this archive is a big step going forward, congratulations!

pixelpoet• 8 years ago

I recently went to the Internet Archive in SF, got a tour of their operations etc. Absolutely amazing place, very forward thinking, and they indeed are very serious about prevent something like the destruction of the Library of Alexandria from happening again.

krylon2• 8 years ago

There is a (great, IMHO) science fiction novel, "The Mote In God's Eye" by Larry Niven and Jerry Pournelle that describes humanity's first contact with another intelligent species.

Due to their biology, this alien species experiences periods of massive population growth that eventually lead to all-out war and collapse of civilization. Over uncounted thousands, maybe millions of years, these aliens have accepted this vicious cycle as kismet, and deal with it by building "warehouses" filled their most advanced technology to jump start civilization after the next, inevitable, collapse.

I hope humanity will never have to deal with such a collapse, but I given our collective tendency towards self-destructive behavior, maybe we should build such an archive as if it was meant for future cavemen to jump-start them into a new Anthropocene. Even if that collapse never happens (I am keeping my fingers crossed!), the resulting tome of knowledge would be a suitable monument to all the incredible things that humanity has accomplished, as well as an insurance policy in case we manage to mess up on a monumental scale.

EDIT: A more positive perspective would be The Library from David Brin's uplift saga, a humongous collection of knowledge acquired by many, many species over millions, if not billions of years.

EDIT: typo

agumonkey• 8 years ago

very nice

swaggyBoatswain2• 8 years ago

I didn't see any articles relating to how it was decentralized. Is internet archive going for something similar to a blockchain setup, because that would actually be one of the few cases it would make sense

I am not familiar with this new dweb subdomain, what is unique about it?

Archive has actually lost some of the archives I stored on it, which is weird, because I ran multitude of backups on it a few years back when the site got taken offline

justin_2• 8 years ago

I wanted to submit an article about this but couldn't find anything either.

My understanding is that this loads content from various protocols (listed at the top of the page), many of which support replicating data in a decentralized way. As far as I know, there's no blockchain involved in anything here yet.

If you browse to the Community Video section and choose a video, you can see peer information as though downloading through WebTorrent. If I disable WebTorrent and look at a video, I don't see the peer information and it seems to fall back to HTTP. Pretty cool! It looks like almost everything is only seeded by the Internet Archive right now, but hopefully they want to encourage more people to participate.

swaggyBoatswain2• 8 years ago

Hm that's interesting, I imagine the internet archive has lots of seedboxes/CDNs/datacenters distributed globally. I have no idea how much information the internet archive is currently backing up though, but its growing at an increasing rate.

I would love to see a write up of how their infrastructure works though

rolandog• 8 years ago

Do you think it's built on top of PeerTube Social or is it something completely different?

PhantomGremlin• 8 years ago

Archive has actually lost some of the archives I stored on it

Who currently controls the domain? One of the ways stuff can be lost is if a new domain owner fiddles with robots.txt. (The Archive has recently changed their policy about that)

We have also seen an upsurge of the use of robots.txt files to remove entire domains from search engines when they transition from a live web site into a parked domain, which has historically also removed the entire domain from view in the Wayback Machine.

https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

textmode• 8 years ago

Is there a delay between the time the robots.txt changes and the time when the content becomes inaccessible via Wayback Machine? How often does the archive.org_bot crawl robots.txt?

Can a script check robots.txt periodically for changes and if changes are detected, then download the content from Wayback Machine before it becomes inaccessible?

Additionally, can a script check the domain registration for an anticipated expiration date, or perhaps monitor domainname "drop lists"?

mitraardron5• 8 years ago

There's a video of the talk I did with Arkadiy (from IPFS) and Feross (from WebTorrent) at the Dweb Summit at https://archive.org/details/youtube-eO6pYYWZBs8?start=10447 it goes into quite a lot of detail about how it was implemented and the challenges we faced along the way.

drc0• 8 years ago

on firefox it hangs past http.

mitraardron5• 8 years ago

Its still an experiment to see what was possible with a big site, in today's browsers using some of the emerging tools (IPFS, WebTorrent, GUN). We only test on absolutely current releases of Chrome & Firefox. If you run into problems on the site, there is a feedback button, otherwise you can report directly on https://docs.google.com/forms/d/e/1FAIpQLSe7pXiSLrmeLoKvlDi2... or open an issue on https://github.com/internetarchive/dweb-archive/issues

nikisweeting• 8 years ago

Aww I was going to add almost exactly the same thing to https://github.com/pirate/bookmark-archiver but IA beat me to the punch! I still hope to add decentralized storage and lookup mechanisms to BA eventually, but considering one BA's archive outputs is the Internet Archive, it's less pressing now.

ezequiel-garzon3• 8 years ago

I strongly believe IA or any serious project working on permanent persistence must provide an option (opt-in if you will) to make the published material irremovable, à la arXiv [1].

[1] Red parts of https://arxiv.org/help/license

quickben• 8 years ago

And that is your right. However most people would like to remove slander with a court order.

ezequiel-garzon3• 8 years ago

Pretty sure arXiv would remove slander with a court order as well. It's an unrelated issue.

marknadal6• 8 years ago

Thank you for bringing this up, it is another reason we need mutable data structures in the decentralized web - for social reasons, not just technological.

In another thread, I also mention that immutability creates algorithmic complexity that makes it difficult to scale: https://news.ycombinator.com/item?id=17693920

bo1024• 8 years ago

What is it? My browser (firefox) sees nothing* even after enabling javascript.

*Except the sentence "The decentralized web is everywhere, but we have to find it." and a Name form that does nothing.

trumped2• 8 years ago

the problem is not firefox... it's probably addons or your internet connection that blocks content

superkuh2• 8 years ago

It is also nothing for me in a Firefox fork with almost all add-ons disabled. Error console says there's a bunch of "class is a reserved identifier" and syntax errors of missing semicolons (before main() can even run).

My guess it's trying to use some bleeding edge emcascript stuff which borks the parsing in non-bleeding edge browsers.

Internet Archive, decentralized

Related Stories