Thank you for helping us increase our bandwidth

https://blog.archive.org/2020/05/11/thank-you-for-helping-us-increase-our-bandwidth/

by edward • 6 years ago

677 199 6 years ago

blog.archive.org

archive.org feels like an irreplaceable treasure, the Wayback Machine alone is a time capsule of our digital history.

I donate to them monthly and know a lot of other people do as well, so I don't worry much about their financial stability. I'm more worried about external pressures taking content down. I hope the data is backed up six ways to sunday, and that somewhere there's a plan to make it all accessible if Internet Archive can't continue to play the role it does.

chrisseaton2• 6 years ago

archive.org feels like an irreplaceable treasure

Yes I save some dead sites that I think are critically important resources to tarballs that I'm storing separately in case the Wayback Machine disappears.

aesh2Xa1• 6 years ago

That's useful for personal use. I think that the IPFS is designed for this purpose as well, but meant to be accessible to anyone.

im3w1l• 6 years ago

While that may work for your personal usecase, there is a significant issue around trust, if you were to share these tarballs with anyone else. I wouldn't trust you to give me originals in the same way I trust the wayback machine. So even if there are backups lying around here and there, their utility is decreased a lot.

dmd3• 6 years ago

I'm very worried about its backups and what happens when the next Big One hits SF.

As far as I've ever been able to determine from talking to anyone at IA (e.g., Kahle, Scott), they don't really have any sort of backups that could actually be restored from in a disaster situation.

britmob• 6 years ago

There is http://iabak.archiveteam.org, but it’s not exactly large.

enneff• 6 years ago

I wonder what the cost would be to build out a secondary backup site on a different continent.

robocat• 6 years ago

Do they have distributed backup?

Surely a lot of people would be happy to donate unused space on their drives (I know I would) - especially since backup wouldn’t use much bandwidth...

bane3• 6 years ago

Whatever happened to their copy in Canada?

https://blog.archive.org/2016/12/03/faqs-about-the-internet-...

erichocean• 6 years ago

They should team up with Backblaze...

city413• 6 years ago

They have lots of copyrighted content. Such as virtually every commercial retro video game ever made it seems.

I personally think it’s great but surely companies aren’t too pleased? How does archive.org avoid being sued into oblivion?

judge20202• 6 years ago

It's a non-profit library and they don't publicly provide a way to download most of the copyrighted content, so rights holders aren't too concerned. I imagine most people, even copyright lawyers, personally support the free archiving of everything made in the current age as long as it doesn't detract from current business operations and the copyrights are respected for their duration.

beerandt• 6 years ago

As far as I know, they limit public access to basically any material with a complaint or request, but keep copies. They may do more, but that seems to be the default response.

Which seems smart enough: minimize litigation costs while not losing content permanently via court orders or dmcas or similar threats.

But it's frustrating for the wayback machine- where sites like Snopes and some newspapers have opted out of having their history published (after being accused of ghost edits to articles).

I don't know what the right answer is, but even if they didn't display the page, it would be nice to see the diffs (ala wikipedia), or at least show if and when changes to a page were made. Maybe that's beyond their mission scope, though.

Paul-ish2• 6 years ago

There are exemptions to DMCA for certain content.

Computer programs and video games distributed in formats that have become obsolete and which require the original media or hardware as a condition of access. A format shall be considered obsolete if the machine or system necessary to render perceptible a work stored in that format is no longer manufactured or is no longer reasonably available in the commercial marketplace.

https://www.copyright.gov/1201/2003/index.html

Aeolun3• 6 years ago

It’s the only organization I actually donate to. Partially because it was so easy to set up a monthly donation ;)

Cthulhu_2• 6 years ago

They could set up a torrent that people could download parts of, I think it's an ideal system for a distributed backup.

But it probably needs tweaking for that purpose; for one you need to ensure that the data is evenly distributed, and second you're dealing with data that is appended to regularly.

But I think it can be done.

Qub3d• 6 years ago

It exists! The ArchiveTeam had a project, you can see its status here: http://iabak.archiveteam.org/

Here's an overview: https://www.archiveteam.org/index.php?title=INTERNETARCHIVE....

Sadly, it seems the project has fallen out of repair.

bane3• 6 years ago

I've thought for a long time why I absolutely agree with this sentiment and the best I can come up with is that the Internet Archive feels to me like the embodiment of the old internet, not this marketing driven, data steal-and-sell, VC backed cyberpunk dystopia Internet that we have everywhere else.

It's the same sort of feeling as I why I enjoy this site, wikipedia, and so on. Bonus, IA is also a bit of an organizational mess, but rewards the adventurer with rich treasure.

(I was recently on a Korean history kick and came across not only 1, but several entirely different first hand books written by visitors in the late 19th and early 20th century, scanned in, freely accessible/downloadable, in a variety of formats, and with an excellent on-web reader. These books are so out of print I checked with three local counties for copies and none of them even have references to any of them in their catalog -- treasure!)

lunchables• 6 years ago

There are only two sites I donate to: wikipedia and archive.org. I honestly don't know what I'd do without them. I have been able to find web sites from 20 years ago on archive.org, it's an absolute treasure.

SteveNuts2• 6 years ago

I'm really surprised they don't use more CDN for this. Anyone know the reason why it isn't served by something like cloudflare?

ajayyy• 6 years ago

It could be philosophical

1f60c• 6 years ago

That's what I'm thinking.

If they wanted to use Cloudflare, I'm pretty sure they'd jump at the opportunity.

adtac• 6 years ago

Please don't make the whole internet basically cloudflare. they've banned my VPN endpoint (Hetzner server) and as a result a huge chunk of websites already don't work for me despite me having done nothing wrong. I've heard reports of Tor users being restricted as well.

SteveNuts2• 6 years ago

Okay, any one of the other CDNs then. It seems like the bandwidth costs and equipment are prohibitively expensive for them. There must be a reason they don't use them which is what I want to know.

judge20202• 6 years ago

It's more likely that website owners are blocking the ASNs of hosting providers since those are often used for content scraping and exploit type attacks [since they're cheap]. There are public and private lists of hosting provider ASNs you can use to block them all if you generally only want visitors with a business/residential IP.

manquer4• 6 years ago

It is usually nothing specific to your VPN setup or IP. It is mainly either for (D)DoS protection or Content Licensing related ( VPNs are easy way to bypass Geo-locked content). Many CDNs and services will allow you to access using only non commercial IP range

mattl• 6 years ago

So many of the VPS companies have bots galore doing credit card attacks on smaller sites that it’s very easy and convenient to simply block those VPS providers.

highclass2• 6 years ago

At the bandwidth levels they are using they would need to use Cloudflare Enterprise, and in my experience that is way more expensive than other CDN providers.

Also, since archive.org has so much content, the caching ratio is going to be very bad and kill CDN efficiency while still requiring lots of direct bandwidth. Cheap direct bandwidth in their case looks best(which is what they seem to be doing).

highclass2• 6 years ago

Another aspect is they are actually using bandwidth to max and have a cap on usage spending. No CDN I know of allows capping bandwidth usage.

The way they use bandwidth means the most efficent GB/dollar cost although at the cost of poor performance.

unityByFreedom• 6 years ago

Cloudflare does not have bandwidth limits.

EE84M3i• 6 years ago

The geographic distribution benefits for archive.org seem minimal to me - for the actual content high latency isn't a big deal.

fomine3• 6 years ago

It's really benefiticial for outside of US.

iofiiiiiiiii3• 6 years ago

Geographic locationa affects far more than just latency. Even though you may have a 100 Mbps link on both client and server , only 10 Mbps might make it across the Atlantic Ocean. Connections between countries/ISPs are not unlimited in capacity!

jcrawfordor• 6 years ago

I don't foresee use of a CDN saving them much or any money. Cache hit ratio would probably be relatively low and the amount of storage required at the edge to cover an appreciable portion of traffic would be very expensive. E.g. their Akamai bill would be in the five figures per month if not six figures, depending on what kind of volume discount they could arrange, just based on outbound bandwidth. They're not the cheapest but also not the most expensive.

Serving up that large of a media library at mid scale just isn't really a great use case for a CDN, those that have to due to transit costs becoming truly enormous (e.g. Netflix) make an enormous investment in hardware at the edge that probably isn't affordable to IA (or necessary at this point).

chronomex• 6 years ago

They serve everything in-house and are their own AS; this allows them to provide stronger guarantees of reader privacy than sites that use a CDN.

Stratoscope2• 6 years ago

They serve everything in-house and are their own AS

Forgive me, but what is an AS?

jonah-archive4• 6 years ago

Sibling comments are exactly correct -- we're a library. We don't track or record the activity of our patrons (see e.g. https://archive.org/services/docs/api/views.html#footnote-wh... ), and since most CDNs cannot offer the same guarantee, we can't use them.

chx3• 6 years ago

I have revoked my donations until they close their pirate library. Sorry but let's call a spade a spade. They admit:

multiple readers can access a digital book simultaneously

with the only caveat being it is borrowed for two weeks but let's face it, most value of a book comes from its first reading -- and what stops you from "borrowing" it again, anyways.

It's been a gigantic disappointment for me to see them do this and not back but try to placate the authors with weasel words and an opt out.

toomuchtodo5• 6 years ago

I’m sorry to hear this. I will make an additional donation on your behalf. I can appreciate how authors feel, but they must also realize copyright laws are asymmetrical (tilted heavily towards copyright owners) and in these times, many have no access to their local public library. They are essentially robbed of access to the book materials their tax dollars (which pays an author) have paid for during this time.

It’s been a gigantic disappointment to see authors respond negatively to this effort (considering this once-in-a-century event), and I will never buy a book again from one of these authors.

chx3• 6 years ago

I also find it amusing that sofware developers do not recognize any more how we are on the same side in this fight. Imagine if we were still selling boxed software and they decided that in light of COVID-19 they just hand it out with a two week limit. It's an imperfect comparison but the current situation is not a reason to just discard the law. It might not be the best law, the place to fight that is in Congress and not on the back of authors most of whom are not in the best financial state to say the least.

bzb3• 6 years ago

Culture wants to be free.

therealcamino5• 6 years ago

Writing wants to be uncompensated?

jeromegv2• 6 years ago

The wayback machine has been essential through COVID as many government just publish the numbers and data "of the day" and the only way to compare to the day before is to look at IA.

makomk• 6 years ago

Governments aren't the only ones overwriting old information with new either. The BBC has developed an annoying habit of completely overwriting old articles about Covid-19 with new, semi-related ones, meaning that the only way to see what they were saying earlier in the pandemic is through sites like the Internet Archive.

There's also at least one outright correction that only seems to exist on the Internet Archive now: https://web.archive.org/web/20200428214306/https://www.bbc.c... "Correction 25 April 2020: An earlier version of this article incorrectly said that France had conducted just under 140,000 tests a day by 21 April. The figure of just under 140,000 refers instead to the number of tests it had carried out weekly."

endgame• 6 years ago

Here's another one: "some of our old tweets were wrong so we just quietly deleted them lol":

https://twitter.com/voxdotcom/status/1242537366620966912

pier25• 6 years ago

Wikipedia is a good source too, at least for the countries I've been following.

simonw2• 6 years ago

Do you have any examples of government sites that are doing this?

I have a side-hobby of setting up scrapers which pull scraped data into a git repository, precisely for this kind of thing. I'd be happy to set a few up.

Some of my posts about this technique (which I call "git scraping"): https://simonwillison.net/tags/gitscraping/

Endlessly2• 6 years ago

Not sure if you’re asking for:

(1) US coronavirus data in general,

(2) examples of sources that do not log prior days data, or

(3) source that “edit” prior data without noting the edits.

(4) something else

If (1) this page in the table under the column “sources” links to where the data came from:

https://www.worldometers.info/coronavirus/country/us/

jeromegv2• 6 years ago

Government of Ontario has been doing it. They now have some APIs that link to the number of cases per day, but for the first month of the pandemic, they were literally overwriting the page every day and when you look at github there was tons of researchers that came up with their own parser in order to save this data before it is deleted.

I've also been running bash scripts every day to save an archive of some of those pages https://github.com/jeromegv/covid_data

dublinben• 6 years ago

For the US at least, the COVID Tracking Project records multiple screenshots per day of the official websites where those numbers are published.

https://covidtracking.com/

est313• 6 years ago

60 Gbit/s of continuous traffic is a lot. If I'm reading the graphs right, Wikipedia "only" has 13.4 Gbit/s of outbound traffic [1]. Of course still well below the single-digit Tbit/s traffic of large internet exchanges [2] [3], but still unexpectedly large.

[1]: adding up the outbound numbers for each datacenter: 1.888 + 8.003 + 810 + 1.958 + 807: https://grafana.wikimedia.org/d/000000605/datacenter-global-...

[2] https://en.wikipedia.org/wiki/List_of_Internet_exchange_poin...

[3]: Not 100% correct as thanks to Corona, ix.br now has passed 10 Tbit/s peaks. https://ix.br/noticia/releases/ix-br-reaches-mark-of-10-tb-s...

dpcx• 6 years ago

I wonder what the comparison between served data is for IA vs Wikipedia - anecdotally, I feel like most of the data for Wikipedia is text based (plus some number of images), whereas IA includes (as mentioned in the article) audio, video, etc.

cperciva2• 6 years ago

60 Gbps is about 1/3 of the traffic served by a single Netflix CDN node.

notRobot2• 6 years ago

Source?

est313• 6 years ago

You are comparing to one of the largest bandwidth using services world-wide. Of course archive.org will lose out.

I should have added: 60 Gbit/s is a lot for a donation run service like archive.org. Not sure if there are donation run services on the internet with more traffic.

toomuchtodo5• 6 years ago

single Netflix CDN node.

Looks like they max out at 100 Gbps, or as low as 40 Gbps, depending on appliance model and link aggregation configuration. No argument either way, just thought it was cool info.

https://openconnect.zendesk.com/hc/en-us/articles/3600345383...

PudgePacket2• 6 years ago

I'd say the "bytes" being transferred are very different.

Netflix has a (relatively) small number of large content pieces, and a CDN node would hold the hottest items.

IA has 430 billion individual pieces of data, on average much smaller than a piece of Netflix content.

So each "byte" transferred from IA is "more work" to produce, and less likely to come from some kind of cache.

jonah-archive4• 6 years ago

It's harder when you don't make money, and everyone's not downloading the same things.

ptman• 6 years ago

does archive.org use cdn or is it all served from one location?

httpsterio• 6 years ago

Wikipedia is mostly text with some images. Internet archive offers every hung from websites and text to movies and games, which obviously are much larger and don't compress as nicely either.

simonmales2• 6 years ago

For the video and audio content. Could they take advantage of something like Webtorrent.

toomuchtodo5• 6 years ago

Every item has a torrent file, and can be retrieved with a BitTorrent client.

https://help.archive.org/hc/en-us/articles/360004715251-Arch...

simonmales2• 6 years ago

Brilliant!

Someone1234• 6 years ago

I donate to the Internet Archive (recurring). I just hope they stick to their core mission, and don't get sidetracked like Wikipedia has. They offer a valuable service and I believe will help historians in the future understand this point in the internet's life.

PS - Don't forget to pick an Amazon Smile charity and use the Smile.Amazon sub-domain. Donated almost $20 to the EFF that way.

notRobot2• 6 years ago

How has Wikipedia got sidetracked?

suyash2• 6 years ago

I believe the Author is referring to the phenomenon why Wikipedia has lost core contributors due to poor management. https://en.wikipedia.org/wiki/Wikipedia:Why_is_Wikipedia_los...

gruez• 6 years ago

Most of wikimedia's budget goes towards non-wikipedia related expenses: https://www.washingtonpost.com/news/the-intersect/wp/2015/12...

Alex3917• 6 years ago

They offer a valuable service and I believe will help historians in the future understand this point in the internet's life.

It's also essential for non-historians. A huge percentage of the info people need just to do their jobs or pursue their hobbies is only found on people's personal webpages, and those presumably all go away once the person dies.

PaywallBuster• 6 years ago

They already kind of have no?

They go from archiving the web (simple websites), itself a huge feat, to archiving a bunch of videos from justin.tv as long as they have 10 views?

There's more cases of websites shutting down, and AI feeling the need to archive everything.

Do we really need it?

tgsovlerkhgsel• 6 years ago

They increased the bandwidth and traffic immediately filled the new capacity. While this could be just a faster web site attracting more users / users were giving up due to slow loads before, it could also indicate that they have a lot of automated traffic that will consume whatever resources are available and can be throttled without endangering their mission.

I hope they'll look into it, find a way to identify that traffic, and then (ideally) classify it accordingly and continue to serve it only after all human traffic has been served (i.e. if they run into a bandwidth crunch due to bots, only the bots suffer, and they don't need to upgrade as quickly).

manquer4• 6 years ago

It could also be they are limited by the capacity, For me Wayback machine is always slow , although my bandwidth is otherwise not been a bottleneck. Perhaps increased capacity is getting utilized by both increased usage and higher performance of their services .

textfiles4• 6 years ago

We don't save logs of who hits us, but we watch who hits us. It's not bots. But good thinking.

twicetwice• 6 years ago

How do you know? Not doubting you, just curious as to how you figure out if a request is from a bot or not.

ufo• 6 years ago

The article doesn't mention why archive.org demand went up so much after COVID. Does anyone know?

gkoberger• 6 years ago

I'm not sure, but this article suggests people might just be online more and reading more?

http://blog.archive.org/2020/05/11/what-it-means-to-be-a-lib...

manquer4• 6 years ago

Not just reading, general increase in internet use, for example they have a lot of old archived games, many official covid health pages only track current numbers so wayback machine is the only way to track first / second derivative data points etc.

sp3323• 6 years ago

I don't know if this explains all of it, but they launched a National Emergency Library, where they pooled the DRM'd versions of ebooks from lots of libraries and removed one little bit of the DRM restrictions. http://blog.archive.org/2020/03/24/announcing-a-national-eme...

social_quotient• 6 years ago

What if there was a feature to load the pages without images? I use IA a lot but I rarely need the media stuff, at least initially. Speaking for myself I’d be happy to get the page without images and then click to optionally get the images. (Lazy lazy loading)

gojomo2• 6 years ago

Inline images (even across all Wayback Machine usage) are likely a tiny factor, compared to the rich-media downloads (audio, video) dominating this usage.

suyash2• 6 years ago

A suggestion around Donation, see if your employer will match it that way you can double the contribution: https://help.archive.org/hc/en-us/articles/360002059712-How-...

afpx• 6 years ago

What's the easiest way to seed the most popular content ala torrent?

__s• 6 years ago

ipfs: https://betanews.com/2018/08/09/decentralized-archive-org

ghastmaster• 6 years ago

I would love to donate bandwidth/storage. I haven't a clue how. I wish there was software I could throw in my server, set how much storage and bandwidth I can donate and run 24/7.

lowwave• 6 years ago

What if to have every item on archive.org also have a webtorrent. I know that webtorrent leaks your private IP, but if an item is requested a lot then it will at least save them bandwidth and be shared amongst people who are requesting the content.

wglane• 6 years ago

Donated.

VectorLock• 6 years ago

What a refreshingly simple and wholesome announcement.

x3blah2• 6 years ago

Archive.org works surprisingly well as a general purpose web proxy. Just prefix the URL, e.g., http://example.com, with https://web.archive.org/save/, e.g., https://web.archive.org/save/http://example.com

The aesthetic intrusiveness of the archive.org header and footer are minimal since I use a text-only browser that has no Javascript engine.

Sometimes I get "This url is not available on the live web or can not be archived." However this happens for only a surprisingly small minority of websites.

Rarely I find that /save is unsuccessful in which case I can still find past copies using something like

   curl -o 1.txt "https://web.archive.org/cdx/search/cdx?url=http://www.example.net&fl=timestamp,original" ;
   sed -i '/^[12][0-9]* h/!d;/^[12][0-9]* h/{s/^/http:\/\/web.archive.org\/web\//;s/ /\//;s/\r//;}' 1.txt

The limitation with past copies versus /save is that archive.org will not usually crawl past page one on websites with many successive pages, e.g., http://example.com/?page=2, http://example.com/?page=3, etc.

Has anyone ever considered mirroring archive.org, or parts of it, to other geographic locations.

Could this be done. Why or why not.

Springtime• 6 years ago

I use a custom browser keyword search to find existing archived pages before saving one, personally. Eg: ar <URL> for:

  https://wayback.archive.org/web/*/%S

I'd imagine it would be useful for IA to implement some message for scenarios where a page has already been saved within a certain timespan and provide both a link to the already saved version and offer to save again. As this would mitigate mass savings of an identical page that can occur when some popular link is accidentally shared with the /save/ URL instead of the static URL or when it's a popular page that people want to archive.

Archive.is displays such a message (to the effect of, 'this page was archived <date>, if it looks outdated click save') and also redirects to the most recent copy.

x3blah2• 6 years ago

That's a good point. I mainly use it for browsing websites that change daily as well as ones with many successive pages, e.g., 1, 2, 3, etc. that IA does automatically not crawl. Thus, not many existing copies if any.

HAproxy changes the Host header and modifies the URL. I can either use the text-only browser's http-proxy option or I can direct the request to the web.archive.org backend by adding a custom HTTP header to the request. If I am not mistaken, the so-called "modern" browsers do not have built-in capability to add headers.

S_A_P• 6 years ago

Not trying to brag or virtue signal here but this is one cause I really feel great supporting. This is potentially our archive of human history and collective knowledge.

fireattack2• 6 years ago

Kinda off-topic, but I noticed that it seems the owners of the sites can ask Archive.org to remove existing archives. I've lost a few I "created" myself this way.

Not say it's not reasonable, but how do we work around this? Any alternative services that are more "resilient" in this regard?

mmmBacon• 6 years ago

Maybe my sense of scale is warped but I’m amazed at how little BW the archive uses. Since this data appears to be averaged over a 1 minute window, I wonder what their p99 usage is. The article says they bought a 2nd router. Were they only using a single router before (no redundancy or protected BW)? I realize the Archive probably has a small budget but one can buy a 3.2T switch relatively cheaply these days.

PudgePacket2• 6 years ago

I'm curious about the outgoing bandwidth, for crawling :) !

sergiotapia2• 6 years ago

What's the endgame for archive.org? Are they expecting for there to be a breakthrough in storage technology?

I read somewhere that data creation is exceeding storage solutions' pace. Is this true?

What about a mesh of some kind where every person who install an application hosts bits and pieces of random data and serves it to whoever asks for it?

teekert4• 6 years ago

I clicked this article to look for ways to contribute.. bandwidth. But there is no torrent/ipfs like method to help, just cash. I'd be happy to run a service that would help Wikipedia, archive.org, ddg (any service I believe in) decentralize. I always keep my Ubuntu torrents open for launch days and some beyond. I have a server running and I don't need my 50/50 fiber all the time.

If I was a better software dev, I'd try to make a daemon which I can feed a list of websites which I want to support with my caches that can be upgraded (like the DNS system?). Like a folding at home for bandwidth.

TeMPOraL8• 6 years ago

just cash

This is almost always the best and most efficient way you can help a cause. You donating infrastructure means donating them extra work - they'll need to integrate and keep track of it, as well as manage a relationship with you (in particular, a risk of problems with you or your service). Meanwhile, with extra cash, they can buy what they need and what integrates well, or pay the most effective specialists they need.

It's an universal principle. Cash is the best gift, if you're gifting to help. That's why e.g. Red Cross frowns at people donating stuff - it creates a huge logistical problem for them as well as depriving them of opportunity to boost the markets in the disaster-struck area. Or why it's better for you to donate money to your local homeless shelter rather than volunteer to work in it - if you work your job for extra hours instead, and donate that money, they'll hire workers better skilled for that task.

teekert4• 6 years ago

But I have bandwidth to share, not cash. I'm already at the lowest subscription level with my ISP. I understand the argument, but if there was a general technical solution this would be nice, right?

iofiiiiiiiii3• 6 years ago

Yeah, I am in the same boat - I have boatloads of unutilized bandwidth and storage space that I would love to contribute to a worthy cause like archive.org.

Ideally, I would like to be able to point at some RSS feed of torrents of relevant archive.org collections (whatever the curators believe popular enough to benefit from peer-to-peer distribution). I could then set a torrent client to donwload and seed everything on this feed.

whanamura3• 6 years ago

Hey, we are working with Filecoin.ai on a pilot program where their new minors in Filecoin Discover will host some of our open data sets like Prelinger Films and .Gov web pages.

It will be an additional back up to our copies. We're hosting a DWeb Meetup to share the latest in Filecoin and Storj (two decentralized storage providers we're experimenting with.)

It's Wed (tomorrow) 5/13 at 10 am Pacific, 5 PM UTC https://www.eventbrite.com/e/dweb-meet-up-virtual-decentrali...

whanamura3• 6 years ago

Sorry, "we" being the Internet Archive.

jl6• 6 years ago

Mixed feelings here... I love the IA but I feel people should not use them as a streaming service. Their mission is preservation, and access is obviously part of that, but I would feel bad hammering them every day for general TV watching purposes.

sradman• 6 years ago

I assumed that their bandwidth needs were due to their web crawler rather than serving cached content. I guess I’m basing that assumption on my own usage patterns.

lihaciudaniel• 6 years ago

Sometimes I wonder if people keep track of internet history. For example, we have the writing how Rome and Republic of Rome was only because of Cicero otherwise we wouldn't be able to comprehend. I think the modern day Herodotus is this website right here (assuming it will survive centuries hopefully)

lkramer• 6 years ago

Crazy idea: Why don't ICANN hand over the running of .org to archive.org? It's a bit outside their expertise, but they do have knowhow on running tech to scale, and the extra revenue could be used to fund a very good cause.

Also, I'd trust these guys not to mishandle .org

(yeah, I know it's not going to happen)

jonah-archive4• 6 years ago

Also, I'd trust these guys not to mishandle .org

As much as I like the idea of running a TLD, if anyone gives me a TLD, I'm gonna put an MX record on it and be jonah@org. That's a promise.

EDIT: To be fair, I would probably be overwhelmed with the usual sense of responsibility and not do this. But, the temptation

Jleagle• 6 years ago

Would be a nightmare with all the crappy email validators out there..

sktrdie2• 6 years ago

Why don't they put their data dumps in an SQLite database and indexed for full-text-search https://sqlite.org/fts5.html

Then put the file in a torrent. Let the users seed it.

Users can use sqltorrent (https://github.com/bittorrent/sqltorrent) to query the db without downloading the entire torrent - essentially it knows to download only the pieces of the torrent to satisfy the query.

Every time a new dump is published by internet archive, the peers can change to the new torrent and reuse the pieces they already have - since SQLite is indexed in an optimal way to reduce file changes (and hence piece changes) when the data is updated.

I talk a bit about it here: https://medium.com/@lmatteis/torrentnet-bd4f6dab15e4

Would save Internet Archive lots of bandwidth and hassle

contravariant• 6 years ago

Seems that you're being downvoted at the moment because it looks like you're making fun of archive.org, but that article is actually rather interesting.

How does this compare to something like IPFS?

Aeolun3• 6 years ago

A 50PB sqlite database? I don’t think any filesystem would be happy with that.

sktrdie2• 6 years ago

Indeed. But nothing nothing stops them from storing torrents of torrents. The search index would be just that, the text index. Which would point to another torrent storing the actual content. Assets would point to yet other torrents.

Would be interesting to learn how it's currently partitioned. I would mimic the same portioning system but use torrent instead so users can help with hosting. And use sqltorrent to serve queries efficiently.

sersi2• 6 years ago

I wish that Archive.org would start collecting funds to finance an external backup site. I strongly feel that archive.org is an irreplaceable treasure and that making sure that it would resist natural disasters should be a priority...

driverdan• 6 years ago

That doesn't require a special fund, donate to them. They'd make backups if they had enough donations to cover it.

sersi2• 6 years ago

I actually already donate about 250 usd a year to them... But would much prefer if there was a special fund I would donate that would make that happen in the future

oknoorap• 6 years ago

We need an open source competitor of archive.org using IPFS technology

iofiiiiiiiii3• 6 years ago

Or an IPFS transfer/mirror mechanism built into archive.org.

whanamura3• 6 years ago

Hey,for those who would like to help the Internet Archive with their excess storage, we are ginning up a project with Filecoin.ai to store some of our open data collections like Prelinger films, .Gov data, some audio and texts.

You can get an update on decentralized storage at our DWeb Meetup tomorrow (5/13 at 10 AM pacific, 5 PM UTC)

https://www.eventbrite.com/e/dweb-meet-up-virtual-decentrali...

Thank you for helping us increase our bandwidth

Related Stories