Waybackpack: download the entire Wayback Machine archive for a given URL

https://github.com/jsvine/waybackpack

by ingve • 10 years ago

261 40 10 years ago

github.com

robbiemitchell• 10 years ago

I was looking for something like this literally two hours ago. Thanks!

bakztfuture3• 10 years ago

Wow this is amazing!!! I've built my whole site around showcasing wayback content: http://www.StartupTimelines.org/

This definitely makes it a whole lot easier, wish I had access to it from day 1. Great work guys/gals!!

Shameless plug, if you found it interesting, please consider donating:

https://www.tilt.com/tilts/startup-timelines-support-fund

http://archive.org/donate/?

Zekio• 10 years ago

Surprising, how few changes some sites have over the years

WA• 10 years ago

My design is 5 years old. I changed a few small things, and while I'd love to make a redesign, I also know that users HATE such changes.

Also, a few new users even compliment me on the great design. I guess it isn't that bad ;)

brassattax• 10 years ago

First visit...

"Sorry, you've already viewed 3 Startup Timelines"

bakztfuture3• 10 years ago

Sorry about that - this is a known bug we're trying to fix. If you refresh or view incognito it might let you browse after a minute or so. This is no way intentional to try to get you to sign up

TazeTSchnitzel• 10 years ago

How much load does this place on the Internet Archive? It'd be a shame if this thing's access patterns caused them trouble.

nacs• 10 years ago

Agreed. From a quick look at code, it seems it just fires off every fetch request immediately after another compeletes.

Hopefully it gets patched to have a built-in rate limit (X requests per minute/hour).

derefr• 10 years ago

If it's already serial, and only works with one backend (rather than being an arbitrary mirroring tool like wget), then the Wayback server can easily "express its preferences" for rate-limiting by adding artificial delay to request-responses that pass the rate-limiting threshold. Backpressure shouldn't be the client's responsibility.

(It only is traditionally, because so many sites do nothing to protect themselves from "being too nice", so arbitrary-backend mirroring-client devs allow their users the option to ask for less than they want. This isn't a sensible protocol design, on either side; it doesn't optimize for, well, anything.)

greglindahl5• 10 years ago

If I read the code correctly, it's one-at-a-time? Which minimizes the stress; if we're slow, it'll slow down.

It'd be nice if it had identification in the UserAgent, so that we could complain to the right people if it was a problem.

jsvine2• 10 years ago

Hi, Greg! Library author here. I'd be happy to add a configurable UserAgent. Perhaps the default would be a generic "waybackpack" but could be configurable to add contact info for the user. Does that sound about right? Prefer a different approach?

And, yep, the library is intentionally designed only to request one snapshot at a time.

anc84• 10 years ago

IA is very supportive of automated access, they even have a post about how to use wget to batch download lots of items: https://blog.archive.org/2012/04/26/downloading-in-bulk-usin...

Granted, that is not the Wayback Machine but I am sure they love people using them.

nxzero• 10 years ago

Always been confused by how the wayback machine works. Feel like if they were able to partner with browsers to anonymously hash of content and discover new content combined with do a better job doing version control that there index would be a lot big and granular too.

greglindahl5• 10 years ago

The Wayback Machine crawls stuff based on popularity (Alexa top million), search engine metadata (donated by blekko), the structure of the web, and the desires of our various crawling partners, ranging from the all-volunteer ArchiveTeam to 400+ libraries and other institutions who use our ArchiveIt system. And, finally, there's always the "save page now" button at https://archive.org/web/

There are big privacy issues to getting data from browsers. A lot of websites depend on "secret" URLs, even though that's unsafe, and we don't want to discover or archive those. That means we need opt-in, and smarts.

We do have a project underway with major browsers to send 404s to us to see if we have the page... and offering to take the user to the Wayback if we do.

sp332• 10 years ago

That last feature is great... And reminds me I need to install Resurrect Pages on my new PC!

toomuchtodo2• 10 years ago

Is there a JSON API call that can be made to archive.org to archive a provided URL and get a success/fail response back?

MoSal• 10 years ago

Are there any plans to support archiving Web 2.0 pages?

More and more people are starting to rely on "archive.is" as it handles Web 2.0 content without issue. But I'm concerned about the survivability of that survice, and whether it can handle big growth.

pjc50• 10 years ago

Does this let you get at material which is hidden due to the current robots.txt, even though it wasn't in force when the site was crawled?

mirimir• 10 years ago

That would be wonderful!

Also stuff that's been censored for other reasons.

speeder• 10 years ago

I wish I had an OS now that I could run this... I wanted a tool like that for a long time, so I can reconstruct some of SimCity series documentation (Maxis had a sort of tradition of team members writing as "PR stunt" detailed accounts of their work on the game, and the few bits I could scavenge from Archive.org, because this is not available on EA site anymore, had been great to help modding efforts, and in my effort to "restore" the SimCity 4 game to work on modern OSes)

chungy• 10 years ago

What OS might you be running that you can't run Python on it?

Negative1• 10 years ago

Is there a way to download content using this? There is a zip I'm trying to get from a particular site and it keeps failing due to some kind of download cap.

Using --continue with wget doesn't work (I'm guessing they turned it off).

shaunpud• 10 years ago

I've been using this with great success too; https://github.com/hartator/wayback-machine-downloader

sanbor• 10 years ago

Is it there any way to download the assets of the website too? Right now the html has urls pointing to archive.org.

genop• 10 years ago

Internet Archive has an HTTP header called "X-Archive-Wayback-Perf:"

I can guess what it means but maybe someone here has some insight?

It certainly looks like their Tengine (nginx) servers are configured to expect pipelined requests. It has no problem with greater than 100 requests at a time. See HTTP header above.

Downloading each snapshot one at a time, i.e., many connections, one after the other, perhaps each triggering a TIME_WAIT and consuming resources, may not be the most sensible or considerate approach. If just requesting the history of a single URL, maybe pipelined requests over a single connection is more efficient? I'm biased and I could be wrong.

However their robots.txt says "Please crawl our files." I would guess that crawlers use pipelining and minimize the number of open connections.

I have had my own "wayback downloader" for a number of years, written in shell script, openssl and sed. It's fast.

IA is one of the best sites on the www. Have fun.

hartator• 10 years ago

Ha fun, I've made a similar tool not so long ago: https://github.com/hartator/wayback-machine-downloader/

Kinnard3• 10 years ago

You're halfway to a blockchain!

j_s2• 10 years ago

One area I'm interested in is a legally binding verification of content at a specific point in time - for example: tracking changes to a breaking news article on CNN.

I'm not sure what technologies would be required to implement something like this, but I feel like the Internet Archive would be important and BitCoin might be away of encouraging verification from a globally distributed network of 3rd-party verifiers.

rakoo• 10 years ago

This idea has already been explored, here's one implementation: https://proofofexistence.com/

Kinnard3• 10 years ago

You should check out zeronet: https://zeronet.io You should also check out 21's new pay-per-call api features:https://21.co/features/

greglindahl5• 10 years ago

People use the Wayback Machine like that in court cases all of the time, mainly for patent prior art.

Waybackpack: download the entire Wayback Machine archive for a given URL

Related Stories