TheTechRobo

Browsertrix (and Webrecorder tools in general) also violate the standard by modifying response data. It's supposed to be the raw bytes as they are sent over the network (minus TLS).

The entire WARC ecosystem is kind of a mess.

Unfortunately the Raspberry Pi version isn't supported anymore because their version of Wget isn't well-tested on ARM. I'm surprised it's still there in the repo.

You can get Docker to emulate x86, though.

That doesn't really confirm that, IMHO. The line before claims the opposite, in fact: "The Internet Archive has been around since 1996, and while that does not guarantee anything, it shows continuity of support and strong commitment to digital preservation with as much access as possible."

Everything transmitted over HTTP. That includes cookies, passwords, etc. You need to be careful when writing WARCs.

Yeah, the format really needs an update. For starters, WARC only officially supports HTTP/1.1. Webrecorder has started faking HTTP/1.1 data in WARC files in order to save other versions, but I don't think faking data is great for an archival format, especially if it isn't standardized.

ArchiveTeam's general idea is to archive it first, and make it usable later. Wayback Machine is generally how stuff is "made usable", but in cases like this, a searchable index might need to made. (That's been done before, though - it's not like it's never happened.)

Yes, what is your point?

Archive Team hasn't really done much yet - all they've done is an ArchiveBot job, which isn't getting anywhere because of DPReview's ratelimits, and some discovery scrapes. This is probably a different issue.

You select the project you run at any given moment. You can also select Archiveteam's Choice, which is usually the most urgent or the one with the most IPs needed, but if you do want to select it yourself, you pick the project.

Similar: github.com/InternetArchive/warcprox

There's github.com/ArchiveTeam/grab-site, but unfortunately it's not maintained very well.