Tell HN: My 'mystery project': I couldn't sleep, so I backed up GeoCities

http://www.reocities.com/

by jacquesm • 17 years ago

250 65 17 years ago

jacquesm20• 17 years ago

For all of those that were wondering what my 'mystery project' was, it's an all out effort to backup all of geocities.com in 6 days before closing time.

It was a lot of hard work, with the help from Abi and some others.

I think we got most if not all of it.

The account restoration process will probably take the better part of the week, there is just too much raw data to do it all in one go. There are still a whole pile of integrity checks to be done and broken links to be repaired.

Please be kind to the server, it is still doing a lot of very hard work in the background. On the homepage there is a status indicator that shows you how many accounts and files have been restored. How many accounts and files eventually will be restored I can't tell you right now but my guess is that we've managed to save a very large portion of geocities.

abyssknight• 17 years ago

Great idea! My wife was just lamenting over the loss of her highschool web-years. Hopefully they'll live on through reocities. Nice design too, especially considering the time crunch.

jacquesm20• 17 years ago

Abi did an awesome job, he did the whole thing under 5 hours.

I have no idea how long it will take to process all the raw data, it's spread out over a whole pile of machines right now, I'm pulling it in batch by batch to integrate it in to the main site.

But it's as close to a 'drop-in' replacement as I could think of.

RyanMcGreal• 17 years ago

> To fix links pointing to old GeoCities pages, we provide you with a small Firefox Greasemonkey script.

This is a nice touch.

textfiles4• 17 years ago

Hi, jacquesm.

Huge props to you for taking this initiative, but are/were you aware of Archive Team and our process of backing up Geocities since April? We found ways around the bandwidth limit months ago, and will be mirroring/distributing the data as well. We own http://geociti.es, for example.

We're somewhere in the 1TB range of data, and we're still finding new stuff, by the way.

jacquesm20• 17 years ago

I think I have 2+T all together right now, and I did that in 6 days, makes me wonder how there can be such a big difference ?

Anyway, feel free to 'complete' your set from mine, but let's please coordinate that so we don't kill this poor little server :)

textfiles4• 17 years ago

Well, there's no way we're going to coordinate without a rsync being somewhere in the mix. There's no rush, we can discuss it after the screaming dies down. I'm interested as well. At worst, it means we saved even more data, which works for me.

tibbon2• 17 years ago

I remember Jason (of the Archive Team) telling me that Yahoo! had very little bandwidth that they were allocating toward Geocities and that downloading was horridly slow. Getting even 1MB/s was nearly impossible.

I'm wondering if Yahoo! increased the available bandwidth (or maybe everyone else just stopped using it, increasing what was apparently available) so that when you got to it then it was nice and zippy compared to when the Archive Team hit it earlier in the year.

scumola• 17 years ago

http://archiveteam.org and http://archive.org have been working together since May backing up http://geocities.com http://archive.org has done two full crawls using a shared seed list that http://archiveteam.org and http://archive.org have been modifying and sending back and forth since there isn't an index for geocities anywhere to be found. I'm suprised that the http://reocities.com guys got a full crawl done at all in only 6 days.

jacquesm20• 17 years ago

Let's say it's 2T of data, that's 16,000,000,000,000 bits.

At 50 Mbits / second that's:

16,000,000,000,000 / 50,000,000 = 320,000 seconds.

That's 88 hours. I used a little more than that because I was off to a slow start and messed up on the filenames (I never expected them to be case insensitive), and there is the bloody bandwidth limiter at Yahoo that keeps tripping you up.

I really don't want to see the message 'Service Temporarily Unavailable.' ever again.

david927• 17 years ago

the reocities guys

I think that describes Jacques pretty well; he's a substantial team unto himself.

marcamillion• 17 years ago

Good stuff Jacquesm. That's one of the most beautiful things about the internet, things can literally live on forever.

Simple as you take it, in this one thread alone there are at least two copies of significant portions of geocities, and I am sure there are others out there (not even aware of HN) that have done the same.

Oh how I love technology.

teeja2• 17 years ago

WP mentions a couple more sites doing some archiving:

http://en.wikipedia.org/wiki/Geocities#Closure

jonknee2• 17 years ago

I wish I could remember what my Geocities site was called. It was before Yahoo! bought them, so it has been quite a while.

jacquesm20• 17 years ago

give me a segment of text that was in there and I'll scan for it.

ars2• 17 years ago

I don't have a geocities page, but that is an awesome offer!

So many times I hear people saying: I had a page, but can't remember what it was.

soult2• 17 years ago

I remember that I wrote "Under construction" and "Please visit my guestbook" as a link on my site. Can you find it?

jonknee2• 17 years ago

I've been racking my brain and it's a lost cause. I think it was my first dive into websites and I seriously doubt there was anything of consequence (I was in middle school). Certainly nothing I can remember other than I know I had an account and remember having to choose a neighborhood. I appreciate the offer though.

Batsu• 17 years ago

Mine was in the Bunker! It was related to Red Alert 1 and had lots of Under Construction images :)

mahmud• 17 years ago

Hah!

Here is the war journal:

http://www.reocities.com/newhome/makingof.html

:-)

[Edit:

Quote of the day.

"It doesn't matter what you do with apache, if there is a problem you can always solve it with mod_rewrite. The question is how." -- jacquesm]

recurser• 17 years ago

That was a great read - nice work jacquesm!

teeja2• 17 years ago

Interesting!! Bookmarking to see what other gremlins popup!

milkshakes3• 17 years ago

Very interesting. I'm unclear about one thing though -- were the bandwidth caps per account, or per user/session + account?

If it's the latter, did you consider distributing the crawlers to other users? Some sort of system where people with spare bw/clock cycles can do your work for you and free up your bw/clock cycles to receive and parse the data? Would writing something like that up have taken more time than it would have saved?

Regardless, congratulations on your accomplishment. It really is impressive.

jacquesm20• 17 years ago

Per account, the same user session would be able to see other accounts.

tibbon2• 17 years ago

Is there any small chance that you could release some of the code that you used to scrape it? I'm interested in archiving some site and wondering what you used to execute it? Just scripting a lot of wgets?

jacquesm20• 17 years ago

Yes, just a bunch of wgets. That's the principle anyway.

But it is quite a bit more involved because you somehow have to avoid duplication and retrying of stuff that simply doesn't exist. Then there's the problem that the urls weren't case sensitive, which causes wget to retrieve much more than necessary.

The code I wrote is pretty geocities specific, I highly doubt it has any value outside of that (other than a sustained DDOS maybe ;) ).

flooha• 17 years ago

Just a heads up: The "some interesting pages" link (http://reocities.com/tablizer/) on this page (http://reocities.com/newhome/makingof.html) returns a 404.

jacquesm20• 17 years ago

Don't worry, I've got them.

Restoring all this is going to take some time, it's spread out over a number of machines right now.

This is the master copy:

http://org.reocities.com/tablizer/

But that does not include all the other boxes, just this one.

edit: Ok, it's fixed now. Thanks again!

tomjen2• 17 years ago

Super nice work.

I wonder about two things:

Did anybody try to contact Yahoo to get a copy of the server content?

If you where running out of machines, why not try Amazons ECC? Would be pretty sweet having web 1.0 saved by web 2.0 tech.

jacquesm20• 17 years ago

Because it's pretty expensive for bandwidth intensive applications.

It they were anywhere near competitive I would have signed up a long time ago, as it is there really is no point.

diegomsana• 17 years ago

Softlayer.com cloud servers have free incoming bandwidth, and outgoing traffic is cheaper than ec2.

lloydarmbrust2• 17 years ago

I love the internet for things like this. Awesome job. I remember back in the day hoarding hundreds of geocities accounts to store and distribute MP3s . . . and of course I owned the copyrights to all of those. . . .

pbhjpbhj3• 17 years ago

If those files are still there then jacques is going to be serving those from his own pages very shortly ...

lloydarmbrust2• 17 years ago

Ha. That's great. Too bad I didn't have gmail back then, because my site names would totally be archived.

ComputerGuru2• 17 years ago

FYI: In "Making of" you're linking to http://reocities.com/tablizer which is a 404.... probably want to fix that ;-D

jacquesm20• 17 years ago

Ok, I bumped that one in the restoration queue.

Should have thought of that before, there was another mention of it below but I didn't think I'd be able to work around without messing things up.

thanks!

ComputerGuru2• 17 years ago

Awesome work. Seriously, mad props are in order for a bang hack job.

Great project, mate!

jeroen• 17 years ago

Quite an achievement in such a short timespan!

Nitpicks: the frontpage says "an verification method"; should be "a". And, of course, validation: http://vldtr.com/?key=reocities.com

jacquesm20• 17 years ago

Hey Jeroen,

I fixed that thanks!

As for validation, I'm painfully aware of it, that's entirely my doing not Abi's. I will fix those errors asap but I have to concentrate on getting the user data in there right now, which is still quite a job.

The design was imported in a great hurry and I absolutely suck at CSS and anything else that is design/formatting related.

Give me tables any day :)

But I will get around to it.

If you feel like helping out shoot me an email ;)

_cbsz• 17 years ago

You can scroll back for a bit in screen. C-a [ (or, apparently, C-a Esc) goes into copy mode, in which you can scroll using the arrow keys. Exit copy mode by wailing on Esc until it gives up. Be careful not to leave a screen sitting in copy mode and expect the process in the screen to keep running.

zandorg• 17 years ago

I think the Internet Archive got involved in this too. I asked the head of the IA if they could just get Yahoo to give them the hard disks and stuff them wholesale in the IA. But in the end, the IA put a 'spider' page on one of the main Geocities FAQ pages on Yahoo, which is not bad I guess.

The IA is distinct from Archive Team BTW.

pbhjpbhj3• 17 years ago

What's the copyright position here?

In the UK transitive copies for the purposes of display, caching &c., have been cleared as non-infringing actions. Googles caches link back to the original author by way of attribution for example. But archiving and reproduction without attribution in any way?

Also whilst Google may have been given a pass in robots.txt (were sub-sites allowed individual robots files? I never had one) to crawl the site declaring oneself as Googlebot in order to spider and archive the whole site could well show bad faith?

Just wondered if you'd discussed the copyright position, perhaps with Geocities. Maybe there was a disclaimer that effectively released content as PD, I doubt it though.

A brief discussion on webmasterworld, http://www.webmasterworld.com/foo/3898789-2-30.htm , but more interestingly an idea to "rape" geocities for content for ad serving sites, see http://ducedo.com/free-content-geocities/ .

Someone on webmasterworld considered whether Google might back-rate based on content so that highly rated content pages that disappear from Geocities (&c.) could be given a boost in the SERPs.

OT: did you use the current username based addressing too or are you only linking the old "campus" names, can't remember mine, was in RT somewhere IIRC.

aminuit• 17 years ago

Do you plan to host reocities indefinitely, or is this just a stopgap measure before you can donate the collection to another organization?

jacquesm20• 17 years ago

I can host it just about forever, I own & operate ww.com, which has a fairly large traffic bill anyway.

I figure if I put some 'friendly' ads on it the thing should pay for itself and that's good enough for me.

The kind of corporate superstructure that Y! puts on top of its products is what makes it inviable, not the concept of free hosting by itself.

If you figure that bandwidth in bulk costs around $3 / Mbit / month then you can serve an awful lot of pages to make back that 3 dollars.

Geocities pages weigh in at about 25K a piece from my meagre sample, so based on that cost / Mbit that's 13 million pageviews + a bit thrown in for server depreciation.

I'm not too scared about doing that.

If the need arises I can put the whole thing in a foundation to keep it alive forever.

PostOnce• 17 years ago

I'm paranoid, and my first thought is that if you put 'friendly' ads on there, it'll only be so long until someone comes along and calls what you're doing profiting from copyright infringement. What's the deal with copyright on geocities stuff? I'm not well-versed in this matter.

schindyguy• 17 years ago

it looks like someone already pointed it out, but you should def talk to the archive team on irc http://www.archiveteam.org/index.php?title=IRC_Channel

They helped me out with an idea I had a while back on shortened urls. The same thing needs to happen with shortened urls cause once that shortened url service goes down, all the links are lost...

bitwize• 17 years ago

Now do you need to buy more print cartridges?

http://www.penny-arcade.com/comic/2002/7/1/

Tell HN: My 'mystery project': I couldn't sleep, so I backed up GeoCities

Related Stories