Come and help save Posterous from oblivion

http://jacquesmattheij.com/come-help-save-posterous-from-oblivion

by jacquesm • 13 years ago

305 95 13 years ago

jacquesmattheij.com

dasil0032• 13 years ago

Stoked to help with this. Will put my new BT fibre to the test.

ersii19• 13 years ago

Bandwidth in itself is not per say needed, in this project. IP's however are very useful.

Feel free to stop by our project IRC channel at #preposterus (Note: It's not spelled pre-posterous) or our main channel at #archiveteam on EFNet

Samuel_Michon3• 13 years ago

"Bandwidth in itself is not per say needed"

I guess so; I have a 200Mbit fiber connection but surprisingly little is happening.

I started the instance 10 minutes a go and it has only uploaded 20MB. I've set 'concurrent items' and 'rsync threads' to the max.

jaryd• 13 years ago

BT fibre sounds pretty sweet :) I'll throw in my crummy broadband as well ;)

ersii19• 13 years ago

That 'crummy broadband' is very much welcome, and so are you.

In this effort, bandwidth in itself is not that important. Feel free to read some of my other posts in this thread regarding bandwidth and/or come join us at IRC (#preposterus at EFNet)

Zirro• 13 years ago

The large "Run your own ArchiveTeam Warrior!" link is currently just linking to a fragment identifier. I think you'd like to change it.

jacquesm9• 13 years ago

Good point, thank you, fixed now.

frasierman2• 13 years ago

I think I saw a comment somewhere about your IP being banned after an hour... anything we should do to avoid this? I'd hate to be scanning for 15 minutes only to be banned and not be able to help anymore.

jacquesm9• 13 years ago

I got banned after doing about 4.7 G, if that is done a few thousand times because of this article it would make a huge difference. And they eventually would have to lift the bans.

sp3326• 13 years ago

The bans nominally last 24 hours. There was a point where there were so many IPs running (from AWS servers) they overflowed the ban list and the bans were shorter!

frasierman2• 13 years ago

I didn't even think about AWS... someone should put together an article on how to set them up... I'd be happy to launch some instances for the cause

ersii19• 13 years ago

Everything does however count. And if you run a single instance on your home/work connection - that'll be a little.

Some is better than none :-)

subsystem• 13 years ago

Depending on your ISP, you can get a new ip by changing your mac address.

wheaties• 13 years ago

You're my new hero. I don't use it and I never used geocities but this is still awesome. Historians will thank you in years to come. Sociologists and such will praise what can be mined. And the lists go on...

sp3326• 13 years ago

Here's one of the best blogs about the Geocities data: http://contemporary-home-computing.org/1tb/

runarb• 13 years ago

The screenshot archive that page mention is fantastic: http://oneterabyteofkilobyteage.tumblr.com/

Bring back so many memories about the early days :)

btown2• 13 years ago

For those of us who might want to donate cloud computing time but have weak/memory-limited laptops, is there an EC2 image that we could fire up for the cause?

veeti• 13 years ago

The OVA image is a standard format. I'd imagine EC2 would support importing it.

btown2• 13 years ago

Sadly, it doesn't seem to (ERROR: Unknown disk image format: OVA). I wouldn't know the first thing about how to reliably convert the format.

ersii19• 13 years ago

There actually is an EC2 AMI available. I won't mention it here though. I'd rather like that you join us up on #preposterus on EFnet over ol' school IRC.

I'm using caution because we don't want to sink Posterous, since it's a very fragile beast, which we're blowing away the caches off of.

We're working on getting a FAQ section up on the project page. (http://www.archiveteam.org/index.php?title=Posterous)

jdrock• 13 years ago

@jacquesm please send me a list of URLs to crawl (10M+), and I'll set up an 80legs job to do this. shion - at - 80legs - com.

jacquesm9• 13 years ago

please ask on irc://efnet/#preposterus that's where the archive team guys hang out. I don't have a list of seeds but they may be able to figure out a way in which you can put 80legs to good use.

ersii19• 13 years ago

The problem is that Posterous is hard to crawl. For one; They'll continously and automatedly ban your IPs, even if you rotate over a lot of them. Two: Posterous can't take all of the requests.

We've (ArchiveTeam) unfortunally made Posterous unresponsive multiple times. So please be careful to not completely bring it down if you're doing a solo effort.

Please also bear in mind that it's not just to "chuck it into the downloader"..

ersii19• 13 years ago

Also, please use a sensible format if you're crawling/archiving this.

We're using WARC (Web Archive) which is an official ISO File Format standard - which the Internet Archive's Wayback Machine can use. It's also a pretty good and nice format for archiving web pages in general.

zopticity• 13 years ago

Stop being a necromancer and resurrecting a dead service. There's got to be a good reason why they killed Posterous, so let it die. Let it go; stop holding onto the past.

sp3326• 13 years ago

Twitter made the decision to kill Posterous when they bought it. It wasn't the founders' decision, or the users' decision, or the visitors' decision.

Nux2• 13 years ago

I've never visited posterous.com until today, but I fail to see why Twitter would buy it and kill it.. can't be that horrific. It's like buying a house and setting it on fire. What am I missing? Are they just killing off competition?

jacquesm9• 13 years ago

Please read the intro to the blog post and think again. I've seen these comments by the boatload when we took on saving geocities and you are simply wrong.

saraid216• 13 years ago

Is there a place I can find this Geocities archive? I apparently missed something I wanted to keep when I archived my own stuff, and I'm wondering if I can find it again.

ersii19• 13 years ago

This isn't resurrecting a dead service nor is it about that. This is archiving it and making the previously public data stay public.

Why would someone do that?

Well, a lot of people have poured their hearts out and made content that lives on Posterous. They might miss the "sunsetting" (asshole term) of the service and lose their content.

Think of all the dead links that'll be around after the service have died. Wouldn't you be able to read something great that was linked from HN a year ago? From the Wayback Machine or similar?

There's plenty of reasons to archive the web and the content that goes up (and down).

bjoernw• 13 years ago

You'd be preserving a piece of history with many interesting pieces of thought on it.

kevinalexbrown• 13 years ago

Whoa, instead of just lamenting the shutdown and how talent acquisitions are horrid, and how VC's won't do the right thing and companies like Twitter are killing innovation (you might believe these things to be true; that doesn't change my point), these guys (Jason Scott and the Archive Team and jacquesm) did something about it. And when Jacques couldn't do it himself he organized other people.

My favorite kind of leadership, and an example of the (double-edged) sword of instant communication: people can be rallied around time-sensitive causes, like SOPA or posterous shutting down.

I know this seems a little obvious, but it's striking how rare it actually seems to be. I'm curious why. Maybe it's just my perception and it's happening all the time. Certainly people are doing great things, but I'm curious why we haven't yet seen more specific, directed actions like this. Does it depend on relatively homogeneous communities like Reddit (SOPA) and HN (this)? If there is a proliferation of such communities, say subreddits or otherwise, could we expect this to happen more frequently? Do we want it to happen more frequently, or do we run the risk of, say, DHS running a pro-search campaign like China's 50 cent army? I'm just curious about why this seemed so striking to me.

This also fits in with an article I'd been meaning to read by Fukuyama[0] on social capital written 15 years ago.

The vice of modern democracy is to promote excessive individualism, that is, a preoccupation with one's private life and family, and an unwillingness to engage in public affairs. Americans combated this tendency towards excessive individualism by their propensity for voluntary association, which led them to form groups both trivial and important for all aspects of their lives.

Perhaps we'll see more of these spontaneous actions as voluntary associations are easier to make as the infrastructure that supports them (e.g. reddit) becomes more well known and fine-tuned.

[0] http://www.imf.org/external/pubs/ft/seminar/1999/reforms/fuk...

jacquesm9• 13 years ago

Jason Scott and the Archive Team should get all the credit here, they are the heroes.

hkmurakami3• 13 years ago

Well I personally wouldn't have found out about this effort if not for this post of yours, so you certainly deserve some fraction of the credit in my book :)

(I have everything up and running according to your short guide and am working on the posterous project, though my contribution will probably limited to under 100GB due to bandwidth cap considerations. Btw, how much disk space does posterous in its entirety take?)

dasil0032• 13 years ago

I'd say the reason it doesn't happen more is because it takes a ton of work and people are mostly self-involved. Activism is not what it once was (in the States anyway).

barce• 13 years ago

I wish more actions like this would happen, too. I'm glad that they at least happen. The technology to share work is there, and here's work that might be overshadowed at times, the BOINC project: http://boinc.berkeley.edu/ . Use your unused CPU cycles for the greater good.

MatthewPhillips• 13 years ago

I find this particularly disturbing:

> I made an offer to continue to host posterous.com and all the stuff on it but never received an answer.

Has Twitter completely lost touch with the web community? Is it possibly to get a response from them these days if you're not an advertiser?

corin_• 13 years ago

As someone who spends reasonably large amounts of money on digital media buys, I've found that most companies like Twitter (though actually I have no experience with Twitter, except as a user) certainly are a lot more friendly when you have money to spend.

Fun story about Facebook: when I first wanted to start spending decent amounts with them (decent not huge - $10,000s/month) a few months ago I literally could not get in touch with a single person. Even completing contact forms I wasn't hearing back from them. A friend who used to work for an SEO/SM company told me a name of someone to contact, I used LinkedIn's InMail (a paid feature) to message him, and 24 hours later I had 3 account managers (including a technical expert, a media strategist and an overall account manager), who answer calls to their mobiles at any time of day. Now I have a nice route to get answers on any topic, not just paid advertising, thanks to my spending. (Was actually shocked about how hard it was initially to make contact and give them money, too.)

danparsonson• 13 years ago

Here's a link to VMWare's OVF converter tool if you use VMWare instead of VBox: http://communities.vmware.com/community/vmtn/server/vsphere/...

sp3326• 13 years ago

The VMware Player (and Workstation, I'm assuming) will import it automatically. The only issue is that you have to choose a different connection for the second virtual disk. Not sure why that's a problem but moving it works fine.

niggler• 13 years ago

"I made an offer to continue to host posterous.com and all the stuff on it but never received an answer."

Have you tried making a public appeal (say, over twitter) to the Posterous owners? They may be able to talk with Twitter and arrange to capture the content directly

jacquesm9• 13 years ago

I will definitely make another one but it was here on HN to one of the former posterous owners.

hedgehog• 13 years ago

Torrent for the VirtualBox appliance:

magnet:?xt=urn:btih:b1b3df637bf9bb78f32e1667944535b60c9d37c1&dn=archiveteam-warrior-v2-20121008.ova

jacquesm9• 13 years ago

c8c86a1e225bb28ccdd229f6f27fe39ad65c9831

Sha1 of image, in case you want to verify.

lutusp• 13 years ago

This event, and many like it, only remind me that the modern Internet is about which groups you belong to, not who you are as an individual.

Websites owned and operated by individuals are now vanishingly rare, while aggregations of people -- Facebook, Twitter, et. al. -- have become the norm.

Example individual website: http://arachnoid.com/

ersii19• 13 years ago

Yes, and in my mind, it's a bit unfortunate to have it all centralized to a few major portals.

Especially since nothing lasts forever, like we clearly see here. When a giant falls..

Samuel_Michon3• 13 years ago

Well, this is exciting: we've just now passed the halfway point!

items: 1470951 done, 1467514 to do

[url to leaderboard deleted upon request]

(NB: it does look like this thing has been underway since Feb 27)

ersii19• 13 years ago

Please don't link to the leaderboard (the link in parent post), it is a bit loaded and fragile. That is used for all the ArchiveTeam Warriors.

Also, unfortunally - those numbers aren't totally up to scratch. Please hang on, we'll have a FAQ describing that in a moment.

Samuel_Michon3• 13 years ago

Thanks for the heads up. I've removed the link.

toomuchtodo• 13 years ago

SIR! (or mam) Permission to help scale out the leaderboard! (I'll hop on EFnet to ask)

guelo• 13 years ago

In the long-term web startups will be dead because it will be more and more obvious that it's imposible to trust your data to for-profit companies that simply cannot maintain your interest in mind no matter what promises they make. Everything important should be on open-source, community run, nonprofit platforms.

ricardobeat• 13 years ago

Or be hosted with a company that has a sustainable business model from the get-go.

calebegg• 13 years ago

"The long run is a misleading guide to current affairs. In the long run we are all dead." -- John Maynard Keynes

savrajsingh• 13 years ago

Awesome! Love the spirit of the effort, running a Warrior now for kicks. Just curious, isn't it possible for someone at Google to press a button and make this happen? :)

ersii19• 13 years ago

Most likely, yes. They probably have most of Posterous cached already.

It's however very unlikely that they'd just release that data. They probably have their own archive format and things as well..

dpcan3• 13 years ago

How does one opt out? I've migrated my blog to my own system so if I want to make changes to an old post I can, but I won't be able to control my content that you are archiving which is a problem for me. It's actually rather bothersome to me. I figured I'd just go make sure my blog is deleted, but it could be archived by now.

craigmc• 13 years ago

That is the world wide web. You published it and anyone could save a copy of what you published (right-click, save as) at any time. This is no different surely?

eli• 13 years ago

Sure, but it's still polite to observe robots.txt restrictions (or whatever analogous mechanism could work here)

kmfrk• 13 years ago

I can also see your deleted reddit comment, but that still betrays some understanding of privacy.

skeletonjelly• 13 years ago

And Facebook not actually deleting your photos when you click the "delete" button is ok too right? It's published forever?

dpcan3• 13 years ago

This archive is taking one more step - right-click, save-as, "re-post to the web".

jami• 13 years ago

Everything that goes online is kind of online for as long as the Internet Archive and Library of Congress decide it should be. But I started worrying about where my account info might end up (I should really worry about that before signing up for every dang thing...), so I just found this info on deleting one's account: http://posterous.uservoice.com/knowledgebase/articles/36544-...

Maybe you can get to it before the Wayback Machine does.

dpcan3• 13 years ago

Thank you, this method worked and my account is permanently gone now. I still run my blog at another location so I can maintain my content which I prefer to have some level of control over - even if it isn't total.

AlexMuir• 13 years ago

This is a great effort. But it makes me furious that the founding team can be so disparaging to their users.

Sachin Agarwal, you used this community to enrich yourself and further your own career. In return, at the very least, you owe an explanation for why such a convoluted effort has to be made to get this content off the servers.

It's also exceptionally discourteous to ignore emails from upstanding community members, but perhaps you missed these. But I know for sure that you will read this thread, so I'd love to hear why a database dump can't be provided, or a couple of IPs whitelisted to just rip through a scrape.

bherms2• 13 years ago

Why is it so important to save all this information? Seems like projects like archive are just contributing to more information clutter. We generate more information every single day, what makes a few million peoples blog posts so damn important? We can't just keep saving shit forever, though the progression of technology I guess makes it easier and easier, but eventually we're going to hit a limit.

markdown• 13 years ago

> We can't just keep saving shit forever, though the progression of technology I guess makes it easier and easier, but eventually we're going to hit a limit.

Yes we can. Through the 'progression of technology' as you put it, I believe we will never come close to any limit. Just look at how much data is stored in DNA, and you'll see that our current data storage technologies are relatively primitive.

ersii19• 13 years ago

How are we to know what's important and not? Surely, there's interesting content available at Posterous. Just to mention an example, CloudFlare's blog is hosted there.

Sure, there's plenty of spam accounts and crappy content - but that might prove worthful in the future. Maybe someone would study what kind of content we as a race were contributing to that kind of platform, in this day and age - maybe someone is researching the automated spam.

This is not really taking up all that much space, in this day and age. There's around 2.2 TB downloaded - it's mostly text and images. That's half a single 4TB drive. Not really storage capacity to fight about in my opinion.

bherms2• 13 years ago

Yeah I guess you're right about the storage piece, however, I don't think it's useful at all. We always live in the moment of "right now is the most important moment in history", when really most of the content we're saving is junk, and, as more and more of it compounds, more and more junk will just accumulate on the pile. I'd assume that 90% of what's in posterous is worthless, the other 10% is just people reiterating good points, but the key word is _re_iterating. Do we really need tens, then hundreds, then thousands of years of files of things people said on personal blogs in the past? Absolutely not.

cedricd• 13 years ago

I love it. This is creating a benign botnet.

mortenjorck• 13 years ago

Benign merely denotes non-maliciousness – I would call this a benevolent bonnet.

tectonic• 13 years ago

Are you guys archiving photos / images too? When I just wget the site those don't generally come down. I don't know if they're being loaded via JavaScript or a plugin.

To preserve my own blog I just saved it as a pdf with wkhtmltopdf, but it'd be nice to have a full HTML version.

http://blog.andrewcantino.com/blog/2013/03/12/archive-a-pdf-...

itsnotvalid• 13 years ago

I am guessing that if Posterous and the ones who is responsible ultimately, should be giving the data to archive.org directly. This is the only sensible thing to do. Of course, they have to clear copyrights first.

doki_pen• 13 years ago

Something like this would be cool as a chrome app. I'm not sure how complicated the job is, but if it's just a matter of hitting apis and queueing data for upload, it should be possible with local storage. You could even use subdomains to defeat storage limits.

mseebach• 13 years ago

It took me a bit of fiddling to get running on a spare Debian box, so I thought I'd share:

    $ sudo apt-get install virtualbox-ose
    $ wget http://archive.org/download/archiveteam-warrior/archiveteam-warrior-v2-20121008.ova
    $ tar xf archiveteam-warrior-v2-20121008.ova 
    $ VBoxManage import archiveteam-warrior-v2-20121008.ovf
    $ screen
    $ VBoxHeadless --vnc --startvm archiveteam-warrior-2

Hit Ctrl+A, D to exit screen and leave the VM running. From a non-headless box:

    $ ssh -L8001:localhost:8001 you@yourserver.com

Point a browser to http://localhost:8001

EDIT: Added 'screen' to steps. You're gonna wanna use screen.

ersii19• 13 years ago

Thanks a lot for writing that up. I'll be sure to add it to the FAQ/wiki on http://www.archiveteam.org/index.php?title=Posterous

jdboyd• 13 years ago

You could use:

    VBoxManage startvm archiveteam-warrior-2 --type headless

which would replace VBoxHeadless and remove the requirement for screen. You will need another VBoxManage command to setup VNC if you desire to use that.

josephlord• 13 years ago

Ethically this is far better behaviour than those who are shutting down the service and there is altruism rather than profit motivation BUT legally isn't this epic scale copyright infringement of millions of works created by thousands of people?

sp3326• 13 years ago

It's been discussed. Because it was published in a public forum, fair use is certainly a consideration. Is it legal to copy stuff from websites without permission? U.S. courts haven’t made a clear determination. Andy Sellars, a staff attorney at the Citizen Media Law Project, says he would argue that it counts as “fair use” under copyright law. However, he notes that the Archive Team’s torrents don’t offer a mechanism for copyright holders to demand that certain material be taken down, which could hurt its case in a court. http://www.technologyreview.com/featuredstory/426434/fire-in...

Come and help save Posterous from oblivion

Related Stories