jvaleski

check your facts by reaching out to Twitter directly.

I'm Jud Valeski, CEO of Gnip (jud@gnip.com). So we're all running on factual data, apparently a problem these days,...

Twitter's value added resellers (e.g. Gnip and DataSift) must... - sell filtered data at no less than $0.10/1k Tweets consumed. Gnip makes this price explicit, whereas DS, apparently (I can't fully tell so check w/ them directly) rolls that number into their costs, making it opaque.

- sell only to firms who will not display the Tweets in "public".

- sell only to firms who will not programmatically re-syndicate the raw Tweets further downstream.

the solution I used leveraged non-blocking I/O in a roundabout way. 150 separate procs (quick and dirty) per instance. no threads. I let the OS manage the I/O in that regard. My thinking was that if I had "a lot" of procs trying I/O... I'd get the same effect. again... quick/dirty.

I can confirm billions of files/keys. I went the quick/dirty route to parallelize the DELETEs and just let the OS manage the async I/O for me at the proc level. Each instance spun up 150 Ruby procs that issued the DELETEs (tight, serial, while-loop style), and ran them in parallel.

I spent quite a bit of time working w/ the S3 team at Amazon to try and come up with a better model, but we all agreed on my approach (mind you, it could likely be done more efficiently) as option 'a', and option 'b' was to delete the account and just let Amazon reap the data.

classic problem. xmpp was designed for this, but we've found it non-trivial (not rocket science either, but non-trivial) to setup/use. ejabberd is solid once it's up and running, but it's a black box once it's going (with exception of some logging).

IM is one of those things that once you start, you'll be endlessly piling on features. sure you can bootstrap IM with what you've got, but think about adding presence, file transfer, status/away messages, rostering, roster add/remove permissions/confirmation/approval.

I helped define the product spec for me.dium's sidebar and associated IM client/roster. massive time-sink, but, IM was core to the experience.

challenge is everyone's expectations around IM these days are defined by Adium, and AOL IM clients, and those are very feature rich products that have taken years to build/get right. that's what you're up against if you start building your own. xmpp tries to bridge that gap.

I'd make very sure you want to walk down this path at all. After doing that, if the answer is yes, then I think you should leverage xmpp and bit the bullets that it winds up firing at you; rather those then reinventing the wheel. you might want to look at openfire as well, and cisco/jabber.com can sell you hardened clustering solutions if/when you need them.

I'm now at http://gnipcentral.com and we do xmpp for message routing (not IM). we've been using ejabberd for awhile (in a very limited manner).

haven't looked at meebo's community IM thing, but that sounds like it might be spot on.