TweetRex launched: Twitter friend recommender built on the Directed Edge engine

http://blog.directededge.com/2009/03/24/announcing-tweetrex-twitter-friend-recommender-built-on-the-directed-edge-recommendations-engine/

by wheels • 17 years ago

28 15 17 years ago

blog.directededge.com

uuilly• 17 years ago

Must be flooded. It hung for me for a while...

wheels6• 17 years ago

It takes a while to pull the data from Twitter. It actually runs around 500 worker threads spidering Twitter, and once the count gets up high some of the requests start timing out, so yeah, the whole process takes a bit.

Unfortunately, since it's a limitation in how fast we can pull data from Twitter, just throwing more hardware at the problem wouldn't help much. Performance may get better as we have a large portion of the Twitter graph cached.

We expected that it'd be a little rough today when it first goes up, but on usual days it should be able to handle a more casual load.

We debated if we should post it here since we were pretty sure it'd swamp the connection, but probably things will level out a bit by the end of the day.

Edit: Now it's hitting the VM pretty hard since once all of those threads are actually pulling, well, data, things slow down. I may set up another VPS with more RAM here in a bit and do a DNS swaperoo. We'll see how things go.

hajrice• 17 years ago

This really encourages my team and me to continue developing a major engine that'll parse content. It's so awesome that this product was built, but I'm not sure how popular it'll be. From my perspective, it has extreme pottential.

wheels6• 17 years ago

Up on a second server now:

http://tweetrextoo.directededge.com/

apgwoz• 17 years ago

There's an OAuth beta now. I'll wait until they make use of it.

wheels6• 17 years ago

Originally I wanted to do all of the password wrangling in javascript so that it never hit our servers. We only need it to make the follow buttons and the little viral message work (we ask for confirmation before posting and say exactly what it'll post to minimize the evil), but using HTTP basic authentication with Javascript seems to be a no-go, so we just bounce the request off our server. All of the data that we pull is just public data that's grabbed with our white-listed account.

I'll look into OAuth in the future. At the time I just saw in the Twitter API docs, "For the time being, HTTP Basic Authentication is the only supported authentication scheme."

FictiveEvan• 17 years ago

What problem did you run into doing HTTP Basic Auth with javascript?

dhimes• 17 years ago

When I clicked someone to follow, then came back to your page (on too), I had to log in again. I was away for only moments.

I selected don't touch my stream* because I had already tweeted.

sandGorgon• 17 years ago

how does directed-edge compare to LinkedIn's OSS Project Voldemort (<http://project-voldemort.com/ >) ?

champion2• 17 years ago

Cool, looks nice. There are several other friend recommenders out there: whoshouldifollow.com (which I helped build), Twubble, MrTweet.

I'd be curious to know more about the recommendation engine -- my recommendations skewed toward popular entities like "Whole Foods", "Zappos" and "DiningFever". Interestingly, although I follow a lot of tech/web/rails people, I didn't see any obvious folks in my recommendations.

wheels6• 17 years ago

We spider using random graph walks, and it makes sure that some x users in y directions and z steps away from you are spidered. However, this was the first time we'd tried to run the engine on an "incomplete" (meaning that we don't have the whole graph) data set and there are some quirks in the recommendations. It was hard to tweak the algorithm, because until there was a bunch of data, we weren't sure what tweaks would be necessary.

The skewing happens because we first check those x and y to see if we've already got enough data before running the spider. If we happen to have already spidered heavily in one subset of your friends, then you'll see the results skewed towards those until we end up spidering the others.

The other skewing, which other works better on complete graphs is that it tries to do some weighting based on how many incoming and outgoing edges users have (friends, followers), which if your tech friends have a whole lot of friends and followers and you don't, might be part of why they'd be less highly rated.

For the moment we're not really trying to compete with Mr. Tweet (though, when I tried their recommendations they were pretty bad) and whoshouldifollow (which does pretty well) -- it was just a neat way of showing off some of our engine and applying it to a different area.

champion2• 17 years ago

Thanks for the reply. Like the idea of making is simple for sites to incorporate a recommendation engine without reinventing the wheel.

epi0Bauqu• 17 years ago

A few suggestions:

1) I would have sent the tweet about your site IF I could have revised it, which wasn't working for me (not sure if it should have).

2) It would be great if you could expand people's stream to see more than one tweet (if wanted).

3) It would be great if you could show the score, and at the same time give many more recommendations (perhaps with a more button). That way, you can still claim it wasn't a great match (by showing the score) but still giving more to those who want them (like me).

wheels6• 17 years ago

All good ideas and things I had to fight off in one-more-thing-ism mode before we put up the beta. Those should all appear in a soon-ish update.

TweetRex launched: Twitter friend recommender built on the Directed Edge engine

Related Stories