Our product was ported and re-skinned inside Twitter's analytics offerings.
On the technology side a lot of our core tech is being used across different teams here. Storm is being used to help power applications for analytics, revenue, content discovery, anti-spam, search and more.
Also we're currently hiring engineers to work specifically on that technology so feel free to reach out to any of us from the BackType crew to get involved.
Cool, great to see you're making use of EDB. Would love to hear more about how you're using, how the transition was, etc. mm@backtype.com.
Fixed.
It's there now. Had to make some changes to distribute some of the work.
Getting throttled a bit right now by the github API — should be better soon.
We use Python for all our data collecting, general scripting, and some of our stream processing (though stream processing is moving to a more generalized framework that is Java/Clojure based).
Also, our underlying API (that serves results to all our products/API/etc) is a collection of Python servers written with Twisted and Thrift.
Thanks! If you're in the city come by the Lisp/Clojure meetup tonight — we're talking about Clojure at BackType, the tools we've open sourced and demoing our new stream processing framework.
Here's a really short slide deck with some details on how it fits into our workflow at BackType: http://www.slideshare.net/michaelmontano/designing-for-massi...
Will have lots more on our tech blog next week.
Wow, that is sketchy. That's on Twitter's end, something must have broke recently (today) because I've never seen that before on any of their auth pages. (mike from backtype)
SOMA, San Francisco — BackType
Hiring is the most important thing a startup does. We care so much that we created an app to help us reach hackers and let our friends explain why you should join us:
We're very flexible — it could be a standard school term, the summer break, or start tomorrow and last into the spring. Shoot me an email mm[at]backtype[dot]com if you have any more specific questions.
BackType, San Francisco.
Small team, hard problems, massive opportunity.
http://www.backtype.com/jobs or email me mm at backtype dot com
BackType (YC08) is hiring brilliant back and front-end engineers.
Why join us?
- we're very early on (3 hackers w/ revenues)
- tackling hard problems with big data at big scale (TBs of data, 100 node cluster, almost 1B monthly requests)
- chasing a massive opportunity (helping companies understand their social impact)
Our technology blog has more about what we're building and the tools we work with (http://tech.backtype.com). You can also learn more by checking out our presentation next week at the Hadoop summit, where we're talking about our tech alongside Facebook, Yahoo, Netflix and Twitter.
Email me (mm at backtype dot com) with links, a github profile, and why this sounds like fun to you.
The bot tweeted it last week: http://www.backtype.com/page/www.maximumpc.com%2Farticle%2Ff...
Heh — I noted in the post that your project placed so highly because of a bunch of bitly URLs being cut off by Twitter (you did well grabbing bit.ly/bk): http://www.backtype.com/page/github.com%2Fholman%2Fbrightkit...
I only queried for projects being shared in 2010. Looks like a Hacker News post actually drove a bunch of tweets for linenoise: http://www.backtype.com/page/github.com%2Fantirez%2Flinenois...
BackType (YC08) is hiring engineers to work on both the front and back end of our analytics product. We work mostly with Clojure and Python and use a mix of technologies from Thrift and Hadoop to Cassandra and graph databases. We're three people, we all code, and we work with TBs of data, manage a cluster of 100 machines, and build and scale products that serve hundreds of millions of monthly requests.
Checkout our tech blog at http://tech.backtype.com and contact me directly at mm@backtype.com
BackTweets has actually been around for a while too -- and was always a part of BackType Connect (which shows you other reactions including hacker news, digg, reddit, friendfeed). We even personalize results to show a viewer which of their friends shared a link: http://www.backtype.com/connect/danieltenner.com%252fposts%2...
We have a similar chicklet to encourage retweets; it differs in that the colors are completely customizable and it integrates automatically with google analytics. Another one of our widgets shows only the interesting tweets: http://widgets.backtype.com/backtweets?url=http://danieltenn...
Hi Brad, thanks for getting in touch with us -- we had an issue related to your domain specifically that is being addressed right now in addition to some corrupted indexes that were affecting all of our counts today. Both of these will be fixed very soon and our counts will return to being accurate.
BackType is hiring software engineers. Our tech blog has a small glimpse at some of the tools we're working with: http://tech.backtype.com
We're also contributing to some cool projects: http://github.com/nathanmarz/crane
We opened up an API for backtweets a while ago: http://backtweets.com/api
What would you build with it? Some people are using it in cool ways right now, e.g. hype machine uses it to power a twitter music chart.
There are a bunch of coworking spaces in the city. I think all have monthly fees, but you can drop in and check them out:
http://citizenspace.us/ http://www.parisoma.com/ http://www.sandboxsuites.com/
Panera in SOMA is right beside the caltrain and has lots of outlets.
You're right, it does increase the delay. It's more of an example of how Gearman can be used.
Also, I _hate_ the idea of running an MTA on 30 machines that shouldn't be sending mail.
Perhaps I'm getting hung up on the semantics. We use it more to farm out function calls, rather than passing content or delivering large messages. The one time I tried to use it for large amounts of data I did have issues with memory and latency.
For simple RPC-style use (millions per day) though, our server has been stable (no change in memory or latency) for months without a restart.
To clarify, Gearman is not meant to be used as a message queue, but instead it's a job server. The example in the post is an interesting use case (and convenient for us because we were already using it as a job server), but not really what it's designed for. I'll follow up with a post on how we use it to do work.
Are you now using Beanstalkd instead?
What kinds of jobs were you submitting? I've also run into problems using the python client handling errors, failover, and reconnects. But no issues with the C API, processing about 5 million jobs a day.
Absolutely, we're going to be regularly deploying improvements to our search. BTW, we're hiring, so if you or anyone else is interested in helping us solve these problems please get in touch.
Aditya, thanks for the feedback -- if anything this latest release reflects that we are much more focused on solving search problems as they relate to both consumers and businesses. Previously BackType was viewed as a toolset for comment authors, bloggers, and businesses -- features like subscriptions, claiming your own comments, etc still exist but they are more buried for those power users. We're working hard on improving our search :)
Yes, our API is live and documented here: http://www.backtype.com/developers/comments-connect
Also here's a great app that was built by a user of our API: http://convotrack.com/I1
In terms of widgets that pull all the conversation together, we just released a WP plugin that does that for posts (twitter, digg, reddit, hacker news, friendfeed, and related blog posts) http://www.backtype.com/plugins/connect. There is also a tool called ConvoTrack (it was built using the BackType API), it's a bookmarklet that shows you the conversation around any page. It can also be embedded on a page as a javascript button. http://convotrack.com/