How to spot first stories on Twitter using Storm

http://micvog.com/2013/09/08/storm-first-story-detection

by nkurz • 12 years ago

53 6 12 years ago

philip1209• 12 years ago

Having such a volume of input tweets streamed in real-time, FSD seemed like an ideal use case for Storm framework which can scale up the work by adding more resources.

This is not coincidence - Twitter open-sourced Storm.

peregrine• 12 years ago

Well technically Backtype open sources storm as part of the twitter buyout of backtype.

nathanathan• 12 years ago

For the problem of filtering out false alarms, this paper seems to have made some significant advances over the fscore for baseline nearest neighbor approach: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.221.... They report an average f-score of 0.541 on a standard TDT corpus. For comparison, their implementation of the NN algorithm had an f-score of 0.450. There's still a long ways to go on this problem.

shriphani• 12 years ago

I remember a similar talk by Miles Osborne @ CMU. Any chance you're from his lab?

mvogiatzis• 12 years ago

Yes, I implemented Petrovic's paper (http://www.aclweb.org/anthology/N/N10/N10-1021.pdf) from scratch but scaled it with Storm in a distributed fashion.

sytelus• 12 years ago

The method of Locality Sensitive Hashing using random projection is better described here: http://www.slaney.org/malcolm/yahoo/Slaney2008-LSHTutorial.p...

How to spot first stories on Twitter using Storm

Related Stories