mashraf

Is Google departing from just throwing white papers over the wall and let community figure out the implementation details? blaze white paper was dropped a while ago and there are already two clones in Pants and Buck at Twitter and FB. It would be interesting to see how far off clones are from original implementation.

I have never used Samza but have build similar pipelines using Kafka,Storm,Hadoop etc. In my experience you almost always have to do your transformation logic twice one for batch and one for real time and with that setup Jay's setup look exactly like Lamda Architecture with your stream processing framework doing real time and batch computation.

Using stream processing framework like Storm maybe fine when you are running exactly the same code for both real time and batch but it breakdown in more complex cases when code is not exactly the same. Let say we need to calculate Top K trending item from now to last 30 mins, One day and One week. We also know that simple count will always make socks and underwear trend for an ecom shops and Justin Bieber and Lady Gaga for twitter(http://goo.gl/1SColQ). So we use count min sketch for realtime and a sligtly more complex ML algorithm for batch using Hadoop and merge the result in the end. IMO, training and running complex ML is not currently feasible on Streaming Frameworks we have today to use them for both realtime and batch.

edited for typos.