Twitter's Storm (complex event processing system) is now open source

by harrigan • 15 years ago

461 56 15 years ago

harrigan• 15 years ago

Some background: http://engineering.twitter.com/2011/08/storm-is-coming-more-...

mhd• 15 years ago

I just knew that someone would use the "a storm is coming" quote. Let's see if this piece of software will shake the universe…

jevinskie• 15 years ago

I'll be taking a look at this today. I'm most excited about it's fault tolerance features. If this sufficiently abstracts out the details of providing robust fault tolerance, it could be a great tool to use with cloud computing.

DevX101• 15 years ago

What are some potential applications of this?

tptacek2• 15 years ago

Any big data application where the majority of the working set changes continuously and requires more processing than just "bucketing".

The example Twitter gives here is for trending topics, but Storm is basically to message queues what Rails is to CRUD web apps, and so you can draw use cases from eg everything Tibco and JMS are used for today.

Storm is nothing you couldn't have done a year ago, or ten years ago, with a message-oriented architecture. But it has a very attractive feature, which is that it bakes in all the fiddley details and problem solving Twitter did while scaling it to their architecture. Systems like this tend to be easy to prototype and a nightmare to mature and manage, so that's not a small feature.

ethank2• 15 years ago

By their very nature, decentralized processing systems have exponential complexity, so this is very welcome to see released. But oh how I hate having to go back into Java :)

scotto• 15 years ago

Yeeeeessssss! Thank you Twitter!!!

michaelbuckbee• 15 years ago

It is a little buried in the github wiki, but this 'Rationale' page is a good overview of the project - https://github.com/nathanmarz/storm/wiki/Rationale

andrewcooke• 15 years ago

thanks. given that, can anyone answer the following?

- if there's no intermediate queuing, how are "overloads" handled? what if the system can't handle a transient load? is there some kind of flow control signalling back up the path to control the rate (i assume not!)?

- what happens if a bolt accepts data then crashes before processing. is that data resent? if so, how does the system handle bolts that are not idempotent? (the problem is that if you assume that data is not handled until the handling process terminates then you may send duplicates; if you assume that data are handled if they are received then you lose data on crashes).

- how can "fields grouping" scale? surely if you add more bolts they won't receive any messages unless a new field appears.

- is there some kind of automatic restarting? how are failures handled? [ok, this is handled by supervisor nodes and is described in the link above. although it's difficult to understand how they can be stateless. what happens to the managed processes if a supervisor dies?]

thanks again! [edit: thanks for the great answers; should have guessed the consistent hashing one ;o]

nathanmarz11• 15 years ago

You generally feed a Storm topology with a queueing system at the source. For example, we use Kestrel to feed Storm topologies. There's a setting called "TOPOLOGY_MAX_SPOUT_PENDING" which controls how many tuples can be pending on any given spout task at a time. This is more than sufficient for controlling bursts of data. If things back up, they will queue on the queueing server (s). In the future (not for a few months, at least), Storm will have auto-scaling where it will automatically scale the topology to the data.

If a tuple fails to be processed, the tuple(s) that triggered that tuple are replayed from the spout. See https://github.com/nathanmarz/storm/wiki/Guaranteeing-messag... for more info on that. In that sense, Storm is an at-least-once delivery system (but messages are sent more than once only in failure scenarios). It is up to you to architect your systems to handle this. The approach I take is to build systems using a hybrid of batch processing (Hadoop) and realtime processing (Storm). With batch processing, you can run idempotent functions even with duplication, which lets you correct what's happening at the realtime layer. In that sense, Hadoop and Storm are extremely complementary. Here are some slides from a presentation I gave about this technique: http://www.slideshare.net/nathanmarz/the-secrets-of-building...

Fields grouping uses consistent hashing underneath. So if you redeploy with more parallelism, it scales naturally and easily.

If a supervisor dies, it starts back up like nothing happened. Most notably, nothing happens to the worker processes. All state is kept either in disk or in Zookeeper. All daemons in Storm are fail-fast, and the Supervisor uses kill -9's to kill workers, so that makes things extremely robust.

I hope that answered your questions!

wh-uws• 15 years ago

Hey Nathan, you should incorporate this page in the Readme markdown so people can get an overview on the front page

buss• 15 years ago

If you don't want to read through the wiki, here's what I've gathered (though I may have misunderstood what they're doing).

This looks like a workflow management system, where you define a dependency graph and their system automatically puts messages in queues, pops them, and executes a step. It seems like it solves the boilerplate part of distributed computing - managing message queues and fault tolerance. Please correct me if I got this wrong or missed something.

ChrisArchitect• 15 years ago

"distributed computing boilerplate" ha, I like it

bergie• 15 years ago

Sounds a bit like http://en.wikipedia.org/wiki/Flow-based_programming

I'll have to read through their stuff to see if they have some interesting ideas for my FBP implementation :-)

kragen• 15 years ago

It does, doesn't it? I still haven't made it through the FBP book, and I'm having a hard time with the author's savior-of-the-world attitude.

narrator• 15 years ago

Hmm... So you're saying it's a light weight version of Apache Service Mix/Camel? The same way Kestrel is a lightweight implementation of JMS.

grantjgordon• 15 years ago

This is fantastic! My mind is spinning with the industries that you could benefit from this, but didn't have the time/resources/focus to roll this sort of (very difficult to scale) system on their own.

phren0logy• 15 years ago

Woah. Launch a Storm cluster on AWS with one line:

https://github.com/nathanmarz/storm-deploy/wiki

All of these storm projects with their project.clj files betray the Clojure roots (using Leiningen as the build tool, which is amazingly great). Here's to hoping for more Clojure examples/docs.

nathanmarz11• 15 years ago

The Storm deploy is awesome. It's built using an awesome tool called Pallet: https://github.com/pallet/pallet

I'm going to try to get more documentation on using Storm from Clojure in the next few weeks. I write all my topologies in Clojure using a small Clojure DSL that ships with Storm:

https://github.com/nathanmarz/storm/blob/master/src/clj/back...

nathanmarz11• 15 years ago

Hey all, I'm the author of Storm. Just wanted to point you to a few resources:

I've written a lot of documentation on the wiki, which you can find here: https://github.com/nathanmarz/storm/wiki

There's a few companion projects to Storm. These are:

One-click deploy for Storm on EC2: https://github.com/nathanmarz/storm-deploy

Adapter to use Kestrel as a Spout within Storm: https://github.com/nathanmarz/storm-kestrel

Starter project with example topologies that you can run in local mode: https://github.com/nathanmarz/storm-starter

Feel free to ask me questions here or on Storm's mailing list ( http://groups.google.com/group/storm-user ), and I'll answer as best I can!

aaronblohowiak• 15 years ago

Thank you for releasing such thorough documentation with your project; it is a great indicator of professionalism and passion. ^.^

nathanmarz11• 15 years ago

Thanks! Good documentation is a critical part of having a usable project, and that will always be a focus of Storm. If you have any feedback for how the documentation can be better, please let me know.

varworld• 15 years ago

How does Storm compare to Kafka?

pp13• 15 years ago

Why did you guys go with clojure, instead of something maybe like scala?

nathanmarz11• 15 years ago

I've had a ton of success using Clojure with projects like Cascalog and ElephantDB. It's a phenomenal language.

tom_b• 15 years ago

Just to point out, Nathan will be speaking at the Clojure/Conj conference this year. Talking about Cascalog.

I have a schedule conflict this year :( but would recommend the conference to everyone - the first year was informative with contagious enthusiasm. The other speakers also look cool.

anand_nalya• 15 years ago

Hi nathan, how does this compare to S4?

nathanmarz11• 15 years ago

The biggest difference is that Storm guarantees that data will be processed, whereas S4 can lose data. I also think Storm is significantly easier to use.

zaph0d2• 15 years ago

Nathan, Congratulations on the awesome work, Storm looks really great.

A minor nit, I was looking at the code, and all the hanging parens make it look really sad :-(

zaph0d2• 15 years ago

Bah, why the downvote? I was pointing out a major Lisp convention.

nathanmarz11• 15 years ago

And one more resource... here are the slides from my launch presentation of Storm:

http://www.slideshare.net/nathanmarz/storm-distributed-and-f...

schiptsov2• 15 years ago

Someone want to implement something like that without 100 jars of dependencies and 8Gb of memory required to just run, in old-fashioned C/Lisp way (or more modern nginx-way)? Update: on top of Plan9?! ^_^

Or, at least, in more suitable Erlang? ^_^

Isn't it an obvious startup-idea?

nathanmarz11• 15 years ago

Storm does not have 100's of jars of dependencies, nor does it require much memory to run. In fact, it's quite straightforward to spin up a Storm cluster (see https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-... ) and Storm has a local mode where you can develop/test topologies on your local mode completely in-process.

schiptsov2• 15 years ago

As far as I know, even clojure-contrib requires lots of jars, and I guess Scala is also very costly, but let it go.

I appreciate your innovative idea and amount of work you have done, so this small efficiency issue does not really matter.

btw, who cares about resources when hardware is so cheap and purchased in ocean containers? ^_^

mdaniel• 15 years ago

Is it common practice for a corporation to release code via one user's Github account? I would expect that if Twitter were open-sourcing something, it would show up as something like https://github.com/twitter/storm (that link is 404, to save you the trouble).

frisco• 15 years ago

This is really more nathanmarz's Storm more than Twitter's Storm. He wrote it at BackType with the intention of open sourcing it, and when Twitter bought BackType they apparently didn't ask him not to. So, it makes sense that it's being released through his Github account. If Twitter uses it significantly internally, they'll clone it to their organization account, like they did for most of the projects that are on Twitter's Github profile.

beagledude2• 15 years ago

nathanmarz does the work at BackType, Twitter gets the credit

xtacy• 15 years ago

BackType has been acquired by Twitter.

beagledude2• 15 years ago

we know that, but all the work was done pre-twitter aquisition.

kloncks• 15 years ago

keep in mind that through the acquisition, nathanmarz got something out of it as well :)

gfodor• 15 years ago

This is awesome, really excited to see this get released!

sherkund• 15 years ago

Just out of curiosity, to what degree was Clojure chosen because of the ability to use Java libraries vs the language design + community?

nathanmarz11• 15 years ago

Clojure is an amazing language, and part of that is the access to a huge breadth of Java libraries. Storm makes use of Java libraries/projects like JZMQ, Zookeeper, and Thrift.

mstanley• 15 years ago

Hi Nathan! this is awesome. I'm really excited to dive deeper.

some questions:

- I'm trying to understand the relationship between ZeroMQ and Kestrel in your architecture. is ZeroMQ used for message passing? and Kestrel used as a stream source/sink - aka a sprout? in other words, my assumptions are: zookeeper helps manage node discovery and coordination while message passing between nimble managed bolt processes' are through zeromq. kestrel queues are used for external integration (data stream sources). Is this correct or am I missing something?

- do you have any tutorials on using cascalog with Storm? are they compatible or have you developed a different clojure programming model/DSL for working with Storm?

thanks and again - nice work!

nathanmarz11• 15 years ago

You have it correct. ZeroMQ is used for message passing between components, and Zookeeper is used for node discovery and coordination.

Realtime processing is fundamentally different than batch processing, so you can't maintain the same semantics of Cascalog on top of Storm. Storm has a small DSL for writing topologies in pure Clojure, but it's not a higher level abstraction like Cascalog is. I've started thinking about what a great higher level abstraction would look like, but what that should look like is still an open question.

ldng• 15 years ago

Really glad that you guys now release and than announce ! the best way to avoid the let down of ending not opening something announced earlier (whatever the reason). Keep the trend going !

Twitter's Storm (complex event processing system) is now open source

Related Stories