Storm's first birthday

http://nathanmarz.com/blog/storms-1st-birthday.html

by nathanmarz • 14 years ago

115 24 14 years ago

nathanmarz.com

dmix• 14 years ago

Are there plans to integrate Storm with Datomic?

nathanmarz3• 14 years ago

Not by the core team, though I think you could make a pretty cool Trident state implementation for Datomic.

encoderer• 14 years ago

Keep up the good work, Nathan. I presented Storm as a tech-talk at Formspring and no question, you made my job easy by building a platform that makes it easy to get a topology running without a ton of boilerplate.

sanswork2• 14 years ago

Nice to see this. I started trying to implement storm towards the beginning of this year since it seems like something I've wanted/reimplemented partially so many times and just found it to be so time consuming trying to get good resources or find a community for misc support that I gave up and reimplemented simple parts of it as I needed to.

I'm off the buy the books and bookmark this page for future reference. Great work Nathan.

nathanmarz3• 14 years ago

Sorry to hear that you had trouble finding support for your issues. The mailing list is very well trafficked so hopefully you have more luck there next time.

sanswork2• 14 years ago

No need to apologize, it was quite early in the development of the project(and expected). I'm just glad to see the progress you've made in a year and look forward to implementing Storm to solve a few of our issues in the near future.

floppydisk• 14 years ago

I've used Storm for a couple of projects and really liked it. It significantly simplifies building real time data pipelines and makes deploying them KISS easy. I'm really excited to see some improved metrics coming out as well--if I had one gripe with Storm .7 it was that I wanted some more metrics to monitor cluster performance. Congrats on a year and keep up the good work, Storm is awesome!

dkersten2• 14 years ago

I've been using Storm for a few weeks and it works great, especially Trident, which I've been using for the past 2 or so weeks. Nathan has also been very helpful answering my n00b questions :)

designatedInit• 14 years ago

I know this has been repeated ad nauseum, but I'd really like to see blogs put what their product is somewhere on the page.

marshray• 14 years ago

LMDDGTFY https://duckduckgo.com/?q=storm

Hmmm...

scotty79• 14 years ago

https://github.com/nathanmarz/storm/wiki

paperwork• 14 years ago

so trident is apparently a high level api for Storm, but I don't see any mention of it in the Storm book. Is that right?

I am very interested in Storm, but wondering if the book is already out of date.

mattmcknight3• 14 years ago

It is a little bit out of date, in that it doesn't cover anything in 8.0, but most of the basic stuff didn't change, it was more of new features being added.

djb_hackernews3• 14 years ago

Even if you have no use for storm, I'd strongly suggest you check out the wiki on github and browse the source code. It is IMO the nicest, cleanest, most exciting OSS project out there.

https://github.com/nathanmarz/storm

chubot• 14 years ago

Can you expand on that? What's nice and clean and exciting about it?

Someone else posted link to Apache Kafka developed at LinkedIn.

http://incubator.apache.org/kafka/design.html

It seems like they address similar use cases... it would be interesting see a comparison.

djb_hackernews3• 14 years ago

Oh easy questions:

The code itself is nice and clean, take a look yourself (chosen randomly) [1][2]

It's exciting because if you've ever done any real time processing of data streams at scale or even been daunted by the idea of it, and you read the wiki, it's clear storm is an exciting option. It does for Big Data Streams what Hadoop did for Big Data.

Kafka is interesting, but they don't really address similar use cases. They are more complimentary and in fact storm has a spout implementation for kafka[3].

[1] https://github.com/nathanmarz/storm/blob/master/src/clj/back... [2] https://github.com/nathanmarz/storm/blob/master/src/jvm/back... [3] https://github.com/nathanmarz/storm-contrib/tree/master/stor...

newobj• 14 years ago

What I'd like to understand at a glance about Storm is durability, idempotency, and specifically those two things in the face of all your standard failure modes.

Is Storm something that is going to get 99.9% of my data there which is good enough for some side-channel processing like ad targeting, or is it something that's D durable in the strongest sense of the word? If I'm ultimately delivering data to something that is 5 9's durable, or better yet S3 10-11 9's durable, is Storm going to be my "low point" of durability?

djb_hackernews3• 14 years ago

Read the wiki:

"Guarantees no data loss: A realtime system must have strong guarantees about data being successfully processed. A system that drops data has a very limited set of use cases. Storm guarantees that every message will be processed, and this is in direct contrast with other systems like S4."

https://github.com/nathanmarz/storm/wiki/Rationale

mattmcknight3• 14 years ago

Storm supports transactional topologies that can provide durability. They are dependent on a spout that is repeatable- where you can request the failed input again. https://github.com/nathanmarz/storm/wiki/Transactional-topol...

nathanmarz3• 14 years ago

Actually transactional topologies are superseded by Trident, which is much easier to use:

https://github.com/nathanmarz/storm/wiki/Trident-tutorial

Here's an explanation on how Trident achieves fully fault-tolerant, idempotent semantics:

https://github.com/nathanmarz/storm/wiki/Trident-state

someone13• 14 years ago

One of my major wishlist items for Storm is an end-to-end tutorial on how to get it working for a non-JVM language. I'm not a JVM coder, nor do I know much about it, and the documentation on how to get it working for Python is hard to follow at best, IMHO.

mattmcknight3• 14 years ago

There's about a 9 page chapter in the new storm book (more of a large pamphlet really) on that. http://www.amazon.com/Getting-Started-Storm-Jonathan-Leibius...

packetslave• 14 years ago

There was a Python interface to Storm announced at PyCon called "Umbrella", but as far as I can tell it was never released or heard from again.

Storm's first birthday

Related Stories