Storm 2.0

https://storm.apache.org/2019/05/30/storm200-released.html

by mnming • 7 years ago

215 238 7 years ago

storm.apache.org

avinium• 7 years ago

Salient line:

While Storm's Clojure implementation served it well for many years, it was often cited as a barrier for entry to new contributors. Storm's codebase is now more accessible to developers who don't want to learn Clojure in order to contribute.

Very common story that I hear from a lot of shops that tried to go Clojure first. Great language, but too little penetration amongst developers. Makes it very difficult to effectively recruit employees/contributors.

At some point, you just accept that it's more important to get a new warm body than to continue pursuing some idealized programming perfection.

maltalex2• 7 years ago

That's not the only point though, is it?

The new Java-based implementation has improved performance significantly, and made Storm's internal APIs more maintainable and extensible.

jsmeaton• 7 years ago

It’s unclear if the author meant that performance is better “because java” or “because rewrite”.

dleavitt• 7 years ago

This is salient too though:

The new Java-based implementation has improved performance significantly, and made Storm's internal APIs more maintainable and extensible.

It's not totally clear if this is because it's been rewritten Java, but the intrinsic qualities of the language do matter; there are real tradeoffs between maintainability and dynamism/flexibility. I feel like this sometimes gets shortchanged in these narratives.

danarmak• 7 years ago

The announcement says they have a new architecture. At the very least it's not a direct performance comparison to the same architecture implemented in Clojure.

Storm 2.0.0 introduces a new core featuring a leaner threading model, a blazing fast messaging subsystem and a lightweight back pressure model. It is designed to push boundaries on throughput, latency and energy consumption while maintaining backward compatibility. The design was motivated by the observation that existing hardware remains capable of much more than what the best streaming engines can deliver. Storm 2.0 is the first streaming engine to break the 1 microsecond latency barrier.

bjoli2• 7 years ago

Idiomatic java is like idiomatic C: pretty fast. Most smaller languages prioritize other things. Clojure has immutability and Haskell has purity.

I have noticed this with every "X produces code faster than C": you begin with two programs that use a suboptimal algorithm, then you take your non-c language and try to write C in it. The result is always awful and removes most reasons not to use C in the first place.

This has somewhat changed with rust and in some sense C++,but for other languages my point still stands. They have a nice idiomatic golden path that is fast enough for most cases. Once you need performance badly enough you have to treat you language as an assembler, and then you will always lose to languages that actually are good at that.

I say this as a scheme/Haskell weenie. Writing really performance scheme and Haskell code often means writing ugly code.

roenxi• 7 years ago

Clojure is marketed as a very fast language (bunch of clever optimisation gets done because data structures are immutable). If the gains were from switching to Java they'd probably have said so less ambiguously.

Surely what they mean is they've rewritten it to be faster, and switched to Java at the same time because they want to open up the contributor pool.

didibus4• 7 years ago

I did find that bit interesting. It seems it's missing a leading "due to the commiters' better familiarity with Java."

Since you can argue knowledge of a language does help with maintenance and extensibility. With this, it also makes more sense for the following bit about finding people able to contribute. No contributors is pretty bad for maintenance.

At my work, we use Clojure, but I do often wonder what would happen if the current Clojure devs left. New devs unfamiliar with Lisps, FP, and Clojure would most likely have a hard time maintaining the code base and extending it, and if they try to rush it before really learning the language, would quickly degrade the quality of the code base, compounding the effect.

I think as long as you continue to have one strong Clojure dev on the projects, you'll be fine, as they can direct newcomers and help them transition to Clojure, but if you lost that, I think a rewrite in a different language would make sense, or there's a risk of the code base degrading quite quickly.

nnnmnten• 7 years ago

Most CS programs have a course on functional programming or programming paradigms including functional programming.

I don't understand why it would be a challenge to pick up Clojure.

Mirioron• 7 years ago

I don't know how it is elsewhere, but I saw many (most?) students struggle with those classes. These are people who have learned to successfully program, but they struggled quite significantly with functional languages (Haskell in this case). Even when they pass, many of them probably wouldn't want to pick up a functional language afterwards.

tanilama3• 7 years ago

What learnt in school is insufficient to meet with all real world challenges. To claim as a developer, you need to understand the internals, that is the ability to poke and trace to the root of the problem deep underneath.

Debugging in FP feels different to most people, where our thoughts is tuned to the procedural model with observing states and form our hypothesis from there.

Another disadvantage with niche programming language is not enough Stack Overflow help there. That alone will drive most people away.

rienbdj• 7 years ago

Many cs courses teach ML style functional programming, not Lisp or Clojure

skywhopper4• 7 years ago

People have limited time in their lives. If you want to recruit volunteers to your open source project it has to be in a language they either already know or are interested in learning. It may not be “hard” to learn Clojure, but the potential contributor may consider the time and mental energy spent fumbling around frustrated with all the off-putting sigils and different brackets to be better used learning a language with brighter prospects.

TeMPOraL4• 7 years ago

It's a sad state of reality, because it drives down the quality of software to the common denominator of fresh junior devs, which is not much.

Another view of this would be: progress in this industry is made by those who succeed while sticking to their guns and not optimizing their infrastructure for the available worker pool.

JustSomeNobody• 7 years ago

So, you’re saying the quality of the code dropped with the move to java?

Could there be other reasons? Maybe they rushed?

pron12• 7 years ago

It's a sad state of reality, because it drives down the quality of software to the common denominator of fresh junior devs, which is not much.

There's no reality here, just myth. That switching to Java would drive quality up makes at least as much sense as it driving it down.

If you're referring to the particular languages, then the reality is that no big effect has been found for the choice of programming language on software quality. As lack of evidence in favor of a big effect is evidence of its lack, the most probable explanation is that no such effect exists (it's possible a small effect exists, but we don't know in which language's favor). And, indeed, while there is a theory explaining (and even predicting ahead of time) why there is no big effect to the choice of the programming language, not only is there no empirical evidence suggesting there is such an effect, there is no theory to predict it, either. That language has a big effect on quality is, at this point, no more than wishful thinking among a minority of developers who are big programming language fans. It's a bedtime story. (I am not saying that language didn't ever have a big effect or that it never will, just that both theory and observation show that there isn't one currently among "reasonable" languages in common use; also, I have used and I like both Java and Clojure)

And if you're only referring to the priority of making the project more accessible, then I don't understand your comment at all. There are many, many more experienced Java developers than experienced Clojure developers, and experienced developers are at least as likely not to bother learning a new language in order to contribute to a project (although perhaps for different reasons). Picking a popular language makes the project more accessible to experienced developers.

What has been shown to drive quality up or down is process. If you have a good process, then making the project more accessible can help, and if you don't then you're screwed anyway.

fauigerzigerk7• 7 years ago

You could be right. But I think there are good reasons to believe that there is more to it than that.

I have a strong suspicion that people smart enough to do advanced meta programming on their own are not smart enough to collaborate with equally skilled coworkers on code that uses all those abstractions to the max. Even collaborating with your own former self is more difficult that writing new code.

So I think _some_ dumbing down is inevitable. The question is how that can be enforced. Choosing a dumber language to bludgeon everyone into submission is the nuclear option so to speak.

I often think of this Dijkstra quote about abstraction: "The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise".

But I think there's something missing in this thought. Abstraction, especially when its precise, is also a form of compression. To understand what a particular abstract expression means in a concrete case requires decompression, which may require a lot of mental effort.

These are admittedly extremely half-baked thoughts...

Jach5• 7 years ago

At another point you just accept you'll have to hire people who don't fit the classic job requirement of "X years experience in Lang" and hire people who don't know Lang at all but want to learn. A growing Clojure shop in Seattle seems to be having success with this, onboarding the language is a small fraction of overall onboarding.

On the more meta level it's weird what people expect devs to learn on the job or know up front. Companies expect employees to keep up with the latest nonsense in web dev, but can't pick up a new straightforward language? Unless maybe another Algol like Go or TypeScript? This Steve Yegge quote captures some of it:

"For some reason, programmers love to learn new stuff, as long as it's not syntax."

amortize5• 7 years ago

A growing Clojure shop in Seattle seems to be having success with this, onboarding the language is a small fraction of overall onboarding.

I can attest to this fact, as part of a unicorn upstart with dev offices in Sao Paulo and Berlin. Clojure onboarding has never been a major roadblock for new engineers. And we never hire asking for "X years of lang experience". Almost everyone in the (~300 strong) engineering team started with zero-to-little Clojure background.

willtim2• 7 years ago

Yes it seems strange to me that a Java developer is willing to learn Spring, a massive ad-hoc framework which exists in part because of Java's weaknesses, rather than functional programming and Clojure (a lifelong skill that will benefit their Java programming).

cwyers• 7 years ago

Well, this isn't a hiring situation, this is a getting contributors to an open-source project. When you're looking for volunteers, the tolerable amount of friction is often less than if you're paying someone for their time.

_cs2017_3• 7 years ago

How long does it take a strong developer to learn Clojure? Is it really a big a barrier to new contributors?

Presumably people who consider contributing to the open source have some spare time and love programming. Wouldn't they find it exciting to contribute while also learning a cool new language?

skywhopper4• 7 years ago

Maybe, but learning a new language by contributing to a large project built in that language is asking for trouble. Programming languages aren’t just syntax and stdlib. The relevant idioms and patterns may be significantly different. Even making small changes can be hard and intimidating.

And then, even if you wanted to learn something new, to a lot of people, it matters what that something is. Unless the person is really set on contributing to Storm specifically, then the barrier of Clojure is still significant. They may prefer to spend their new language learning bandwidth on something with more momentum.

yogthos9• 7 years ago

My experience from hiring and training co-op students for many years is that it takes around 2 weeks to learn with a bit of guidance. when a co-op student with little to know programming experience can learn Clojure in that time I find it surreal that there are professional developers who cannot.

jcadam• 7 years ago

As a Clojure enthusiast who has never been able to find a FT job using the language, I find it hard to believe that the demand for Clojure devs outstrips the supply.

And even if you have to hire devs who don't already know the language I really don't understand the difficulty in learning Lisp - few languages are simpler or easier to grok.

puredanger• 7 years ago

The demand and supply for Clojure both exist, but they are not equally distributed geographically.

emidln• 7 years ago

I didn't have any problems finding work in Clojure when I was last looking (Aug-Sept 2018) with offers in four locales across the US. I was not able to find remote work at comparable salary for a US based org. (Some context) I have 7 years experience using Clojure professionally across 3 orgs (16 years overall) and was looking for individual contributor roles.

Some things I observed about the Clojure companies:

- Their HR seemed slower than the non-Clojure companies I was dealing with. I don't think this has anything to do with Clojure, but I am no longer working in Clojure full-time because the most credible offers took too long and I've found non-Clojure work I love.

- Outside of trading firms, I did the most whiteboard coding questions with Clojure firms compared to shops looking to hire for Python or Elixir. I like whiteboard work, so this was fine for me.

- 6 Clojure companies, 4 different conccurrent processing models and 6 different service frameworks. Neither of these is necessarily bad, but the community was much more fractured than I realized.

- Of the 6 Clojure orgs, 4 were actively hiring (had open roles) and 2 were passively hiring whenever someone materialized on IRC/Slack/Twitter. This leads me to think if you want a Clojure job, you should probably reach out to people you know inside Clojure orgs and, barring that, the HR people at these orgs directly. This isn't a Clojure-specific thing, but worth keeping in mind.

- 4 of 6 didn't seem to have dedicates DevOps, leading to them looking for candidates also familiar with deploying, monitoring, and debugging Linux hosts. I think I was given heavier weight at some orgs because of a background doing this type of work for Clojure stacks. Might be an angle worth looking into if you don't have production Clojure but you do have Linux and Java deployment/monitoring/debugging/tuning experience.

yogthos9• 7 years ago

My team has been using Clojure for close to a decade, and we found the opposite to be the case. While the pool of applicants is smaller, so is the noise ratio. Clojure being niche means that you get people who are willing to look outside the mainstream, and are typically genuinely interested in programming.

We also rarely hire devs who already know Clojure, and we typically just train on the job. It's really not hard to do when you already have people who know the language on the team. In fact, if somebody can't learn Clojure, I would question if they can learn to work on a large project in general. Dev practices, architecture, code styles, tooling, and so on tends to change quite a bit from company to company. The language is only small part of that.

In case of Storm, Apache commons is run by Java devs who have zero interest in learning Clojure. So, it's not surprising they would rewrite Storm in their preferred language.

fnordsensei3• 7 years ago

Same (sure; anecdotal) experience here. Hiring for Clojure has been so much easier for us than hiring for JavaScript.

Granted, these are two different products, and maybe the Clojure one is just more attractive to the right kind of person. Nevetheless, in terms of quality vs. quantity of applicants, Clojure has been massively winning.

lvh7• 7 years ago

FWIW, I have a very different read on this, based on both my experience with Clojure and some of the leading Storm-using orgs (and the Apache process).

Firstly: Clojure isn't actually a problem. We hired people who are highly junior (one who even refused to call themselves a developer) and taught them Clojure. It was fine. It is true that you need a strong dev lead to guide them through that, but honestly, if you don't have a strong dev lead your Python/Java project is screwed too.

Secondly: look at Storm's increased success at Java data processing sweatshop^H^H^H^H^H^Hconsultancies. Of course "can write Java" is what you're going to get if most of your business revolves around the lowest bidder.

Fellshard2• 7 years ago

Same thing I repeat again and again: There is a false cost assigned to learning a language. Developers are too unwilling to even try stepping beyond the boundaries of the first thing they learned. The cost is always lower than they may think, and the benefits far surpassing what they may think. We've got to work at showing developers those benefits early; it's as important to creating software effectively as any other engineer's basic toolkit.

mgkimsal• 7 years ago

Developers are too unwilling to even try stepping beyond the boundaries of the first thing they learned. The cost is always lower than they may think, and the benefits far surpassing what they may think.

and yet... almost every time I've come in to a project where developers got past those boundaries.... they inevitably recreated patterns/idioms from tech 1 in to tech 2, and make tech 2 horrifically bad, and made it even harder for anyone who actually knew tech 2 to come in to the project and help.

adding other languages is more than just 'get over it' - without guidance from experienced people who can understand good/efficient ways to model the problem domains unique to a business idiomatically in the new tech, you're very likely adding even more technical debt that you won't even know about for months/years. And... IME, the "let's use new stack X" are moved on to another project or company by the time the technical debt is apparent to everyone.

Kye• 7 years ago

I suspect programming has something similar to Gear/Plugin Acquisition Syndrome in photography and music. It's easy to let tinkering with new toys stand in for doing the actual work of creating.

At some point, you have to sit down and do the work with the tools you have if you want to get anything done. If I were to get serious about programming, I would use C# or Go because I found them the most accessible after trying lots of languages. There's nothing wrong with finding a niche.

noir_lord• 7 years ago

At some point, you just accept that it's more important to get a new warm body than to continue pursuing some idealized programming perfection.

I think that is why Kotlin and Typescript are doing so well generally, they build on top of already established bases offering good new things without throwing out everything else.

The trade off is they are constrained from how far they can diverge from their roots but on the flip side that is a strength as well.

I like both, they feel productive without feeling totally alien.

moocowtruck• 7 years ago

another interesting part:

The new Java-based implementation has improved performance significantly, and made Storm's internal APIs more maintainable and extensible.

hugi• 7 years ago

At some point, you just accept that it's more important to get a new warm body than to continue pursuing some idealized programming perfection.

Describes my dating life pretty well.

zmmmmm• 7 years ago

A more glass-half-full view of it is that it's a magnificent success for Clojure.

Yes, it got redone in Java, but the fact is that Clojure took it from nothing to one of the world's premier frameworks for distributed computing - something that might have been impossible with the same resources if Java was its starting language. And because of the commonality of the underlying JVM, choosing Clojure allowed the functional approach with far less risk because the transition later was (I am assuming) much less risky and smoother.

Looking at it in that light, this would be a huge boon to people considering starting new projects in Clojure (or possibly other JVM languages): it proves if they succeed and get so popular that moving to Java is the right thing, it can be done.

solussd2• 7 years ago

Sometimes that backfires.

- Clojure dev

zenlot• 7 years ago

"While Storm's Clojure implementation served it well for many years, it was often cited as a barrier for entry to new contributors. Storm's codebase is now more accessible to developers who don't want to learn Clojure in order to contribute." - interesting phrasing. Seems that the authors assuming that new contributors are more willing to learn Java instead of Clojure. And those who know Java find Clojure difficult to learn? Clojure, in fact is quite easy to grasp - especially for someone coming from Java, as you would still be working on JVM and always have Java Interop. I guess same thing goes to someone coming from Erlang to Elixir, where you are still working in the familiar ecosystem.

dvfjsdhgfv• 7 years ago

Seems that the authors assuming that new contributors are more willing to learn Java instead of Clojure.

No, many more people know Java than Clojure, that's it. Whether it's the right decision for the future of the project, time will show, but I don't think it was taken lightly.

twblalock• 7 years ago

Most new contributors already know Java.

cygned• 7 years ago

Clojure, in fact is quite easy to grasp - especially for someone coming from Java

I disagree on that point. In my experience as a programming language teacher, people often struggle to learn functional programming and even more to learn a Lisp, especially if they are used to a C-like language. And learning both at the same time is even more difficult. I liked working with Clojure but I have yet to meet someone who easily grasps the language and - more important - development patterns.

adz5a• 7 years ago

Hello, this is an interesting take. Clojure has been a hobby of mine for some time but now I am a professional Clojure developper (clj/cljs). I knew the language before starting my new job but I found it really different than know the "platform": how a project is layed out, tests are written, some libs, how to pinpoint potential technical debt/suboptimal code accurately and so on... But I guess this would have been true if I had transition to any other new language.

Would you have any example of what your trainees stumbled upon or found difficult in the language? That would be useful I guess when we try to onboard new people on the project.

Have a nice day :)

eecc2• 7 years ago

Hey, once you get over accepting a = a+1; without gagging it’s all downhill from there ;)

kazinator3• 7 years ago

even more to learn a Lisp, especially if they are used to a C-like language

Ah, you're mixing up Clojure and Lisp. The restrictions in Clojure have nothing to do with Lisp tradition.

Someone used to a Lisp-like language will also struggle with Clojure.

Traditional Lisps like Common Lisp are multi-paradigm languages in which variables and most aggregate types are mutable.

They are not hard to pick up for someone with a background in C.

C has pass-by-value functions; so does proper, traditional Lisp.

C has mutable variables, arrays and structures; so does Lisp. Lisp data structures like lists or hash tables are easily understandable by a C programmer with a good background in data structures.

cons is like malloc-ing and filling-in a two-field struct, except that freeing is taken care of.

People have written Lisp systems in C (and written more than one book on that exact subject: Nils M. Holm just put a new one out: http://www.t3x.org/lsi/index.html), with a lot of the internals being written in C over the Lisp library functions. E.g. visiting and printing the elements of a list can literally look like this, in the internals of a C-based Lisp implementation:

   for (val iter = list; list; list = cdr(list))
     print(car(list));

val is some typedef for a pointer to a "struct lisp_object" or whatever. The nil value is represented by null, and so it goes.

agumonkey4• 7 years ago

Also, java 8 and above is getting a lot leaner to use, api, linguistics .. so the gap with clojure is a bit less than before.

kamaal2• 7 years ago

It will take years if not decades for these things to seep into regular programming practice in the Java world.

I still see people writing

    for(i = 0; i < something; i++) {
    }

Java could get cleaner, but habits die/change when programmers retire not when Java makes new releases.

hatch_q• 7 years ago

It's not about the language, it's about the code... understanding Clojure code that someone else wrote is harder than writing your own.

yogthos9• 7 years ago

I don't find that to be the case at all. I worked with Java for around a decade and I never contributed to a single open source project. Every time I'd see a bug or a feature I'd like to add, I'd open the source and it would be a nightmarish maze of classes and interfaces with business logic being sprinkled across them.

Meanwhile, I've successfully contributed to plenty of Clojure libraries I use, and many people contribute to my projects. Clojure code is far more direct and concise making it much easier to understand the intent. And majority of the code is written using pure functions that can be reasoned about in isolation. On top of that you have an editor integrated REPL, so you can just run any code you're not sure about as you're reading it.

cutler5• 7 years ago

A dynamic, functional Lisp built with immutable data structures and inherent laziness is about as far-removed Java's mutable OOP as I can can imagine.

kazinator3• 7 years ago

... and from Lisp. Please, folks, don't assume Lisp is weird, opinionated and hard to use based on trying Clojure.

kazinator3• 7 years ago

That's like saying going from any language to any language on the Unix platform is easy because the POSIX system calls are the same. In any half-decent language on Unix you can get access to a file descriptor and call fcntl() on it, so it's all the same.

agumonkey4• 7 years ago

poor lisp, just like reddit ... always dropped for another language.

sitkack2• 7 years ago

What if they are phases of evolution? A small team that can speak a lightweight language can quickly change velocity (speed and direction), iterating until a PoC is a useful system. Once the interfaces and the protocols fall into place, calcify or harden the design by converting to something more static, more literal.

Imagine going from something like lisp to Ada? Or lisp to Rust? Write quick prototypes in lisp, solve the hard problems of understanding the domain, then convert portions that require correctness to a language that is amenable to correctness checking (property testing, model checking, etc).

I think gradual typing [0] (not Gradual Typing) or language embedding [1] are steps in that direction. In a way, lisp can be thought of as a CASE tool for bootstrapping new systems.

The great thing about language platforms (JVM, Beam, Graal, Wasm, Racket) is that many languages can easily exist in the same system so that in many cases, code can be converted on a function by function basis.

[0] http://mypy-lang.org

[1] http://terralang.org

fulafel4• 7 years ago

The SW world is full of Java/C++ rewrites of products that originally started out in a dynamic & productive language like Lisp or Python. It might make sense in the lifecycle phase when you're past making big changes under make-or-break pressure, and are in the "provide predictable 10 year maintainance roadmap for this large codebaes" stage. You end up employing different kinds of programmers too because many of the original team may be reassigned to something more exciting/important to do.

But it's in no way inevitable, of course, there are many old Lisp, Python, etc codebases in production too.

BTW. I've been trying to rediscover this blog post and can't seem to find it again, anyone? It described different kinds of software projects, the (1) kind that try to deliver incremental improvements (eg 10% cost savings), and the (2) kind that try to make 10x improvements, and the difference in how those projects are run (the 1-type project can't go over budget because it will then deliver negative value etc).

0815test4• 7 years ago

Hopefully we'll start seeing some Rust rewrites in the future then, since the pitfalls of C++ are widely-known by now, and few people would claim that they don't impact long-term maintainability. (And the Rust Evangelism Strike Force would surely approve of such a shift!)

agumonkey4• 7 years ago

also they can still have hybrid clojure jars if needs be

juskrey2• 7 years ago

Apache products have always been a result of a blind majority rule, and thanks to that many of them are a corporate-style incomprehensible mess. They favor the process instead of a result: Apache things need to show constant progress to be afloat.

Indeed, adopting Clojure for the developer product of this style is a dead end. Clojure is a minority rule. A slow tinkerer's and a lazy (in a good sense) practitioner's tool, whose aim is to survive in a real world, not to show a steady movement no matter what.

Shmebulock• 7 years ago

These Talebian references got me thinking. Couldn't you argue that it shows a certain arrogance and non-results oriented "falling in love with ideals and tools" detachment from reality to use Clojure instead of good ol' Java?

krn• 7 years ago

non-results oriented

I think Clojure's ecosystem is more results-oriented than almost any other currently available[1]. It just requires a completely different mindset to understand, which an average software developer lacks. Therefore, Java is a much better fit for an Apache project.

[1] https://news.ycombinator.com/item?id=19728502

jrs952• 7 years ago

Oh I'm sure people make that argument all the time. I just don't think it holds up to reality. It's sort of just a dismissal of better options. The reality in my experience is that Java is just a poor choice for a lot of software. That's why you end up with frameworks built upon frameworks & AspectJ to basically make it behave like a dynamic language and try to hide some of the ugliness of the underlying APIs from you. And it's just because it's corporate approved and "everyone knows Java". I don't see how it can reasonably be considered the pragmatic choice. It's just not productive, and the final product isn't something that I would consider easy to understand or maintain most of the time either.

TeMPOraL4• 7 years ago

Couldn't you argue that it shows a certain arrogance and non-results oriented "falling in love with ideals and tools" detachment from reality to use Clojure instead of good ol' Java?

Since when trying to do better than using a tool designed for mass-producing software with dumb and cheap labor is "falling in love with ideals and tools"? It's kind of like saying that using an excavator over a bunch of people with shovels is "falling in love with ideals and tools". No, it just allows a single person to get more work done faster and better.

Our industry is weird. When recruiting, companies claim they want "the best of the best" and will often test you on ridiculous stuff. But when it comes to actual work, on industry scale, learning, growing professionally, and doing things in a clean and efficient way is frowned upon. Best take the weakest tool available and compensate for its shortcomings with third party services.

ian1321• 7 years ago

Java is statically typed. Clojure is not. To me, this is the biggest difference and the reason to choose Java.

jnwkw• 7 years ago

Is this true or false? I don't understand the downvotes.

pyb• 7 years ago

Many people disagree with "the reason to choose Java".

Rovanion• 7 years ago

While it is true that Java is statically typed and Clojure is dynamically typed it is perhaps one of the less important differences between the two languages. If you in fact would like to add a static type system on top of Clojure there is a way to do that: Typed Clojure.

https://typedclojure.org/

CodeArtisan• 7 years ago

https://danluu.com/empirical-pl/

innocentoldguy• 7 years ago

I'm not going to downvote the comment, but I think the "statically typed" argument is fantastically overrated. I spent years of my life coding in Java and C# and other years of my life writing Clojure, Python, Ruby, and Elixir code. I've never once thought to myself of the latter, "Damn! This would be so much better if I could have static types." It has literally never, not once, ever been an issue for me.

roshan_naik2• 7 years ago

In the critical path many things can matter that ordinarily don't. When working on perf improvements for Storm 1.0 release, profiling revealed hotspots in Clojure code. Adding type hints to avoid dynamic typing fixed some of these hotspots. In other places, Clojure primitives showed up as a bottleneck. These required rewriting some code in Java and invoking it from Clojure.

This was mentioned briefly here: https://hortonworks.com/blog/microbenchmarking-storm-1-0-per...

The Clojure based core has served the community well for some time. It was still powered by Disruptor (a fast Java messaging library). So there was always some back and forth between Java and Clojure.

For Storm 2.0, the Java rewrite happened first. The re-architecture work might have not happened without it (at least for the foreseeable near future)... primarily due to Java expertise being more easily available.

jaimex2• 7 years ago

If you're going to be on the JVM, stick to what the J stands for :)

lvh7• 7 years ago

The newest JVM out of Oracle, GraalVM, has its polyglossy as its central feature. And Clojure has always been a lot closer to Java (in terms of quality of interop) than even e.g. Scala, let alone something like Jython. (Notable exception JRuby.)

rcaught3• 7 years ago

I struggle to see how you pick Java over Kotlin on the JVM these days. It's everything that Java is, minus everything Java shouldn't be, plus everything that Java should be.

jrs952• 7 years ago

It's the lowest common denominator. It's the same reason I haven't been able to adopt Kotlin at work. Everyone acknowledges it would be better, but "everyone knows Java". Seems almost like "nobody ever got fired for buying IBM" to me.

cutler5• 7 years ago

... or, worse, "everyone here uses Java <= 8".

ivan_gammel4• 7 years ago

Syntactic sugar does not solve programming problems. Kotlin does not offer any paradigm changes which could make switch worth it.

vbezhenar7• 7 years ago

The only Kotlin feature that it's really hard to live without is properties. Otherwise sure, it's better than Java 11, but I can live without all those niceties. And (non)nullable types even get in my way!

I just don't understand where there's syntax sugar for propeties in Java. There are lot of unneeded stuff added, that I don't really care about. But my classes are still full of autogenerated getters/setters. I recently tried to find some JEP and I did not find one. There was some talk around Java 7, but nothing since that. Nobody even wants to add properties support for Java?

rcaught3• 7 years ago

Kotlin's null safety alone is enough to prove you wrong here.

jackpeterfletch• 7 years ago

Correct null handling genuinely helps you write better, more correct, less bugged code.

It's not just syntax sugar.

tanilama3• 7 years ago

Except there is no need to do. They already biased for maturity and long term maintainability. I don't think Kotlin is bad per say, but it feels untterly unnecessary.

p2detar2• 7 years ago

As someone that uses both languages, I can't help but think that the OpenJDK will (hopefully) start implementing much of the Kotlin paradigms. Perhaps at some point Java & Kotlin would not differ too much. Let's see what the 6-month release cycle brings. That remains to be seen.

pjmlp8• 7 years ago

I guess you mean Java here, and yes some of them have a JEP defined.

fulafel4• 7 years ago

They want to use a language that has a large pool of interchangeable "x years of experience in language y" labour available.

pjmlp8• 7 years ago

Because until JVM turns into KVM, Kotlin is just another guest language on the JVM, without much to add as a company pushing their agenda and trying to use the JVM as a lever.

KVM might well turn out to be ART, but that is the extent of it.

rnikander• 7 years ago

As someone who just started using Kotlin again, I'm curious what you mean by "agenda". A typical company trying to make products for money, or something else?

zubairq• 7 years ago

We did the same with our open source project. We ditched Clojure and Clojurescript for NodeJS and JavaScript, for the same reason as it was a huge barrier to entry for new users

unlogic3• 7 years ago

To me, this makes total sense as the project moved to Apache. Obviously, much more people will be able to consider contributing when it's in Java. Apache goal is sustainability and long-term viability, and Java would work better for that.

I also consider this a success story for Clojure. It gives Clojure another usecase: a "production-ready prototype" language where the resulting "prototype" can last for eight years and benefit thousands of developers until it gets rewritten to something else when all the hard questions are answered, and most experimentation/wandering is over.

sitkack2• 7 years ago

See my response inspired by your comment and agumonkey here https://news.ycombinator.com/item?id=20074532

I am drawn to the idea of scalable languages or languages with zero friction for interop.

This thread on Pallene is interesting https://news.ycombinator.com/item?id=18038619

usgroup4• 7 years ago

I think it’s noteworthy that whatever the benefits of clojure it didn’t outweigh the adoption issue.

It’s quite a mature language so I’m not sure that bodes we’ll for its prospects this late in the game.

stingraycharles2• 7 years ago

What if Clojure is a language that bodes well especially early in a project’s lifecycle?

unlogic3• 7 years ago

Whatever the benefits of Clojure are, they were enough for Nathan Marz and first contributors to write Storm in it and succeed. Note that it's not an argument about impossibility of writing something like that in Java (see Spark, Flink, Heron, etc). But at that exact moment in time, a Clojure Storm was created and became useful to many people for years. There were many more Java programmers than Clojure programmers then and now, yet the Java coomunity didn't produce their own Storm first.

I think it speaks of something.

lvh7• 7 years ago

If you can have the foresight to know how your project will be used many years down the line, sure, make technical decisions based on that :-)

(Personally, I think the change is rather evidence of the primary drivers behind Storm development in 2019, and... that's not a good thing.)

yogthos9• 7 years ago

A Java shop inherited a Clojure project and rewrote it in their preferred language. It's silly to read more than that into this.

amortize5• 7 years ago

I also consider this a success story for Clojure.

It is a success story for Clojure, but this move is a big negative feedback for the language. A team starting out on an open-source project will be mighty reluctant to start it with Clojure; because it might get rewritten not so much into the future. That is not good news for Clojure

fnordsensei3• 7 years ago

If, five years (or what-have-you) down the line—when you've pretty much enumerated all use-cases for the project, solidified what it's supposed to do and how, learned from your mistakes, and so on—you can't come up with a better design, something is off. With all that knowledge, there should be plenty of space for performance optimizations and pruning of vestigial stuff for a non-trivial project.

As for switching to Java, that makes perfect sense to me if that is what the contributors prefer. If you're going to make a big rewrite, you might as well change the language to one you prefer when you have the opportunity.

As someone who spends 100% of dev time in Clojure, I'm quite happy to see this development. Dipping into Java isn't uncommon for optimizations in Clojure, so this is like someone taking the time to do a massive under-the-hood optimization from the point of view of a Clojure user. I doubt it changes the project's status much in the Clojure community. Perhaps it'll even see an uptake in use.

What it doesn't say much about is Java vs. Clojure in my opinion. Different contributors, with different amounts of pertinent knowledge, and years apart. I'm not sure how you would control for those variables in a comparison. It should be read more as, "we put in a heck of an effort to make this thing faster and more useful", and kudos for that.

neilv• 7 years ago

(I think the HN title was edited since your comment; I think an earlier title highlighted the move from Clojure to Java.)

It gives Clojure another usecase: a "production-ready prototype" language where the resulting "prototype" can last for eight years [...] until it gets rewritten to something else when all the hard questions are answered, and most experimentation/wandering is over.

That general story, first heard from Paul Graham, about projects starting with Lisp and being successful that way, before eventually being rewritten, works for me.

I've been hoping at least a couple college startups would be inspired by this to use Racket initially, but if they have, I haven't heard of it. (I speculate that the current FAANG feeder emphasis among CS students hasn't helped. Who has time to play with a Lisp, when the now-all-important whiteboard interviews won't use it.)

Obviously, much more people will be able to consider contributing when it's in Java.

If you have a company, and you want to hire Lisp people (whether it's Clojure, CL, or Scheme/Racket), I think you can probably hire people, because Lisp people like getting paid to use Lisp.

If you're looking for unpaid/volunteer contributions to an open source project, there's all sorts of things that affect that, and it's not unusual to get little-to-no contributions.

stingraycharles2• 7 years ago

I think the HN title was edited since your comment; I think an earlier title highlighted the move from Clojure to Java.

This is correct, it was “Storm ditches Clojure in favor for Java”.

cutler5• 7 years ago

Pegging Clojure as a prototyping language relative to Java is hardly encouraging.

unlogic3• 7 years ago

Oh no, I'm saying it's another usecase, not the only one. And yet, it is a pattern I follow quite often in my mixed Clojure/Java projects: Clojure code is much easier to tinker with, take to new directions, rapidly modify and see the results. But once I settle with an approach/algorithm, I rewrite some things to Java for performance, or for easier interop with other Java/Scala teams, or because the imperative mutable code looks much better in Java.

So, you can say that Clojure among everything else is my IDE to write Java code. Is that a bad suit for a language? Depends on your perspective, but I personally am very happy that I have Clojure by my side.

hanswesterbeek• 7 years ago

Clojure just seems "too weird" to too many people, albeit for all the wrong reasons. Its syntax looks like a 6ft hurdle to newcomers, especially if they have lots of experience in C-synthax-languages. But anyone who just /tries/ to jump it finds out the hurdle really isn't that high at all.

This dynamic will probably never change. Clojure will remain an acquired taste, which is fine, I guess. Those 'in the know' will happily continue to develop effectively with it.

keymone• 7 years ago

Bet they had to reimplement half of Clojure when rewriting storm core.

juskrey2• 7 years ago

Yep, at last a lot of progress to report to Apache Board of Directors

eecc2• 7 years ago

I see what you did there

readme3• 7 years ago

Apart from the language part , i am super excited about trying storm-2.0. In many ways the buzz in the ecosystem has been that storm was dead!. We run lot of prod workloads on storm and spark , storm has worked like a wonderful workhorse without quirks. The ecosystem has moved on , but i think storm may still continue having loyalists!

scribu• 7 years ago

The ecosystem has moved on

Just ouf of curiosity, what tools are people using instead of Storm?

tanilama3• 7 years ago

Flink is a strong competitor. Spark itself also claims to be a runner up with minibatching.

But streaming model is more widely adopted outside of frameworks. AWS Lambda or Fass platform can be viewed as streaming model with simple topology.

nimrody• 7 years ago

Flink is popular. I believe Samza also fills a similar niche.

There's also Kafka Streams which is suitable for many cases that do not warrant a full cluster with all the management and trouble required.

grkg8tr2• 7 years ago

I thought Heron was the Storm follow-up. I'm surprised to see a Storm 2.0.

fulafel4• 7 years ago

cutting off 1.0 immediately sounds quite harsh, I wonder what kind of user base it has?

"EOL for 1.0.x With the release of 2.0.0 the 1.0.x version line will no longer be maintained."

Hendrikto• 7 years ago

It is designed to push boundaries on throughput, latency and energy consumption while maintaining backward compatibility.

JanecekPetr• 7 years ago

That's because they'll be maintaining the 1.1.x and 1.2.x branches...

kabhwan3• 7 years ago

Many big projects tend to manage around 3 version lines. Since Storm 2.0 is out, there're three version lines for 2.x, 1.2.x, 1.1.x. 1.0.x could be EOL-ed.

mrkeen2• 7 years ago

In this thread:

* Non-Java JVM languages are too different for people to use.

Also:

* It's not worth switching from Java to other languages because they're just Java with syntactic sugar.

skywhopper4• 7 years ago

Clojure and Kotlin are two very different languages. I don’t see anyone making the broad statements you are calling out.

mrkeen2• 7 years ago

No, of course not.

I misread

Syntactic sugar does not solve programming problems. Kotlin does not offer any paradigm changes which could make switch worth it.

and

Also, java 8 and above is getting a lot leaner to use, api, linguistics .. so the gap with clojure is a bit less than before.

pjmlp8• 7 years ago

Sad to see it happen, however it is yet another example how platform languages always win in the end, even if due to their nature and market size they tend to be more conservative regarding feature adoption.

skywhopper4• 7 years ago

I dare say it may be because they are more conservative regarding feature adoption.

gigatexal• 7 years ago

How am I just learning about this? This seems to be rather novel. Real-time bolt on for any database? Pretty cool stuff.

grego• 7 years ago

In the old days a project would often start in Lisp (Common Lisp for example), and then later be rewritten in C.

This is the same pattern, and one should embrace it. Write the first cut, explore the problem space and prototype in a dynamic language. Once the design is proven, rewrite in statically typed language for performance and to get the possible remaining bugs out.

If the design was perfect from the get go, you could just write it in assembler to start with.

amortize5• 7 years ago

While Storm's Clojure implementation served it well for many years, it was often cited as a barrier for entry to new contributors. Storm's codebase is now more accessible to developers who don't want to learn Clojure in order to contribute.

It is interesting to contrast this with state of affairs in Apache Spark. Spark has thrived well in spite of being a Scala project; Scala arguably has a higher barrier for entry compared to Clojure (although the flavour of Scala used within Spark closely resembles Java).

_cs2017_3• 7 years ago

Why change the title of the post?

Almost every one of the ~100 comments is in response to the original title that Storm switched from Clojure to Java. Now this context is lost.

lvh7• 7 years ago

Because the previous title was editorialized? Furthermore, perhaps the conversation is about the old title because it was, and now the conversation can go be about Storm itself :)

_cs2017_3• 7 years ago

I think the rule about not editorializing needs an exception so that someone can point out an important or interesting fact using an article whose original title is too generic.

Another example would be "Press release from XYZ". If it talks about XYZ reaching some major breakthrough, the title should be allowed to reflect that. It's a waste of everyone's time to post the article titled "Press release".

And arguably this situation should fall under such an exception.

jacobtracey• 7 years ago

What exactly are you doing?

ww520• 7 years ago

While most comments touch on the developer issue, the significant performance gain deserves more discussion. I guess Java's static type, pragmatic mutability with less garbage produced, and efficient parallelism give substantial boost to performance.

It would be interesting to see what the upcoming blogs have to say about the performance gain.

yogthos9• 7 years ago

My guess would be that it's always easier to make a rewrite performant because you already have a working version and know exactly where bottlenecks are. I imagine rewriting it in Clojure, or any other language, would've resulted in a significant performance boost as well.

grkg8tr2• 7 years ago

How does this relate to Apache Heron? I thought Heron was essentially Storm 2.0. Which one should I be using?

roshan_naik2• 7 years ago

Not well known is that Heron's performance claims in the "flying faster" blog and SIGMOD paper were made by comparing against a fork of an older (pre apache) version of Storm even though much more performant version was already out.

IMO (biased?) Heron makes surprisingly poor design choices for their key distinguishing features like threading model and backpressure. And it shows when you benchmark it. More specifics: https://github.com/apache/storm/pull/2241#issuecomment-31787...

kabhwan3• 7 years ago

Given there're so many streaming frameworks with different characteristics, you should be the one who understands the difference and select one fits for your business use case. There's no silver bullet.

didibus4• 7 years ago

Can someone enlighten me on the inner working of Apache?

Do they hire developers to maintain and work on the projects? If not, is there community leads in charge of each one? How are they appointed?

geodel2• 7 years ago

http://www.apache.org/foundation/governance/

didibus4• 7 years ago

Interesting, where would you go to see the list of committers for a given project and find information about them?

I'm curious if the commiters are Clojure developers who chose to switch to Java, or if they are Java devs who decided to rewrite the project to a language they are more familiar and comfortable in. Or if they are Clojure devs looking to transition the project to other commiters and they struggled to find other Clojure devs willing to take it over.

snorremd• 7 years ago

While Storm's Clojure implementation served it well for many years, it was often cited as a barrier for entry to new contributors. Storm's codebase is now more accessible to developers who don't want to learn Clojure in order to contribute.

While I appreciate that programmers have limited free time and may not want to learn a new language to contribute to open source, I feel like learning Clojure could be of value to the programmer in itself.

The supposition that Clojure acts as a big barrier against contribution from developers might be true, but I really feel like developers who only write code in the ALGOL family of languages are missing out. Learning a lisp language is not an impossible task and can improve your skills as a developer.

didibus4• 7 years ago

It is interesting to read the history of Storm from the original author (Nathan Marz) himself: http://nathanmarz.com/blog/history-of-apache-storm-and-lesso...

Clojure allowed him to build the initial Storm release by himself in only 5 months.

I made all of Storm's APIs in Java, but implemented Storm in Clojure. By keeping Storm's APIs 100% Java, Storm was ensured to have a very large amount of potential users. By doing the implementation in Clojure, I was able to be a lot more productive and get the project working sooner.

Also interesting is that Nathan has now founded a startup working on something which sounds like the next evolution of Storm, and he's building it in Clojure again:

https://redplanetlabs.com/

They recently got 5M in funding and are hiring. Building a Clojure team for it.

Personally, I feel the Storm 2.0 release rewrite to Java is really just a case of the new maintainers not knowing Clojure very well, and being primarily Java devs, looking for contributions from people using Storm, which tend itself to be mostly Java shops. Clojure is not quick to pick up, and very different from Java, so unlike Scala, Kotlin, C#, Go, etc. Someone with a Java background and no Clojure or Lisp experience won't be able to contribute as easily. And that's fine, and possibly best for Storm's future now that Nathan has moved on. Unless a Clojure dev had stepped up and adopted Storm, this change isn't at all surprising.

fulafel4• 7 years ago

For anyone interested in the Clojure history, there's this interview of Nathan Marz, originalcreator of Storm, from 5 years ago: https://www.infoq.com/interviews/marz-lambda-architecture/

kabhwan3• 7 years ago

DISCLAIMER: I'm one of PMC member of Apache Storm.

Even though Clojure has been Nathan's first-class language, not everyone in contributors and even committers couldn't become native on Clojure. (Though I guess part of committers could feel Clojure as native.) It might not be a problem when Nathan builds most of things in Storm (I'll link his blog post which describes lesson learned from Storm on his perspective http://nathanmarz.com/blog/history-of-apache-storm-and-lesso...), but when Storm becomes popular and also no longer his toy project, Clojure matters for scalibility of the project.

Personally, I had a hard time understanding plenty of macros and even chained macros - that might not be a problem if I were allowed to have plenty of times to study Clojure and get used to it, but possible contributors don't get a chance when they're ready. In many cases, the first time they read the code on what they're using is when they encounter issues on stage/production and have to dig with stack trace. For me, I had to investigate on Storm because my team suffered from lack of documentation and also lack of knowledge on internal which ended up just taking workarounds. It was pretty hard time learning language to just read and understand the details to track down the issue. I even couldn't think of writing code on Clojure.

When I decided to contribute Apache Storm, I started from connectors and CI build which wasn't written on Clojure. After some months I was able to read and understand the code to get into details but I still couldn't be native for Clojure, so still required N times of efforts to read and write the code even after I became one of PMC member.

Don't get me wrong. I'm not saying something is better than some other one. In scalability point of view on project, I guess we had to do this, even though it ended up teaching me some lesson that migrating language is not the thing which can be considered easily, especially for a big project. Lots of efforts were spent here.

Someone could claim for Apache Spark's case for Scala. Yes, I'm also contributing to Apache Spark and I had to learn Scala (fortunately I had a chance to do that before contributing) but the case is different. If you read Databricks Scala guide in details, you might realize that the doc is actually not encouraging contributors to use advanced features (a.k.a the features only Scala geeks might feel natural) on Scala. Same goes on reading codebase. Apache Spark community has been trying to keep necessary knowledges on Scala to contribute Apache Spark low enough, so that non-expert of Scala engineer could easily try out contribution. If they have been requiring hard understanding of Scala to accept contributions - Spark may not be able to get its amazing reputation.

Storm 2.0

Related Stories