How I want to write Node: Stream all the things

http://caolanmcmahon.com/posts/how_i_want_to_write_node_stream_all_the_things_new/

by nua • 12 years ago

107 114 12 years ago

gfosco• 12 years ago

This sounds intriguing. I wish there were some more complex examples. Part of the greatness of Promises is taking a big chunk of pyramid code and turning it into a set of simple steps... I'd like to see how this would handle that.

ctcliff• 12 years ago

Loads of examples at http://highlandjs.org/.

caolanmcmahon5• 12 years ago

Good idea, I'll definitely post a follow-up with some real code done using async/callbacks and highland/streams. The comparisons usually start to look more favourable with longer examples, due to the Highland API being so composable.

qubyte• 12 years ago

Nice. I need to compare code written with highland to code with async.series/.each etc. (but not waterfall, I don't like that dude ;)).

ixmatus4• 12 years ago

Forgive me for being "That Guy" but I really think Javascript is ill-suited for this paradigm!

Streams, honestly, are hard to keep straight when the program gets big without a stronger type system. IMHO.

Some really sharp people have been working on stream computing software in Haskell for a while - Gabriel's Pipes package is a good example of generalized stream computing with strong equational reasoning as its foundation.

Maybe if you really want to try and do this in Node you can gain some inspiration from his journey: http://hackage.haskell.org/package/pipes

jkrems5• 12 years ago

Streams are being used in node.js already with a lot of success. Both basic byte chunk streams and object streams. As long as components are well behaved and only emit uniform data (everything they emit is of the same type), it's not better or worse than any function call.

ixmatus4• 12 years ago

Right, but the burden is on the programmer to ensure those types. If the stream composition is of simple things, like:

    cat some.txt | sort | uniq > yay.txt

Then it isn't a problem - it's obvious and simple, but I think there will be difficulties as the programs get larger and type-level awareness occupies more space in the programmer's brain vs. it being handled by the compiler...

aegiso3• 12 years ago

Here's the main enlightenment of becoming a node.js guy:

Node follows the unix way. Everything is a stream. It's just Buffers and JS objects flying around. It's really stupid, and sometimes it's nasty. This isn't helped by Javascript's warts.

But there's an enormous upside to this: following the stupid Unix way means that no matter what you need to do with your data, there's an npm module for it. Just .pipe() your stream in and your code is done. This is amazing. And it's possible only because of how bare-bones and loose the Buffer stream API is.

Strong typing has its place, but it would ruin node's biggest selling point. It's hard to realize this without trying it.

hamburglar8• 12 years ago

Yes, and "the unix way" isn't just "everything's a stream", but rather "everything's a text stream." In that regard, node's version of "everything's a stream" is actually a half-step up in abstraction.

ixmatus4• 12 years ago

I don't know node very well and only Javascript minimally but I understand what streaming I/O is and have used generators / iteratees for a long time in many languages.

I understand the benefits and that's why Pipes in HS is such an exciting thing because it gives us a formally reasoned and general set of stream computing tools - you can compute anything with type-level guarantees. It's just as flexible and general as, say, Unix pipes but better because there are guarantees of the library's tooling and there are guarantees of the programs you produce! You can't say that in Node / JS, Python, Ruby, etc...

JS's lack of strong typing limits your ability to reason about streams (a lot more than just streams, too) and further limits your ability to write performant stream computing software. Pipes, in Haskell, give you the big three: Effects (I/O), Composition (function composition with fusion), and Streams (generators and iteratees); because of the type system Haskell (and some nudges here and there by the library author) can fuse and optimize that code to a ridiculous degree in addition to all of the other nice guarantees you get from the type system (separate of I/O from pure code, etc...)

I personally don't think dynamic typing is a selling point, ever - I write software faster and with fewer bugs in Haskell than I ever have before in Python, Ruby, Erlang, or Scheme. But that's a totally different topic and I don't want to derail this one.

Don't misunderstand me as being aggressive, please. I fully respect what people decide to like and work on, I'm just trying to expand the awareness that there are tools in existence that do it better.

zmanian• 12 years ago

I'm a Haskeller but I'll defend Node a bit.

The absolute most compelling thing about Node is how it has hacked organizational dynamics in large companies. Walmart and PayPal basically used it to completely liberate their frontend groups from their backend systems using facade system with huge improvements on customer systems.

What is sad is that all the other high concurrency systems are going to end up implementing much of the GHC runtime without the reliability of Haskell...

glenjamin• 12 years ago

If you're doing Node.js, Caolan's async library is pretty much part of the standard toolkit.

I know Caolan's been thinking about this and reworking it for a while, so I'll be interested to see whether it manages to see significant takeup.

onestone• 12 years ago

Indeed caolan/async has been a de-facto standard until recently. But for me personally it has become obsolete, with the standardization of generators and promises.

rubiquity4• 12 years ago

This. The best part of Harmony is to wipe libraries like this off the face of the planet. I dread the days of trying to pick between 10 different flow control libraries. A few years ago 5 new flow control libraries were coming out every month.

nailer6• 12 years ago

From Great British Node Conf, maybe two months ago: a speaker talking about promises asked what people use for flow control:

- promises: about 20% of the room.

- async: about 80% of the room

For me async.waterfall([list of functions]) is little nicer than 10 chained .thens().

And people advocating promises still keep saying it keeps things flat. No it doesn't, we're already flat because we're all using async. Stop pretending async doesn't exist and isn't massively popular.

And way, way better documented. Q.spawn what? And this is the best promises library?

Stack Overflow question: Simplest fs.readFile example with generators and Q?

Current only answer:

  Q.spawn(function* () {
      …
      var data = yield Q.ninvoke(fs, "readFile", somefile);
      …
  });

Answer from Highland docs:

    var data = _.wrapCallback(fs.readFile)('myfile');

- What's Q (yes it's a module, but what does it mean? Is it supposed to a misspelt queue or something else?

- What does 'ninvoking' something do?

- Shouldn't I just be able to to put the variable declaration outside of the scope?

- Why do competing Open Source implementations of the same standard exist? Can't there just one reference implementation?

That's not the future.

I might be really ignorant here. I probably am - I could read a shit tonne of docs to work out what this strange beast does and technically someone can probably do a better job answering that Stack Overflow question. But nobody has, because very few people know how to operate the current state of the art generators/promises setup.

From the Q docs: "If you have a number of promise-producing functions that need to be run sequentially"

No, I don't have a number of promise producing functions. Nobody in nodeland has that. I just have functions. I could read about turning them into promise producing functions, and calculate whether this abstraction layer is adding value, but then again, I could do productive work with async.

And from the looks of it, Highland too.

jonny_eh• 12 years ago

Bluebird is the best promises library: https://github.com/petkaantonov/bluebird

jkrems5• 12 years ago

async:

    async.waterfall([
      fn1, fn2, fn3
    ], function(err, result) {
    });

    resultP = [ fn1, fn2, fn3 ].reduce(Q.when, void 0);
    resultP
    .then(function(result) {});
    .catch(function(err) {});

If you chain thens in Q, you are not doing it right (imho).

phleet• 12 years ago

Promises aren't just about keeping things flat.

The biggest value to me is being able to avoid the passing around of callbacks and them relying on varying conventions (some async are function(args, ..., callback(err, data)), some are function(args, ..., callback(data), errback(err)), some are function({success: callback, error: errback})).

Promises solve this problem by not passing around callbacks _at all_. Instead you return the promise and let the consumer attach the callback itself. And once we have promises widely available and part of the standard library, the calling conventions will be standardized too.

EDIT: I agree that as it stands, the lack of standardization of promises (jQuery's are mutable, for instance) is a pain, and that documentation could certainly be better.

spion3• 12 years ago

  var Promise = require('bluebird');
  var fs = Promise.promisifyAll(require('fs'));

  Promise.spawn(function* () { 
    var data = yield fs.readFileAsync(somefile);
  });

Documentation: https://github.com/petkaantonov/bluebird/blob/master/API.md

My favorite example is doing a diff using an async diff service which provides a function `svc.diff(string1, string2)`. But also imagine that you need to preprocesses the files using the sync function `removeEmptyLines(buffer)`. This is how the function looks like when implemented using Bluebird:

  function diffTwoFiles(f1, f2) {
    var file1 = fs.readFileAsync(f1).then(removeEmptyLines),
        file2 = fs.readFileAsync(f2).then(removeEmptyLines);
    return Promise.join(file1, file2).spread(svc.diff);
  }

I'd love to see someone come up with a better example using callbacks and async.

gruseom6• 12 years ago

I have a question about the async library: what does it do if an asynchronous function or a callback raises an exception? The word "catch" doesn't appear in https://github.com/caolan/async/blob/master/lib/async.js.

I ask because I wrote some Lisp macros (I work in a Lisp that compiles to JS) to implement a few async patterns I need, and making sure that exceptions are trapped and threaded into the callback chain correctly was the most complicated part.

jhrobert3• 12 years ago

  Boxon.cast( fs.readfile, 'myfile' )
  .then( function( content ){ console.log( content ) })
  .catch( function( err ){ console.log( "Error", err ) });

Boxon is promise implementation agnostic, works with Q, bluebird, etc... See https://github.com/JeanHuguesRobert/l8/wiki/AboutBoxons

alasdair_• 12 years ago

Agreed, almost everyone at Groupon is using Async over the alternatives.

cjf4• 12 years ago

Really can't wait for "all the things" to stop being used as an acceptable replacement for "everything."

badman_ting3• 12 years ago

I get that arrays are meant to stand in for more asynchronous sources of data but those things seem so different to me that I don't understand why you'd want a library that treats them the same. If you need to map an array that's taken care of. I know I'm being dense, I just don't get it.

hamburglar8• 12 years ago

This is highly intriguing, but I must ask why the JS community has this fascination with obscure identifiers like "_". It decreases readability when what should be a logically-chosen descriptive identifier for your class is replaced with a single character that visually recedes into the language syntax. Edit: I've grudgingly given jquery a pass on this because of its ubiquity, but come on, a stream library? Not to mention the fact that a reasonably popular lib already seems to have squatted on underscore.

caolanmcmahon5• 12 years ago

var Highland = require('highland');

hamburglar8• 12 years ago

Yes, clearly you can do that, but it's obviously not your intent. All the docs use _ as the class name, and that's what you use internally. The class is named "_" and that's what people using it are going to expect to see if it becomes popular. You could make a better choice.

e12e• 12 years ago

var ‾ = require('highland'); // Meet overline, the new underscore

wcummings3• 12 years ago

In my codebases, literally 1/10 lines or more contain at least one call to underscore.js, frequently more than one (not even including async). Its for brevity. If you wrote out underscore.* or Highland.* your code would become hard to read. These are utility functions that are used heavily.

I'm excited about this, as someone who uses underscore.js and async together, heavily.

hamburglar8• 12 years ago

underscore.js is a pretty good sized collection of utilities, which is why you call it a lot. I am having a hard time believing you're going to be constructing so many Highland streams that you need a one-character identifier for it to increase readability.

ryanatkn• 12 years ago

JS devs rely heavily on libraries to do things that most programmers would expect the language to handle. $ and _ have been adopted as toolkit identifiers and are littered throughout most JS codebases. Highland does much of what one would expect from _ via Underscore/Lodash. I'd bet most people think they improve readability.

badman_ting3• 12 years ago

Just seems like a silly thing to get hung up on.

hamburglar8• 12 years ago

Naming is important for code readability. I think a lot of people recognize that. Which is why I'm so baffled by this trend of "screw it, I won't even bother with a name! Call everything underscore!"

Cthulhu_• 12 years ago

I agree; what I personally wonder is why Underscore and similar libraries don't make use of Javascript's prototype business and add methods to the array and object prototypes? Probably things I'm overlooking here, but, [1, 2, 3].map(stuff) is much nicer than _.map([1, 2, 3], stuff) and the like.

b3n• 12 years ago

It's considered bad practice to add add methods to the prototypes of built in objects because it has the chance of breaking other people's code.

jashkenas2• 12 years ago

Ha ha.

You're describing the state of affairs before Underscore existed, back when functional-ish programming in JavaScript was ruled by Prototype.js:

http://prototypejs.org/

... which added a lot of useful methods to native prototypes.

While handy in controlled and limited environments, mucking about with native prototypes quickly becomes extremely dangerous and difficult — once you have two third-party modules on the page that expect different versions of your patched prototype method ... once you have a new version of a browser that implements one of your previously-extended functions, but does it differently — you're pretty well screwed. Both of those things tended to happen in large sites.

owenversteeg• 12 years ago

Most people are very hesitant to add methods to array, string, etc. because if more than one person does that you'll end up with all sorts of problems.

rubiquity4• 12 years ago

Underscore uses the "_" variable because, well the library is actually named "underscore." Highland uses the "_" because... I don't have anything nice to say.

eplawless• 12 years ago

I'm curious which features highland provides which RxJS doesn't. From what I understand, composable streams from any data source with backpressure support is pretty much the definition of Rx.

Sometimes simplicity is a feature, too, though.

caolanmcmahon5• 12 years ago

Rx doesn't handle back-pressure or laziness, so it's for only really for handling events.

platz• 12 years ago

since `fastSource.map(slowThing)` automatically pauses the source while the slow thing is processing, how is this different from iteration? It also states that in the case of a non-pausable source it will buffer the data. How does it know when to pause vs when to buffer.

Is there some way to know what kind of source you've got, or are the the sources constructed in a way that chooses which behavior you get?

caolanmcmahon5• 12 years ago

RxJS advocates are unhappy with this comment so I'm going to qualify it a little. Apologies for any misunderstanding...

Rx doesn't handle automatic back-pressure (like Node Streams) but does have mechanisms to avoid overwhelming slow consumers. Rx also has delayed subscription which you can call lazy, but not by turning the stream into a pull-stream (allowing you to sequence actions in the way Highland does).

If any of the above needs further qualification or comment please weigh in on the issue by commenting here... but for now I'll leave it at that. I actually list RxJS in the blogpost because it's a good example!

ilaksh• 12 years ago

I use async.map when I need to do a bunch of file operations or something asynchronously and wait for them to finish. I have largely avoided using stream specific syntax because the event model was more familiar to me. I have used ToffeeScript instead of async.waterfall or async. series because ToffeeScript is cleaner.

With these improvements to streams in Highland making things more convenient and broadly applicable I expect to be using Highland streams for certain things.

syntern• 12 years ago

Most of the things he describes can be done with Dart's async and collections libraries. Compiles to js, works on server side, used in production. Even if someone wants to reinvent the wheel, they should take a look at those libraries and see how it is done there.

nailer6• 12 years ago

@caolanmcmahon: consider using methods?

- We have object.defineProperty() in ES5 to avoid enumeration.

- You can use user-specified prefixing to avoid future conflicts.

Eg:

    {foo: 1, bar: 2}.hlPairs();

rather than:

    _.pairs({foo: 1, bar: 2});

albertoleal• 12 years ago

This is amazing! And it's something that I wanted for some time.

LazyJS just received stream support, but I'm pretty much sold on to highlandjs.

kimjotki2• 12 years ago

It looks like this whole notion of 'stream' is just a syntactic sugar for javascript guys who lost themselves in a bunch of nested stupid callbacks, which is worse than lisp parens. Instead of nesting whole callbacks, create a stream in the middle; execute first half of callback chain and dump the result into the stream so that next half of callback chain can be executed later.

Why is this so an enlightenment for node guys? UNIX does it right since epoch - simple programs perform simple tasks and connected via pipes. Python has gevent, so you don't even need 'a stream' or other bullshit, you just write the code as-is and greenlets provide the concurrency needed.

The real enlightenment comes from 'programming properly'; you start with C and torture your brain with function pointers and realize why it is a good idea to treat functions as a first-class objects. then you learn some 'proper' functional programming languages like lisp or something to learn how to think in functional way. which is the only guaranteed and proven path to prevent yourself from shooting your own foot by writing 20+ nested callbacks. If you start with binding an anonymous function to a <button>'s click event and think you can do this to do real programming, you'll never get it right.

woah3• 12 years ago

Wise words of wisdom from kimjotki2, the only real programmer on the internet. Bow down, bitches.

coldtea• 12 years ago

This is not reddit.

tjholowaychuk• 12 years ago

yup! node does streams horribly wrong

jhrobert3• 12 years ago

I am considering the idea of spending some time studying node streams. Your remark worries me. Is there some sources I can consult about node stream potential weaknesses? Thanks.

mattgreenrocks• 12 years ago

What is it about JS that causes people to re-invent everything and call it progress?

lukasm3• 12 years ago

I really like Node, but I want a language with sane semantics. What's the best option apart from JS and CoffeScript. TypeScript is awesome, but I'm afraid that there will no good community, because of MS stigma.

codygman2• 12 years ago

You can use Haskell for the same semantics if I understand you correctly. Actually coming from Node, Go might be a better fit.

lukasm3• 12 years ago

In other words I would like to use language that runs on Node, but it's not JS or CS

rubiquity4• 12 years ago

When did sane and static typing become synonymous?

lukasm3• 12 years ago

sane has nothing to do with static typing. Python is a language which is dynamic and has a sane semantics. e.g. in JS {} + {} is NaN and many other examples.

cgag• 12 years ago

Clojurescript sane and pretty mature now. http://himera.herokuapp.com/synonym.html

outside1234• 12 years ago

Definitely a good community, have a look at all of the type libraries alone:

https://github.com/borisyankov/DefinitelyTyped

michaelwww• 12 years ago

Google Dart and TypeScript are pretty good alternatives to JS in some cases and Dart has the notion of streams.

https://www.dartlang.org/docs/tutorials/streams/

_greim_• 12 years ago

Perhaps Dart?

mcgwiz• 12 years ago

Along these lines, John Resig recently created http://nodestreams.com/ as a way of composing and conceptualizing stream processing. Pretty nifty.

How I want to write Node: Stream all the things

Related Stories