Node.js Stream Playground

http://ejohn.org/blog/node-js-stream-playground/

by jeresig • 13 years ago

130 15 13 years ago

jedireza• 13 years ago

Wow. nodestreams.com is great. It's really helpful to see the results of different chained combinations instantly. Thanks for making this John.

marknadal2• 13 years ago

Anybody know how to dynamically zip things on the fly? Like if you want to generate a download for a user that has files and folders... how can you do this fast and in one go? Currently I have to literally write to disk, then zip it, and only then can I finally send a response. Any help?

jeresig• 13 years ago

Absolutely! Poking around I found one module that could do it - I'm sure there are more (and better ones, too): https://github.com/wellawaretech/node-zipstream

In the example there you would .pipe() out to an HTTP response rather than a file, presumably.

marknadal2• 13 years ago

Sweet, thanks. I think I had found this a while ago but lost it, looks great!

...holy smokes, John Resig just replied to me. Thanks so much for jQuery, it has changed my life and is what made me become a programmer. (I now do client work for 20<20Thiel Fellows, and my startup has been in WSJ - none of which would have been the case if I hadn't started programming with jQuery!)

maxogden• 13 years ago

for an example of doing this with tar instead of zip check out https://npmjs.org/package/dir-tar-stream

contrahax• 13 years ago

node-archiver is the best ive used so far... imo

k3n• 13 years ago

My only complaint about Resig's blogging is that he doesn't do it enough! It's rare that I read one of his posts and don't come away having learned a lot.

Not as much to learn today, but with the really neat tool I can certainly learn from it, so I can't complain.

ilteris• 13 years ago

another useful resource for getting familiar with streams: https://github.com/whatwg/streams

ilaksh• 13 years ago

Great tool. However, since this is an extremely obvious example, I have to attempt once again to point out that textual programming languages are far from ideal representations in many problem domains. The inability or refusal of programmers to accept visual and interactive tools is the main thing holding back software development. I think the main reasons are related to false outdated beliefs about the nature of programming and a fear of being labeled as a novice, since interactive graphical tools are always categorized as being for novices. http://stackoverflow.com/questions/461796/dataflow-programmi... http://noflojs.org/

lambda• 13 years ago

The problem with every visual programming language that I've ever tried is not that it's aimed at novices and I'm afraid of being labelled that way. It's that there's a whole wealth of tools for dealing with text based programming that just can't be used with visual programming languages without a huge amount of effort (if at all). Visual programming languages don't work well with revision control; they usually don't have diffing and merging tools. They don't work well with grep.

They tend to have a fairly low information density on the screen. When looking at a simple program that all fits on a single screen, that's fine. But when working with an actually real world code base, when I don't understand exactly what it's doing, I need to be able to see as much on screen as possible at once to keep it all in context, while browsing through to figure out how all of the pieces link together. All of the empty space taken up in a visual language makes it a lot harder to get as much information on the screen at a time.

And the programs can frequently be quite hard to follow. Tracing all of the wires between different nodes, figuring out what's stacked on what.

There's a reason that despite the very obvious circuit diagrams you could represent electronics with, hardware engineers for integrated circuits don't use a visual tool for laying out their hardware, they use Verilog or VHDL, only compiling it down to the actual netlist and laying it out at the very end.

I have never seen a visual programming language that I would say could scale to anything beyond a few basic demos, which could just as easily be accomplished in a page of code.

austinz• 13 years ago

Xilinx actually has (had?) a visual designer tool for their FPGAs in their IDE, allowing you to connect signals between blocks and do other stuff. It didn't work well and made for unmanageable designs. I've heard similar things about complex LabView programs becoming unmaintainable spaghetti code, although I don't know if that's actually true or not (never used it myself).

siddboots• 13 years ago

The inability or refusal of programmers to accept <paradigm> is the main thing holding back software development.

Visual programming with streams is nothing new, nor is it maligned or ignored. Pure Data is a good example of a domain where it has been a popular choice for several decades.

There are limitations that come with the flow abstraction, and there are strengths that plain text will always have over other approaches.

Having awareness of the diversity of programming paradigms and tools is a good thing, but looking for a silver bullet in any particular approach is surely not a good thing.

hackula1• 13 years ago

...since interactive graphical tools are always categorized as being for novices.

Tools aimed at novices tend to ignore some of the fundamentals of solid software development, such as automated testing. In my space, GIS, there are lots of visual modeling tools, and these inevitably grow to a certain size, then get ported over to a textual programming language when maintenance becomes a nightmare.

There is a place for visual programming like there is a place for picture books (graphic novels even push into some pretty serious territory), but at the end of the day text wins with both prose and code, since it is far more expressive where it comes to expressing reasoning and logic. Doubly so perhaps, because unlike stories or art, code is typically not up for subtle interpretation.

gobengo• 13 years ago

Those who enjoyed this and streams2+ in general may find useful this library we've made to use objectMode streams in the browser. http://github.com/Livefyre/stream There are quite a few tests if you want to get a feel for how streams work: https://github.com/Livefyre/stream/blob/master/tests/spec/re...

For a concrete implementation, check out our SDK. We use streams to abstract away the details of paging out Content from our APIs. There is an example in the README. https://github.com/Livefyre/streamhub-sdk

These streams power the bottom 5 apps you see here: http://apps.livefyre.com/

rashthedude• 13 years ago

I started looking into Streams 2 days ago and boom this headline greets me on hacker news today. Thanks a bunch Mr Resig.

Node.js Stream Playground

Related Stories