Pipelining – A Successful Data Processing Model

http://blog.stuartowen.com/pipelining-a-successful-data-processing-model

by stuartaxelowen • 11 years ago

57 8 11 years ago

blog.stuartowen.com

misterdata• 11 years ago

In fact, pipelining is supported in a lot of systems already. RethinkDB (http://rethinkdb.com) is built around a query language of chained data operations. Warp (OS X tool) provides a user interface for pipelining and executes its queries on regular SQL databases (https://pixelspark.nl/2015/warp-a-query-by-example-analysis-...)

dougabug• 11 years ago

Oracle has had pipelined table functions since version 9.0.

agibsonccc• 11 years ago

Disclaimer: Committer on this library. We actually built canova[1] for unstructured data. Docs aren't quite there yet but I'd love to float the concept out there for people to get feedback on.

In ML we like to term this as vectorization rather than pipelining. Same idea though.

Many libraries like binding the ML with the feature vector transformations. Our goal was to isolate this process in to something composable.

General idea is file -> feature vector/matrix

[1] https://github.com/deeplearning4j/Canova

thmcmahon• 11 years ago

F# and R both have good ways of doing this. See http://cran.r-project.org/web/packages/magrittr/vignettes/ma...

sebastianavina• 11 years ago

R is one of those heavily misunderstood languages because it uses features so advanced yet so common once you are a seasoned programmer that nobody cares to explain or even make a blog post about.

graffitici• 11 years ago

Somebody rediscovered functional programming, or am I missing something?

aidenn0• 11 years ago

Sounds a lot like http://en.wikipedia.org/wiki/Lucid_(programming_language)

visarga• 11 years ago

You mean what we have already been doing for decades with head, tail, cat, cut, grep, sort, uniq and awk (or inline perl)? Only problem is that it doesn't validate types and has a weak error system.

After learning about functional programming, I was amazed how advanced is unix the command line. The level of composability is famous. To think that the common unix pipe is related to FP, blows my mind ...

Another place where this pattern is widely used is in jQuery, where the pipe is much smarter. Also, see Lazy.js for pipe-like lazy data processing.

Pipelining – A Successful Data Processing Model

Related Stories