Prismatic's "Graph" at Strange Loop

http://blog.getprismatic.com/blog/2012/10/1/prismatics-graph-at-strange-loop.html

by janaboruta • 14 years ago

106 45 14 years ago

blog.getprismatic.com

a-priori• 14 years ago

Very interesting, needs more examples though.

It seems like this fits a similar purpose to Nathan Marz' Storm framework. Could you do a bit of a compare/contrast thing between Graph and Storm?

programnature• 14 years ago

Graph is not distributed, at least not in the way that Storm is. Notice the post doesn't talk about assigning individual components to individual machines. AFAICT its intended to run on a beefy box, and then horizontally scaled.

tlrobinson• 14 years ago

You could probably write a higher order function (like "observe-graph" in their example) to distribute the work across multiple machines. Maybe even map it onto a Storm cluster automatically.

w01fe8• 14 years ago

Thanks! There are some more complex examples in the slides, I just didn't want to make the blog post any longer.

The main similarity between this and Nathan Marz's Storm framework is that they both rely on a declarative expression of the structure of the computation graph.

However, beyond that there are many key differences. Storm is a graph computation framework, which compiles your specification into a distributed real-time computation pipeline. In contrast, Graph is just a library for expressing composition structure, but says nothing about execution strategy.

In principle (with a lot more code and some more annotations), one could compile Graphs into distributed real-time topologies like Storm. For now we've been using Graph in-process for real-time processing. But because Graph is so simple and close to the language, it's very easy to apply to new situations and build new abstractions on top of. For example, we also find Graph useful for expressing the composition structure of our production services, which are built up from many components.

jwr• 14 years ago

But, from what I understand, if I structure my software using Graph, it should be much easier to run it using Storm, even without automated tooling. That way Graph becomes a stepping stone: structure your software, and if you need Storm's parallelism/scalibility, define a topology using the same functions.

aleyan• 14 years ago

Nice. Assuming that the computation of each node is memoized, can we 'dirty' a single node, so that during the next computation only the needed parts of the graph are recomputed? This would make multiple computation of models where only a few parameters are changing much more efficient.

Also, has this at all been inspired by SecDB's powerful graph oriented features?

w01fe8• 14 years ago

We haven't had the need for that application yet, but it's certainly possible -- that's why declarative specifications are great :)

I haven't heard of SecDB, but I'll definitely check it out -- thanks!

gajomi• 14 years ago

This is definitely a use case I am interested in. I have been playing around with computer algebra in Clojure for awhile where it is critical to keep track of these dependencies (I have a hacky implementation which relies on metadata so that the resulting data structure is out of site). But of course, when you actually want to evaluate something, this is when you need to get sophisticated. There can be multiple paths in the graph to arrive at the same node and you have to decide. With memoization at nodes being an important consideration. Anyway, I would be very happy to see an open source version of this!

P.S. As I understand it all the graphs are dags. In my line of work (which involves doing stuff at the interface of probability and statistics) there are quite a few recursive functions that are difficult to unroll explicitly, or are otherwise undesirable to unroll for the purposes of readability. Can we expect to see circular dependencies at some point. Well, I know, the halting problem and all, but still...

jpadvo2• 14 years ago

This sounds very useful. I'm having trouble understanding the nature of the system, though, because I don't really understand lisp / clojure. Is this a pattern that could be applied in any language with first class functions, or does it require a lisp to be practical?

w01fe8• 14 years ago

Sure! This particular way of expressing declarative structure may be specific to Clojure, but the basic ideas should be useful in any (especially dynamic) language with first-class functions.

The first idea is to take a complex system and express the set of components and their relationships declaratively, rather than procedurally. This is an old idea, and makes it possible to reason about the system, mock out components for testing, map over the components for monitoring, and more.

The second idea is simplicity. By making this declarative language as simple as possible, it becomes trivially easy to do all the things I just mentioned using the existing tools in the language, without needing to write complex library functions to support each use case.

For another example of a declarative system for composition you could look at 'react' in JS, which can be used in similar ways to Graph (and also supports things like async composition): https://github.com/jeffbski/react

wonderzombie• 14 years ago

react looks interesting, although man, the stringly-typed bit sorta reinforces a lot of the Lisp quotes about being able to modify the language, etc. (Disclaimer: I've only just started studying Clojure in earnest over the last month or two. No zealot like a convert, eh?)

jpadvo2• 14 years ago

Thanks! That makes a lot of sense, and react looks like a really interesting project. I'll have to go back and reread the article with an eye for applying its ideas in Javascript / Coffeescript.

grayrest2• 14 years ago

You can do it in any language with first class functions and reflection.

Here's a stupid simple js implementation of their example:

https://gist.github.com/3874826

Edit: Loaded the page and got interrupted by lunch. Shows me for not refreshing.

w01fe8• 14 years ago

That's awesome -- thanks for posting

jessepollak• 14 years ago

Just a heads up, it looks like something is up with your CSS:

http://cl.ly/image/241X1C2D3O35

I'm on 10.8 with the latest version of safari.

In other news, this looks pretty cool.

janaboruta3• 14 years ago

Are you using a Retina Macbook?

renderkid• 14 years ago

I have the same issue and i'm using a Retina MBP.

salimmadjd2• 14 years ago

Same issue on iPad 3 that has retina display

marbemac• 14 years ago

Yup looks like a Retina problem.

abp• 14 years ago

I had this same problem on my Android phones Chrome browser and went to my laptop for reading.

pnathan2• 14 years ago

If I understand the nature of this system, it's essentially a higher-order function composition tool. Correct?

Nit, defnk? what about def-kw-func or something? not wholly readable as defnk.

w01fe8• 14 years ago

Yep, that's correct.

Point taken about the names, thanks for the feedback. Since Clojure already uses defn and fn, defnk and fnk seemed natural. But maybe defn-keyword and fn-keyword would be better.

pnathan2• 14 years ago

I really like this idea. Do you have a Common Lisp port of this, out of curiosity? If not I might have to see what I can come up with. :-)

I like to have fully expanded names, ala Common Lisp. My editor will autocomplete those on demand after I use a name once, so it's not a hindrance to reading or writing.

My goal for identifiers is that I can read without consulting my internal lookup table.

dkincaid• 14 years ago

There was a defnk at one time in one of the Clojure libraries (I think it was even created by Rich himself), but it disappeared at some point. Nathan Marz created a defnk macro in Storm that he uses for the same thing. Maybe it's time for Clojure to bring defnk back if it offers some advantages over the :keys destructuring.

salimmadjd2• 14 years ago

I love reading prismatic's blog. You have some really smart people there. However, it seems like you're trying to solve everything from a purist view and I think your product is suffering as a result. I have made the same suggestion consistently that would make me use it more without any reply. Basically, I've stopped using the product as the UX does not match the reason I would want to use the product.

janaboruta3• 14 years ago

I'm sorry about that. If you're still interested in giving us UX feedback send us an email feedback@getprismatic.com. I'll personally respond to it. Please understand that we're still a small team and we get a lot of emails on a daily basis. :-)

dmix• 14 years ago

I agree with the author, the UX is definitely under-developed and I've also given many feedback msgs.

We only say this because we care :)

tensor• 14 years ago

I'm curious what your complaints are, do you mind sharing? Especially because you invoke "purist view", which is rather confusing talking about UX, a very subjetive area.

For myself, I feel that the main feed could use a lot more whitespace and have less information about each story. For example, I wouldn't mind being able to toggle off the social quotes completely and perhaps even the images (images are nice for some sources, but not others).

emeraldd• 14 years ago

Graph reminds me of working with Spring (from what I remember of that). The concepts sound very similar to Spring's auto-wiring.

DennisP• 14 years ago

Happy to see you're opensourcing. Are you planning to do the same with Flop and Optimize?

w01fe8• 14 years ago

Flop is definitely on the list (with about 10 other libraries), but I'm not sure about our optimization code. We plan to start a discussion of what we should prioritize soon, so stay tuned.

tel2• 14 years ago

This feels like lazy evaluation bolted on to Clojure (cool!). There's further benefit since you get some dataflow abstraction—which then reminds me of the Par monad.

http://community.haskell.org/~simonmar/par-tutorial.pdf

mattdeboard• 14 years ago

There's no need to "bolt on" lazy evaluation since (future) and (delay) are parts of the core language (not to mention lazy seqs)

moomin3• 14 years ago

Well, future, delay and lazy sequences are bolt ons to an eagerly evaluated language. I'm not convinced the laziness is Haskell is worth it, tbh, but its true that you really wouldn't need this project.

blaines2• 14 years ago

Is it just me or is the prismatic blog text being cut off on the side? https://twitter.com/blaines/status/256545302926995456/photo/...

moomin3• 14 years ago

Happens on the iPhone to everyone, I think.

blaines2• 14 years ago

Appears this may be a retina mbp thing...

xtrahotsauce• 14 years ago

happens to me on an iPad as well

moomin3• 14 years ago

This is great, but I wish they'd take it one stage further. At the moment, the first example they give (monolithic let) is pretty idiomatic and readable. The transformation from one to the other is pretty automatable. So for most cases, why not just put a macro on top? e.g. defgraph-asfn

Storm requires some manual intervention because you need direct control over how things are distributed.

rafaelferreira• 14 years ago

Pretty cool project, like a functional reinterpretation of dependency injection frameworks. The basic idea behind dependency injection is to make all dependencies for each unit explicit and externally supplied.

DI frameworks then build upon the pattern offering a declarative DSL to declare dependencies and wiring. They also offer a selection of alternatives to build dependent objects (e.g. build a fresh one when needed or supply a global instance for every dependee) and a smidgen of generic functionality via impersonation (e.g. monitoring, logging, timing, transaction acquisition/release).

Interesting that untyped functional programming and macros allow for a lighter expression of the same basic idea.

Also interesting that maps of functions are a fundamental part of the solution. There are definitions of object-orientation that amount to basically records-of-lambdas. I've long thought that there is a lot to gain by using this basic abstraction as it naturally provides seams for testing/observability/auditing/etc.

Prismatic's "Graph" at Strange Loop

Related Stories