Scalable Program Architectures

http://www.haskellforall.com/2014/04/scalable-program-architectures.html

by mightybyte • 12 years ago

131 79 12 years ago

quchen2• 12 years ago

Gabriel is also the author of the popular Pipes Haskell library, and nice person in general. Both of these make me heartily recommend his other blog posts about various Haskell topics, which often portray advanced features remarkably well.

Examples:

Scrap your type classes: http://www.haskellforall.com/2012/05/scrap-your-type-classes...

Hello Core: http://www.haskellforall.com/2012/10/hello-core.html

Coding a simple concurrent scheduler yourself: http://www.haskellforall.com/2013/06/from-zero-to-cooperativ...

Tutorial as part of library doc (!): http://hackage.haskell.org/package/pipes-4.1.1/docs/Pipes-Tu...

the1• 12 years ago

and quchen is trusted authority. you can like whatever quchen likes, too, until there's heartbleed.

dllthomas2• 12 years ago

Pardon?

gamegoblin• 12 years ago

Agreed. I sent him an email asking for help on proving some category theory laws for some datatypes, and he wrote back a several page email answering everything in detail.

briantakita2• 12 years ago

This is one reason why you should learn Haskell: you learn to how to build flat architectures.

I suspect that most people learn the advantages of flat architectures with experience using any programming language. For me, I started to "get it" with Javascript & Ruby. So Haskell hardly has a monopoly on this.

I understand that Haskell has a different take on this. It seems the author has found a pattern in Haskell that he frequently uses.

Ideally, I can use patterns in multiple languages, so my experience can seamlessly transfer.

This often means there's a lowest common denominator to a particular pattern. I'm afraid that if I learn a pattern unique to Haskell, that it is not applicable to any other language. I suspect that there are underlying & common principles that are expressed differently with Haskell.

jroesch• 12 years ago

I think his argument is that these patterns are the easy thing to do in Haskell, not that they are necessarily unique.

The Monoid pattern is universal, but most languages don't give you tools to talk about it as an entity.

It is harder to enforce something like implementing a Monoid Ruby or JavaScript but you can always implement mappend, mempty and use "duck typing". Haskell's advantage in encoding these patterns is static checking, and type classes.

T-R2• 12 years ago

These are not patterns unique to Haskell. In fact, it's just the opposite: these are mathematical abstractions, and so they can be used anywhere that you have use of function application, in stark contrast to the alternatives. By using these patterns instead of Ruby or Javascript-specific, Application-specific, or Object Oriented-specific patterns, you can be more confident that your abstractions will be consistent with eachother, and as a result, more composable.

Case in point: I'm currently developing in Javascript, but I use these patterns (Monad, Applicative Functor, etc) to manage things like asynchronous code and error handling, because they're far more flexible/composable and reliable (in terms of not having side effects) than the other offerings on hand, and I know that will always be the case, because these abstractions were designed with the constraint of composability in mind: in fact, they are, by definition, the minimum set of constraints required to get these emergent properties. The benefit that Haskell itself offers is that it can check at compile time that these constraints hold true.

briantakita2• 12 years ago

I'm currently developing in Javascript, but I use these patterns (Monad, Applicative Functor, etc) to manage things like asynchronous code and error handling

Thank you for this comment. I'm interested in knowing which these fundamental patterns that Haskell emphasizes.

Unfortunately, time is limited & currently, I'm not able to allocate enough to learn & use Haskell in a meaningful project. I recognize that there are some important abstractions & I'd love to learn all of the useful abstractions that are applicable to other languages & environments.

aaronlevin4• 12 years ago

Ideally, I can use patterns in multiple languages, so my experience can seamlessly transfer.

In many cases you are not guaranteed via the type system that one individual's implementation of the Observer pattern (for example) is not full of bugs, contains side effects, or closely adheres to the pattern's spec.

I believe Gabriel is hinting at the theoretical underpinnings of what it means to be composable, as opposed to just talking about how things can be mashed together (via pattern or interfaces or contracts or testing or whatever).

twic6• 12 years ago

In many cases you are not guaranteed via the type system that one individual's implementation of the Observer pattern (for example) is not full of bugs, contains side effects, or closely adheres to the pattern's spec.

Nor are you in Haskell, of course. Nothing about that language prevents code being full of bugs, nor failing to adhere to a specification, nor, given the existence of unsafePerformIO, having side effects.

Certainly, a competent Haskell programmer will not write bugs, will adhere to the specification, and will not introduce hidden side-effects. But then, nor will a competent programmer in any other language.

twic6• 12 years ago

The idea that this composability is unique to Haskell is false. Java's streams are an example of exactly the same thing in another, non-functional, language. The Gang of Four "decorator", "composite", and "chain of responsibility" patterns are further general examples.

It is true, perhaps, that in Haskell, programmers reach for homogeneously-typed composition more readily than programmers in other languages. Good for them! But it's either arrogance or ignorance to assert that this is a special Haskell thing.

Furthermore, i am dubious that this really is a good strategy for building large programs. The idea that you can combine lots of parts of some type to build a bigger part of the same type is extremely appealing. But in my experience, the bigger part often has slightly different properties, behaviours, or uses which warrant a different type with more features. Unless you want to impose those features on the smaller types too.

For example, consider a batch application which processes files through a number of stages (i realise it's the 21st century, but apparently we still need to do this). There is clearly a type for a stage, with values for things like uncompressing, validating, renaming, parsing, etc. There is probably going to be a type for a chain of stages, with values for various uses of the application. A chain looks like it should have the same type as a stage - ultimately, both take a file in, and spit a file out.

But then, it turns out that we want to move the file through a sequence of directories, one for each stage, as we process it (the operations guys are really keen on this). Furthermore, we need to be able to report on what files are currently at which stage. So, a stage knows which directory it owns - presumably, it has a property of type directory for that. But a chain owns all the directories of its component steps, so it owns several directories - it's going to need a property of type collection of directory. So what do you do? Report a single directory for the chain, and somehow expose the rest through a backdoor? No, that's a kludge. Have every step report a collection of directories, which will mostly be single-element collections? No, that's weak, because the type of a step no longer fully describes its the constraints on it. Use a higher-kinded type parameter, so the chain can have a collection of directories, while the steps have a single one? Mad wicked, but racks up the reader's cognitive load. Use different types for steps and chains? Well, actually, since that's simple and doesn't have any practical drawbacks, probably yes.

Lambdanaut• 12 years ago

Why do you need a type for chain? The steps are perfectly composable as they are.

mightybyte3• 12 years ago

Or you could just make a sequence of several stages itself be a stage, collapsing the preponderance of abstractions like this post talks about. Then, instead of having a mutable property holding a directory, you might just use a function that takes a directory and returns a stage. You could have another function that takes a file and returns a stage. Guess what...this is starting to sound an awful lot like a monad.

Departing a little from the OP's topic, I think it's interesting that your outlined design relies significantly on mutable state ("a property of type directory"). I think one of the biggest advantages of Haskell is its purity, which discourages you from thinking about things in terms of mutable state. This has far ranging positive effects. It makes APIs much easier to understand and use. That in turn results in Haskell programs having MUCH more code reuse than other languages.

twic6• 12 years ago

I think it's interesting that your outlined design relies significantly on mutable state ("a property of type directory").

The design does not rely on mutable state. The property doesn't have to be mutable. It doesn't even have to be a field - in an object-oriented language, it could be a method that collects the directories from the child steps. You appear to be dressing me in a strawman's outfit.

Then, instead of having a mutable property holding a directory, you might just use a function that takes a directory and returns a stage. You could have another function that takes a file and returns a stage.

What? Could you expand on this? The thing i'd like to do would be to take a chain and make a list of the files in its various directories. How would i use your functions to do this?

danidiaz• 12 years ago

Come to think of it, Maven maps plugin goals to execution phases in a "monoidal" way. Each plugin has a number of goals, and each goal can be mapped to one or more phases: https://maven.apache.org/guides/introduction/introduction-to... In a way, each plugin is a "mini-lifecycle" in itself.

When more than more goal is mapped to a phase, the order of execution corresponds to the order of the plugins in the POM (which means that "plugin aggregation" is associative).

loup-vaillant15• 12 years ago

The idea that this composability is unique to Haskell is false.

Haskell is one of the rare language/community where composability is the default. Even purely technically, few languages do composability as well as Haskell.

The Gang of Four pattern you mention most likely work around weaknesses of Java. As such, they demonstrate the shortcomings of Java, not it's strengths. If you need a pattern that badly, make it a language feature, dammit. (Even without Lisp-style macros, source-to-source transformation is quite viable.)

But in my experience, the bigger part often has slightly different properties, behaviours, or uses which warrant a different type with more features.

I'd say most of the time, you just failed to capture the commonalities. More specifically, you seek generality through exhaustiveness, when you should use genericness instead.

If you stumble upon something that looks like a monoid, except for such and such detail, then the details probably need to be either removed (they're could be design bug), or integrated into a generic bucket or something.

For example, […]

Those specs suck. Whoever asked you this don't understand their own needs, or are struggling with a legacy behemoth that should eventually be replaced. (EDIT: or, as mightybyte suggested, you just approached the problem the wrong way. Which strengthen my point about failing to see the commonalities.)

tel4• 12 years ago

I'd say most of the time, you just failed to capture the commonalities. More specifically, you seek generality through exhaustiveness, when you should use genericness instead.

THIS is one of the most important things I've ever learned from programming Haskell[0]. The way to make things generic is to whittle away all of the sharp edges and implement them as compositions atop base patterns. Those base patterns arise out of polymorphism, not exhaustiveness.

You don't make lists generic by realizing that you need a list type to have all the properties of a string, a tree, a vector... You make it generic by realizing that much of its structure arises by forgetting what's inside.

[0] Which is not to say you can't learn it elsewhere, or that you cannot do it elsewhere. Instead, Haskell has the facility, the philosophy, and the community to do it really effectively all the time and so if you read Haskell code you'll be floored by some of the great examples possible there.

twic6• 12 years ago

Those specs suck. Whoever asked you this don't understand their own needs, or are struggling with a legacy behemoth that should eventually be replaced.

That may or may not be true, but presumably you're not suggesting that Haskell is unsuitable for writing software for users whose specs suck, or don't understand their needs?

jbooth2• 12 years ago

The gang of four book predates Java the language. FYI. The book's written in Smalltalk.

pjmlp• 12 years ago

The Gang of Four pattern you mention most likely work around weaknesses of Java.

Except it was written with Smalltalk and C++ in mind.

T-R2• 12 years ago

The idea that this composability is unique to Haskell is false. Java's streams are an example of exactly the same thing in another, non-functional, language. The Gang of Four "decorator", "composite", and "chain of responsibility" patterns are further general examples.

Certainly the concept of composability isn't unique to the Haskell community, no one's implying that it is. In fact, part of the point is that even these specific abstractions aren't unique to Haskell, but available anywhere, as they've been borrowed from mathematics. Composability and consistency are things that the mathematics community does exceedingly well, and the Haskell community has chosen to learn from them, while most other language communities do these things poorly, as they've chosen to reinvent the wheel. The GoF Design Patterns, in particular, are a rather verbose reinventing-the-wheel of well-known concepts like partial application, but with inconsistent vocabulary and implemented with regards to the limitations of some specific language's semantics, and do a poor job isolating the minimal set of constraints necessary to get the intended emergent properties. Calling a person or community arrogant for trying to highlight the utility of learning these lessons isn't helpful to anyone.

rapala• 12 years ago

So stage is some transformation of the file and a working directory

  data Stage = Stage (File -> File) Directory

If we want to construct chains of Stages where we can still inspect any individual Stage, a monoidal composition wont cut it for the reason you stated. But this raises the question. What is the monoidal composition of Stages?

Well first of all, we must do something about the parameter Directory because the second type parameter must be a monoid too (the transformation function all ready is). But lets assume that it is, and see what we get.

  mconcat (Stage f dir1) (Stage g dir2) = Stage (g . f) (mconcat dir1 dir2)
  mempty = Stage id (mempty :: Directory)

So so if we can create a new directory from two directories, we can compose two Stages to a new one that uses the newly created working directory. Or if the second parameter of Stage is a list of directories, the new Stage uses two working directories. They aren't the idea of chains that you had in mind, but they still sound useful, at least to me.

quchen2• 12 years ago

(Minor nitpick, your definition is `mappend` [1]. `mconcat` is `foldr mappend mempty` [2], i.e. concatenates a list of monoids.)

[1] mappend :: Monoid m => m -> m -> m

[2] mconcat :: Monoid m => [m] -> m

DougBTX• 12 years ago

It seems that each stage will need to know about at least two directories, an input directory and an output directory.

If it was implemented with a chain type and a stage type, the "run it" command would presumably have to read the directory for the second stage, then pass that into the first stage as an argument when processing the chain.

Alternativly, each stage could have a directory property and a next stage property. Then when the stage runs, it could ask its next stage for the output directory. There wouldn't be a chain type as such, only links in the chain of stages. The "status report" query would return a list of statuses by prepending its status to the "status report" of the next stage. Joining two chains would compose by joining the last stage of one with the first stage of the other. To run the chain, there would only need to be a reference to the first stage.

The linked-list of stages might have too much flexibility, for example there would be nothing to stop you having a particular type of stage which branches out to multiple other stages, and there may need to be safeguards against loops in the chain, say, by making the next-stage property immutable - though that would force the chain the be built up in reverse, which may be inconvenient.

dragonwriter• 12 years ago

It seems, as DougBTX points out, that in your example each stage has one input directory and one output directory. This is true, also, of the chain itself (the chain's input directory is also the input directory of the first stage, while the chain's output directory is the output directory of the last stage.)

So, it doesn't seem like there really needs to be a difference in structure at all.

eternalban• 12 years ago

An calling it "architecture" is a bit of stretch as well.

gregw134• 12 years ago

If anyone wants to contribute to a similar project, I've started a concurrency framework for Java: https://github.com/Gregw135/Simple-Java-Concurrency-Framewor...

dkarapetyan2• 12 years ago

I think the general principle here is combinator based approaches to program structure. Combinators are just as easy to use in object oriented languages as they are in functional languages especially languages that allow some form of operator overloading. The following is all valid ruby code

  f > g
  f | g
  (f | g) >> lambda { ... }
  (f > g) >> lambda { ... }

Taking these ideas a little bit further you end up with mini DSLs purpose built for expressing things very concisely. In fact if you squint a little bit you could imagine the above code expressing some form of BNF grammar as Ruby code and there are several parser combinator libraries out there that do exactly that.

I find it a little annoying that the general principles get lost behind smoke and mirrors like monoids and monads when things are in fact much more accessible and do not require anything other than some basic understanding of abstract algebra. It's the algebraic and not the fancy static types approach that has paid the most dividends when it comes to how I structure my code for readability and maintainability.

prutschman2• 12 years ago

The important part of what I guess you could call algebraically structured combinators is that they must follow certain laws. A type system can certainly help enforce at least a subset of those laws, but you're right that you don't have to have one in order to make use of the structures.

Where you lose me is in saying both that a basic understanding of abstract algebra is helpful and that monoids are part of the "smoke and mirrors"; I'd consider monoids a part of a basic understanding of abstract algebra.

dkarapetyan2• 12 years ago

Poor choice of words on my part. Yes, monoids are indeed a very basic kind of algebraic structure but the tutorials for these structures are always in a language like haskell where the actual simplicity of the concepts gets lost behind the type system. Type systems in my mind are logical structures and even though there is a very close relationship between algebra and logic one doesn't always have to tie the algebraic structures to some kind of type system to make sense of them.

Scalable Program Architectures

Related Stories