Structuring Clojure applications

https://yogthos.net/posts/2022-12-18-StructuringClojureApplications.html

by nefreat • 4 years ago

198 63 4 years ago

yogthos.net

TacticalCoder• 4 years ago

A similar post, also mentioning protocols and integrant, was posted here a few days ago and may also interest some:

https://mccue.dev/pages/12-7-22-clojure-web-primer

dwohnitmok• 4 years ago

It's interesting to see a lot of FP communities independently arriving at the same architectural structures. See e.g. Haskell's "handle:" https://jaspervdj.be/posts/2018-03-08-handle-pattern.html

seanc• 4 years ago

Convergent design is certainly a thing, but it's also quite possible that communities are cross-pollinating ideas. Either way, when you see those recurring patterns that's certainly a strong indicator that one should pay attention.

logistark• 4 years ago

For me, protocols i tend to not use it, because it makes it harder to understand the code and Cursive cannot find instances that implements the protocols.

For testing purposes is easier to redef a function than implementing a full new test protocol.

yogthos8• 4 years ago

I agree that in most cases with-redefs works fine, and I tend to use protocols sparingly myself for the same reasons.

I thought it was worth mentioning that you can use them to encapsulate any effectful code since they can be used as a tool to help enforce a bit of discipline. If you're just calling functions that can cause side effects then it can get tricky to figure out what all the functions that need to be redefined are. You basically have to read through all the code to know what you need to mock. If you stick all the side effects in a protocol, then you're being very explicit about what needs to be mocked out.

haolez3• 4 years ago

I'm an experienced developer and I'm getting the feeling that advanced languages are getting less relevant for most applications, since you usually just need a little glue code to glue together mainstream solutions or managed services. I don't need the power of Clojure to connect SQS to Lambda with some extra custom logic.

But Clojure does look amazing :)

yogthos8• 4 years ago

For me it's mostly a quality of life issue. I find Clojure workflow is far more pleasant than most languages because it's interactive, and I like Clojure as a language because it's small and focused. I find the main feature of Clojure lies in its simplicity as opposed to advanced features. Clojure code tends to be very direct where you're just passing data through a series of transformations, and you apply a few common patterns to solve a wide range of problems.

ballpark• 4 years ago

Interesting point. I've recently taken on a client who insists on using C# with their cloud solution. It's killing me. Though you're calling Clojure "advanced", what I'm missing is its simplicity.

haolez3• 4 years ago

Maybe a better classification would be "amazing languages" vs "good enough ubiquitous languages" :)

austinbirch2• 4 years ago

I can see why Clojure is seen that way ("advanced", "powerful"), but a big piece of it’s design is Rich Hickey’s view that "we can make the same software we're making today with dramatically simpler stuff."

Learning Clojure has made working in other languages much harder for me - pretty much all of my Clojure programming is functions plus transforming data structures (mostly maps). So few concepts to keep in your head!

The majority of my work is in TypeScript these days, and I always write the most Clojure-y TypeScript I can manage.

haolez3• 4 years ago

Why not use ClojureScript and transpile it to TypeScript/JavaScript? That's part of my point. It's usually not worth it.

nerdponx2• 4 years ago

But you need to write the Lambda function itself in some programming language. The type of "glue" is less relevant than in the past, sure, but the thing being glued needs to be written as well, and that's often critical business logic.

dig1• 4 years ago

I am not fond of the multimethods because they can easily tangle the code and give you a false sense of scalability. For example, in a blog post, "handle-action" is nicely decoupled with 3 different actions, but let's imagine how that will look after someone adds 20 new actions. Good luck debugging that.

Also, I saw numerous cases where people will copy/paste multimethod arguments without knowing what they are used for.

I still find case/cond more readable and way more performant, especially since the author uses the same type for a multimethod dispatch, but YMMV.

Folcon• 4 years ago

case/cond is pretty readable, but I wouldn't say they provide a comparable api to multimethods.

From my perspective, I reach for multimethods when I want to provide my caller with the ability to declare their own "actions" as it were. The goal is to offer an open interface.

Without that need, yes, case or cond can be sufficient.

escherize• 4 years ago

Case and cond are nice, but they suffer from the expression problem. When you want to let other people add methods to your code without modifying the source you need to use multimethods (or protocols).

If your method only needs 1 argument then why should the other ones matter? I don't see a problem here. clj-kondo will guide people to name them as _ or _thing anyway so you don't even need to think about it.

fulafel3• 4 years ago

Multimethods also have some problems vs reloading that need working around.

twawaaay5• 4 years ago

I love Clojure and also Common Lisp (basically, Lisp in general). But I also observed every single corporate Clojure project I had any connection with to fail spectacularly. Typically as a complete unmaintainable mess. Some as unmaintainable mess that is very slow and unreliable.

My theory is that this is result of no guardrails on how to structure your application. Clojure to be productive must be used by people who absolutely know what they are doing when it comes to structuring your app.

When it comes to Java you get hordes of devs still producing passable results. The structure is largely imposed by the frameworks (mostly by Spring/Spring Boot) and available help, literature. Even some antipatterns at the very least achieve some level of convention/predictability that is needed to be able to find your bearing around the codebase.

I would say, if you have a normal-ish project, care about productivity but don't have really stellar and mature developers -- skip Clojure.

Choose Clojure if you know how to use all that additional power, have need for it and understand what your added responsibilities are.

If you don't know how to wield Clojure's power there is very little you can gain by choosing it but a lot to loose.

dustingetz• 4 years ago

i think you're measuring the economic climate of the last two decades and it's impact on management practices, hiring/recruiting etc. which is basically: scale scale scale money money scale, oh and fuck people. Clojure is not for those teams. Clojure is for those of us who reject that

rockyj• 4 years ago

Same problem exists for JS/TS projects btw. You can write anything, anywhere in any style suited to the developer. I guess that is why coporates like Java so much, it commoditizes developers, but can also rock the boat on the other side - cargo-cult programming with no original ideas or innovation.

bcrosby95• 4 years ago

In general I find Clojure's "power" to be its simplicity.

As you mention, if you're building websites/services, I think the biggest "problem" with Clojure is the community hasn't rallied behind any particular framework.

I'm reminded a bit of companies. The point of a large, successful company is to slow its employees down enough so they don't kill the goose that laid the golden egg. The point of a startup is to find the goose.

When it comes to programming, Java is the former. There's nothing particularly special about Java that keeps you from coloring outside the lines, so to speak. It just makes both coloring outside and inside the lines harder.

Of course, in steps the standard frameworks everyone uses.

I contrast Clojure a bit with Elixir. I find both to be similarly "powerful", in similar and different ways. But the community has rallied behind a web framework (Phoenix) and it's pretty easy for the average web dev to just read a book and be told how they should do everything.

slifin• 4 years ago

I can't remember where but Rich Hickey at one point said himself web development isn't a solved problem

If you agree web development isn't a solved problem, do you want to couple your entire language to a massive framework that is committed to solving an un-solved problem in the general sense using technique X that presumably will need to be replaced Y years down the line? Or not work in situation Z

Personally I don't want that for Clojure, instead I see lots of different approaches in Clojure (and some outside) that could be the next big thing:

- https://www.hyperfiddle.net/

- https://github.com/whamtet/ctmx

- https://github.com/leonoel/missionary

- Unison

Ultimately I want to go long on my programming language and short on "the one true framework" until we have a one size fits all approach for solving the general problem of web development, until then give me low coupling libraries that I can mix and match

And that's exactly what Clojure is doing right now, maybe in the end it will be something like chatgpt just mix and matching the approaches for you but for now I'm personally not looking for a framework to be the only way to use Clojure

cultofmetatron2• 4 years ago

I built my startup on elixir.

Once you get past the syntax, a lot of the day to day thinking about how to model your solution in the context of the system is similar. to the point that I'd say elixir is Clojure with a ruby like syntax. elixir has let bindings but they are implicit and blend into the language. you don't really think about them. At the same time, you have all teh standard fare of a functional enough language without all the type ceremony.

Id say its more convenient to work with elixir day to day. Less tracking of parenthesis and the code naturally comes out looking more uniform. There is the price that macros are harder to write in elixir. That is not to say its difficult but lisps in general make writing a macro so damn easy that anything else feels difficult in comparison. I think its a good compromise. Its just difficult enough to make avoid creating macros until you REALLY need them.

mattgreenrocks• 4 years ago

When it comes to Java you get hordes of devs still producing passable results. The structure is largely imposed by the frameworks (mostly by Spring/Spring Boot) and available help, literature...I would say, if you have a normal-ish project, care about productivity but don't have really stellar and mature developers -- skip Clojure.

Regardless of whether I agree, I'm struck by this thinking as a sad commentary on the state of our industry: devs can't be trusted to actually architect the code for maintenance, so they should be forced to color by number with frameworks.

twawaaay5• 4 years ago

Faced with decision overload, most people get more productive where most of the decisions are made for them by somebody else.

If you had to consciously make every single decision about everything you do you would die. That's where habits and patterns come to reduce the number of decisions you have to make to a reasonable level.

Most software development projects really are the same as a lot of other projects. You get API services that basically shuttle data between database and web interface, you get frontends which basically shuttle data between APIs and UI, etc. There is no need to invent everything for every project you do, it is much more productive to just focus on things that are specific to your project and adopt the rest from either your experience or some ready made guidebook.

It is not necessarily a lack of trust. It is just that when I hear some development manager to start on a journey of reinventing everything for a pretty mundane backend project it immediately suggests to me a lack of common sense or prioritising personal goals over good of the project.

turtledragonfly• 4 years ago

sad commentary on the state of our industry

Honestly, I find myself structuring my own code to be dummy-proof and paint-by-numbers, even when I am the only user of it (:

I don't see it as a bad thing, necessarily; sometimes you want to "bake in" structure and conventions in various places so you can use your creative energy elsewhere.

Though I do take your point that this is not necessarily (or at all) communicated to junior devs, who are sometimes plopped into a little artificial coding cage and discouraged from reaching outside of it.

Depends on your perspective and intentions, I suppose.

lairv• 4 years ago

That goes against the "Beating the average" lesson [0] but I guess this thing is already 20 years old, I wonder if it's still relevant now

[0] http://www.paulgraham.com/avg.html

twawaaay5• 4 years ago

There exists no single tool that is best for everything.

Also, not every company can hire best developers (even though most claims so).

Imagine you are project manager for project X which is pretty mundane backend with pretty mundane API and you are given a deadline to develop it.

Will you:

a) try to find absolute best developers on the market and then use absolutely most powerful language (Lisp) to get it done?

b) try to find some developers that are available and use technology that is adequate enough that they can work with not shooting their feet off?

Take into account that you might not be able to hire best developers and even if you do, they might quickly leave not being satisfied with the mundane job you gave them.

jimbokun2• 4 years ago

Doesn’t beating the average require above average developers according to that article?

yogthos8• 4 years ago

I've had the opposite experience myself. At my last job we switched from Java to Clojure and found that similar types of projects were much faster to develop and easier to maintain. My team also hired a lot of coop students and junior devs that we trained up.

I find the mistake people often make is to get clever with Clojure. Using macros when regular functions would do is a good example of that. It is absolutely possible to write impenetrable Clojure if you start doing weird things just because you can.

However, I find the beauty of Clojure is precisely in the fact that you can write simple and direct code that solves the problem in a clean way without the need to get clever.

My biggest advice for structuring maintainable Clojure projects is to keep things simple, and to break things up into small components that can be reasoned about independently.

jwhitlark3• 4 years ago

Of the two billion-plus exits I've been (peripherally) a part of, the most recent one was Clojure, and the one 10 years ago was Python. I've also seen several Clojure projects and one Scala project fail. With very rare exception, your tools will neither doom you nor ensure success.

It all comes down to execution, in the end.

cultofmetatron2• 4 years ago

your tools will neither doom you nor ensure success.

This holds true as long as you're not buidling your startup on something truly outlandish like brainfuck or piet

thelazydogsback2• 4 years ago

Of the two billion-plus exits That's a lot of exits :)

beders2• 4 years ago

Maybe it is you :)

But I understand where you are coming from. Understanding an existing clojure code base is exploratory in nature. You'll REPL into it and run the functions you have difficulties with.

This is a very different activity than chasing object references and hunting down method calls.

jb1991• 4 years ago

I worked on a large Clojure app for a small company for a few years and it eventually died for similar reasons as you describe. Then, I was hired to work for a startup also developing a large Clojure app, and I must say, it was a very challenging onboarding. The main problem is that "anything goes" in a language like Clojure, which can be great or devastating. My personal take is that the lack of types makes it very hard to understand a large codebase you are exposed to for the first time, unless, as you say, it is structured in a way that was very clear and strict from the outset. Trying to trace the ideas through a large Clojure codebase can become extremely difficult when all they are is a bunch of functions with untyped parameters. Liberal use of spec can help to some degree (though that is also a challenge to enforce, much like optional typing in some languages) but it doesn't change this fundamental paradigm of the language, and I've seen more businesses sunset Clojure than move to it. It's a shame, really, since the language is vastly more flexible and powerful than most, but it goes to show that there are some things that have, in the wild, outsized influence on productivity for the 80%.

twawaaay5• 4 years ago

Lack of types -- yeah, I forgot to mention it.

Another is -- lack of good IDE support (which is connected to lack of strong typing). In something like Java you can always understand what is the thing under your cursor.

But an experienced developer knows how to deal with these problems. Lack of strong types means you have to be building clean, well specified interfaces. Lack of strong typing in language does not mean you don't have types in your program -- it just means that it is now on you to make sure it is easy to figure out what is the exact specification of piece of data at any point in the program.

I also found that spec rarely helps. One of the major points of something like Clojure is to be able to create general purpose functions that can do useful operations on very wide range of data and then use those functions to compose your program. Spec really hinders your possibilities here.

It's a shame, really, since the language is vastly more flexible and powerful than most, but it goes to show that there are some things that have, in the wild, outsized influence on productivity for the 80%.

It is the curse of Lisp. It is the most powerful language that can exist and yet it will never be mainstream because it requires a whole other level of development mastery to really get the productivity benefits.

I personally use Clojure for my rapid prototyping which is to say -- "as long as I am sure nobody else will ever need to work on it with me".

I did a very efficient algorithmic trading framework proof of concept in Common Lisp and it was a joy to work with. But then when it came to actually productionising it I was forced to rewrite it in Java at the cost to performance and heaps of boilerplate code.

mullr• 4 years ago

I had the opposite experience. I worked for a company with a bunch of Clojure projects, written by people of varying levels of experience. I had to do some cross-cutting changes and feared the worst. But when I actually got down to it, everything more or less made sense.

Why did this happen?

- We had small common framework that everybody used, at the very highest level (think application lifecycle management). That imposed some amount of consistency at the most basic run-the-program stage.

- The devs communicated openly, a lot, so there was some general consensus on what to do, and what not to do.

- The team at large was very suspicious of introducing new macros. You could do it, but you'd better have a really good reason.

- When I went to make the changes, I didn't have to worry about spooky-action-at-a-distance kinds of consequences anywhere NEAR as much as I do in other languages. Being strict with your state management, as Clojure strongly encourages, REALLY pays off here.

The actual problems I had were entirely related to the overall build system, the fractured nature of the source control, and figuring out who was responsible for what code once we were 3 reorgs deep. The code itself was remarkably resilient to all this nonsense.

nerdponx2• 4 years ago

We had small common framework that everybody used

Python applications seem to benefit from this as well, and I've encountered a surprising amount of resistance to it from other developers. I think everyone has been burned at least once by "over-designing", building too much of the wrong abstraction. But the result is that they are never willing to commit to a common internal framework even long after the need for one has become painfully obvious.

Usually this happens among developers who have been solo developing a project for a while, and see themselves as YAGNI zealots fighting the good fight against excessive abstraction and overengineering.

I understand and sympathize with the sentiment. But when every design decision is ad-hoc and as-needed, it makes it really really hard for external contributors to make changes to the existing codebase. It discourages contributors from "big-picture thinking" and eventually leads to the dreaded Ball of Mud design, with some combination of:

• meandering flow control

• poorly-defined or nonexistent interface boundaries

• inconsistent naming

• lack of documentation, or incorrect documentation and/or comments

• redundant safety checks, or absent safety checks

• poor runtime performance

• difficult-to-test code that freely mixes I/O and business logic, requiring complicated test fixtures, tests that are difficult or impossible to change, and poor test coverage (antipattern and code smell: "idk how to test that, don't waste your time. did you run it on the QA environment and check that it worked?")

I think it arises as a misunderstanding of the forces that lead to overengineering and building incorrect abstractions in the first place. The problem is usually one of expanding the scope of an abstraction or framework too early, not of building the abstraction or framework in the first place.

agumonkey• 4 years ago

Interesting take. How much of these architectural problems came from

    - process / workflow (everybody goes on hacking a topic for weeks and then integrate ?)
    - team size (I can foresee how clojure would make large team a problem, but small teams happier)

mtrimpe• 4 years ago

The same applies to most Scala projects from the early days. It attracted many developers who just wanted to have a go at writing their own custom half-baked versions of their favorite parts of category theory for a simple line-of-business CRUD app.

I think both eventually found their niche and now have plenty of developers that stuck around write appropriate code in it. But when they were shiny and new they indeed both created their fair share of failures.

silver-arrow• 4 years ago

I initially had the same thoughts as you when I got on my first Clojure project. Previously, I used strongly typed languages with "approved" frameworks. Developed in Java for over 20 years before getting put on a Clojure project.

It because apparent over time that the lack of types and the other features provided by Clojure resulted in much smaller and simpler codebases compared to those previous languages and frameworks that I toiled in for so many years.

It has really highlighted to me the value of simplicity for better productivity and maintainability. I wouldn't even want a framework to build web apps using something like HTMX for example. Clojure handles HTMX in almost magical ways with a simple library or two such as hiccup.

Recently, I was doing some work on a Java / Spring project and was dismayed with the proliferation of classes and packages; really the complexity of it all. And remember, I am solid with Java experience, so it is a result of those types of languages and architectures IMO.

twawaaay5• 4 years ago

It because apparent over time that the lack of types and the other features provided by Clojure resulted in much smaller and simpler codebases compared to those previous languages and frameworks

Totally agree. My very initial motivation that caused me to get interested in Clojure was my study of various API clients generated I think by Swagger Editor? All clients were littered with generated boilerplate -- except for the Clojure one which looked clean and exactly as you would imagine some intelligent programmer writing it. As I already had experience with Common Lisp I immediately understood what is happening here.

Recently, I was doing some work on a Java / Spring project and was dismayed with the proliferation of classes and packages; really the complexity of it all. And remember, I am solid with Java experience, so it is a result of those types of languages and architectures IMO.

I also have over 20 years of experience with Java and I also share your experience.

Recently I have started mixing OOP with functional and FRP. For example, most of my code is now FRP (ReactiveX/Reactor) and for some strange reason Java is superbly suited to it. It is not a panaceum but I found that I can frequently write what would normally be large features even in minutes.

yoyohello13• 4 years ago

I love lisp in general as well, but I think lisps will always tend toward an unmaintainable mess unless the people working with it are very disciplined. With the REPL based workflow and macros, lisp encourages a very 'hack on it until it works' development workflow. Which feels awesome but doesn't lend itself to building maintainable solutions.

samsquire• 4 years ago

I've worked on 3 Clojure projects professionally.

For background I'm a devops and software engineer and I'm comfortable with Javascript, Java, Python and C.

I am implementing my own multithreaded programming language and I have a switch based interpreter and a compiler that codegens for it.

I think other people's Clojure code is unreadable.

Clojure code resembles the author's mental model of the problem being solved. Which is often very different to how I model the problem to be solved.

It's not the complexity of Clojure codebases that I have trouble with, it's the syntax and the approaches to the problems being solved.

I would use LISP for AST generation and codegen but not for direct programming.

fulafel3• 4 years ago

Like many replies I've had the opposite experience. We've had good developers but I think that's also partially due to Clojure, it seems to take some professional confidence in many cases although we've had first-language Clojurists too. In my anecdotal experience the projects do better than the ones going for stack du jour.

nkh• 4 years ago

Any one tried Polylith with the multi-method approach mentioned in the article?

beders2• 4 years ago

This approach is not simple. It complects a business process state with multi-methods. Don't do it.

Model state where it belongs: in a database.

yogthos8• 4 years ago

This approach is perfectly compatible with modelling state in the database. The problem this addresses is the data flow, which is a completely separate problem from managing the state itself.

rmuslimov• 4 years ago

This is very informative and beginner friendly write up which came be used as strategy for organizing apps in Clojure. I personally have something very similar which Redis storage used as persistent storage for storing jobs and tasks within workflow are potentially executed on diff hosts. I would recommend to extending this topic and share your thoughts about component/mount like abstraction to the code. For example notify-sender should have credentials to connect to the services. How they are delivered? as input to the :handle-action method or as as component/mount. Interesting to learn about your approach here.

Thanks for sharing!

yogthos8• 4 years ago

Glad to hear this was helpful, and regarding managing stuff like credentials, I like the pattern of initializing the client up front using component, mount. Then passing the client in the resource map.

The advantage over simply passing credentials is that the initializer for the client can validate itself when it loads. If you just pass the credentials and assemble the client when you try to send the message then if some variables weren't set correctly you only find out when you try to use the client at as opposed to when application starts.

shaunparker• 4 years ago

I haven't looked at Clojure in a bit, but shouldn't the (nil? to-info) check in the transfer implementation of handle-action come first? It seems like that would never be reached in the current implementation.

yogthos8• 4 years ago

Oh yeah you're totally right. :)

the-alchemist• 4 years ago

It seems like a lot of the anti-Clojure sentiment boils down to 1) lack of static typing, 2) poor IDE support.

I'm wondering, though, doesn't the same apply to Ruby, Python, and Node projects?

I've over-hauled 80k line Python projects, and the "lack of typing" there seemed to apply as well.

Why don't Ruby, Python, and Node projects suffer from the same critique? Genuinely curious...

fulafel3• 4 years ago

They do get the same themed critique from static language proponents, even more so since JS/Python don't have culture of using schema systems (ala spec and malli). But of course dynamic languages have a lot of upsides as well, it's just a tradeoff.

Structuring Clojure applications

Related Stories