Tour of our 250k line Clojure codebase (2021)

https://blog.redplanetlabs.com/2021/06/03/tour-of-our-250k-line-clojure-codebase/

by todsacerdoti • 3 years ago

86 45 3 years ago

blog.redplanetlabs.com

dang• 3 years ago

Discussed at the time:

Tour of our 250k line Clojure codebase - https://news.ycombinator.com/item?id=27385208 - June 2021 (227 comments)

valcron1000• 3 years ago

Sounds like a great way to burn money: the company was founded in 2019 and they still haven't released anything, probably because they've spent most of their time building other stuff. Their devs sure are having a lot of fun though.

_halgari3• 3 years ago

That was my issue for a long time. I even talked with their founder several times on twitter back a few years ago. Each time I was greeted with buzzwords, that I knew the meaning of but I think they assumed I didn't.

They would claim grand things like having solved the issues with continuations and delimited continuations, distributed process migration and a whole host of other very hard problems that haven't been solved in the past. I would ask their founder: "right, so you know that delimited continuations have problems with accidental captured scope, they run poorly on the JVM, how did you solve this, have any papers I can read?", and all I ever got was that Clojure, immutable data, X and Y would fix these issues and you just had to wait and see what they were cooking up.

That's when I knew they had no clue what they were doing. I'm all for pushing the boundaries of tech, but if you're doing something attempted many times before, you at least need a good elevator pitch as to why it's solved now.

I think a great example of this done right is Rich talking about Clojure. People would ask "isn't immutable data expensive to reclaim and allocate". And his reply was always that the JVM's GC was just that good that the benefits to be gained from immutable data outweighed the marginal performance penalty of the amount of garbage collected. What changed since the old lisp days? Well we now have GCs that are super fast and JITs that can optimize dynamic code well.

That's the sort of laser focused vision I never saw from Red Planet labs. You gotta get that problem statement and the solution out early, refine the elevator pitch and be able to articulate to people who know what they're talking about how you're going to succeed where others have failed for decades.

eduction2• 3 years ago

I see your overall point which is a good one and I know this is a nitpick, but I thought Rich’s core solution to the cost of immutable data structures was to find a way to get the cost down by extending some existing research by Phil Bagwell.

From https://dl.acm.org/doi/pdf/10.1145/3386321

“I then set out to find a treelike implementation for hash maps which would be amenable to the path-copying with structural sharing approach for persistence. I found what I wanted in hash array mapped tries (HAMTs) [Bagwell 2001]. I built (in Java) a persistent implementation of HAMTs with branching factor of 32, using Java’s fast System.arrayCopy during path copying. The node arrays are freshly allocated and imperatively manipulated during node construction, and never mutated afterwards. Thus the implementation is not purely functional but the resulting data structures are immutable after construction. I designed and built persistent vectors on similar 32-way branching trees, with the path copying strategy. Performance was excellent, more akin to O(1) than the theoretical bounds of O(logN). This was the breakthrough moment for Clojure. Only after this did I feel like Clojure could be practical, and I moved forward with enthusiasm to release it later that year (2007).”

Or you tell me what I’m missing. Big fan of your work in core.async if this is the same halgari.

zilti• 3 years ago

Scala has delimited continuations since some time now, so it can seemingly be done in a way that's performant enough.

nathanmarz2• 3 years ago

Another hypothesis is what we're doing is so hard and so valuable that it really requires that much effort.

I suggest reserving your judgement until you've seen what we've built, which will be soon.

Mawr• 3 years ago

People would ask "isn't immutable data expensive to reclaim and allocate". And his reply was always that the JVM's GC was just that good that the benefits to be gained from immutable data outweighed the marginal performance penalty of the amount of garbage collected. What changed since the old lisp days? Well we now have GCs that are super fast and JITs that can optimize dynamic code well.

It's a little odd to see this deeply mistaken belief dating back to the early Java days being advocated for today. GCs are very constrained by the tradeoffs they make, there's no free lunch.

Much in the way an F1 car is only fast on a race track specifically made for it, the only reason a massive rate of allocations can have a merely marginal performance penalty is if the GCs in question have been specifically designed to handle it. But in doing so, they must have made sacrifices elsewhere, e.g. to memory usage.

Code that's not written with the underlying machine that will ultimately execute it in mind will never be fast, no matter how much we jiggle tradeoffs around. Therefore, while the gains of immutable structures may still outweigh the performance loss, the loss cannot possibly be characterized as marginal.

jasonjackson3• 3 years ago

The founder has released successful large projects in the past, e.g. Apache Storm used by Alibaba, Yahoo, Twitter, etc. Probably just need to wait a bit longer.

adrianmsmith• 3 years ago

the company was founded in 2019 and they still haven't released anything

According to this tweet the author has been working on the product/company for ten years! https://twitter.com/nathanmarz/status/1636050500553699330 (And presumably still hasn't released anything in those 10 years?)

chii2• 3 years ago

The thing i want to know is how is this company funded? What puts food on the table? Or is it just living off personal wealth? Or is it investor funded, in which case, how come the investors are not clamering for a return on investment?

Edit: just found https://medium.com/red-planet-labs/introducing-red-planet-la... , so i guess they're using investor money.

bokchoi• 3 years ago

Now up to 345k:

https://twitter.com/nathanmarz/status/1616577265139355655?

bschwindHN• 3 years ago

Gotta be honest, this sounds like a nightmare. Clojure felt like it got out of hand in my own _personal_ projects, I can't imagine going through 250k lines of it without a strong type system backing it up.

And they're building a "programming tool". Hmmmm

We have about 400 macros defined through our codebase, 70% of which are part of source code and 30% of which are for test code only.

jasonjackson3• 3 years ago

You can write 'type signatures' easily in clojure code, there's dozens of options, they're just only checked at runtime (and usually relegated to developer builds). if you combine that with good code coverage, it stops being a weakness and starts being a strength, because you declare data types with far more specificity & precision than a static type system.

As rich likes to say, the type signature of your stereotypical statically typed language, e.g. reverse :: [a] -> [a], doesn't have a much information content.

I write clojure, and still 'think in types' & category theory.

MrBuddyCasino• 3 years ago

if I type ctrl+space, will it give me the correct auto-completion? despite what type-system galaxy brains might think, this is 95% of the value

grumpyprole4• 3 years ago

the type signature of your stereotypical statically typed language, e.g. reverse :: [a] -> [a], doesn't have a much information content.

It's a little bit unfair to imply any alternative to bleeding-edge clojure is going to have a simple 70's Damas-Milner type system. Today, static types can describe a lot more. Refinement types (e.g liquid Haskell) and/or dependant types (e.g. Idris or Agda) can describe and enforce the desired properties for reverse statically.

riwsky2• 3 years ago

“Measuring programming progress by lines of code is like measuring aircraft building progress by weight“—Bill Gates

96593• 3 years ago

which is really a good way to do it. you design it to be so many pounds, when you get it all together you should be there (or close to it).

conductr2• 3 years ago

You start a project knowing how many LOC it will take?

I want to assume you’re being sarcastic but it really doesn’t sound like it

sokoloff• 3 years ago

“This airplane design will be done much faster if we use lead and tungsten in key places.”

samsquire2• 3 years ago

I feel that every codebase built by one person is understand by that person really well and nobody else gets to the same level of understanding, unless the codebase is embedded, it shall only exist as long as the original person is around.

baby2• 3 years ago

I want to see a whole discussion based on that comment. This is a real and big issue, and the big reason why it’s hard to scale the team.

samsquire2• 3 years ago

I don't believe code _is_ the documentation.

I've thought of information systems for code, I once used OpenGrok to parse a large codebase and try work out how it works. But it's undirected learning.

I think question answering would be useful:

"Where should I change the code to insert behaviour that changes X, Y, Z behaviours"

chii2• 3 years ago

This is a real and big issue, and the big reason why it’s hard to scale the team.

it's also why java as a language is good, not because the language is good, but that it's easy to have someone not originally involved in adding/maintaining the code base - you can't get too clever with java.

kaba03• 3 years ago

See Naur’s Programming as Theory Building.

lionkor• 3 years ago

So, does it do anything? Is it just pretty code that does nothing? I sure dont see it doing anything, and every sentence on the website is incredibly vague. The CEO just claims 100x more features on twitter, and things like that.

layer8• 3 years ago

How many K parentheses might be more interesting.

Tour of our 250k line Clojure codebase (2021)

Related Stories