Beware the Complexity Merchants

https://chrlschn.dev/blog/2025/05/beware-the-complexity-merchants/

by kiyanwang • 1 year ago

42 38 1 year ago

This isn't just applicable to code or development; there's a huge B2B marketplace built on the idea that the more complex your business is, the more profitable it will be. Hence why every business must be harvesting as much data as possible, analyzing it with different tools, across different platforms, different storage methods, different transformation and retrieval tooling, different vendors and formats and product suites and integrations.

It drives leadership nuts, but I find asking the "why" or "what problem does this solve" questions helps the organization remain focused on what actually delivers value versus what's just hype we can ignore or throw away.

jmugan• 1 year ago

I think about this from the angle of administrative burden. Sure, the tax code might be better if you make a tweak for zucchini farmers, but now you have to ask every taxpayer whether they have engaged in that activity.

userbinator• 1 year ago

I suspect a lot of accidental complexity also comes from those who don't really know what they're doing, so instead of focusing on the result, they take a very roundabout path until they stumble upon a solution. AI and "vibe coding" is going to make that problem much worse.

For an interesting example of the outcome of strongly optimising for essential complexity, look at the demoscene. They have done some amazing things that were thought to be impossible.

patapong• 1 year ago

Interesting. I also think a lot of complexity stems from developers preferring building systems over building products. It is much more satisfying to build somehting that can address many use-cases, even if we are only working on a single use-case. But, this also introduces the cost of complexity as the user/downstream developer is pushed into making the choice of which of the many available paths they want to take.

See the eternal urge of gamedevs to build engines instead of building the game.

brookst• 1 year ago

Yep — and it’s hard to argue against. Option value is valuable. Is it worth spending 20% more and de-optimizing for the core use case to support unknown future use cases? I’m not sure there’s a general purpose answer.

And if you do stick to purpose-specific design and management asks for a seemingly minor expansion of scope, the “this is a small change, why did we make the system so rigid that it’s expensive to update?” is an uncomfortable conversation.

pan69• 1 year ago

Experienced developers understand the start-up cost associated with building a software product. So, when developers think ahead and architect and design their software for a possible future roadmap and that roadmap doesn't eventuate, they have introduced accidental complexity and they are considered idiots.

CharlieDigital2• 1 year ago

I suspect a lot of accidental complexity also comes from those who don't really know what they're doing, so instead of focusing on the result, they take a very roundabout path until they stumble upon a solution.

I suspect that you are right on this; it often appears as a lack of depth of understanding of the problem being solved for and whether the proposed solution actually solves that problem and solves it in the simplest way possible.

eddythompson80• 1 year ago

I agree with the general sentiment. The article is vague on the specifics as to what counts as "added complexity" though, which I found to be the central issue.

Everyone loves a bit of abstraction that encapsulates a given complexity and make it a single simple concept. This is not: "a complex bit of technology that has pros and cons and will lock you in a particular architecture and vendor and would almost always necessitate 2 architectures because not all tasks/scenarios would be suitable for it". It is a "Lambda". So simple. Just a `handle(event) {}`, how much simpler can this be? No complex runtimes, VMs, environments, etc. It's just a `handle(event) {}` no complexity to see here. good luck.

On the other hand, sometimes people who "champion" simplicity are also the same people who don't care about the engineering part of it. The "Hey, I added a full text search. It's implemented as a:

    select * 
    from posts 
    where column1::text like $1
          or column2::text like $1
          ...
          or columnN::text like $1

Lets push it. Oh out database CPU is burning, search is so slow. What a surprise this is. I guess it validates the popularity of search. Can we optimize it? Maybe we cache the result? Maybe we add an LLM to normalize queries so we can cache results more effectively? I'm trying to keep things simple"

There is also the requirement or ask that is trying to hide a particular complexity from one side by shoving it all under a rug somewhere. Once the complexity has been sufficiently shoved under the rug, we can think of the rug as just a rug. We can move it around and use it like a rug. Surely the shit underneath will never have any impact on anyone any time.

aleksiy123• 1 year ago

I feel like all these articles have this issues.

The hard part is in knowing which is which and I don't think anyone real has a tried and true solution for it.

I think because its really about correctly predicting the future in a n continually evolving environment.

Those "simple" solutions can be piled on top of each other turn into an absolute monstrosity over time.

And then engineered solution can be overengineered and overly complex when it turns out the problems it solves never ended up happening.

On top of that people don't even think of simplicity in the same way. One person's simple is another's overly complex.

What seems to work for me is try give yourself outs/alternative paths.

Hedge your bets by overengineering slightly on where you think there is more risk. And keep it simple where there is less risk. Then reevaluate.

ashayh• 1 year ago

Complexity is also about empire building and job security.

If every dev took the path of least resistance, and did the easiest thing that did its task, we would lose 80% of all tech jobs. This would only funnel more money to billionaires and share holders instead of the employees. Why would we want that.

kevmo314• 1 year ago

This is an underrated reason why startups can be successful. Many large and even mid-sized companies' employees are encumbered in complexity indirectly because they need to engineer something to justify their jobs. I think a lot of devops is basically this sort of persona because if devops really did their jobs correctly, we shouldn't see so many devops jobs.

bob1029• 1 year ago

While accidental complexity can arise innocently enough from well-intentioned engineers, the darker side is that it can manifest from ego and self-preservation.

I am seeing a huge uptick in this style of "accidental" complexity.

I recently had the pleasure of observing a perfectly functional T-SQL script be refactored to use entity framework, resulting in a ~50x decrease in throughput.

Ignorance is something I am willing to entertain. Arrogance is way, way more difficult to deal with. One is an educational opportunity. The other is game of thrones.

mrkeen2• 1 year ago

[flagged]

mrbungie5• 1 year ago

It is a feelpiece, but I don't think that's necessarily a bad thing, at least it invites discussion.

Also, framing it as "us vs them" misses the point. For instance, in my world (Data & Analytics), there's no universe where Spark makes sense for <1TB data or using Kafka for what could be solved using with more simple and predictable batch processing, yet we still see them used in wild implementations that clearly don't need that scale. Those implementations can end up needing a lot of headcount and budget for support without adding any value whatsoever and sometimes even ending up in a more brittle platform. Complexity often comes from misguided choices, not just ego.

I've seen enormous data teams at startups, by that I mean bigger than WhatsApp engineering team in its prime pre-Meta (~30 people). That's a red flag, especially when they're processing far less data.

moritzwarhier5• 1 year ago

Inviting discussion through underspecified warnings or appeal to feelings of superiority, safety/caution or "best practices" is a very pervasive issue in software.

I admit I did not read the article. But I have done so before I guess, I remember the headline.

The headline is enough to provoke thought. See

https://world.hey.com/dhh/merchants-of-complexity-4851301b

And after clicking, I see, this article is very recent. Strange...

mrkeen2• 1 year ago

Also, framing it as "us vs them" misses the point. For instance,

We would never use Spark or Kafka for small data, not like them. Their choices are misguided.

Their data teams are too big, ours are the right size.

Fwiw, I use Kafka regardless of scale. It's the only mainstream system out there that doesn't encourage throwing away your receipts. It's the programming equivalent of "accountants don't use erasers".

sublinear• 1 year ago

"not invented here"

gwern• 1 year ago

Likely AI-written. OP sanitized some of the most obvious tells, or perhaps the blogging software added those spaces to the em-dashes, but when a dev who is as gung ho on GenAI as https://chrlschn.dev/blog/2024/10/im-a-gen-ai-maximalist-and... is, writes an ending like that, and the whole thing is about as intellectually nutritious as a bag of Doritos, we all know what they did. A waste of time.

---

Out of curiosity, I asked ChatGPT-o3 what LLM it thought wrote the ending, and I agreed with its rationale for guessing that 4o wrote it:

"The prose is archetypal ChatGPT rhetoric—second-person apostrophe (“dear traveler”), triadic repetitions, and a moralizing close. / Vocabulary inflation (“complexity merchant”, “moat of self-importance”), syntactic parallelism, and the cadence of stacked, comma-separated clauses are hallmarks of GPT-4’s temperature-controlled completions. / No concrete anecdotes, data, or links—just abstract exhortation—exactly what an LLM defaults to"

But more interesting was the rationale for it not being Claude or Gemini-pro: "those models tend to hedge (“may”, “might”) and inject more safety language. This text is bolder and more florid—typical of GPT-4 with style-boosting prompts."

I'm not sure I would've identified that as stylistic tics of Claude/Gemini. I'll have to look closer for those in the future.

CharlieDigital2• 1 year ago

I am the author, my blog is an Astro template and you can see the GH repo here for this post: https://github.com/CharlieDigital/chrlschn/commits/main/src/...

And of course, the full commit history of all of my writing: https://github.com/CharlieDigital/chrlschn Even a simple look would have led you to the repository with full commit history of my fixes for grammar, spelling, etc.

I've also kept a blog long before GPT existed: https://charliedigital.com [0] so I would point you to that and my long history of writing (e.g. https://chrlschn.medium.com/) that this is hand rolled content. Rather I'd say AI has copied my style of writing.

Please feel free to peruse and use your bulletproof analysis to let me know which of these spanning back to 2005 are AI written. I hope that your takeaway from this is that your AI spidey senses are completely broken and realize the irony of your intellectual laziness here relying on an LLM instead of just following the links in my profile or clicking the link to my GH on my blog. If you have any sense of righteousness, please retract your flag and let the mods know of your error.

[0] Exaxmples:

https://charliedigital.com/2020/11/13/principles-of-speed/

https://charliedigital.com/2020/10/29/how-leaders-fail-their...

https://charliedigital.com/2020/09/20/in-praise-of-simplicit...

https://charliedigital.com/2016/04/11/recipe-crunch-time-exe...

tomhow• 1 year ago

We ask people to avoid making accusations like this in the comments, because the hurt and damage caused by a false accusation outweighs the benefit to the community of making the accusation publicly.

Instead, please flag the post and email us so we can take a look and see if we think the content and the comment thread are HN-worthy, as that's ultimately all we can assess. The originality and quality of the writing is always an important consideration, regardless of how it was created.

(We also discourage pasting LLM-generated output into comments. In this case it doesn't really prove anything either way and takes the comment further into off-topic territory. More broadly, we want to keep HN for discussion between humans.)

Michelangelo11• 1 year ago

https://grugbrain.dev/

wcfrobert• 1 year ago

Past a certain scale, complexity is mostly unavoidable.

Accidental complexity is compressable, but essential complexity is not. At some point, you cannot compress further without losing nuance.

In compiler design, there's a concept called the waterbed theory of complexity which states that you can try to abstract complexity away, but it'll just show up elsewhere.

tikhonj• 1 year ago

I don't like talking about "complexity" any more because while almost everyone agrees it is "bad", people have radically different ideas of what it means. "Complexity" in software conflates several distinct concepts:

• plain poor design

• complexity in operation rather than design (or technology)

• inescapable real-world complexity

• lots of moving pieces and details, straining working memory

• abstract or novel concepts that are hard to learn up-front, but easy longer-term

Some of these are practically opposites! At the pareto frontier of good design, there is a pretty fundamental trade-off between having more abstract concepts that are harder to learn up-front, and exposing more details that make systems hard to work with on an ongoing basis. People just call both of these "complexity"! These are two concepts that absolutely should not be conflated.

I've seen lots of other patterns that are misleadingly described as "complex". For example, some of the most effective software I've seen has been situated software[1]; that is, software built for, and largely in a specific social context. Think seemingly messy code that is incestuously coupled to a single person's or team's workflow. This software might actually be perfectly simple in context, but it's going to seem painfully baroque to anybody outside that context.

[1] https://gwern.net/doc/technology/2004-03-30-shirky-situateds...

Given all I've seen, I've come to the conclusion that generic exhortations about "complexity" are actively harmful. If you're going to write a universally applicable rant, just write about how bad design is bad and good design is good! At least that's something that people will disagree with—I've met far more people who insist there is no such thing as "good" or "bad" design than people who insist that complexity is actually better than simplicity.

burnt-resistor• 1 year ago

Talking ability "complexity" is about as useful as talking about "government" in hivemind, abstract terms that have little meaning or purpose except to justify attitudes or decisions.

moritzwarhier5• 1 year ago

Because that's the other side of this. There are rarely high margins in actually selling someone something that they then own. Much better to rent it to them. They'll own nothing and you'll convince them they should be happy.
By what sorcery? Fear, mostly. Vanity, sometimes. Sloth, occasionally. Pride, definitely. The more insecurities the merchants of complexity can trigger, the easier the sell.

https://world.hey.com/dhh/merchants-of-complexity-4851301b

You can also find blogs and podcasts about this subject, when DHH coined the phrase.

Think about the topic what you will, but the submission looks like a regurgitation to me.

readthenotes1• 1 year ago

Those sorceries--

https://en.m.wikipedia.org/wiki/Seven_deadly_sins

moritzwarhier5• 1 year ago

All in service of greed. Another one.

Thinking about it: it's interesting bridge to utilitarianism (the other deadly sins enable the greed of the merchant).

And also, the title ironically is itself an overgeneralization and appeal to fear/scrunity (though certainly not generally unjustified)

gmuslera• 1 year ago

Complexity is the mind-killer

loadingcmd• 1 year ago

This article labels complexity as bad. I think the central challenge is not whether complexity is inherently good or bad, but rather whether it is justified and well-managed, whether it adds value and whether the team is open to alternatives and looking for continual improvements.

Duanemclemore• 1 year ago

There's an often missed distinction between the complex and the complicated. Complexity can emerge from the interaction of two (or more) simple things. Think of the double slit experiment[0] or Penrose's observations of quasitiles[1].

I practice and teach computation within design - mostly architecture. When talking to my students I make sure to highlight this distinction. Creating complicated things is one way to create complexity. But complexity as an outcome of carefully chosen simple ideas, techniques, or phenomena is another, arguably more powerful way.

Brooks' ideas of essential and accidental complexity are precisely in line with this. Managing complex effects is itself a challenge (an exciting and rewarding one). And the more "complicated" a system, the more difficult this becomes - and exponentially so I would argue. So both in my own work and in helping students understand how to design for managing complexity but minimizing complicated-ness.

I would be remiss if I didn't mention Murray Gell-Mann's notion of Plectics, which combines simplicity and complexity in one[2].

This is a really awesome subject area to explore, and I look forward to more discussions around it.

[0] https://en.wikipedia.org/wiki/Double-slit_experiment

[1] https://www.youtube.com/watch?v=th3YMEamzmw

[2] https://onlinelibrary.wiley.com/doi/epdf/10.1002/cplx.613001...

belZaah• 1 year ago

Complexity is a known problem in systems engineering. There are useful metrics, that correlate well with both cognitive and financial metrics. Prof. De Weck and his group at the MIT are doing great research on this

sigsergv• 1 year ago

Complexity could be reduced to safe levels using proper documentation and documentation processes.

Beware the Complexity Merchants

Related Stories