Against Essential and Accidental Complexity

https://danluu.com/essential-complexity/

by weinzierl • 6 years ago

246 102 6 years ago

pron7• 6 years ago

I normally agree with Dan's essays, but this time I very much disagree. As someone who first started programming around the time Brooks's article was written, my conclusion is that not only was Brooks right, but, if anything, that he was too optimistic.

In the late 80s and early 90s, a lot of software development was done with various RAD languages (Magic, Business Basic -- my first programming job used that -- and later VB). We fully expected the then-nascent Smalltalk, or something like it, to become dominant soon; does anyone claim that JS or Python are a significant improvement over it? Our current state is a big disappointment compared to where most programmers in the 80s believed we would be by now, and very much in line with Brooks's pouring of cold water (perhaps with the single exception of the internet).

My perception is that the total productivity boost over the past three decades is less than one order-of-magnitude (Brooks was overly careful to predict no 10x boost due to one improvement in language design or programming methodology within one decade), and almost all of it comes from improvements in hardware and the online availability of free libraries (Brooks's "Buy vs Build", which he considered promising) and information -- not from changes in programming methodology or language design (although garbage collection and automated unit-tests have certainly helped, too). The article also mixes hardware improvements and their relationship to languages, but we knew that was going to happen back then, and I think it's been factored well into Brooks's prediction. Moreover, my perception is that we’re in a period of diminishing returns from languages, and that improvements to productivity Fortran and C had over Assembly are significantly greater than the gains since.

The best way to judge Brooks's prediction, I think, is in comparison to opposite predictions made at the time -- like those that claimed Brooks's predictions were pessimistic -- and those were even more wrong in retrospect.

I would also add that if you want to compare the ratio of essential and accidental complexity in line with Brooks's prescient analysis, you should compare the difficulty of designing a system in an accidental-complexity-free specification language like TLA+ to the difficulty of implementing it in a programming language from scratch. I find the claim that this ratio has improved by even one order of magnitude, let alone several, to be dubious.

Brooks states a bound on how much programmer productivity can improve. But, in practice, to state this bound correctly, one would have to be able to conceive of problems that no one would reasonably attempt to solve due to the amount of friction involved in solving the problem with current technologies.

I don't think so. Although he stated it in practical terms, Brooks was careful to make a rather theoretical claim -- one that's supported by computational complexity results obtained in the 80s, 90s and 00s, on the hardness of program analysis -- about the ability to express what it is that a program is supposed to do.

mikewarot3• 6 years ago

It is my observation that if you want to write a GUI, the tools peaked around Delphi and Visual Basic 6. GIT is way nicer than making manual pkZIP backups, and there are nice refactoring tools in Lazarus... but it's not THAT much better.

What was really surprising to me is the lack of GUI building IDEs for python, etc. Now I know why they are scripting languages, and not application languages.

paulryanrogers2• 6 years ago

How maintainable were creations on those old RAD tools? Because in my limited experience problems quickly outgrow them or become nigh impossible to comprehend monstrosities.

jcelerier• 6 years ago

What was really surprising to me is the lack of GUI building IDEs for python

https://www.learnpyqt.com/tutorials/first-steps-qt-creator/

ajuc• 6 years ago

I remember writing stuff in Delphi 6 and making GUI resize correctly wasn't particularly convenient.

On the other hand Qt3-era QtDesigner was great at everything Delphi was good at and much better at layouts.

dominicjj2• 6 years ago

Agreed. And Dan is doing nothing with ANY essential complexity here. Seriously: he's copying logs and generating plots from them. That's no different from copying logs and generating plots in 1986. Only the underlying software and hardware infrastructure has advanced. Conceptually it's an identical problem. And touchingly he thinks that having a grasp of how to use sophisticated tools to achieve what is a very simple task is somehow essential complexity. It isn't. It's still just copying logs from one place to another and generating a plot from them.

And you and Brooks have been absolutely right: there was no one improvement in language design that gave a 10x boost in productivity.

sanxiyn3• 6 years ago

I think you misunderstood. Dan agrees there is no essential complexity. On logs: "this task ... is still nearly entirely accidental complexity". On query and plot: "I think it's unbelievable that essential complexity could have been more than 1% of the complexity I had to deal with".

Brooks claimed in No Silver Bullet that 2x programming productivity improvement is unlikely because now essential complexity dominates over accidential complexity. Dan is trying to demonstrate that it is completely untrue.

scroot• 6 years ago

Our current state is a big disappointment compared to where most programmers in the 80s believed we would be by now, and very much in line with Brooks's pouring of cold water

People not exposed to this era almost have a hard time believing it. My litmus test is this: how easy is it for any person to make a button in a given computing system? On a Mac in the late 90s this was so easy that non-programmers were doing it (Hypercard). Today where would one even begin?

tl2• 6 years ago

What Dan got right:

Brooks clearly missed the boat when he said, "Well how many MIPS can one use fruitfully?". I'm not sure where the fault lies on that one. A recent article [1] bemoans the slowness of modern software.

What Dan got wrong:

Both problems discussed were created by computers. A large collection of machines generating logs is possible to analyze, but that possibility comes from massive investment in tools designed to parse, extract and report on linear and structure data. Brooks covers this in "Buy vs. Build", although his example is closer to Excel than ggplot.

Also, if you are dashing off a python script in a single wall-clock minute, you either write insane amounts of Python for purposes similar to this or are way smarter than the average engineer. In Advent of Code (where Python is popular), the fastest solution to the easiest puzzle (Day 1) this year was 0:07:11. [2]

[1] https://news.ycombinator.com/item?id=25386098

[2] https://adventofcode.com/2020/leaderboard/day/1

Olreich• 6 years ago

I just tracked my time for doing that Advent of Code day 1. It took 10 minutes from start to finish. Across the two parts of the question, I spent approximately 2 minutes building the scripts, 1 minute determining an algorithm, and 7 minutes reading the prompt. I suspect similar breakdowns for the 7:11 you quote.

If Dan excludes the prompt reading and solution determination because he's seen this sort of problem before, 1 minute using tools he's comfortable with doesn't seem unreasonable. Considering all it is is a list of hosts, a loop, a try/except block, and a scp command, using much more than a couple of minutes writing would be surprising to me.

Jtsummers• 6 years ago

There was a 6 minute delay in getting the inputs for day one for many people. The fastest part one solution was 35 seconds. And the fastest part two (after finishing part one) was also about 30 seconds.

If there hadn’t been a server issue I suspect many of the top 100 would’ve finished both parts in under two minutes

crispyambulance2• 6 years ago

I think Dan has braggadoccio'd his time estimates, or his task is somewhat different from what he describes. I mean, the guy talks fast, like really fast, so I suppose he's quick but mere minutes for something like this doesn't seem realistic unless it's extremely routine. Instead, it seems ad-hoc-ish and exploratory to me, it seems like something that needs to be considered and planned out rather than done between 2 sips of coffee. (I am considering his whole task here, not just the scp'ing of files).

He's talking about log files from a few hundred THOUSAND servers that results in several terabytes of data that have to be parsed. He doesn't say exactly what he's looking for, but the point is he's trying to answer some questions about performance for more than a few different applications. Are these simple questions, or involved ones which spawn other questions? We don't know, but even if they're easy questions, there's many applications involved and many servers.

Right off the bat, for something THAT BIG, I think it's reasonable to figure out what you're going to do with a sample of logs before downloading "home depot" onto your hard-drive. So this is definitely a multi-pass kind of job: start with a survey, then try a bigger chunk, if everything's OK do the rest.

Next, I think it's advisable to consider factors about the servers themselves: the application versions, whether or not the applications were running (and why not), the hardware, the role of the server, whether or not the server was up (and why not). Is this metadata about each server available (can you trust it?) or is it something that has to be queried each time on each server? Dan says this supposed to be a couple of years of data, has each server been through upgrades? when? Is that relevant? We don't know any of these, but they would have to at least be considered for someone doing this task.

After the data is parsed there's slicing and dicing to do for the purpose of graphs. That takes lotsa of time-- I am assuming he's not just talking about extracting one figure for each application and plotting it.

For someone that is all set-up and on top of things, this seems like something that is a day's work and easily more, not counting follow-up work and validation to further investigate the additional questions that would inevitably (in my opinion) be raised on such a big dataset.

adwn3• 6 years ago

What Dan got wrong: Both problems discussed were created by computers. A large collection of machines generating logs is possible to analyze, but that possibility comes from massive investment in tools designed to parse, extract and report on linear and structure data.

I don't understand your argument. In what way does that disprove Dan's point?

taeric2• 6 years ago

It isn't one user's work. This would be a literal technical debt of the modern user.

Edit: I meant this as a question. Used wrong punctuation and don't want to ninja edit.

tl2• 6 years ago

Analysis of data coming from computers (logs coming from servers in this case) is guaranteed to have a structure that makes it easier to process, compared to data coming from more organic sources (reports collected by people, measurements from instruments, etc...) which are closer to the domain Brooks would have dealt with.

When dealing with these problems today, the greatest challenge isn't writing a script; it's deciding whether data points that don't fit the model invalidate the data or the model.

Moreover, we've spent five decades systemizing the former. Doing the latter is more challenging than ever and fraught with controversy.

loeg• 6 years ago

For leaderboarders, and especially the easier challenges, total AoC solve time is largely prompt comprehension. If you already know what you want to do, and that thing is straightforward, it is not unimaginable to bang out a short python script in a minute.

amadeuspagel• 6 years ago

Technological improvements can also make it easier to deal with conceptual complexity, by allowing us to express ourselves better.

giovannibonetti• 6 years ago

It is easy to see that the total complexity has been severely reduced in the last years when you look at how many small companies are doing things that required an army of people a few decades ago.

For example, running the technology for a real-time banking system nowadays can be done by perhaps even a single developer (with enough experience) thanks to open source, cloud computing and advances in software and hardware. Now we even have distributed relational databases that are relatively straightforward to operate, like CockroachDB.

pron7• 6 years ago

First, Brooks wasn't talking about that. He was talking about productivity boosts due to programming language design and programming methodology. More specifically, he did not exclude improvements to both kinds of complexity overall, but only those that drastically change their relative proportion. Second, in the early 90s I had a summer job at a company whose old ERP, which had been written in a language called Business Basic and ran on a minicomputer, was maintained and operated by a single person.

jasode3• 6 years ago

, Brooks wasn't talking about that. He was talking about productivity boosts due to programming language design and programming methodology.

Brooks was also talking about non-programming language advancements as a possible "silver bullets". See the pdf and look for the following sections that are not about programming syntax:

  Hopes for the Silver
    Artificial Intelligence
    Environments and Tools
  Promising Attacks on the Conceptual Essence
    Buy versus Build

He underestimated AI machine learning. With hindsight, we see that the deep-layer neural net combined with GPU beat human rule-based programming for automatic language translations, DeepMind AlphaZero beats expert programmers hand-tweaking IBM DeepBlue, etc.

EDIT reply to : ", nor have they reduced the overall cost of programming by 10x even after more than three decades."

I'm not convinced of that. The issue with your conclusion is that it omits the increasing expectations of more complex systems. I'd argue we did get 10x improvement -- if -- we hold the size & complexity of the system constant. E.g. write 1970s style text-based code to summarize sales revenue by region and output it to green bar dot matrix printers. This was tedious work in old COBOL but much easier today with Python Pandas or even no code tools like MS Excel.

The invisible factor we often overlook is that our demand for software to do more complex things will always outpace the smaller productivity increases of the tools. This makes it look like our newer programming tools never gave us 10x improvement when it actually did.

paulryanrogers2• 6 years ago

Total complexity must then exclude the layers of hardware, cloud infrastructure, and FOSS contributions that involved many other people.

coldcode• 6 years ago

Complexity in just what I and doing right now is impossible to comprehend. Imagine a mobile app with dozens of dependencies taking on a "secret" project involving a dozen of those dependencies that are drifting far way from the rest, with a dependency manager unable to function in this circumstance requiring manual guessing of what goes with what. Now imagine trying to develop with this Frankenstein set of dependencies and have any hope of getting work done. Of course the problem is not technical, its institutional; yet it's an unreasonable well of complexity that belies the demands placed on its workers. Technological complexity is not the only kind to worry about, it's the difficulties of managing complexities that are not obvious and cannot be explained to non technology savvy executives leading to painful results.

Now add a wealth of server side complexities with webs of micro and not-so-micro services interdependent on each other made by teams further and further removed from each other leading to more instability and unpredictable behaviors.

Managing systemic complexity (or not doing it well) seems endemic to programming today much worse than when Brooks wrote his essay (and I actually predate).

CraigJPerry• 6 years ago

In 1986, perhaps I would have used telnet or ftp instead of scp.

In 1986 you would probably have used rcp (the thing that inspired scp). A lot less secure but with one benefit - when Dan did this mass scp i bet his local CPU was absolutely pegged well before network saturation?

Ssh imposes a hefty cpu overhead because of encryption that i didn’t appreciate until around 2005 when i was updating an rcp centric script to use scp.

Modern scripting languages didn't exist yet (perl was created in 1987 and perl5, the first version that some argue is modern, was released in 1994), so writing code that would do this with parallelism and "good enough" error handling would have taken more than an order of magnitude more time than it takes today

You could have used ksh (a popular scriptable shell that heavily inspired Perl).

rcp errors are denoted by return code and thats also the natural error handling approach in ksh.

As for the grep / sed / awk part of the problem, it’s possible the 1986 local machine was uniprocessor - it might have been faster to do the processing locally on each node (via rsh) then only network transfer the result set back to the local node.

EDIT - job control (the thing that makes parallelism trivial in ksh) was only added in 1989. To easily get parallelism you would have had to write the script in csh - life’s too short for scripting in csh (or its descendants) so i conclude everything i said before is wrong and in 1986 i’d have said sod this and went and brewed a cuppa tea instead :-) (not strictly true since i was coming up on 2 years old at that time...)

earthboundkid3• 6 years ago

Yeah, my thought on the problem was that there’s probably an AT&T promo video from 1986 on YouTube somewhere that shows Brian Kernighan logging into 10 machines and downloading phone records and then processing them with AWK. He just wouldn’t be able to download quite as many and do as much with it.

mikewarot3• 6 years ago

In the advent of code, I solved day 23 (the crab and the cups) doing brute force, and even with today's hardware it would have taken many hours, if not days, to complete. A tweak of the algorithm got it down to 0.248 seconds. That tweak required an 4 megabyte array in RAM.

I back ported the solution to Turbo Pascal 7, it used a "file of longint" in place of the array which you can't do in MS-DOs... it finished in about 10 minutes, because I have an SSD. Otherwise it would have been about 50,000,000 IO operations at 10/second --> 50+ days.

We can and do use the heck out of the hardware, but it's not infinite, and there will always be orders of magnitude performance to be gained by seeking and using better algorithms.

mark254• 6 years ago

Well, using "himem.sys" you could make use of RAM above 1MB... I used it to write a sort utility that only uses temp files when RAM was exhausted.

On my 386 with 4MB total RAM I could use about 3.7 or so, the rest was a shadowed BIOS copy. (Never tried to deactivate that, perhaps it would have been relocated and then made available?)

hbrundage• 6 years ago

A missing colour in the discussions around complexity today is the ever growing ambition for the problems we're all trying to solve. My grand unified profoundly unserious theory is that we're always going to complain about accidental complexity because we will keep discovering new reams of it by extending what we're trying to do out into the unknown parts.

If we continued to play in just the known parts of the problem space, we'd see it melt away further and further into Brooks' model of the world. Instead, we built distributed systems, we work in absolutely giant teams, we deploy to the strange heterogeneous runtime we call the web, etc etc, and all of that generates new complexities to abstract away before we get back to just the essential stuff.

eternalban• 6 years ago

In this example, let's say that we somehow had enough storage to keep the data we want to query in 1986. The next part would be to marshall on the order of 1 CPU-year worth of resources and have the query complete in minutes. As with > the storage problem, this would have also been absurd in [1986], so we've run into a second piece of non-essential complexity so large that it would stop a person from 1986 from thinking of this problem at all.*

If you are given a bucket to drain a lake, the enormity of task at hand is not "accidental complexity".

OP is asserting that doing x one time or a billion times is still x, with an inherent complexity of using a bucket to (dip, draw, transfer, drain). This is true -only- if we willfully insist on ignoring that our 'conceptual model' (a reservoire, a bucket, and a worker) is intimately bound with available technology and is a leaky abstraction. The actual conceptual [model] (move liquid from one reservoire to another) says very little about "inherent complexity". (Can you point it out?)

It seems it is more accurate to note that the complexity of materializing abstractions tends to increase with scale of operations.

Our goal then, as a discipline, should be to insure this relationship approaches linear [or better] to the extent possible. Continuing the example, the actual task (drain a lake) could materialize as an army of workers, each with a bucket, all converging on a sink to drain their bucket. Is managing the workers really an "accidental" complexity?

What above then implies is that the notion of "essential complexity" is a faulty paradigm if considered to be a static measure indepdendent of du jour technology. The essential complexity of a task is held to be inherently related to the magnitude of task, and thus inherently addressable by, and related to, advancements in technology.

And finally, since nearly all of the modern software systems are merely a scaled up ("web scale") version of earlier, "less complex", systems, it seems reasonable to question the industry's singular obsession with languages as the remedy to what is inherently a problem of scaling well understood tasks.

eadmund• 6 years ago

In 1986, perhaps I would have used telnet or ftp instead of scp. Modern scripting languages didn't exist yet (perl was created in 1987 and perl5, the first version that some argue is modern, was released in 1994), so writing code that would do this with parallelism and "good enough" error handling would have taken more than an order of magnitude more time than it takes today. In fact, I think just getting semi-decent error handling while managing a connection pool could have easily taken an order of magnitude longer than this entire task took me (not including time spent downloading logs in the background).
Today, we have our choice of high-performance languages where it's easy to write, fast, safe code
In 1986, there would have been no comparable language

Shell existed in 1986, and IIRC ftp could actually be scripted, a bit, so I think that it could have been used to run the download portion. Maybe I misremember?

Lisp existed in 1986; the first edition of Common Lisp the Language was written in 1984. It certainly could have been used, and even in the 80s I think it would have good enough performance for log parsing. Certainly it would now, but that is after four decades of optimisation.

His point about libraries, though, is excellent. The rise of open source libraries radically changed the software world for the better.

I should note, though, that Plan 9 — which was developed in the late 80s — removed a ton of the accidental complexity in dealing with e.g. FTP servers. We could have much less accidental complexity, but we choose not to.

pydry3• 6 years ago

His point about libraries, though, is excellent. The rise of open source libraries radically changed the software world for the better.

This was one of the first points made in no silver bullet (except he said buy not "download for free").

It was a lot less obvious in 1986 than it is today and arguably it's not even all that obvious today.

kazinator• 6 years ago

The reality is that Common Lisp would not fit onto a Unix workstation from 1986 in a light weight way, like firing off another /bin/sh or /bin/awk job.

Machines had memories measured in single digit megabytes, often closer to zero than to 10.

karmakaze• 6 years ago

The fault in this discussion and what I was hoping to learn about is the reduction in complexity of evolving systems. This post and the original cited work is very much about one-shot waterfall development. This is not what is done today and has little bearing.

How much of the effort in fitting a new requirement into an existing system is due to choices in the way it was decomposed and structured? I'm convinced there are much better than 2x better ways of keeping this effort low by choosing seams carefully and putting in the work when some are found to be in messy places.

Another way to say this might be, any system could (for sake of argument) be built with 2x essential complexity, but an evolved system typically has a much higher factor due to its lineage. Finding ways of keeping this near 2x should be the focus.

taeric2• 6 years ago

My gut is that the claim translates better to percentage of processing power per user. With such an excess of power today, you can scrape a ton of use out of it.

That said, the use of scripting languages kind of proves the point. There is little attempt made to maximize compute. Only gets more true when you consider the absurd amount of memory being used.

pydry3• 6 years ago

Fred Brooks suggests in 1986 that future productivity improvements focus more on:

Exploiting the mass market to avoid constructing what can be bought

In 2020 danluu says that:

In 1986, there would have been no comparable language, but more importantly, I wouldn't have been able to trivially find, download, and compile the appropriate libraries and would've had to write all of the parsing code by hand, turning a task that took a few minutes into a task that I'd be lucky to get done in an hour.

IME this demonstrates remarkable prescience by Brooks. The only real difference between Brooks' hypothetical future where software is assembled from bought components and the one danluu lives in is that the components are, in this case, free.

Package managers might seem obvious and necessary today, but I don't think they were at all obvious in 1986 (or even in 1996). Fuck, golang didn't even think they were necessary in 2009.

Also:

Looking at the specifics with the benefit of hindsight, we can see that Brooks' 1986 claim that we've basically captured all the productivity gains

I don't believe that this was his claim. He claimed that productivity gains from improved programming language design would be incremental rather than game changing (hence: no silver bullet).

Some people probably will argue that there is a 10x difference in language productivity due to language but I don't think that is the case. In fact, I think there's a class of people who keep getting proved wrong and wrong again about the relative importance of package management ecosystems over "game changing" language features like non nullable types that are incrementally better, but not an order of magnitude better.

ncmncm3• 6 years ago

I see that you are unaware of the large collection of programming language improvements accumulated in recent decades, which I will guess is because your favored language mostly does not implement them.

pydry3• 6 years ago

Which improvements caused a 2x+ productivity boost that my favored language does not have?

BenoitEssiambre• 6 years ago

This is a pretty poor argument.

This does not represent productivity in solving new problems, it just tells us it's worth reusing pieces for already solved problems.

Yeah carpenters might seem very productive if they started with increasingly already built houses like programmers do.

Solving the part of the problem that is novel still has irreducible complexity.

coldtea3• 6 years ago

This does not represent productivity in solving new problems, it just tells us it's worth reusing pieces for already solved problems.

That's neither here nor there.

First because most (if not all) of new problems can be solved by reusing existing parts (from languages and libs to widgets and cli tools).

Second, because we tend to re-work on the same problems over and over for the most part, anyway, so improvements on those are still greatly important (even more so than novel problems actually, which are about potential uncertain future markets, whereas solved problems are what drive existing global markets, so dozens of trillions of dollars).

vannevar2• 6 years ago

I don't think that Brooks was arguing that any given fixed programming task could not be made easier over time; clearly, even in 1986, there were programming tasks that had become trivial by use of existing code but were originally quite challenging to code on their own. I think Brooks was talking about the work of the individual programmer: given two identical programmers, with the same available libraries of existing code, is there a language or IDE that will make one 10x more productive than the other? And I think the answer is still, no.

brabel• 6 years ago

Exactly, this blog post missed the point completely. Unbelievable.

joshuafcole• 6 years ago

I think the lines between "available library of existing code", language, and IDE are much murkier than they might appear. If one programmer is given raw assembly and the other Python, you're probably going to see at _least_ a 10x productivity bump to the python user _even if they use no additional libraries_. Language and IDE choices impact the vocabulary available to express solutions in the same way.

vannevar2• 6 years ago

You can certainly contrive a counter-example---even in 1986, you could say much the same thing about assembly and something like C or BASIC. But once that leap of abstraction was made, we seem to have hit a plateau of expressive power. Is the jump from C to Rust as empowering as the jump from assembly to C? I don't think it is.

jnwatson• 6 years ago

You're added premise is an implicit point of Luu's article. Or perhaps the point is that Brooks' framing doesn't really apply when you can avoid programming altogether.

The silver bullet is open source. The ability to utilize the millions of hours of other people's work to accomplish your programming task is the game changer, and is what makes formerly complicated tasks trivial.

The essential complexity of the software I deal with is about two orders of magnitude more complicated than stuff I dealt with 20 years ago. I'm not sure that 20 years from now it will be another two orders of magnitude, but perhaps with utility computing and higher level orchestration concepts, it very well might be.

astine2• 6 years ago

Broooks framing does apply when you can avoid programming altogether. He specifically mentions that as is prefered solution to the problem. Rather than coding systems from scratch, in which case you have to deal with the essential complexity inherent in the problem, you can either buy software or share components between developers and avoid having to deal with the problem altogether.

ozim3• 6 years ago

I have different idea about essential complexity and accidental complexity. I think examples in the article are all just accidental complexity.

Essential complexity from how I read Fred Brooks is a domain knowledge like what an accountant does and what features he is using. All frameworks, servers and doing scp or queries in that is accidental complexity.

Accountant does not need any React or Angular front end, they don't need a server.

Where angular brings in reduction in accidental complexity since by using it I don't have to build a lot of stuff from scratch, it does not make tax laws any simpler. Where tax laws are essential complexity that needs to be coded in accounting software.

The same with servers, going to cloud so reducing my accidental complexity by removing need to deal with servers is not making tax laws simpler or easier to grasp.

This way in my opinion, essential complexity and accidental complexity holds really well as general idea.

sanxiyn3• 6 years ago

Eh, you and Dan are in agreement? "This task ... is still nearly entirely accidental complexity", etc.

ozim3• 6 years ago

Not really, because I think that tasks he picked as "essential complexity" are still fully "accidental complexity" in terms of how I understand Brooks writing.

I cannot agree with Dan, because I feel that all "computer stuff" is accidental complexity in Brooks writing context.

Some other replies change context into "from software dev perspective that is an essential complexity", now we are discussing what Fred Brooks wrote in 1987. If we change that context then I can agree with Dan, but if you want to refute argument you have to operate inside the same context.

What I see that Dan argues, is accidental complexity is not getting easier throughout the years. He does not touch anything about essential complexity being well "complex" and that we cannot do much about it. So I don't agree that you can refute whole idea based on those examples.

unabst7• 6 years ago

In your example, the tax code is simply the abstraction layer one level down that you have no control over. So to translate your example, HTML specs would be essential complexity and all authored HTML would be accidental complexity to a front-end web programmer.

But this makes complexity a matter of context.

A browser is built using a programming language and uses interfaces provided by the OS and other libraries. This makes the browser accidental. And you can take this all the way down. Assembly is accidental with respect to the processor being essential.

But now we see how complexity flows through each layer. Just as the accidental complexity of the tax code becomes essential complexity for the accountant, the accidental complexity of HTML becomes essential complexity for the web programmer.

And a popular pass time is complaining how much simpler everything could be, only if.

(not sure if this is what the original article had in mind but this model seems accurate)

reificator• 6 years ago

So to translate your example, HTML specs would be essential complexity and all authored HTML would be accidental complexity to a front-end web programmer.

The HTML specs are not essential complexity, are you joking?

Accounting software that handles tax codes is the essential piece. HTML is just one approach to building an application.

Even though I whip out web dev quite often, I find it scary that more and more people view it as the only option or the default approach. To say it again, the HTML specification is not essential complexity in the context of an accounting app.

ozim3• 6 years ago

My main point is that all "computer stuff" is accidental complexity as I believe people in 1987 would think. It might be hard to go along with it when one is software dev and much of our lives are on the internet. But that is what Fred Brooks was writing about back then.

Essential complexity idea is about real world being inherently complex and that we cannot remove that complexity by making bigger better computers. Dan is not attacking this at all, he only argues that accidental complexity is not getting easier.

You argue the same thing as Dan. You want to move essential complexity to a different place. But that does not refute "Essential and Accidental complexity" in context of book it is just arguing that accidental complexity is not getting easier. Fred Brooks argues that even if you make accidental complexity easier you will not make real world easier. He was not saying that accidental complexity will get easier or that it is some kind of law that it always gets easier.

archibaldJ• 6 years ago

Agree.

Factoring in the business process (esp a somewhat formalized one) as well as the notion of economic value would help to introduce a pragmatic separation between essential and accidental complexity.

I guess what gets interesting is when the business process is fussy itself (e.g. in a start-up setting) and that is when the direction of programming and software development at large takes in a more reinforcement learning discourse v.s. a pure top-down reductionistic one.

Perchance that is also when the categorisation of essential vs accidental complexity becomes more ambiguous in the human process.

mc10• 6 years ago

I think this assumes a fine distinction between "domain" and "non-domain" knowledge that often doesn't exist.

If you're writing a trading strategy, the technical details of how the code's executed often dictate what kind of trading you're planning on doing. e.g. How do I balance speed vs how sophisticated the algorithm is?

Also, if your "domain" is technical in nature (e.g. building a new database), it's not clear at all what parts are "essential" and what parts are "accidental". Is the file system accidental? Is the underlying data store (SSDs, spinning disks) accidental? Often the purpose of the project is intrinsically tied to the technical capabilities of the platform it's built on top of.

pm• 6 years ago

I personally refer to these concepts respectively as inherent and incidental complexity, what should be classed as inherent complexity is anything that directly supports the description of the problem. Everything else is incidental complexity (and we mostly write incidental complexity).

The big thing that I don't hear often is about where complexity lives. Complexity is like a pile of receipts you need to hunt down to do your tax - you can have it distributed all over your house, you can sweep up into a pile, or you could file it. The first is by far the most complex, and I've worked on codebases that looked like that. Even the most trivial of changes takes a long time.

jariel• 6 years ago

I think the bifurcation might be more clearly described as 'problem space' and 'solution space' complexity.

The 'complexity of the problem' is inherent, by definition.

Unless accountants want to literally change the requirements.

The complexity of the solution space is something else entirely and it may be hard to derive theoretically ...

But with sufficient tooling, such as AI, maybe it becomes trivial.

If we have 'AI Apis' that can immediately handle complex problems and dissolve them into something really simple, then many apparently complex domains could be reduced to something simple.

As long as we're allowed to use 'other software' and not have it 'count' as part of the complexity we are measuring, well, the then complex of the 'app' tends towards 0.

You can already see this a little bit with CRUD apps, which often feel like just gluing some forms together.

megameter• 6 years ago

From a philosophical standpoint, the starting line for "complex" is in something you take it upon yourself to pursue. The world we inhabit has a complex ecosystem, physics, etc; our study of it is far from complete. But every time we learn something about it a technical advance becomes possible.

Nobody has to keep the gears turning in this vast society. But someone at some point decided it was worth it to create those gears, without answering all the how's and why's: basic science doesn't tell us how to engineer things, only that we have ways of dong so.

So the "accident" of it emerges like a freeway pileup, from different domains of study colliding. And with computing being the workhorse technology for everything now, the crashes are so frequent as to be unremarkable.

dustingetz• 6 years ago

Hypothesis: Constraints on time is the cause of accidental complexity. Given infinite time I would think longer, yak shave more, rewrite everything simpler, recur until perfect. In real life what constrains time? The business, its competition and available resources. Traded off vs speed of human thought and communication.

heisenbit• 6 years ago

Brooks:

The essence of a software entity is a construct of interlocking concepts: data sets, relationships among data items, algorithms, and invocations of functions. This essence is abstract, in that the conceptual construct is the same under many different representations. It is nonetheless highly precise and richly detailed. I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared to the conceptual errors in most systems.

I think Brooks by focusing on the reality of the world reflected into topology of systems focuses on the essentials for the complexity. Yes, we can abstract some complexity but we also know these abstractions are leaky - heaping too many of them on top of each other and it topples.

Even more important there are limits to how much most of us can juggle in their minds - exceed that threshold and we need a team with all the overhead that accompanies that. Once inter human communication is part of the process any optimization on tool level can only improve the situation by so much.

neolog• 6 years ago

Dan, I'm curious how all those people listed at the bottom read the post early. I sometimes want early feedback on my writing and you seem to be getting it from a lot of well known people. How do you do that? Do you just email them all and ask?

astine2• 6 years ago

This essay badly misinterprets Brooks original essay. Brooks was talking specifically about the problem of building new systems from scratch.He was speaking about programming productivity. This essay is talking about operational tasks. Of course operational tasks can be speed up with improvements to software and hardware, that's the whole point of writing new software and developing new hardware.

Furthermore, Brooks doesn't actually argue that there is no way to tackle essential complexity when writing new software. What he says is that there is no single technique that will speed up the process of writing code across the board, that is, there is no silver bullet. However, he does make allowance for techniques that allow us to break up complex systems into smaller systems which can then be solved independently. He specifically mentions object oriented programming as a tool which makes it easier for programmers to write software as components but CPAN would have been a better example. He just doesn't allow that OOP by itself would allow for an order of magnitute improvement (something which I think that experience has born out.)

The author of this essay, points out a couple of operational tasks which involve programming were easier for them than they would have been 34 years ago because faster hardware noow exists and because software that already takes care of a subset of the problem now exists when it didn't before. Partially, this is confirmation of Brooks' idea that breaking down problems it the way to go long-term, but more critically, it's a failure to realize on the part of the author that they're speaking of productivity in their role as the user of a computer system whereas Brooks is speaking about productivity in designing new software products.

edit:

I think that the fundamental root of the essay's problem is that the author seems to have misssed what Brooks was referring to which he discussed accidental vs essential complexity. Brooks specifically pointed out that he was referencing Aristotle when he used those terms, and in this case essential doesn't mean unavoidable, but a part of the essential character of the problem itself. A more modern terminology would be intrinsic vs extrinsic. Complexity is extrinsic to programming if it comes from something other than the nature of the problem itself. If my computer is too slow to allow me to write a program, then that's probably not due the the nature of the program itself (except indirectly) and so represents a form of accidental complexity. Feature requirements however, represent essential complexity. You can manage essential complexity according to Brooks by dividing the problem up or avoiding altogether, but it's not something that will go away simply with better tooling or a better methodology, (unless that better tooling replaces software that you otherwise would have written.

zmmmmm• 6 years ago

I found the example given very contrived and the whole article slightly disingenuous because of it.

Brooks was talking about building long-living applications serving complex business requirements. The article talks about a one off script with no ongoing life. It's not even really "software" at all in the terms Brooks was describing.

It's like someone describing absolute limits on fuel efficiency for vehicles and then I get on a scooter and coast down a hill and jump off at the end so that it crashes into a wall and explodes, and using this example to claim they are wrong.

daotoad• 6 years ago

The difference between a one off script and a production-grade system with documentation, instrumentation, and monitoring is pretty big.

Here's less than a minute's worth of questions I would want answered for a professional version of the log fetching example:

* Is that list of a thousand servers static? * How is it updated? * What format is it in? * What do you do when the resource isn't available? * What other systems are impacted when the resource can't be contacted? * Is it acceptable to use a cached server list until it the resource can be updated? * How do you handle servers that don't respond when polled? If we retry connections, what scheme do we use? Linear or exponential back off? * After a service interruption, how do we resume data collection? Do we keep rolling forward or do we request older data? How quickly do we retrieve backed up data? Is service resumption plan different for the case when a single system is down vs when the log fetching system has an outage?

mchusma• 6 years ago

Agree. Nearly all the work of programming I have done is related to the task of trying to figure out the business problem in enough extreme detail to provide a solution.

The main lever to improve productivity for me as a developer are open source tools, commercial APIs, and similar things. This isn't reducing the amount of programming, it's just moving it around.

Magically 1,000x faster computers would maybe help me be 20% more productive.

0xbadcafebee• 6 years ago

Let's say you're building a home. Foundation is poured and it's time to frame it. If you hire some random guys with power tools standing outside a home depot, it'll take weeks to months for them to finish, and might well end up crappy. But hire two 55-year-old men with hammers, nails and a single skil-saw, and they can have it up in 3-4 days. Or grab your Amish neighbors and raise a barn in a day.

Productivity is not bounded by your access to technology, it's bounded by the problem you need to solve and the means you use to solve it.

Brooks explicitly dismisses increased computational power as something that will not improve productivity ("Well, how many MIPS can one use fruitfully?", more on this later), but both storage and CPU power (not to mention network speed and RAM) were sources of accidental complexity so large that they bounded the space of problems Brooks was able to conceive of.

Yet here in the future, our excess of storage, CPU power and network bandwidth have bounded our ability to be fruitful with them. If you add up the MIPS we use today and compare it to what solutions we're implementing with it, we are vastly less productive today. Brooks was right, but in a weird way. There is a bound on programmer productivity, and it is the programmer.

In the past, people wouldn't have set themselves up to need to collect a billion billion logs to analyze some tiny metric among them. Anyone would've seen that as poor planning and a horrible waste of resources. Why would you bury needles in billions of haystacks, and then need to hastily build a robot that finds needles? Why not just keep the needles in a pile of needles? Do you really even need all those needles?

Thinking that the tools you personally use are as good as it gets is an easy trap to fall into.

This actually seems like a hard trap for most to fall into. Nearly every software engineer I've met either thinks they could make a better tool, or just wants to try for the hell of it. On the other hand, most engineers think that building a better mouse trap will solve all their mouse problems.

ncmncm3• 6 years ago

The only way Brooks's argument makes sense is that we naturally tackle problems as big as the resources available to us permit. So, our daily load of programming hardly changes, because as we get better equipped, more is asked for.

But in that interpretation, none of his supporting detail applies.

sontek• 6 years ago

Maybe off topic but this site might have convinced me to move to Firefox. Are all of you reading this page on mobile or a small screen?

I have a large monitor and this page expands full width of the screen and the lines are extremely hard to follow. It turns out Firefox has a "Reader View" that makes sites like this be put into a nice readable column.

Here is a comparison of the two:

https://i.imgur.com/plV0e7l.png

I find it interesting that this post reached to the top of HN with such an unreadable layout and so I was interested in if everyone has some tricks for reading content like this or is Firefox/Reader Mode the best way?

konjin• 6 years ago

Going to hard disagree.

What he's saying there is that today we have free implementations of libraries which solve incidental complexity and those libraries work (nearly) everywhere (nearly) the same.

I worked in finance on K and the name makes in impossible to use google/stack overflow to find anything about the language. Which meant I had a paid for help and tech support from a company that's been largely unchanged from the 80s.

It was fucking phenomenal.

Once I gave up the idea that I knew best and started working in the way the language expected me my productivity was an order of magnitude higher than it would have been in a cobbled together python pipeline like the one he uses there. And because someone was responsible for the language and tooling I could use scripts from before I was in primary school, and they worked as expected.

I dare you to try and do the same with any language that isn't C.

Against Essential and Accidental Complexity

Related Stories