A Generation Lost in the Bazaar (2012)

https://queue.acm.org/detail.cfm?id=2349257

by zbentley • 5 years ago

108 69 5 years ago

queue.acm.org

Hokusai• 5 years ago

That is the sorry reality of the bazaar Raymond praised in his book: a pile of old festering hacks, endlessly copied and pasted by a clueless generation of IT "professionals" who wouldn't recognize sound IT architecture if you hit them over the head with it.

I think that all the article is self disqualifying. Maybe this was acceptable language in 2012, but it was still having a bad effect on people, arrogance cannot substitute argumentation.

A Bazaar in conjunction with regulations is more sensible. As an architect at a medium sized organization my job is not too build pretty cathedrals, but to allow the teams in my department to row in the same direction. I set non functional requirements, I set direction, but I cannot know the systems that each team is developing as well as they do. Bazaars require rules, but they evolve dynamically with the needs of the organization.

I love the sagrada familia un Barcelona, but it's still under construction after 150 years. Building cathedrals maybe not the way.

felipemnoa2• 5 years ago

I love the sagrada familia un Barcelona, but it's still under construction after 150 years. Building cathedrals maybe not the way.

Well, they are not jus nailing two pieces of wood.

Also, if I were to build a cathedral today I would use 3d printing and CNC milling machines as much as possible.

TheOtherHobbes2• 5 years ago

Like this?

https://www.bbc.com/news/av/technology-31923259

WJW• 5 years ago

For something like a cathedral, having it take a long time is a feature. Of course you could slap something of the same size together from prefab concrete blocks and some 3d printed stuff, but you would "just" get a soulless building with no effort from the faithful invested in it.

A real cathedral is a monument for the ages, bigger than any individual who worked on it. Something for entire generations to pour their effort into so that at the end of construction you can say: "my grandfather started building this, my parents worked on it, I worked on it and during the time of my great-grandchildren this building will still be as magnificent as it is now".

Trying to build it as quick and cheap as possible is simply not the point for a cathedral.

analognoise• 5 years ago

I think if everyone is simply rowing in one direction, you end up with a favela - functional, but not great in an earthquake or flood.

tempodox• 5 years ago

I once talked to a CEO who expected and wanted an important piece of software to be rewritten every few months so it could use the app framework du jour.

I wonder how many other shops there are where implementations aren't designed to last long, especially in web development.

joshz404• 5 years ago

Agreed. People don't flourish inside cathedrals, no matter how attractive they look.

mihaic2• 5 years ago

Agreed. People don't flourish inside cathedrals, no matter how attractive they look.

Nor do they flourish inside supermarkets, these buildings are designed for something else. Walking in St. Peter's in Roma gives a sensation of awe even to devout atheists, something I can't imagine the local Walmart doing.

TheOtherHobbes2• 5 years ago

Barring apocalypse, Sagrada Familia will still be standing five hundred years from now. UNIX will be mostly forgotten by then - in the same way blacksmithing and carriage building are mostly forgotten today.

That's the whole point of cathedral projects - make it beautiful and build it right because you're building for the future. Not to scratch your own personal boredom itch, or to nail a bit of wood to another bit of wood because why not.

dnautics2• 5 years ago

That has zero to do with development style, and everything to do with medium. There have been plenty of cathedral software projects that are already lost, much less lost 500 years from now.

the_af• 5 years ago

UNIX will be mostly forgotten by then

UNIX may not be the best example for your argument: it's often criticized because of how long its legacy has endured, with some people arguing it's time to move on to more modern visions of what an operating system should be. So UNIX, for good or bad, is indeed the software equivalent of an ancient cathedral which endures from ages past.

simiones4• 5 years ago

Still, looking at ancient constructions, 150 years is a huge amount of time. Most roman constructions that have survived to today were built at most in a single life time - at an extreme end, it seems that the Hagia Sophia was built in 5 years. The more than 4500 years old Great Pyramid of Giza was built in a single pharaoh's lifetime.

Which is to say, engineering culture and goals can absolutely degrade over time in terms of productivity, even with great increases in construction technology.

simiones4• 5 years ago

I love the sagrada familia un Barcelona, but it's still under construction after 150 years. Building cathedrals maybe not the way.

Interestingly, this may also be a sign of declining engineering. The Hagia Sophia, the largest church in the world for 1000 years, was apparently built in 5 years (532- 537).

In general, it seems Rome was able to build fantastically persistent buildings in fractions of the time required later. Even today I wonder if we actually have the engineering and logistics practices required to build buildings that would survive thousands of years in a under a decade - of which the Romans had several.

felipemnoa2• 5 years ago

>Interestingly, this may also be a sign of declining engineering.

Or it could simply mean that not enough resources were committed to the project.

titzer• 5 years ago

Maybe this was acceptable language in 2012, but it was still having a bad effect on people, arrogance cannot substitute argumentation.

I find it refreshingly honest, vivid, cathartic and spot on. Complaints from the peanut gallery just make me love it even more.

In 9 or 90 years, maybe they'll think all of our 2021 eggshell tiptoeing to be completely useless drivel devoid of insight and dump it down the memory hole. Sensibilities change!

mihaic2• 5 years ago

Things seem to be getting even worse, and as I hate to iterate on all the points often repeated on HN (the loss of older industry best practices, CV-driven development, bloated FAANG departments, the issues of funding OSS, etc.), I just want to notice one thing: Most developers and architects seem to have lower and lower confidence in what is technically achievable without some complex dependency web, and that becomes a self-fulfilling prophecy.

As believing something is possible is a pre-requisite to actually approaching it, I expect things to continue getting worse.

PragmaticPulp2• 5 years ago

Most developers and architects seem to have lower and lower confidence in what is technically achievable without some complex dependency web, and that becomes a self-fulfilling prophecy

One of the biggest problems I’ve had with hiring engineers out of big companies is that they’ve developed a sort of learned helplessness. The longer they spend in big, slow companies the more they’ve developed an idea that everything they do must adhere to mountains of process and approvals and documents.

It can be a challenge to un-teach these ideas from people after hiring them into a company where they don’t need to put together presentations and seek multiples levels of approval before doing anything.

One surprise that I didn’t account for is that many people come to like these process-heavy systems with approvals everywhere and no progress until the rules have been followed. It becomes a safety net to pull as many different people and process into a decision before action is taken. If something goes wrong, no one can blame you because you followed the process and got all of the approvals, right? I think this leads a lot of people down the road of preferring high-process, low-action environments because following the process is a type of low-effort, low-risk work whereas producing something new is high-effort, high-risk.

stinkytaco2• 5 years ago

I've worked in environments like that (in and out of software), and this is almost always driven in reaction to something, which is why it's hard to unteach. In my current office there is a culture of consensus building and feedback. If someone feels they were not given sufficient opportunity to participate in a decision, they will complain, resist or otherwise cause an issue in the future. Sometimes they are correct, which is why it's not as easy as simply firing them. Sometimes the red tape is in response to problems in the past. Policies that were poorly enforced that resulted in negative consequences for customers (support failures or discriminatory behaviors). Sometimes it's there to protect low level staff from accusations of the above or to provide clear guidance when dealing with a high stakes/high stress situation. Sometimes it's to protect customers, such as regular security reviews.

So no, I do not think this is about "low effort", though that can be a side-effect. No one sits in an office wanting to find ways to make their coworkers lives difficult. More often than not the policy is a response to pressure to do something to prevent whatever failure prompted the policy. The people who like the process heavy approach, in my experience, anyway, are often remembering a time when failure to follow a procedure resulted in negative consequences for someone.

morpheos137• 5 years ago

Complex process creates more work, sets barriers to entry for competitor workers, and reduces individual responsibility for something that goes wrong.

So, from the employee perspective complex process means a lot of work to do (job security), higher pay (even if the domain problem is not too difficult process complication makes it difficult reducing the available supply of qualified labor) and less responsibility if something goes wrong.

From the business point of view compexity means, being current, keeping up with the latest tech, signaling you are not a dinosaur. Keeping up with security because the shoddy earlier layers had holes because software engineers do not output a finished work product like other engineers. Even if some dinosaurs may do things cheaper and better.

From the customer perspective the incentives are pretty often the same. Many customers want a touch screen instead of a knob or button because the latest tech all has touch screens. They want to use Slack or VS Code because that's what the cool kids do these days.

From the outer space perspective it looks like humans are building a new tower of babble with their tech stacks. Ever higher layers of complexity not actually going any place or doing much new.

Eventually it might all crumble into a pile of confusion.

Communitivity• 5 years ago

There is truth in this article, but I do not feel as pessimistic as the author seems to feel.

I think we need to push for more quality in Open Source software, and in commercial software. That quality can come from a team, though it seems more likely to be driven by a charismatic BDFL. The quality though comes from tests, attention to detail, QA, etc.

herio• 5 years ago

I do agree with PHK on his feelings about autotools. It is a cargo cult dumpsterfire, unfortunately it mostly works and nobody wants to go in and fix it. Also suffers from changes breaking stuff for somebody somewhere who insists that modern software should run on his ancient OS running on even older hardware.

For me personally I have a bit of a dislike for GNU software in general because of the same type of things. It's all autotools and legacy code being dragged around. Don't get me wrong, I fully appreciate the GNU project and how much they have done; I just don't like their code very much.

On a larger scale the Bazaar suffers from not having managers paying people to do boring things. This leads to boring things generally not being done as much, people on their own wants to work on interesting things.

kryptiskt• 5 years ago

Yes, there's a lot to be said for blowing away the cruft every now and then. At least autotools has been in a steady retreat since this was written and I don't see it too often now. It's perhaps noteworthy that this article coincided with autotools development winding down, after 2012 it didn't have a release until 2020 (https://lwn.net/Articles/834682/).

taeric• 5 years ago

Nothing particularly against managers, but that idea hits me a bad way. Managers have a tendency to want more and more in their management pool. Either people or products, if they can't get clients. So... I don't see that being a good path.

Granted, developers often do this, too. If you can find a way to align incentives with users, I'm all for it.

ho_schi• 5 years ago

Take a look at Meson[1]. A lot of GNU, Gtk, GNOME and Freedesktops related projects have migrated or are migrating. The C, C++ and also Rust people use it.

I've started reading the Autotools manual and turned away, to much bloat and hard to read code. Meson popped up and I immediately went that direction.

[1] https://mesonbuild.com/

the_gipsy• 5 years ago

the Bazaar suffers from not having managers paying people to do boring things

That does definitely not work, see enterprise software

dec0dedab0de• 5 years ago

Funny thing is that the cathedral was meant to represent GNU, vs the Bazaar that was Linux.

jandrese2• 5 years ago

My favorite thing to see when I download a project is a build system that consists of a single 6 line Makefile. Granted this usually means the project only has a handful of files to manage and it does run out of headroom as the project becomes more complex, but those build systems always work on the first try and finish quickly.

Sure it means I would run into complications on a 30 year old AIX system with some old defunct C compiler, but that's a tradeoff I'm willing to make to simplify the build process.

lmm2• 5 years ago

As someone who used FreeBSD and Interix/SFU/SUA for a while, those were always the projects I hated the most. Just get over yourself and use autotools. Sure, your build will be a little slower, but guess what: it works.

jnwatson• 5 years ago

That works fine if you’re only building for Linux. If, however, you want to run on BSD, you need something like autotools.

phendrenad2• 5 years ago

I recently came across a project with no external build system at all. Just a perl script which mimicked enough of the functionality of a build tool to get the job done. Basically "which files need rebuilding, just rebuild those". It's actually rather trivial when you look into it. Plus Makefile syntax is so bad it actually makes perl look like a hot new language.

joshuaellinger• 5 years ago

Hard to take the analysis seriously when it (a) take UNIX as an example and (b) pretends Linux doesn't exist.

The Linux kernel looks to be a successful Bazaar project with a single person (Linus) having nominal responsibly.

I recently return to UNIX after a 20 year break. Linux in 2020 is much better than Solaris in 1998. However, Windows made giant leaps as well. Both still have massive amounts of deadwood but Windows does a better job of hiding it.

h2odragon• 5 years ago

Linux in 2020 is much better than Solaris in 1998

Depending on your purposes, Linux in 1998, was better than Solaris in 1998, even on Sun hardware.

shadowgovt2• 5 years ago

Windows has the vested interest (customers and money) to make their deadwood compatible.

There are some great stories about the engineering they've done to continue to support legacy applications (because those applications matter to customers).

beprogrammed2• 5 years ago

Well he's a BSD guy, so he refers to the entire bunch as UNIX.

"One of Brooks's many excellent points is that quality happens only if somebody has the responsibility for it"

Linux is a prime example of this with Linus. I fear the day he's not around as a guiding force.

tored• 5 years ago

I don’t think the author says that a Bazaar project can’t be successful.

jrm4• 5 years ago

Overly dramatic headline is overly dramatic, and sub-headline is so obvious that it should be self-evident.

I can't help but think all articles like this are fighting a very old and outdated strawman. I'd venture that literally no one is arguing that "Free Software is magic" anymore.

Look, sometimes Free Software is the thing that works well, sometimes non-free but open, sometimes proprietary is what you have to use. There are a lot of things we can work on and do better, and concrete solutions and experiments are important to consider: but this sort of big picture whining isn't helping anyone.

ChrisMarshallNY5• 5 years ago

I love that article, but it made me sad. It is just as relevant today, as it was, nine years ago.

I'm a huge proponent of improving software Quality. I have found that complaining about it doesn't actually do much good. My complaints tend to get downvoted into the Mantle, and I just get pissed off.

So, my approach, is to do the very best I can, in my own work, and limit the amount of other people's work on which I rely. When I do use it, I am quite careful, and vet the work. Even so, I have had to back out several highly-rated packages that I had thought were good, but my apps started crashing, as soon as I incorporated them.

Just mentioning that, has often proved incredibly unpopular. It seems to be considered blasphemy.

It does reduce the scope of what I can accomplish, but I am quite happy with what I do.

yashap3• 5 years ago

Re:

I have found that complaining about it doesn't actually do much good. My complaints tend to get downvoted into the Mantle, and I just get pissed off.

IMO creating real change in an organization is almost never done through complaining, it’s done through pushing practical solutions in a positive/constructive way. It might not be the content of your argument, but the way you present it. This comment, for example, instantly biased me against your argument, because the tone felt complain-y and self-righteous. I think regardless of what your argument is, if you present it with this tone, you won’t be persuasive.

ChrisMarshallNY5• 5 years ago

Oh, I wasn't talking about in my organization (which was actually famous for Quality). I was talking about making posts in tech communities, like this one.

Software Quality is not a popular topic for discussion. HN isn't too bad, but some other communities react to Quality discussions, as if I was insulting God.

I've learned to just keep my opinions to myself. Even then, when I talk about my own experiences, and my own process, it is not received well.

linuxhansl• 5 years ago

Yeah, yeah, autoconf sucks, so does maven (IMHO), software is complex, and, yes, we could do better... But, to reason from these examples about thousands of projects is disingenuous.

The rumors of open source's demise have been greatly exaggerated.

I use open source both privately (Linux, some open source games, FreeBSD a long time ago), and professionally (mostly Java and database stuff), and while sometimes I can't help rolling my eyes, on the whole open source works incredibly and surprisingly well; especially compared to the alternative of closed-only corporate software.

mgrennan• 5 years ago

If I could up vote this 100 times I would. I love open source. But 40 years in computing (hardware to apps) has left me with questions. Like, Why could I write an entire Point of Sale system in 64K in 1987 (Apple - The Store Manager) and now you couldn't do it in 64G.

vessenes• 5 years ago

Commenters here I think are missing what’s most interesting about this essay - it’s a sort of paean to an age we’ve now forgotten, an age that had ended in 2012, written by a high priest (published in the ACM!)

Poul Henning-Kamp has been around long enough to see the end of his system and the rise of what Raymond talked about; high priests like Henning-Kamp just don’t matter as much as they used to, compared to the insane productivity of the entire world pumping out stuff whenever and however they want, gluing together whatever they need.

There is a lot of shit being made, sold and given away in the Bazaar, but there are a lot of gems. A tech startup cost millions of dollars to start in say, 2008, the height of the dotcom one boom. For reference, discount hosting providers charged on the order of $30,000 a month for a single Sun server to serve your website.

Today, a solid startup can glue together hundreds or thousands of open source libraries and launch product for free. Even if you ignore free time from the AWSes of the world, many a startup can structure their infra around serverless functions and be paying in the 10s of dollars a month. That’s the power of the bazaar, or whatever system we have now, which is more complex than Raymond described.

I think the vast majority of users of this site do not want to return to a world where releasing products takes someone of Henning-Kamp’s intelligence and skill - there just aren’t that many of him in the world, and we would have far fewer awesome things in that world.

bsedlm• 5 years ago

as I see it, it comes down to ownership.

Something about treating something which ain't yours (such as a software system with millions of lines of code) as if it were.

Something about maintaining a complex software system which you are responsible for, but somehow are never allowed to make simpler for your own sake. You are only paid for work which increase the "value" (money, profit, whatever) of the real owners of the business. It's not really worth their money to pay you to make your own job easier.

Quality happens only when someone is responsible for it.

The easy way for this to happen is to be able to own, really own what you do. This just doesn't happen in the 'software industry'.

shadowgovt2• 5 years ago

Fully agree. This is one of the implicit advantages of most corporate structures; the ownership question has default answers.

Who owns the product? The company.

Who maintains the product? Whoever the company pays to maintain it. Usually, companies will build out trees of responsibility so some sub-group is responsible for smaller pieces of large products (with ultimate responsibility "rolling up" back to company ownership).

How can I contribute? Your boss will tell you what needs maintenance or improvement. Do it or you're fired. If you have something else you want to work on, a good company will respond to your feedback and shuffle responsibilities around (you get better work when employees are satisfied), but ultimately, someone needs to maintain the build tools and if that's what they hired you to do, that's what you're doing until you decide to stop taking their money for it.

There are lots of disadvantages to the corporate model, but (like the old chestnut from Thomas Paine's writings), it has the advantage of simple ownership.

diegocg• 5 years ago

The bazaar meme advocated by Raymond, "Just hack it," as opposed to the carefully designed cathedrals of the pre-dot-com

The pre dot com days were not a golden age of software quality.

He also has this idea that Unix is a "beautiful cathedral". No, the descriptions of the original environment where Unix was designed resembles to a internal AT&T version of a bazaar. Multics was the well architected, beautiful cathedral.

kazinator• 5 years ago

The bazaar meme advocated by Raymond, "Just hack it," as opposed to the carefully designed cathedrals of the pre-dot-com years, unfortunately did, not die with the dot-com madness, and today Unix is rapidly sinking under its weight.

A level-headed comparison of actual historic Unix sources, which are publicly available for study, reveals this to be false. Unix was always "just hack it", and later Unix is, overall, an improvement over earlier Unix.

It seems I have not come across test cases in old Unix code bases.

Can someone point to some circa 1980 Unix code base where there test cases? For anything: system calls, shell, yacc, cc, ..

I think the idea may have been that if Unix rebuilds from scratch and you get to a system prompt where you can type "ls", the toolchain has been validated.

floatingatoll• 5 years ago

Is the book mentioned in this article worth reading?

bitwize• 5 years ago

Unix was invented long before modern dynamic linking schemes were. PHK seems to think it was possible, around 1990 or before, to wave a magic wand and have all vendors agree on a way to build and link dynamic libraries. There are a number of decisions to be made before you even get there. First, which binary format -- COFF? ELF? Something else? Secondly, how do you account for the fact that the library will be loaded into different programs' memory spaces at run time? Do you:

1) insist on PIC (what Linux does)?

2) replace all pointers with dummies and fix them up at load time (what Windows does)?

3) always load a given library to the same place in virtual memory (what Linux did in the a.out days)?

Before you have a single compiler flag for dynamic library generation, you have to answer questions like this. And no single answer existed in the Unix environment of 1990, which wasn't a bazaar -- it was more like a struggle among competing cathedrals! Hence libtool, whose messiness reflects the messiness of the Unix scene at the time. GNU actually made monumental efforts to clean up the different Unixen and make them more consistent.

Fortunately, today, all widely used Unixen these days seem to have settled on ELF format and option one -- all, that is, except macOS!

clankyclanker• 5 years ago

I occasionally contemplate this article (and am glad I found it again!), mostly because of this quote:

there is no standardized way to build a shared library in Unix. Instead of standardizing how to do that across all Unixen—something that would take just a single flag to the ld(1) command

I've never looked into it, but I'd love to know what ld flag PHK was thinking about. Could the entirety of libtool still be replaced by that flag?

whatever_dude• 5 years ago

This is a great article that I find myself re-reading and referencing all the time.

I think it's unfortunate that people only skim over it and then use it to complain/defend Linux/windows/m4/autoconf/etc. More important is the main point that "quality only happens when someone is responsible for it". This explains a mistake I see time and time again in all manners of software projects: anything by committee, anything overly democratic, can only aim to be mediocre at best. You need an owner/architect/champion/tyrant if you hope to achieve anything above average (not guaranteed though).

A Generation Lost in the Bazaar (2012)

Related Stories