Long term support considered harmful

http://www.tedunangst.com/flak/post/long-term-support-considered-harmful

by luu • 11 years ago

207 140 11 years ago

fidotron• 11 years ago

Upvoted for some interesting points, but it's quite wrong for the simple reason that on a commercial timescale one year is nothing. There are reasons Microsoft end up being dragged into supporting old OS versions, and it's because running an operating system is just a means to an end, which hopefully is doing something useful. "Upgrades" have, understandably, become synonymous with unnecessary pain and breaking things.

Maybe OpenBSD really is more rigorous about quality control, but (as an example) if you were to just accept every Ubuntu update it wants to install you'd be wasting a significant amount of time just ensuring your system works properly.

deathanatos• 11 years ago

on a commercial timescale one year is nothing.

I've worked in commercial environments where common invocations of `tar` did not work, because `tar` was a decade out of date. I had to learn things and habits that had died before I even started programming. It wasn't pleasant. Do not underestimate the age and stubbornness to upgrade of some environments.

Recently, I've helped migrate software from Ubuntu Precise to Trusty, and the amount of differences make things mildly frustrating. We don't get to just run one or the other, and we can't just drop everything and move to Trusty. We have to continue to support the old while we build support for the new, briefly support both, transition to the new, then tear out the support for the old. Migrated. It's a lot of work when the changes are huge, but much more manageable when I can take the changes more piecemeal (it's one if statement, as opposed to many, that I need to manage at any given time). That's in a production environment.

I run Gentoo at home. I much prefer its rolling releases to Ubuntu and Debian which I ran alongside and before, respectively. Things break every now and then. It's a tad annoying, but it gets fixed sooner or later.

zokier2• 11 years ago

Isn't 6 month release cycle and support for past 2 releases essentially also the Fedora release model? From what I've heard (and used) that hasn't been massively successful.

No one upgrade is likely to end in disaster because there simply isn’t enough change for that to happen.

That might be true for OpenBSD, but in Linux land the rate of change certainly is great enough to cause major breakage even at 6 month cycle.

craigds• 11 years ago

"Considered Harmful" Essays Considered Harmful: http://meyerweb.com/eric/comment/chech.html

I think the OP is being a bit naïve if they expect all users to upgrade to a new OS every year. Upgrades of Ubuntu and OSX are usually quite painful endeavours, fraught with UI-breakage and new bugs to solve, and there's usually little incentive for users to upgrade in every 6-monthly release.

IMHO Ubuntu has a good two-pronged approach - short-term support for most releases, but a LTS every couple of years for those who don't want to handle the pain of upgrading all the time.

Anecdotally, even upgrading to stay up-to-date with the LTS versions can be difficult. A company I deal with is currently scrambling to ditch Ubuntu 10.04 before it loses support in April this year. That's 5 years old now. Companies don't upgrade for fun.

chollida1• 11 years ago

A one year support window isn’t too short; it’s too long.

Wow, I'm in very strong disagreement with the author on this point.

I don't think I could find a single person, technical or otherwise who didn't use the same operating system for more than a year. In fact its the default for almost everyone who isn't technically inclined to upgrade their os only when they get a new computer.

What about every single car produced today. I doubt you'd want to upgrade them every 6 months.

As a ubuntu user, I find that I cross my fingers every time I run sudo apt-get upgrade. About half the time I get broken builds and once a year my OS will just flat out crash from it, or fail to reboot in virtual box.

This view, while idealistic is just so laughable I can't begin to believe the author is serious.

I've worked with some mission critical systems( stock exchange's) They take more than a year planning a big OS upgrade. To turn around and tell them that they should be doing this every 6 months is just so out of touch with reality, I don't know how to respond to the author:(

Many companies won't install a new piece of software until its been proved out in production for 6 months to a year by someone else....

lomnakkus3• 11 years ago

From a different perspective...

... perhaps if we were to actually get serious about doing small/incremental updates all the time, we'd get better at actually doing them without incident.

I think one of the major obstactles is that the software stack has become ridiculously (and hideously) complex over time. If we could converge towards a world more like what's described here[1], then I think we'd be in a better place overall. (Granted, this the video is talking about deploying services, but AFAICT there's little essential difference between that and a modern GUI system with all its DBUS interfaces and whatnot.)

[1] https://www.youtube.com/watch?v=GaHzdqFithc

rodgerd2• 11 years ago

... perhaps if were to actually get serious about doing small/incremental updates all the time, we'd get better at actually doing them without incident.

That would require more developers to actually give a fuck about the user experience not breaking from release to release.

vinceguidry• 11 years ago

Small, incremental upgrades are something I'm finding are essential to maintaining sanity as the only dev at my company doing what I do. Waiting to upgrade only increases the pain. Now instead of having one potential issue to troubleshoot every now and again, I have dozens when I finally do buck up and upgrade.

So I get in the habit of upgrading the dependencies of all the apps I work on, every time I work on them. Issues happen, but only to one dependency at a time. It's manageable.

What I would love is to eventually have a CI server do it for me. Every single day, it would run bundle update, run the tests, and deploy unless there's a problem. If there is a problem, it drops me an email with the trace and fixing it becomes part of my morning routine.

If subtler problems surface this way, then I've discovered an oversight in my test coverage, or an overly complicated architecture that I need to remove dependencies from.

I'll probably implement this sometime this year. I'm thinking I'll want to redo deployment instead of relying on Capistrano, then finally growing my own CI solution. I'm slowly moving away from big monolithic apps to smaller, homegrown solutions that do only what I want them to do. I've already reimplemented provisioning and configuration management. I believe in DevOps as code.

Sanddancer4• 11 years ago

The problem is that OpenBSD has had some pretty significant user-facing changes in only the past year. The past few releases have seen apache replaced with nginx, and the next version is going to be getting rid of nginx for their own httpd. If you're just using the basic webserver, that's some pretty significant change right there. While they're advertising clean and simple, they can and do introduce potentials for heavy breakage in nearly every release.

nathan_long2• 11 years ago

They take more than a year planning a big OS upgrade.

That sounds painful.

I think the author is really advocating for a faster feedback cycle. Imagine if instead of a big upgrade every X months, you had a tiny upgrade every day, which didn't require rebooting and happened in the background.

Breaking changes could surface the same day they were shipped and be rolled back or fixed.

Now imagine that conservative players also constantly upgrade, but always stay 6 months behind the bleeding edge, getting only releases marked as "tried and true". Eg, if something had to be rolled back or patched, they skip over the bad version.

Wouldn't that make upgrades much less painful for companies like you describe?

tormeh• 11 years ago

You do realize he's talking about a stock exchange? Even a millisecond downtime is unacceptable. Some things don't even need backup, others need testing on identical hardware in identical configuration before deployment. That's just the way it is.

jkot2• 11 years ago

I know group who does REBOOT once year, just not to forget how it is done.

pekk• 11 years ago

You find the author's view laughable, but I find your view laughable. Reasonable deployments of Ubuntu or Windows get upgraded with security fixes all the time, browsers update themselves all the time. Waiting a year between security fixes is just foolish.

Silhouette4• 11 years ago

The problem isn't the security fixes. Everyone agrees that pushing security fixes promptly is generally a good idea. The problem is all the other changes that you may or may not want that get bundled in with those security fixes in a forced-upgrade model. Certain web browsers are among the best examples I know of this user-hostile behaviour.

phunge• 11 years ago

I've held jobs at both ends of the spectrum, from bleeding-edge to death-by-legacy. There's a good deal of pain with both approaches, that's the reality. Would your org rather deal with security vulnerabilities and no upstream support, or deal with things blowing up once in a while?

What I like about this article is it's talking about what the global optimum is -- what's the best approach for the entire ecosystem? Not just for one individual or one org. What's Pareto optimal? (Or maybe it's a little less general -- what's the best approach for those who depend on FOSS, and don't pay someone like RedHat for long-term support and CYA insurance).

And basically I agree. The world's upgrade cycles could stand to be a whole lot shorter than they are. Every time I think about all the hardware in this world that'll spend its whole lifetime vulnerable to heartbleed, or shellshock, or a gazillion smaller vulns, my skin crawls.

stonemetal2• 11 years ago

In fact its the default for almost everyone who isn't technically inclined to upgrade their os only when they get a new computer.

Most new computers come with a warranty that is 90 days. Why should software support be 5x longer?

Silhouette4• 11 years ago

Most new computers come with a warranty that is 90 days.

Where? In the UK, consumer protection laws would probably force a minimum of two years in most cases. Even after that, if an expensive device mysteriously failed on day 2x365+1 in a way that wasn't reasonably expected, the purchaser would still have some relevant rights.

Why should software support be 5x longer?

It is all about reasonable expectations. No-one paying for basic system software would reasonably expect it to stop working after three months.

Of course if we're talking about Open Source software that you're getting for free then it's your own responsibility to make sure it does the job you need and no-one owes you anything. This is part of the reason that selling support for otherwise free software is a viable, and sometimes highly lucrative, business model.

viraptor• 11 years ago

Depends on the country. In most places in Europe that would be 2y minimum.

PhasmaFelis• 11 years ago

Because hardware warranties and software support have completely different purposes? Seriously, this is a non sequitur.

nailer• 11 years ago

I appreciate what you're saying, but your seem to not be addressing the point of the post: you could reduce risk by moving to rolling upgrades. A year of changes is a big change, smaller updates are not. There's also a huge risk in stasis.

Not that it's relevant, but to respond to your first point: you know people who roll their browser, and a few Linux distros roll their release too. I've heard of some startups who use a Fedora rather than CentOS for that reason.

jms703• 11 years ago

I don't think I could find a single person, technical or otherwise who didn't use the same operating system for more than a year.

You can. Think AWS systems (as an example) where you don't log in and patch, you spin up a new instance instead. This can't apply to everything (stock exchanges, MRIs, etc), but for the software affected by glibc, absolutely.

munchhausen• 11 years ago

As a ubuntu user, I find that I cross my fingers every time I run sudo apt-get upgrade. About half the time I get broken builds and once a year my OS will just flat out crash from it, or fail to reboot in virtual box.

What I'm about to say has been said a thousand times, but I guess we need to keep saying it as long as this myth survives. Are you using PPAs? Are you using PPAs that provide newer versions of software that also ships with the system? Are you using third-party drivers (looking at you, Nvidia)? If so, there's your problem. It's not that difficult to blast away your PPAs with "ppa-purge" prior to running a distribution upgrade. As for regular package updates - "apt-get upgrade" won't break a system, that's not been already severely broken by the user. Period.

I use Ubuntu on the desktop and the server, and have done many upgrades over many versions. I don't recall a post-upgrade broken system in recent years that wasn't due to third-party stuff. Even then, it's trivial to fix - reboot to recovery mode if you can't get your X11 on, "apt-get purge" the offending stuff, reboot and carry on.

My impression is that Ubuntu screwed up on a major version upgrade several years ago, and their upgrade process is still being dinged by folks due to the wave of bad publicity that that incident created. Which is a shame, because Ubuntu post 12.04 is a rock-solid operating system that is a joy to use (upgrading included). Granted, third-party packages can sometimes reduce the overall stability, but then what do you expect? It helps to carefully vet which PPAs you use - PPAs that are done by people who know what they're doing don't tend to cause trouble.

If you want a rock-solid system, do a clean install of an LTS release, don't set up PPAs, and enjoy. I have a side Thinkpad with 12.04 on it which is exactly that, and it's the most stable system I've ever used. And it's being kept current with security patches and bugfixes with regular "apt-get upgrade".

PeterisP• 11 years ago

What about every single car produced today. I doubt you'd want to upgrade them every 6 months.

Tesla cars get automated over-the-air updates far more frequently than every 6 months.

zdw• 11 years ago

The downside of this is that OpenBSD is pretty difficult to upgrade in a clean manner given it's "uncompress the tarball over /, then clean stuff up manually" method.

I really like OpenBSD, but it desperately needs a modern package management system. This is frankly my largest sticking point with it - it's faster and much cleaner in most cases to wipe, reinstall, then apply configuration than to try to upgrade.

jkot2• 11 years ago

How about debian with freebsd kernel?

elektronjunge• 11 years ago

The new pkg tool in freebsd is pretty good and meshes nicely with the ports tree. The marginal improvements for 3rd-party built packages with apt aren't really worth getting rid of the rest of freebsd for. It also doesn't help OpenBSD, which really needs a new package manager.

danielweber2• 11 years ago

As someone who used to help maintain hundreds of OpenBSD boxes, yes.

But don't forget, OpenBSD is designed for the developers.

brynet2• 11 years ago

This is oft-repeated, but how is OpenBSD's package management system "not modern"? It handles upgrades properly, modified configuration in /etc is preserved. Packages are cryptographically signed as of 5.5, files are individually checksummed..

I'd say OpenBSD still handles dependencies and upgrades better than FreeBSD's supposedly "next-generation" pkg.

OpenBSD's package system and ports tree are tightly integrated and work together, and despite the historic naming, OpenBSD's pkg_{add,delete,info,create}(1) are unrelated to the now deprecated FreeBSD equivalents.

brynet2• 11 years ago

The recommended and supported way of upgrading the base system is to boot a bsd.rd kernel and type 'U'. If you don't have a console, the manual upgrade guide on the website is an added convenience.

ams61102• 11 years ago

Package management in OpenBSD is as good or better than any other distro I've used.

Upgrades are: boot bsd.rd, upgrade. Then run sysmerge, and pkg_add -u

IshKebab• 11 years ago

Can we have a ban on "X considered harmful" titles? Have some originality.

jerf• 11 years ago

In theory, I agree. I've often likened code development to muscles; you get good at what you do, and what you don't do atrophies. If you push changes ever hour, you get very good at making sure that works. If you push them every three months, you get good at pushing them every three months, and panic when something requires you to move within days. And so on. This applies to all sorts of development process aspects (test a lot, get good at testing, fail to test, becomes impossible to test later, etc).

In practice, neither what tedunangst nor I think matters, because once you ship something to a customer in any way, they're not going to upgrade it. Offer automated upgrades and they'll demand a way to turn them off. It's not just developer boxes we have to worry about, unfortunately.

gtirloni• 11 years ago

Oh please, all it would take for this "Ghost" vulnerability not to happen was someone at glibc to have made the right call regarding how to treat a buffer overflow in a function dealing with external data. They didn't, so what? The lesson here is that Red Hat & other companies should do additional review if they are shipping something branded as super secure and stable.

Long term support is great. People screw up sometimes. Additionally, it's about time core software components get more attention from companies with the big bucks (and that are profiting from it).

vezzy-fnord• 11 years ago

Personally I'm of the idealistic belief that glibc is really starting to show its corrosion, and should be replaced by a clean-room, standards-compliant and elegant design like musl: http://wiki.musl-libc.org/wiki/Functional_differences_from_g...

Of course, this would be a massive change for the Linux ecosystem at large. It is my hope that projects like Sabotage Linux will eventually procure most major software to become musl-compatible, though.

The source of the GHOST bug was in NSS, which musl lacks by design.

GregBuchholz2• 11 years ago

Speaking of corrosion, maybe we should be building systems with a language which avoids most of the problems of C.

zokier2• 11 years ago

Personally I think a 6 month release cycle is the worst of both worlds. Either give me LTS and I'll do that one big upgrade every few years, or give me rolling release where any breakage should be relatively localized and minor. But with 6 month release cycle you have to constantly be upgrading, but the upgrades are still big "break the world" upgrades.

Sanddancer4• 11 years ago

There are a lot of problems with this argument, especially when you go into the ports section. For example, the version of go between 5.4, 5.5, and 5.6 is different for all three versions, which means you're going to have to rebuild packages, run regression tests, etc every six months to make sure some new feature doesn't run into your code, or some bug isn't introduced. Similarly, in the case of the base install, you are still going to have to run a regression test there to ensure that any deprecated or removed features don't impact your workflow.

These things are expensive from a development and ops perspective, and why most commercial software vendors tend to stick to a releasing for a few platforms they know are going to be supported for longer than a year or two. Yes, OpenBSD upgrades are mostly painless, but still, bugs can and do happen, and expecting everyone to be able to just drop everything and rebuild is plain and simple unreasonable.

learnstats2• 11 years ago

Now on the one hand, this forces users to upgrade at least once per year.

The problem with this article is this assumption which is not true in the slightest.

There's no force which makes users upgrade - some users will upgrade when they're told; some will wait 5 or more years - and may have good reason to.

Nothing kills a bug report faster than “My network card worked in 4.4 but stopped working in 5.6.” Developers aren’t going to bisect five years of changes; you get to do that yourself.

This statement can translate into don't ever upgrade. Is that really the goal?

sarciszewski4• 11 years ago

No. Ideally, the goal is that users will upgrade as soon as a fix is available.

taeric4• 11 years ago

I would think ideally, programmers wouldn't break things that are working. We know that ideal doesn't pan out.

LukeB_UK• 11 years ago

The author makes it sound like the developers won't care about the bug because the user waited from 4.4 until 5.6 to upgrade.

In this case the user is better off not upgrading if the developers aren't doing to do anything about it.

sliverstorm2• 11 years ago

Is that really the goal?

The goal appears to be something more along the lines of not capitulating to the worst-common-denominator. Designing your product and spending your efforts to carefully accommodate people who upgrade every few decades (give or take) results in an objectively worse product for everybody else.

Unless, of course, that customer is willing to front the money to make it worthwhile.

taeric4• 11 years ago

Does it really result in an objectively worse product, though? Subjectively, I can't but agree. But objectively?

Especially in an industry where we are constantly amazed by what people accomplished years ago. I question whether long term planning and acting is truly objectively worse.

marcosdumay2• 11 years ago

There's no force which makes users upgrade

Except for auto-upgrade run from cron every few days.

username223• 11 years ago

Except for auto-upgrade run from cron every few days.

This sort of user-hostile behavior is guaranteed to make me take the time to open a shell and nuke your software from orbit with blasts of "rm -rf". Don't do this.

allendoerfer2• 11 years ago

His reasoning makes sense but, as all developers tend to do, he spots an error in a system and wants to switch to another or invent a new one. You know the xkcd.

Often times the solution is just to work together and fix the old one. For me the logical fix would be to patch all kinds of malicious, undefined or non-spec behaviors in LTS releases in short cycles regardless whether the developer thinks, it is security-critical or not. To make this more feasible you could either pay for it or use a minimal base system separated from the user-space. Both exists today.

JoachimSchipper2• 11 years ago

That's both a lot of work and a lot of churn, though - people choose LTS because they don't want things to change!

allendoerfer2• 11 years ago

In a sense, things do not change, they stop to be changed from a assumed state.

jacquesm• 11 years ago

There is a cost to upgrading and there is a cost to staying behind. As soon as the cost to stay behind outweighs the cost to upgrade I upgrade, but no sooner. Otherwise I'd just be throwing away money and time. The bigger problem here is that the cost to upgrade is only known after the fact. What should be a routine job can easily spiral out of control into a marathon of misery.

thaumaturgy5• 11 years ago

Frequent upgrades amortize the cost and ensure that regressions are caught early. No one upgrade is likely to end in disaster because there simply isn’t enough change for that to happen.

Oh, how I wish this were true.

For what it's worth, it's pretty true as far as OpenBSD is concerned, in my experience. But OpenBSD is the exception here, not the rule. Everywhere else, developers all seem to have embraced "break early, break often".

Eventually you get burned. For me, it was a routine should-have-been-minor web server update where one of the packages I relied on suddenly became unsupported and every single hosted site stopped working. Since there's no way to roll back server upgrades, I had a marathon night involving building a new server stack and migrating all hosted sites there by 8 a.m.

But you can't yell at anybody when that happens, because the answer's always the same: it's not the developers' fault.

Who really believes sysadmins wouldn't update everything all the time if they could? Old, dodgy, out-of-date servers exist exactly because updates are butthole-puckering, because everyone's been burned at least once by a "minor" update, and because once the damage is done, undoing it is horrifyingly difficult.

edofic• 11 years ago

"Since there's no way to roll back server upgrades"

It is if you run a modern filesystem like ZFS or btrfs. You just do a cheap snapshot before upgrading(can be automated) and roll back if there are problems. Even works with lvm.

thaumaturgy5• 11 years ago

Ooh, thanks. That's a good idea. I'd never considered running ZFS on my small servers.

KaiserPro2• 11 years ago

Sadly as zfs isn't as widely available as I'd like, you can use LVM to provide snapshots.

Its not as friendly as ZFS snapshots, but it is at least available in centos 5 https://www.centos.org/docs/5/html/Cluster_Logical_Volume_Ma...

(sadly somethings need long term support)

vel0city• 11 years ago

Or, if you're using VMs, just take a snapshot and roll back if something doesn't work right. For anything that isn't virtualized, there's ZFS.

vacri3• 11 years ago

btrfs still isn't production ready. I went to a talk from a btrfs dev two weeks ago, and it was still "use btrfs with caveats", not "just use btrfs".

davexunit• 11 years ago

Or, you could try out NixOS/GuixSD, which support transactional upgrades and rollbacks for the full system. No need to take a disk image (outside of your normal backup routine, of course).

marcosdumay2• 11 years ago

and because once the damage is done, undoing it is horrifyingly difficult

That's the one reason why I'm looking at NixOS. There's no inherent reason for it to be hard.

ryantrinkle• 11 years ago

Go for it! I've been using NixOS personally since February 2014 and in production since September, and it's been fantastic.

mathattack• 11 years ago

I worked a firm that supported two main branches that forked >5 years apart. It was a nightmare fixing both simultaneously, especially after the fork was 5+ years old. In the end they wound up breaking something with every new functionality.

hermanradtke• 11 years ago

> Frequent upgrades amortize the cost and ensure that regressions are caught early. No one upgrade is likely to end in disaster because there simply isn’t enough change for that to happen.
Oh, how I wish this were true.

Your anecdote is even more evidence that it _is_ true. Most minor upgrades do not end in disaster. Few of them do end up in disaster.

Frequent updates still require some diligence on the part of the person or organization updating.

AlisdairO• 11 years ago

The difference with irregular upgrades is that it's clear when the breakage is likely to occur and you can plan around that.

fluidcruft• 11 years ago

Many times you have to hold back software upgrades on things like MRI scanners to wait for multi-year research studies to complete, and often new studies start up in the interim so that locks you down for even more time. Scanner upgrades change all sorts of things in ways that introduce all sorts of confounds.

Not to mention that in the real world scanner upgrades often break surprisingly fragile clinical workflows. Technically, the engineering and processing of the scanners are improved quite a bit in one aspect or another by the upgrades, but old workarounds need to be replaced by new workarounds etc and documentation is very sparse and quite uninformative.

Figs• 11 years ago

Hmm... what does an MRI scanner update do? I would have thought a system like that would just record whatever it gets from its sensors and any updates needed would only apply to the analysis and visualization software... Do the updates actually modify what the hardware does during scanning? Or, is it all painfully coupled because some sort of interactivity is required during scanning?

yellowapple• 11 years ago

Even with OpenBSD that isn't always true. The time_t fixed killed binary-compatibility between 5.4 and 5.5 on 32-bit systems, which meant that any installed packages had to be uninstalled and reinstalled if one were to perform an upgrade. The OpenBSD project isn't afraid to break compatibility if it means fixing a bug - something that I think is a very good thing, but has implications in terms of support.

On another note, upgrade hell is a pretty convincing argument for systems like NixOS, where upgrades can be easily rolled back, configurations are declarative, etc.

rlpb3• 11 years ago

I had a marathon night involving building a new server stack and migrating all hosted sites there by 8 a.m.

This is because of a shortcut that somebody took when building your server originally, by failing to make the deployment reproducible.

...where one of the packages I relied on suddenly became unsupported

I've never heard of this happening in any Linux distribution. Can you be more specific? Did you choose to use some third party source for a package here? If so, how can you expect your distribution's developers to support you going off-piste in a way that they never claimed to support in the first place?

Old, dodgy, out-of-date servers exist exactly because updates are butthole-puckering, because everyone's been burned at least once by a "minor" update, and because once the damage is done, undoing it is horrifyingly difficult.

You might want to look into "DevOps". The idea is that you script your deployments together with automated tests for them. There are many tools to help you do this now. With this in place, nothing you've stated is true any more.

PythonicAlpha5• 11 years ago

For people with >2 servers, that might be the right solution, but for the others, scripting deployment might be just more overhead.

sorbits• 11 years ago

I've never heard of this happening in any Linux distribution. Can you be more specific?

Not the OP but I run a personal server for mail etc., and I do remember one LTS → LTS upgrade of Ubuntu that removed the DKIM milter that my postfix config depended on.

So it happens.

You might want to look into "DevOps". The idea is that you script your deployments together with automated tests for them

For a personal server the things that in my experience break are related to configuration files.

This can be subtle things, like some new option enabled by default that conflicts in some edge case, it can be upgrade scripts that butcher existing customized configuration files, or it can be a total restructure of how a package has structured its config.

All of this has happened to me, and I don’t think having scripts able to redeploy my server would have been of any help resolving these issues.

thaumaturgy5• 11 years ago

I've never heard of this happening in any Linux distribution. Can you be more specific?

It was a couple of years ago. I know somewhere I have notes on it, but I can't find them just this minute. I remember that it had something to do with one of the components in my apache-mpm-worker--php5--libapache2-mod-fcgid--apache2-suexec-custom--libapache-mod-security stack. It was something like, mpm-worker no longer supported libapache2-mod-fcgid or some such thing. At the time, I was really good about doing regular server updates, so when it happened, I spent some time researching it but eventually found the package had been removed during the update and was no longer supported, with no workaround aside from finding a new way to build an apache server.

Either it got fixed or I'm running a slightly different stack now than I was at the time. Sorry I can't be more helpful.

This is because of a shortcut that somebody took...
You might want to look into "DevOps"...

That somebody was me, and I'm aware of devops. Pretty big fan of it actually. Unfortunately, I'm just a small MSP, the owner and the senior tech and the sole software developer and the sysadmin, and I don't charge enough. The servers exist as an add-on service for my clients, especially ones that have special needs that other hosting companies can't easily meet. They work, the stacks I built a few years ago are robust and efficient (one of my customers was featured on a popular national radio show with a reputation for killing websites, but their site stayed up and responsive the whole time). Reproducible deployments, centralized management, further improvements to automated security, etc. are all on my to-do list -- along with like a couple dozen other things.

I was supposed to rebuild all of the servers during last December, typically a slow period, but instead it was unseasonably, hair-on-fire, brain-meltingly busy, and it still hasn't let up.

PythonicAlpha5• 11 years ago

I also can not recommend the "frequent upgrades" model.

When systems would be more stable and less problems would occur at updates, this would be right. But the reality is simply different. My experience is, that on many updates there are surprises:

- software that once was good, just has gone garbage since the last version

- some problems with not so common software combinations that was not found from package maintainers

- some device drivers that do not cooperate

- desktop environments that do no longer support some options as before or just have gone bad

- legacy data is not supported by newer application versions or some subtle problems with this data occur ...

- ....

Also, when upgrades would be possible without hassle and troubles, that model could work -- but it just is not. For example: I wanted to install a newer Ubuntu version on my hosted server. But the automatic upgrade process explicitly says, that it should not be done via a remote session. So for a hosted system, I have to fall back to a complete new install (backup data, fresh install, complete new configuration, re-install backed up data).

Also, when you get into trouble, it is not possible to easily go back to the last stable version (in this situation, virtualized systems are very useful).

In such a case, it is clear, that I don't want to give up my life to always have the newest stuff on my server.

I regret, that OpenBSD is not for me (unless OpenBSD does never suffer from those troubles).

vacri3• 11 years ago

But OpenBSD is the exception here, not the rule.

It's worth mentioning that OpenBSD doesn't support a lot of the hardware people use, doesn't support a lot of the applications people use, and is a fairly specialist distro. They do some awesome things, but they also benefit greatly from not having to support regular non-techie end-users.

Edit: As an example, the upgrade process that you have to do at least once a year[1] is hardly something that a tech naif would find painless

[1]http://www.openbsd.org/faq/upgrade56.html

4ad• 11 years ago

And what hardware it does not support? And what applications it does not support? I'm tired of this meme. OpenBSD runs on virtually everything you throw it at, and virtually all open source is supported.

In some cases OpenBSD hardware support is better. E.g. I have a few laptops where suspend only works in OpenBSD, but not Linux. Also a few years back my WiFi cards were supported natively only in OpenBSD, not Linux. For other hardware, it might be the other way around, but overall it's pretty unlikely to find something OpenBSD does not run on.

And for software, it's exactly the same. Sure there's some Linux-specific software out there, but the bulk of it, is not.

lsc3• 11 years ago

Eventually you get burned. For me, it was a routine should-have-been-minor web server update where one of the packages I relied on suddenly became unsupported and every single hosted site stopped working. Since there's no way to roll back server upgrades, I had a marathon night involving building a new server stack and migrating all hosted sites there by 8 a.m.

The traditional way to handle this in a cluster is with testing. The idea being that you have the upstream repo, the testing repo, and the production repo.

Note, in most cases 'repo' means "directory tree served via http" and "sync" means "Copy, you know, with rsync or something" - This is not complicated.

10% of your servers point at the testing repo, the rest at the production repo.

every X days, you set a test box against the upstream repo, update, reboot, and run your tests, you know, to catch the obvious stuff. If that works, you sync your testing repo with the upstream repo, and so 10% of your production now runs your new stuff. This is where you are gonna catch most of your problems, in my experience, but if something chokes, you only lose 10% of capacity.

after Y days of the test stuff being on 10% of your boxes, you sync the test repo to the production repo.

Of course, if you are like me, and when 10% of your boxes are down 10% of your customers are down, you want to spend a lot more effort on the 'test before you get customers on it' step.

Also, sometimes there are 'don't sleep until you roll it out everywhere' updates, like this one (oh god, I am so happy that srn is on that now and I didn't have to deal with it) or like shellshock. In that case, well, sometimes you sync the upstream repo straight to production.

xorcist• 11 years ago

Since there's no way to roll back server upgrades,

Apart from the obvious comment about snapshots (at the volume or machine level), really most of the time your problems are because of a specific package.

And while most package managers don't support downgrades per se, you can just remove the offending package and install the old one. Nine times out of ten that would give you more time to fix the problem.

That said, you need to test before you go to production, no matter how trivial the patch may seem. Staged rollouts, a separate environment, or both.

And that's also the elephant in the room in Ted's rant. Sure, if we didn't have to test anything, we could update every 6 months. But we have to, and we can't.

joncrocks• 11 years ago

Eventually you get burned. For me, it was a routine should-have-been-minor web server update where one of the packages I relied on suddenly became unsupported and every single hosted site stopped working.

I'm not wishing to be inflammatory here, but surely you could have exactly the same problem with something that is updated 'slowly'. If you're relying on the software you get being 100% correct/bug free every time you get it, you're building your house on sand.

This is where having testing (ideally a good set of automated testing) is invaluable. Having a robust set of tests you run before you roll out changes to 'production' is important if your 'production' is "we can't afford for this to go down."

Along testing, you need to have a backout strategy. What do you do if it all goes wrong? This is usually very similar to your backup/restore strategy as the problem is generally the same: If your server gets hosed/breaks/fails, what do you do?

Silhouette4• 11 years ago

This is not an argument against long term support. It is an argument against using OpenBSD for any project you need to work for more than one year.

CrLf2• 11 years ago

Forcing users to upgrade frequently has one outcome, and one outcome only: they will stop upgrading, period. If you give users only two choices, they will choose one, but the one that requires less effort.

There is more to life than keeping up with the constant upgrade treadmill. It's already bad as it is. Some Linux distributions are on a 3-year support cycle, which is too short. If you have many servers, especially many servers running a lot of different things, upgrading every 3-years means having people doing nothing more than upgrades.

Once you fix your applications to support the changes in the next OS release, that release is almost out of support...

wmf• 11 years ago

they will stop upgrading, period

Has that been observed with Chrome, ChromeOS, iOS, etc.?

CrLf2• 11 years ago

That's an apples to oranges comparison. To upgrade Chrome a consumer depends only on himself. To upgrade a server OS, the "user" (the sysadmin) depends on a number of other people with different priorities. And that's _if_ the people he depends on are still with the company...

I've been faced inumerous times with impossible upgrades (nobody cares about upgrading besides you, so nobody cares to fix whatever is preventing the upgrade). And when the upgrade is possible, many times it takes years.

That same user that upgrades Chrome on his laptop, is still stuck with IE7 at work because his company relies on it to pay his salary.

slasaus• 11 years ago

It's funny that just today, before I found out about the glibc vulnerability, I reconsidered if I really want to upgrade my Ubuntu 10.04 mail and web servers to Ubuntu 14.04 LTS. I was triggered after reading some bad things about 14.04 [1]. I've looked at the M:Tier binpatches and package upgrades for OpenBSD, looked at FreeBSD, Debian with it's experimental LTS, but eventually I'm still in favor of the 5-year support for Ubuntu server.

I have bad experiences in upgrading production machines in-place, whether it is OpenBSD, Ubuntu or Debian and always install a new machine (vps) side-by-side which is really the only stress-free guaranteed way to go in my opinion. Having to do this only once every 5 years is really a lot nicer then at least once a year. The good security backports of Ubuntu, minimal breakage (auto-security upgrades at my Ubuntu servers have been working almost flawlessly [2]) are the least maintenance, stable and secure setup I can imagine.

OpenBSD having only one year of support, no binpatches of the kernel and having no stable security fixes of the packages are the reason I only use it with anything that can be done by the base system (backup host and nameserver). OpenSMTPd looks very promising, but I would need supported amavisd packages, same goes for httpd that needs php in my setup. Besides it's limited use, I still love OpenBSD and the mindset that stewards it. If only they had longer support and binpatches for kernel and packages :)

[1] https://tim.siosm.fr/blog/2014/04/25/why-not-ubuntu-14.04-lt...

[2] the mail config was overwritten once after an auto security update of dovecot in 10.04, quickly recovered it with etckeeper (/etc in git)

tghw• 11 years ago

Thanks for the tip on etckeeper! I've been wanting something like that for a while.

coherentpony• 11 years ago

"[Useful popular thing] considered harmful."

jrochkind1• 11 years ago

The only thing that would make this realistic is if open source developers prioritize backwards compatibility very highly.

Of course, there will always be some bugs anyway. And it varies across language (or other open source affinity) communities. But in many communities, developers aren't really even _trying_ to ensure backwards compatibility.

If developers committed to backwards compatibiilty with successive versions for X number of years -- then you could update to the latest release for a number of years worrying less about breaking things, rather than needing to freeze the versions you use for that number of years.

Yes, this would significantly increase developer hours on open source projects. Nothing comes for free.

But wouldn't those developer hours be better spent centrally on ensuring backwards compatibility in the projects themselves; as opposed to every developer of every consumer of the project needing to deal with backwards incompatibility on every upgrade? Or every distro needing to backport patches? (This latter is more debatable, which is exactly why we have the status quo)

kbenson3• 11 years ago

Every time this topic comes up, we have people talking about really different things not clearly explaining what they are referring to, and causing confusion and people not understanding the reasoning of others.

I've read the following arguments already, but with portions of them implicit. See if you can determine why the people are talking at cross points:

- "I generate VM server images for all my needs, and deploy to virtualization infrastructure behind load balancers to handle services. I treat he OS like an application like the rest of the stack, and all my data is abstracted to a data layer. I just generate a new image with patches and test it, then deploy if it works in my test suite."

- "I have hundreds of servers in dozens of roles with different software needs, and I need them to be secure and stable in a timely fashion, and I can't spend multiple weeks achieving that. Long term support and back-patching allows me the time to plan needed large changes in infrastructure without having to spend all my time managing updates, software changes and configuration changes multiple steps down the dependency graph."

- "I have a few servers with a few roles, or many servers with one or two roles, and I can manage frequent updates just fine, and it allows me to take advantage of the newest features, get security updates immediately as the software authors fix it, and I don't have to worry about end of life of software."

- "I have an application stack with multiple dependencies, and I just make sure to update my stack as I make changes to the software. I would love/have set up continuous integration software to build and test everything as I go, so I know if it works or not before taking it live."

As someone who's been in all of these situations at different points, often multiple at once, let me be clear: Unless you have an argument that addresses ALL of these situations, you really haven't thought through the issue.

hawleyal• 11 years ago

I think the author is advocating that those scenarios should converge. His argument is that second scenario is an anti-pattern and should be corrected to be more inline. It might be a pipe dream, but it's still his perspective.

kbenson3• 11 years ago

They serve different needs. The second scenario may well be better served by moving towards the first, but that's not a quick project, especially when uptime and reliabiity is important. Even with that, the for the third scenario a full VM infrastructure may by overkill. This whole topic can be summed up with "Examine your needs, examine your options, make the best choice you can at the time, try to make your life easier by leaving pathways to change your choice with the least problems based on expected future needs."

Just because someone looks to be doing something similar to you, doesn't mean their needs and constraints aren't quite different. I think this is the big thing most people miss.

contingencies• 11 years ago

Reading the title I assumed this was a rant about how proper ops processes replaced the need for LTS releases and the demonstrably bad practices they could be seen to encourage. IMHO, this would have been a more interesting and valid point.

The current post is a straw man argument for a few reasons.

First, because vulnerabilities will be exploited whether the window of availability is 1 day, 100 days, or 1000 days (like this one) and likewise, upgrades will be missed, for various reasons, whether or not they are encouraged.

Second, the author spuriously implies that a 6 month sliding upgrade window ("one size fits all") approach is superior, easier and more appropriate for everyone through a combination of broad assumption and vague hand-waving about developer time efficiency. Rubbish.

If you want newer code, then ASAP is the go and BSD-style monolithic releases are bad... something closer to the versionless OS ideal of constant, incremental, package-wise release processes such as Gentoo's portage (BSD-inspired!) where you can even install a -9999 (git HEAD) version of many packages would be ideal. Obviously, with such an approach, stability caveats must be considered (as they are). Thus, both FreeBSD and OpenBSD actually represent a 'middle-ground' between Gentoo and commercial Linux vendor LTS, with OS wide release processes incorporating some greater collective package and package interoperability testing, combined with the package-wise release of things through the ports trees.

What's the real ideal here? A general purpose means to build working, tested, maintainable, secure systems with minimal effort for a broad audience.

How to get there? Clearly, not by whinging about external release window frequencies of volunteer-based projects, as any frequency still creates bugs, and no process is perfect. The answer, I believe, is time-honored. Careful design for failure and minimalism, and good process. Test. Measure. Remove surplus features and components. Iterate.

ak217• 11 years ago

Spoken like someone who has never had to maintain a large production system.

This notion is handily disproved by the market, though. Red Hat and Canonical enjoy commercial success and mindshare precisely because they provide stable long-term support platforms on which others can build software. Most of Canonical's worth is actually embodied in their commitment to LTS releases, which provide a sweet spot between stability and the glacial pace of RHEL.

jeffdavis2• 11 years ago

PostgreSQL has a 5-year support window.

I'm curious what negative consequences the author thinks PostgreSQL suffers as a result from that policy.

silvestrov• 11 years ago

It's a lot easier with a 5 year window for PostgreSQL: Kernel developers have to support all kinds of weird semi-buggy hardware that they can't buy in their country, they have to make the kernel work with badly written GPU drivers , and they have to deal with interrupt-level code which is notoriously difficult to debug.

jeffdavis2• 11 years ago

The author's primary example for the "LTS is bad claim" was glibc, which is not a kernel.

Are you saying the "LTS is bad claim" only applies to kernels and other things very close to the hardware (closer than a database)?

reality_czech• 11 years ago

Yes, we should all step forward into the wonderful world of running brand new software all the time in production. I'm sure it will all be much more secure than relying on year old software that is being actively maintained.

Is the date on this one supposed to be April 1st?

cesarb• 11 years ago

A problem with the lack of long term support is that you have to keep moving.

The most important part of "long term support" distributions is that breaking changes are kept to a minimum. If a "long term support" distribution releases an update to for instance glibc, you can expect that applying that update will change nothing other parts of your software stack might depend on.

The dependencies might be subtle; for instance, a new version of a database server might have optimized its query planner, which happened to make one particular query your software does a couple of percent slower, which led to it using a few more seconds to do its processing, enough to push it over the timeout limit for a different part of the system. So the only sane way to avoid breaking changes is to avoid all changes.

The opposite would be, as advocated in this post, frequent upgrades. "Frequent upgrades amortize the cost and ensure that regressions are caught early", but that means you are dealing with upgrades and regressions all the time. You arrive at work in the morning, planning to write a new feature; but an upgrade has just arrived, and it needs a few changes to your project. You develop, test, and deploy these changes; in the meantime, another upgrade has just arrived, needing more changes, to something you had changed just a few days ago. The day ends, and you didn't even start to develop the new feature. You use more time chasing the upgrade stream than doing productive work.

Long term releases "batch" the changes. When several changes affect one part of your software, you only have to deal with them once. Sometimes, you can even discard that part of your system and do something else, while with a continuous change stream, you might have wasted time adjusting little by little.

A somewhat relevant post from Joel on Software: http://www.joelonsoftware.com/articles/fog0000000339.html

Long term support considered harmful

Related Stories