Upstream vendors and why they can be harmful

http://marc.info/?l=openbsd-tech&m=135220552521529&w=2

by rohshall • 14 years ago

122 104 14 years ago

derefr• 14 years ago

> Either you're a modern linux with pulseaudio and pam and systemd, or you're dying. So much for the pionneer spirit of opensource, where you were free to innovate and do cool things, and more or less have interesting software able to run on your machine...

What a weird doublespeak paragraph. By "innovate and do cool things," the author seems to mean "ship, and rely on, old versions of software."

bkor8• 14 years ago

He means more that some software is not shared (or available to) with everyone. Systemd is very explicit in that only Linux is supported. I guess pulseaudio is also a problem.

A large part of open source is to share the source code so that others can take that work and do their own cool things / innovate. But then it has to be available.

gillianseed3• 14 years ago

Yes but isn't that a core problem, if you want to make the best use of something (in this case Linux) you will want take advantage of all the features it offers.

From what I gather the problem is that other systems (in this case *BSD) doesn't support certain features either because they don't mesh well with the BSD system(s) and/or has a lack of manpower to implement them.

reidrac2• 14 years ago

I think he's complaining about the "Linux-centrism" of some projects.

What I find ironic is that those Linux focused projects are turning more like BSD: a tightly coupled system.

If I want my app to work best with X and I need to interact with some stuff that it's only available on X, that's it. I may provide an alternate way of doing things, but I may not.

For example: I may use DBus to speak with NM to see if I the system has network connectivity, but what if DBus or NM are not available? Some time ago you would put effort on that possibility, now you may not because... DBus and NM are everywhere!

That's what is changing I think, although I'm not sure if Espie means exactly that.

In some cases it's the "brand" thing I can't understand, like "only for Ubuntu" that started about 2-3 years ago.

takluyver4• 14 years ago

> Some time ago you would put effort on that possibility, now you may not because... DBus and NM are everywhere!

Which sounds like a good thing. People have put in a lot of effort on things like DBus and NM, so that applications can use a consistent, reliable API. That's rather a waste if every application author still has to handle the case when they're missing. They're valuable precisely because we can build on top of them.

I'm slightly playing devil's advocate, but that's actually pretty close to my real view.

bkor8• 14 years ago

The development within Linux is going much quicker than anything else can keep up. Within GNOME only very recently we got some non-Linux using contributors. Portability just for portabilities sake is not a goal anymore.

This is somewhat ironic, as excluding OpenBSD feels way too similar to how Linux is/was excluded in favour of Windows. But if almost all the people spend their time to make their Linux distro better, then this seems like a natural thing to occur.

derleth• 14 years ago

The case for Linuxism is that Linux has the underlying tools applications require to deliver the user experience users want, and if OpenBSD (and this is from an OpenBSD list) wants to run those applications it should provide the same tools.

A better case against Linuxism is that portability encourages modularity, which is good for reasons far beyond portability. Having to do some things in a different module because they can't be done the same way on all systems encourages you to separate concerns, which leads to code that's easier to reason about and, therefore, less likely to be buggy.

The historical perspective is "You think this is bad, sonny, be glad you're not trying to convince people to play nice with Eunice." (Eunice was a Unix-like environment running on VAX/VMS.)

caf• 14 years ago

Note that some of these things are grumbled about within the Linux community too - there's plenty of complaining about pulseaudio, and systemd is very unlikely to ever be universally used by Linux distributions.

gillianseed3• 14 years ago

Hmmm... well I certainly see the problem from 'their end' in that they find themselves 'incompatible' with external projects on which they have relied on since they are nowadays increasingly being developed with 'Linux in mind'.

However it's also hard to fault the obvious urge from Linux and the surrounding Linux-oriented ecosystem of making sure components integrate better so the result is a 'full system' which takes advantage of advancements made to the Linux kernel and it's surrounding core components.

I can't say one is right and one is wrong, I guess it's all about which side of the fence you're on.

meaty10• 14 years ago

I agree entirely with the sentiment here. If you've ever had to piece together something external to your distribution you will know the pain involved. This is amplified considerably if you pick a BSD-related operating system and for no good reason.

The Linux kernel itself is a remarkably stable system as far as ABI/API is concerned (Linus you rock here), but the packages gyrating around it are completely impossibly unstable and unreliable.

A fine example of this is trying to get an SSTP client working with NetworkManager recently. The whole NetworkManager VPN thing is an epic pile of hacks with bits of UI and bits of backend sticking out all over the place. If you're using Debian stable for example, there is no chance in hell you can get a working configuration unless you go and fix the numerous broken interfaces all over the place manually and arguing with pkg-config and literally hundreds of dependencies. How the fuck this ever even worked to start with I don't know. There are massive API changes with minor versions of packages all through NetworkManager and Gnome, which is just wrong.

The underlying issue is simply shitty engineering from a large body of open source projects who don't care about upstream or doing their job properly. To name and shame: The entire Gnome project, just about anything Ubuntu have chucked out, the entire KDE project post KDE2.

It's utterly frustrating. I wish they did take a more BSD approach, which is conservative engineering i.e. think about it, get it right and leave it.

For all the people here whinging about innovation, waah waah, ship old versions etc, I doubt you've actually had to deal with these issues or build a project on top of them. There seems to be a large detachment from reality here.

As an example to back my point up, one of the reasons Windows is so damn popular is that the API is incredibly stable and software written in the mid-90s still works fine on current versions without a recompilation or any API changes. A two week old build of NetworkManager probably will break all the VPN providers...

cperciva• 14 years ago

The Linux kernel itself is a remarkably stable system as far as ABI/API is concerned

You're joking, right? There have been cases where ABIs have been broken by security patches, and users have been told that they have to choose between having a secure system and being able to use their third-party binary drivers.

(And yes, I understand the arguments about how binary drivers are evil; but I don't think users should be used as pawns in that fight.)

zimbatm• 14 years ago

He was probably talking of the user-space ABI/API. Drivers are internal to the kernel (or should be) and Linux never made a claim that those APIs where going to be stable.

telent• 14 years ago

> The Linux kernel itself is a remarkably stable system as far as ABI/API is concerned

If we're talking about userland applications, the Linux kernel A[BP]Is have IME historically been more stable than the Glibc interfaces (which are supposedly the "supported" interfaces) layered atop them.

In-kernel interfaces for binary-only modules are a different kettle of fish altogether.

meaty10• 14 years ago

As other people have said, userland ABI/API is solid. I should have clarified that.

There is no reason the internal API would be as to be honest it's the Kernel's job to make the drivers sit behind that API and stay there regardless of the problems. That is the abstraction boundary.

The issue is where the abstraction boundary is. In NetworkManager for example, which is an example I'm probably just picking on, there are many more abstractions above the kernel's abstraction which are unstable.

Again, comparing to Windows, there have been several driver model revisions but the userland API+ABI is stable between them.

ori_b• 14 years ago

Linux does not have an external API or ABI for drivers. Whether you like it or not, Linux drivers are developed against internal interfaces.

The external interfaces that Linux provides are remarkably stable.

npsimons2• 14 years ago

With all due respect, binary drivers can bugger off. As for you, shame on you for conflating userland ABI with driver interface. This has already been hashed out, many times before: http://www.kernel.org/doc/Documentation/stable_api_nonsense....

You can even get driver development for free: http://www.kroah.com/log/linux/free_drivers.html

If you want binary drivers, find another OS. As for userland ABI, people have static binaries from the 1.2 days of Linux that still run just fine.

bkor8• 14 years ago

Aside from being a GNOME release team member I am also the packager for GNOME within Mageia. I find packaging GNOME to be really easy, way easier than I expected actually.

Your argument comes across as that others should work more like the way you want them to work. The "large detachment from reality" is IMO an unneeded comment, easy to make comments from the sideline.

The people who are involved with GNOME from OpenBSD side are really nice and friendly. But some in that openbsd thread have the weird notion that if OpenBSD would drop GNOME that then we'd take OpenBSD more into consideration.

Your bit about NetworkManager: The goal is that you can easily configure your network. The way of doing that at that time and for now is NetworkManager. If it isn't good enough, then promote something which delivers the same goal, but is technically better. That's how to positively and practically change things.

thaumaturgy8• 14 years ago

Mageia is forked from Mandriva, and not very old, and Mandriva already has systemd support. I'm not surprised that Gnome has been easy for you to package for your distribution.

As far as I know, none of the BSDs are packaging Gnome 3 right now. Not one of them. Don't you think that might be signaling some trouble?

meaty10• 14 years ago

In that case as a Gnome team member, please take the comments I made as complaints relating to things that need fixing. I wouldn't say them if I did't stand behind my opinion.

In fact, my argument is that Gnome should adhere to sound engineering principles. This is not my sole opinion - it is a generic opinion which can be applied to any engineering project and is an established norm. "Move the goalposts, expect trouble".

NetworkManager utterly wins at easily configuring your network, but only if you have one ethernet card or a wireless card. For any other circumstance, it falls over miserably and turns into a shitfest of epic proportions which proceeds to bugger you at every available opportunity.

ifconfig is a better tool, particularly on BSD as it handles ALL interfaces equally, ironically due to sound engineering!

My efforts would be better spent writing a new desktop environment which actually cares about UNIX's semantics and architecture rather than reinventing it every few months.

nitrogen2• 14 years ago

If it isn't good enough, then promote something which delivers the same goal, but is technically better. That's how to positively and practically change things.

No it's not. The way to positively and practically change things is to do them right in the first place, or failing that, fix the system you have. The "throw it away and do something new" attitude is the driving force behind everything that is wrong with Linux on the desktop.

fafner2• 14 years ago

> As an example to back my point up, one of the reasons Windows is so damn popular is that the API is incredibly stable and software written in the mid-90s still works fine on current versions without a recompilation or any API changes.

Windows started from a different position. Backward compatibility makes sense when you have all the applications. And it results in you having to maintain a sh*tload of legacy APIs and modules. Microsoft has a different budget and works as a stricter organized company. Linux Desktop lacks the manpower. But on the other hand Linux has to move forward.

It certainly has to settle down a bit in the future. But things like NetworkManager, PulseAudio, Systemd, Wayland are introduced because they solve existing problems. You can certainly argue about implementation aspects but I don't think you can argue about the problems they are solving.

thaumaturgy8• 14 years ago

This submission won't make any sense without a little bit of background. (The comments here at the moment are all predictably confused.)

Start here: http://lwn.net/Articles/520892/

Core parts of Gnome (and probably lots of other Linux-y software, I haven't been following all of it) are all moving to require systemd. Systemd certainly promises a lot of advantages, like better handling of removable devices (and more: https://bbs.archlinux.org/viewtopic.php?pid=1149530#p1149530), but it is also replacing the tried-and-trusted rc approach. OpenBSD leaders in particular are concerned about systemd from a security standpoint; it has a lot of code that runs as root, it is big and unwieldy and the documentation is a mess.

So, not only were the OpenBSD developers faced with either auditing the entire systemd code base -- and then doing it again every time there's a new release -- or attempting to re-implement it in a compatible way, now they're also staring down the very short barrel of a nasty gun in which some really big stuff is going to stop working if they don't deal with it.

It's a real ugly situation, and OpenBSD really doesn't have the resources to spare to deal with this kind of nonsense. I wouldn't be surprised if this mess ends up tying up a couple of their core developers at an upcoming hackathon, where they could instead be working on something more useful than having to replace a huge part of the system that was working just fine up until Linux decided to throw it out. (Obviously there are plenty of people that would argue it wasn't "working just fine", either because they have specific knowledge about it or because "otherwise we wouldn't have needed systemd". That's not sufficient justification for forcing other developers to waste tremendous amounts of effort to re-work what was already working for them.)

Personally, I'm not a huge fan of the switch to systemd. I liked rc scripts. I've cobbled together my own rc daemon scripts from time to time; I'm not sure how that will work with systemd. I'm also seeing lots of ugly and persistent problems in NetworkMangler and whatever they're using for core audio these days; I'm a little gobsmacked that, rather than fixing those problems, some people decided to replace the init system.

darklajid2• 14 years ago

Thanks for a little bit of context.

About systemd itself: I guess so far it's really hard to say if it's worth the hassle. My experience is limited to two installations: One on a (admittedly beta level) Fedora image and running Arch on my Pi.

The latter was really pleasant, _including_ the systemd migration. I never used it before and definitely fell in love with a couple of features (The status, unifying log messages and the service status, for one).

I'd be the first to agree that there's quite a bit of room to improve. Some design decisions are questionable imho (like the stated opinion that systemd shouldn't write a text-only /var/log/messages file by default).

In the end it's always the same though: Some people are having a vision of how things should work and try to promote it. If they are successful, I'd argue that their idea has merit (or everyone else doesn't care).

stephen_g• 14 years ago

"Some design decisions are questionable imho (like the stated opinion that systemd shouldn't write a text-only /var/log/messages file by default)"

I think that may have changed in the latest release (or maybe it's in the next release, but I'm pretty sure Lennart said the default was going to be to write logs).

gillianseed3• 14 years ago

Thanks for the lwn link, it really helped to put things in a larger perspective. And I can't help but feel that Gnome is being used here to enforce systemd dependancy for reasons other than purely practical..

I think systemd is currently the best init system when it comes to potential, however I don't like how it 'swallows' other core components like udev as I think that might have a negative impact on possible alternatives being developed.

I think systemd should compete on it's merits and not by what seems like it becoming a dependency as part of an agenda rather than a practical necessity.

varjag2• 14 years ago

Systemd depends on an httpd. It simply can't be the system with the best potential, I refuse to believe it.

ambrop73• 14 years ago

Concerning NetworkManager, you might like my NCD scripting language designed specifically for programming of network configurations. http://code.google.com/p/badvpn/wiki/NCD

thaumaturgy8• 14 years ago

Wow. You developed a programming language for handling network connections; that is actually really neat. It's way past my bedtime, so I might've missed it, but is there any way in NCD to detect, say, a new wired connection, and then a timeout waiting for dhcp, and then as a fallback, configure the interface with a static ip?

I'm not sure what I think about having to edit a file every time I want to connect to a new network, but on the other hand, it would be super nice not to have to reboot every N times I switch network connections.

pmr_• 14 years ago

The maturity of systemd is at the root of the problem here. I switched (even for ArchLinux standards) relatively early and since then I could see how things changed continuously. There are two factors here: I appreciate active development of software and I would like things to stay compatible with previous releases at a minor cost to "progress". It is hard to get both in such a complex design space. Most of the problems are hard to predict and only come up as soon as there is active use of the system and if you never go public, there will never be active use.

Overall, jumping on the systemd bandwagon was done prematurely (at least by ArchLinux) and I think the large software packages should just give it another few months before migrating. systemd solves a lot of problems now and developers want problems to go as away before they become a feature or to pervasive to the codebase, but a little restraint might to a lot of good here.

There has to come a point where systemd announces feature completeness and stable interfaces so everybody can use it as a standard and then everybody can go for broad use and acceptance. I would hope that this gives the smaller players a chance to catch-up and much needed safety when investing resources into their own implementations.

AnthonyMouse• 14 years ago

>The maturity of systemd is at the root of the problem here.

That is pretty much what I came in here to say. It seems like OpenBSD is trying to have it both ways: Include all the latest and greatest toys and still thoroughly audit everything that changes between releases. But it's new toys, secure, cheap. Pick two.

Is it really the end of the world if OpenBSD bears a closer resemblance to Debian Stable for the sake of security? Why is 'latest Gnome' a requirement? Just stick with older versions of everything until the new APIs stabilize and mature.

lostnet• 14 years ago

Thank you for the background!

I worked on Solaris during the Gnome transition and during the largely overlapping SMF transition. I'm not sure which was more painful or was more broken by design, though SMF surely affected more developers.

If I were making decisions for OpenBSD I would not accept a requirement to port an init replacement, but I also wouldn't be putting any resources into keeping Gnome running that could go into better integration with an alternative.

While it is nice to be able to mix and match everything rather than having choose a silo, having these large market share players like gnome dictate changes on every platform is ultimately rather destructive to having meaningful options.

I would rather run the Linux kernel for practical reasons, and often use gnome for practical reasons though I would be thrilled not to.

Showing me you can keep up with Linux running gnome isn't compelling at all, particularly if it means you are no longer doing much else. But if you can better integrate with KDE or enlightenment (or fvwm and some duct tape,) and give me an alternative to linux+gnome then I'll deal with a new kernel (and not having PAM?!)

pja3• 14 years ago

I don't understand why systemd is suddenly such a hard requirement: it suggests that the Gnome developers are simply punching through the abstraction layer provided by dbus for reasons of convenience. If some part of Gnome needs a system service, it requests it via dbus and parses the response. How that service is delivered ought to be irrelevant.

Since systemd is supposed to offer all its services via dbus, is the problem really "we don't want to implement all these services that a modern desktop needs". OpenBSD et al could offer a dbus layer that implements the same set of services, but they don't want (or have the manpower) to do so. In which case, I'm not really sure they have a leg to stand on: the things systemd offers really are important for a modern well integrated desktop (or any other system for that matter) and they weren't well served by initscripts. OpenBSD is free to implement a service that offers the same dbus interfaces whenever they want, but complaining about Linuxisms is perhaps a little rich: if you want your system to remain stuck in the early 1990s then that's fine, but you can't really expect the rest of the world to hang around waiting for you to catch up.

pja3• 14 years ago

NB If the answer to "why can't you write dbus interfaces" is "we'd love to, but the Gnome developers keep changing the interfaces every minor point release" then that's probably a valid retort...

krakensden• 14 years ago

> it is big and unwieldy and the documentation is a mess.

Only if you compare it to RC and then omit the enormous horrible undocumented copypasta mess that was RC scripts

blaze33• 14 years ago

Wouldn't it be more appropriate to talk about LinuxDesktopism?

There I see part of an explanation regarding the opposite market shares between the server/HPC and desktop worlds.

olgeni• 14 years ago

Starting from small things, it would also be great to stop using #!/bin/bash in scripts for no apparent reason (hint: "it's my default shell" is not a good reason.)

pja3• 14 years ago

I'd far rather people did that than used #!/bin/sh and assumed that sh was bash!

papsosouid8• 14 years ago

It isn't a choice between "do it wrong A" and "do it wrong B". There are a whole host of options, one of which is "do it right". Relying on a non-standard shell to be in /bin is bad, relying on non-standard shell extensions to be in the standard shell is also bad. So either use the standard shell as the standard shell, or use whatever nonstandard shell you want by calling env to find it.

klibertp• 14 years ago

I am not well versed in shell magic, but this one bit me quite a few times because, being on FreeBSD, my bash lives under /usr/local/bin (and I installed it just for the scripts written by other people, too).

I'd like to ask you what can I do with this, ie. how can I modify those scripts so that they work both on my machine and on others'? I don't want (obviously) to symlink bash executable to /bin. I guess that using plain sh, which actually is in /bin wouldn't be good either, because (I suppose) if author wanted bash it was on purpose. How to write shebangs so that they are compatible with Linux and BSD at the same time?

jsnell• 14 years ago

The standard hack for that would be:

#!/usr/bin/env bash

gavanwoolery• 14 years ago

I would love to see a group of hackers get together to create a real (i.e. not a weak ARM device) computer and operating system, for profit, but open source. It seems like people have mostly given up on this notion, as if its impossible to create a competing device. In the 70s/80s, a few people launched brands like Apple, Microsoft, and Amiga; I know things are harder now but it is far from impossible.

Why for profit? So that people have both an incentive and means to work on the product fulltime.

dman• 14 years ago

Indeed. What we need is a yet to be created firm with the engineering ethos of Trolltech attacking the problem that Apple is right now.

mixedbit3• 14 years ago

When OpenBSD was pissed off by proprietary HSRP protocol designed by Cisco, they decided to implement their own protocol (not compatible with HSRP) and use the same IP protocol number 114 without IANA approval. http://en.wikipedia.org/wiki/Common_Address_Redundancy_Proto... This created a serious mess for system administrators and gives a good example of how to handle compatibility and standards related issues ;)

papsosouid8• 14 years ago

I guess IANA should have given them a protocol number like they requested then, instead of playing politics because cisco employees run the organization.

mixedbit3• 14 years ago

Sure, but OpenBSD could have chosen some reasonably safe number that would not be in obvious conflict with Cisco equipment and would make users life easier. Instead they decided to have a childish revenge on Cisco.

fafner2• 14 years ago

The Linux Desktop can't stand still to accommodate some minor platform. I understand that a move to systemd and in the future to wayland will have problematic consequences for BSD support. But the changes are made for good reasons and to bring the Linux and Free Software Desktop forward.

Systemd is much better for Desktop uses. The major reason is that it is fully designed to support hotplug. Which is something you have to constantly deal with on Desktop/Laptop systems. Disks are plugged in, Network connection isn't ready, and so on. It is also much faster and has a lot of other nice features. The lack of systemd support in *Ubuntu might be the reason for me to switch to another distribution for my Desktop.

Wayland finally allows us to get rid off all the legacy crap we have to carry around with X11. There is a reason why it is developed by the Xorg devs.

Pulseaudio had its reasons as well. Such as having useful support for Bluetooth- or USB-audio. Per-application audio settings and so on.

Of course you can argue with certain details in certain implementations. But I don't think that we can argue about the problems those solutions solve. Maybe you are happy with the setup you've been using since the 90s. But for most people the times are changing.

It is certainly sad that the BSDs don't have the manpower to keep up. However we can not expect the Linux Desktop to stand still because of them. The Linux Desktop itself lacks manpower and is fighting an uphill battle. Supporting minor systems will only drag it down.

riffraff• 14 years ago

> The lack of systemd support in *Ubuntu might be the reason for me to switch to another distribution for my Desktop.

are the advantages of systemd over upstart so strong for desktop use?

rlpb2• 14 years ago

> Systemd is much better for Desktop uses. The major reason is that it is fully designed to support hotplug.

This is exactly why upstart was invented, long before systemd existed.

> Which is something you have to constantly deal with on Desktop/Laptop systems. Disks are plugged in, Network connection isn't ready, and so on. It is also much faster and has a lot of other nice features. The lack of systemd support in *Ubuntu might be the reason for me to switch to another distribution for my Desktop.

Ubuntu uses upstart, which handles everything that you have specified just fine. You may well have a good reason to switch away, but you have failed to state your case here.

ciupicri• 14 years ago

> This is exactly why upstart was invented, long before systemd existed.

Except that upstart didn't deliver too much, unlike systemd. Fedora used upstart (RHEL6 still does) and it wasn't very different from the traditional SysV init. I also wasn't able to figure out how to use it to its full potential, it looked too complicated. On the other hand perhaps I'm biased because Lennart provided a series on systemd for administrators.

papsosouid8• 14 years ago

>The Linux Desktop can't stand still to accommodate some minor platform.

The current strategy of charging backwards at full speed doesn't seem like a great plan though. We're not getting closer to linux on the desktop being plausible, we're getting further and further away. Making a typical linux desktop system even more complex, brittle, and fragile than windows is not a good way to compete with windows.

aidenn0• 14 years ago

systemd is not better than init for any use right now. My guess is that it is unlikely to ever be, but predictions are hard, especially about the future. As far as pulseaudio, it's basically a bloated version of esd, which was another project trying to solve the same problems that also sucked.

I'm taking a "wait and see" approach to Wayland. Clearly X needs to die. Many times before people have tried to kill it and it's still here; Wayland seems to have a lot of backing though.

rjsw2• 14 years ago

The changes to X11/Weyland are an even bigger issue for non-Linux platforms than Gnome.

Even if the BSDs had unlimited manpower it would be extremely inefficient to try to keep track of all the changes to what has become a Linux specific project.

zobzu• 14 years ago

I'm a 100% Linux user

I find systemd & others are hurting Linux as much as it hurts BSDs.

In fact, just look at the ArchLinux forums, its FILLED with systemd issues. And most people know by now about the "disable pulseaudio, use pure alsa" trick to solve "all Linux audio issues". Heh.

Upstream vendors and why they can be harmful

Related Stories