Debian project has plenty of money but not enough developers

https://www.theregister.com/2020/09/10/debian_project_address/

by mr_toad • 6 years ago

152 111 6 years ago

tluyben22• 6 years ago

Bit of a weird headline as the article says Debian org has $900k in the bank but that is not spent on the devs; they are volunteers. If they would offer, let’s say, 2k/mo (the article says they need good people who like though problems; 2k is nothing for those types however, many people would like do it because of love for Debian; they do not do it now as they need at least a little food on the table) per full time contributor, there would be plenty of devs joining. But for their goals to be met, they would be also immediately broke. So in fact they do not have plenty of money to achieve what they want, unless people work for free.

dcow2• 6 years ago

Yeah I’m also confused by this. Debian has an opportunity to reiterate that free software isn’t just a pipe dream volunteer effort but rather that there is business being software authorities without the baggage of a slew of software patents and terms and license agreements. Perhaps if they paid core maintainers it would ostracize other contributors who didn't make the cut? Perhaps they don’t want to become a Canonical and have to answer to steak holders? There’s value in being frugal but you also meed to know your own value or others will take advantage of you.

microtonal2• 6 years ago

Some Debian developers tried to get funding to finish the Etch release, but it was met with a lot of hostility:

https://lists.debian.org/debian-project/2006/10/msg00260.htm...

That doesn't mean that it cannot be done. E.g. the FreeBSD Foundation pays developers for specific projects that are important to the future of FreeBSD:

https://freebsdfoundation.org/our-work/projects/

Tepix3• 6 years ago

I think volunteer work is at the core of their culture. Paying a few core full time contributors seems like a good idea - i assume it's happening already.

pabs33• 6 years ago

The only payments Debian does to contributors are to Outreachy interns:

https://www.outreachy.org/

petre• 6 years ago

2k/mo would pay for 37 devs for a year. Rather than that do micropayments for contributions (approved packages, finished tasks). A project such as Debian only needs a few core members or people working on important projects paid full time. It works for FreeBSD.

I really like Debian but volunteers are probably better off packaging for other distros such as Arch or Alpine with simpler packaging/better tooling and less strict rules. One thing to improve would be tooling to make packages.

elipsey2• 6 years ago

I am currently unemployed/independently-wealthy/thrifter/bumming around the bay area. I love debian, and I sometimes joke that if there was such a thing as a minimum wage dev job, I would gladly take it to get out of having to pass another technical interview :/

It would be neat if they could split the difference between free and real job, but I wonder if part of the reason they don't is that they want people who care unreservedly about values and code quality rather than people who need $500 a week?

neilwilson2• 6 years ago

It is of course one of the problems a Federal Job Guarantee would solve.

Clearly working full time on Debian is of value to society, a good use of time and clearly verifiable by committed contributions.

Definitely worth the living wage.

belorn• 6 years ago

One model that is between a free and a real job would be a collective where housing, food, and internet is provided but not much else. Hacker spaces are also similar to this but in my experience rarely ever focus on system development, but instead more on hardware and hardware art.

ahelwer• 6 years ago

You'd probably be interested in part-time contracting. Set your own hours and rates. Often there's a small paid project in lieu of an interview. Lots of interesting projects are willing to shell out for a contractor that aren't willing to hire a full time employee.

nix236• 6 years ago

Maybe the FreeBSD Foundation is something for you, they Sponsor you so you can implement wanted stuff into FreeBSD (just an option not opinionated)

sn• 6 years ago

I don't know Debian's specific history, but having participated in volunteer organizations, getting everyone to agree on how to spend money can be tricky.

Personally I'd love to be making more open source contributions as well, and I've been thinking off and on about taking a sabbatical to make time for it, though that probably won't happen.

I might be able to help find dev work for you that doesn't involve technical interviews, though it may not necessarily be a "job".

est31• 6 years ago

they need good people who like though problems; 2k is nothing for those types however

Many people of that type work at universities for comparable compensations. E.g. 50% E13 position at TU Berlin gives you monthly 2k pre tax. If you spend lots of your time to improve the public good, you'll still need something to put food on your table (unless you are financially independent of course, but many people aren't).

fock2• 6 years ago

basically, if they would offer 50/50-jobs in partnership with public services, where the public service guarantees job security and the Debian project guarantees some oldfashioned hacker-spirit, people would fly at these (or at least I would. People could spend 50% of the time for Debian and push OSS adoption in the public service and develop the basic CRUD in the remaining time.

But.... sadly this is not going to fill the coffers of politicians and consultants alike, so things like these will just be daydreaming until a majority of people is fed up with corruption and not braindamaged enough to believe extremist bullshit.

andy_ppp• 6 years ago

If you pay some of the volunteers does that not basically spit in the face of those not being paid? It would really annoy the hell out of me if I’d been making Debian better for 20 years for free while training up someone being paid to get involved.

bluGill2• 6 years ago

Depends. There are a lot of tedious things that the volunteers know need to be done but don't want to do. Paying to get that done is helpful. (Note that what is tedious varies between personalities...) There are also cases where the volunteer knows he is overloaded and will welcome help.

GuB-42• 6 years ago

The big difference is subordination.

A volunteer gets to choose what he wants to work on and if he wants to work at all. An employee/contractor does what his boss/customer wants him to do and get compensated (i.e. paid) for it.

There are many reasons why you may want to pay someone. The obvious one is to do things no one wants to do for free.

But maybe you also want a proper contract, to make sure some essential job is done fully. Volunteers who work on they spare time will stop working for you once they stop having spare time, and there is not much you can do about it, both legally and ethically.

Sometimes you also need certifications. I don't know how it applies to Debian but certified people usually don't work for free. The certification process can be expensive and they can have legal responsibilities. Even if people can offer their time and skills for free, legal responsibilities are usually off limits.

As an example, I volunteered to staff a few events. None of the regular staff and organizers were paid. However, security was paid. While we had a few burly guys, no one wanted to take responsibility, and probably couldn't anyways.

roenxi2• 6 years ago

Mmm. A million dollars isn't a lot of money these days. That is something like 5 to 18 full time devs, for 1 year. Good luck running the Debian project at market rates.

TomSwirly• 6 years ago

I have some spare time and I'd love to work on Debian, but I got offered $500 to work on another open source project.

The person who made me the offer was very apologetic - as she said, it's not even my day rate - but as I said, I'd probably do it for nothing, no one's making a penny out of the software, it's really useful, and $500 isn't nothing.

I think they could get plenty of people to work on Debian for 10 hours a week for $10k a year, as a hobby.

LockAndLol• 6 years ago

There are thousands of people working opensource for nothing. Why do you or anybody care about market rates when those people (me included) would do it for minimum wage?

I'd happily code all day long on stuff I really care about for minimum wage. And when I day minimum, I don't mean the penance described by the law that forces you to have multiple minimum wage jobs to get by. No, I mean one single job that pays enough to get by.

ashtonkem• 6 years ago

And that’s not even accounting for benefits, which often add a huge amount of cost above and beyond the listed salary.

Theory52• 6 years ago

I'm in the InfoSec field, and bug bounties are a huge thing now (well, for years but it seems to keep gaining steam) for app security. There are people who make a living, or at least generate secondary income off it, or others do bug hunting as a hobby and to help out and don't mind a little bit of reward when it's there.

Maybe it's because I'm not big into the development side of things, but I cannot recall any programs that'd be similar to bug bounties but focus on someone building code or fixing the bugs and submitting that in a freelance type capacity.

Granted, my limited knowledge of development may just mean I don't see obvious reasons it's not done or widely known about, and there would be a few different things thatd need to happen vs how bug hunting works, but its similar to your idea to attract more talent to work on small bits, without going out and hiring full time developers.

rstuart41332• 6 years ago

I'm not sure how Jonathan is defining "enough" developers. It's very easy to come up with a definition that says Debian doesn't have enough developers: just say Debian should package every piece of software out there.

But that makes no sense. In reality, there has to be some sort of decision mechanism over what packages are included and what aren't. If that decision isn't made pro-actively, it will be forced upon you - via the man power problem. Debian is at its heart a do-ocracy, which boils down to something happens only if some individual can raise the energy to make it happen. So a package dies when no Debian member can make (unpaid) time to save it. It's not like there is someone who can order someone else to save it - Debian is a purely volunteer organisation.

It's easy to paint that as having a man power problem, but curating the packages is essential task that must be undertaken. Doing it isn't "a problem", it's essential to survival.

It's also easy to criticise the way this decision is made. Just leaving it to the individual decision making of 1k volunteers sounds so unorganised. But the flip side is if you have paid engineers you are doing the reverse, you are giving someone the power to order other developers around. In an organisation like Debian that attracts membership through it strong egalitarian principles, that's going to be an anathema to most of it's members.

As an side, despite all the whinging here while it's difficult to introduce a new package to Debian, it is not difficult to step up to support an existing package. You don't have to be a developer or a maintainer, and there are any number of existing developers and maintainers who will upload your work if you do it. So if a package dies, it's not too far from a stretch to say it was because no-one on the planet with the skills was that interested in doing the work.

raverbashing• 6 years ago

Debian from the exterior sounds like the anti-corporate organization: flat hierarchy, top-notch democracy, self-directed work with almost no top-down guidance/"orders". And not so surprisingly, that also cause problems. Let's not forget the high barrier to entry for contributors.

It is complicated.

diminish• 6 years ago

I wish Mozilla mimics Debian or Linux development organization. It earns hundreds of millions and can't keep pace with chrome.

dkdk8283• 6 years ago

Mozilla has had management problems for a long time. I don’t think this will change any time soon. Look at the latest 999 and highest salaries.

lmedinas• 6 years ago

I think you are comparing Apples with Oranges. Linux organizations are mostly done by volunteers and most of the infrastructure used is either donated or allocated in organizations which support their development. So any costs they have is for maintaining anything they really have to pay such as: legal, taxes etc...

On the other hand Mozilla has to pay for infrastructure (and here they have to support multiplatforms and multi OS developments), development, legal, taxes, marketing etc... etc.. It's not even comparable in any order of magnitude the costs involved.

If Mozilla would be to survive with volunteers would be even more miles ahead of other browsers. You can see Thunderbird as an example that Mozilla stopped maintaining since years the development pace slowed considerably.

On a side note: I really wish for the sake of the internet that Mozilla gets their s* together and start to improve their products to be really competitors to Chrome and Safari.

tus882• 6 years ago

Actually firefox is faster at javascrupt that chrome.

octoberfranklin3• 6 years ago

That's like saying that emacs's LISP interpreter is faster than vim's vimscript interpreter.... Yeah, so what?

The performance-critical parts of vim are written in C. You can't say that about emacs. Or firefox.

exmozilla• 6 years ago

I think this just means Firefox isn't the priority of Mozilla

raxxorrax• 6 years ago

It is a foundation and chrome is a browser. You mean FF of course and it can compete very well with Chrome. I use both for development purposes and there are some minor strength and weekneses for both browsers. I use FF privately and for non-development purposes.

Generate a HTML doc with tables with 200.000 rows and columns with table cells with non-zero height/width and try to scroll fluidly on a tablet for example. Works in FF, works in Safari, doesn't work in Chrome.

True that there is historical baggage with HTML tables and maybe these optimizations aren't a good focus, but I doubt there are many website where all modern browser are at least competitive.

Tepix3• 6 years ago

I like Debian and have entertained the thought of becoming a Debian developer multiple times. The learning curve for a new developer is steep. Just packaging a small piece of software requires a large amount of documents to consider last time i looked.

qiqitori• 6 years ago

Maybe, but you will be able to use some of the knowledge or intuition gained if you ever have to package something for any other distro.

nix236• 6 years ago

So AUR is the better choice? pkgsrc? At least you don't have to read tons of Docs.

Tepix3• 6 years ago

Packaging something for my own use (or my company's) is quite different.

microtonal2• 6 years ago

It is a tiresome process. I attempted to submit a particular package (and others before and after me). It was either met with a lot of red tape (IIRC document licenses for each file individually) and months of no visible progress when comments were addressed.

This was in stark contrast to e.g. contributing to nixpkgs. Do a PR on GitHub. The package gets built automatically by CI. Get feedback. Make changes to address the feedback. And it gets merged. Of course, PRs sometimes go stale (most projects are dealing with a lack of manpower), but the process is generally fast.

But the outcomes are radically different: I never attempted to contribute to Debian again, but I have become an active contributor to nixpkgs.

I think to attract developers the process needs to be modernized. Put all the package definitions in a Gitlab monorepo. Build each PR in a sandbox on CI. Describe the submission process on a single page.

ylyn• 6 years ago

Debian cares a lot more about licensing than Nix. Debian also has far more developers than Nix, I imagine, and processes that have probably existed for longer than GitHub itself.

wink• 6 years ago

Back when I was eager to join they still had this ridiculous several month or 2 year long process to get in, so after reading a few accounts of that I said screw it and never even tried again, even when they changed that. I guess I wasn't the only one, and in the end I joined other projects where contributing was fun and easy. (And yeah, I'm neither someone to start projects nor spending consistent time every week, but that doesn't mean you can't contribute lots of time).

Still a happy Debian user after all these years, but I'm past the time where I'd spend effort to even try to join.

rleigh7• 6 years ago

The processes are a legacy of distributed development in the era of dial-up internet. No always-available services, no github, not even any requirement for version control. With every package being maintained independently. While most packages do now use version control, every package has the potential to use a different system, and internally could use a different set of tools to do the actual packaging work.

If you were to design the .deb packaging workflow today, you'd do it very differently. You would probably have a single, or much smaller, set of source repositories, and you would have a single tool to do all the core packaging work.

You only have to look at the BSD Ports, MacPorts, Homebrew, vcpkg, and other software collections to appreciate how extremely complex the Debian approach is, and how the other systems benefit from a strictly standard set of functions to do their job, making all packages uniform and easy to work on.

On the other hand, because of these considerations, Debian has vastly better support for complex versioned dependencies, and the SAT solver in apt is best in its class. The other systems manage the collection as a whole and don't need to care to the same extent.

So I would agree some modernisation wouldn't go amiss. But it's difficult to accomplish that when you have a 25 year legacy of doing it a completely different way. And you have over 20000 packages to fix.

secondcoming• 6 years ago

Yes, I gave up trying to package a small utility.

foresto• 6 years ago

On the other hand, those documents are well organized and clearly written, which makes skipping around to just the parts you need rather easy. A wealth of example material and helper tools (e.g. lintian) are also available, and all the policy changes are versioned and described as human-readable diffs[1], so the cost of catching up if you haven't been following progress for a few years is quite low.

I think my first simple debian package was working within a few hours, and I was tackling more complex stuff within a few days. Not bad for a system that can handle the oddball installation requirements of just about any software, while also accounting for many install/configure/upgrade/downgrade/reinstall/remove edge cases.

I'll grant that it is a complicated system, and there's a lot of legacy stuff beneath the higher-level tools. I suppose my point is that the vast majority of the complexity turns out to be avoidable in most cases, and getting started is not as hard as it looks.

[1] https://www.debian.org/doc/debian-policy/upgrading-checklist...

qznc• 6 years ago

I guess the call to action is: https://www.debian.org/intro/help or https://www.debian.org/devel/join/

Maybe also https://wiki.debian.org/how-can-i-help

teruakohatu• 6 years ago

Is $900k in the bank a lot of money for as large and important an organization as Debian? That is how much the Mozilla Corporation earns every 18 hours. Even if the staff are volunteers, having rainy day spending seems prudent.

bluedino• 6 years ago

Debian doesn’t spend millions of dollars on executives

dsr_• 6 years ago

Or on any salaries.

Debian's expenses are hosting, conferences and travel reimbursement -- and not a lot of that.

It's still not really a lot of money, though. As a thought experiment, if Google decided to donate $1 per installed Debian or derivative OS, I think that might double Debian's bank account.

That's a plausible funding slogan -- "A Dollar for Debian".

insertnickname• 6 years ago

This 2019 blog post from a Debian developer describes some of the problems they are facing from a developer perspective: "Winding down my Debian involvement" https://michael.stapelberg.ch/posts/2019-03-10-debian-windin...

bnmaker• 6 years ago

Not liking the bug tracker and mailing lists as well as several other points come across as typical things-need-to-be-done-differently now, which has destroyed the culture and meaningful participation in other projects, like Python.

I'm sure there are many valid criticisms of Debian, the main one for me is overpatching upstream packages.

sitkack3• 6 years ago

Getting in a patch fight with a .deb maintainer is a right of passage that everyone in the OSS world gets to go through. The slicing and dicing that is done to well mannered software that didn't harm anyone is atrocious. PETP (People for the Ethical Treatment of Packages)

Distros and packaging have much lower relevance as operating systems are viewed as applications, rarely are unix systems multiuser, often serving a single software package. Even desktop operating systems are getting containerized, Qubes, etc.

rwmj• 6 years ago

The bug tracker is objectively awful. It takes 15+ minutes to subscribe to a bug, which is ridiculous! Absolutely no one would write a bug tracker today where everything is processed by email[1].

[1] Which is not to say that messaging is bad - a web interface with a proper message queue behind it would be fine. In fact that's how Fedora's systems work. It's that email is not a reasonable messaging system for this purpose.

BiteCode_dev• 6 years ago

I had to create a deb for a web app, it had to setup and populate a DB (from web sources), and manage updates for this db when you upgraded the deb. It had a build process of static files that couldn't be vendored. It has a lot of configuration to be entered prior those would happen, using debconf. It had to vendor a virtualenv. I had dependancies on 2 other web apps that had the same story, and they shared some config files.

I didn't chose this architecture. But I was tasked to package it so that people could just "apt get install stuff" and start using it.

It was HARD.

The tech is very old school, it doesn't give a lot of feedback, it's hard to debug, documentation is sparse and examples in the wild are not up to date, not to mention the state of tooling. And you do all that with shell scripting, the worst language for the job. Sure you can use anything else, if you like zero doc and example.

Then we had to distribute it.

Well, turns out creating an APT repo is not so easy, either.

Now, we did all that by cutting a lot of corners, zero red tape, and only us to please. Also, the consequences of not using best practices, or worst, breaking something, were few.

On a team of 6, only 2 were really understanding what was going on. The rest were lost, awaiting instructions from us because they just couldn't picture were their work would fit in this blob of bash workflow. Now that I'm gone, I have no idea how the team is going to maintain the thing.

I can't imagine doing the same thing with the level of road block working for something as sensitive as the official debian repo would bring.

If debian wants more contributors, they have to make this stuff way, way more accessible. And be friendlier with people telling them it's hard.

Every time I raised the topic, somebody came along to say I shouldn't be doing x or y. Or just that I shouldn't be packaging something and just publish it on the debian repo, letting the debian maintainers do the packaging. It's not going to bring me to work with the community if the first taste of it is the demonstration they completely ignore their shortcomings.

And I say that while I've been typing apt commands for a decade now. I'm also french. Debian is, with VLC, Libre Office and Firefox, one of the 4 pillars of FOSS for me.

sitkack3• 6 years ago

Making deb _is_ painful, the politics of how packages are folded, bent, welded, drilled and smashed is not a simple landscape to navigate.

A lot of Docker is a response to how difficult it is to build packages and how invasive package installation is to the base system (Nix gets this right).

If in the future you need to build packages again, I'd take a look at fpm.

The goal of fpm is to make it easy and quick to build packages such as rpms, debs, OSX packages, etc.

https://fpm.readthedocs.io/en/latest/

https://github.com/jordansissel/fpm

protontypes• 6 years ago

We have developed a tool for projects like Debian to distribute donations over larger projects. By distributing the donations in the dependencies, Debian could even attract new developers in its own dependencies: https://github.com/protontypes/libreselery

kasabali• 6 years ago

No wonder, they have the largest red tape on maintainer application process.

Until very recently they were expecting GPG keys signed by 2 Debian developers, and that could be only done by being developers face to face, or by attending a key signing party on DebConf. Tough luck if you're in a remote location. Not sure how it's being done after pandemic.

octoberfranklin3• 6 years ago

I think it's awesome that they have a fully autonomous cryptographic web of trust. Far too few organizations do.

nottorp• 6 years ago

Why don't they hire Poettering to further make Linux unusable for hobbyists?

octoberfranklin3• 6 years ago

Because getting him do do that does not require paying him!

Badump-ching! I'm here all week, folks. Don't forget to tip your waitresses...

nix236• 6 years ago

He works at RedHat, and i strongly believe they paying him.

nix236• 6 years ago

unusable for hobbyists

Why would any hobbyist wanna use Linux?

yjftsjthsd-h• 6 years ago

? because it's fun and it's free?

mikece• 6 years ago

If LMDE picks up steam I’m sure that could change. Is the Ubuntu distro so different from Debian that those devs can’t simply work more on the upstream distro (or are Canonical devs not really doing much Linux work but userland/app work instead)?

mlatu• 6 years ago

https://nm.debian.org/faq/

"The site gives me 403 Forbidden errors if I disable referers"

yah cool faq, many questions answered.

liveoneggs• 6 years ago

NetBSD (with a lot less money in the bank and similarly bad bug trackers) has hired people using a few models in the past -- hourly, CoD, fixed-rate, etc but it's really the hiring itself that is so much trouble, just like in the real world! Not to mention doing "management" is also a total pain in the butt that someone now has to do for free.

The irony is that the people being hired are mostly already there doing work for free.

There is no way debian (or NetBSD) could pay market rate to compete for the time of anyone working on the project for very long so it's usually a motivational push or justification for sabbatical or "I have three months between contracts".

It's much much easier to buy hardware and things for developers who ask for it.

I suppose these projects are seeking developers with passion, personal wealth, and free time.

bluGill2• 6 years ago

Hiring people who would do it for free allowed them to work faster. It becomes there day job. I know of a few projects in FreeBSD where the author said I'm between contracts in a few weeks, you can pay me to do this work and it is done in a month, or I can find a new contract and the feature "that everyone wants" will be worked on in my spare time and done in 2 years.

moonbug• 6 years ago

if only there was some way to exchange money for labour.

dxxvi• 6 years ago

$900k is not much. It's only enough for 4 developers for 1 year. If you don't spend the money the sponsors give you, the sponsors won't give you anymore.

ilaksh• 6 years ago

With the amount of government and corporate reliance on Debian derivates, the budget should literally be 100 times larger. Like $90 million instead of $900,000. Or maybe just $9 million. Then you have no trouble finding developers. You use money to get them. It works very well.

Part of the issue though I suspect may be the way packages work. I am just guessing but it may be the case that these volunteers are creating packages for lots of third party software and constantly fighting with incompatibility to upgrade them. Which seems like the third parties should be more involved with updating the packages.

But personally I think that even though snaps are so hated, the relative impossibility of keeping all packages up to date and similar issues is making things like Snaps or Flatpak make more and more sense as a general direction.

I don't know if Flatpak has a way to do compression or reuse layers like Docker but I wonder if it might eventually make sense to go in that direction.

Or another crazy idea: maybe it's possible to train an AI to resolve dependency or build issues and create packages. Maybe pretty far-fetched but when there are so many tens of thousands of packages I feel like you should start going for radical ideas.

rstuart41332• 6 years ago

At the 1000ft view RedHat and Debian perform much the same function.

Debian has $900k in the bank. IBM purchased RedHat for $34B. The number's aren't directly comparable, but methinks a factor of 100 understates it.

ausjke• 6 years ago

Unless Debian has systematic discrimination for race or gender or whatever, I don't think there is a need to focus on "diversity" for its developers, NBA is a good example there and nobody complained. Improving certain area is fine, put it a priority for problems non-exist will do more harm than good. You do it because you like it or need it, not because of where you are from or who you are.

And yes $900K should be used to improve the situation, there might be some great developers actually need the money, or the money can get more young developer to start in debian.

credit_guy• 6 years ago

Plenty of money is always a relative term. If you think you have "plenty of money" but you don't have X, in reality you don't have plenty of money. There are exceptions like X=love, admiration, etc, but for most X, if you have plenty of money and a shortage of X, you just exchange some of your loads of money for X and you are good.

Debian project has plenty of money but not enough developers

Related Stories