Xz: A microcosm of the interactions in open source projects

https://robmensching.com/blog/posts/2024/03/30/a-microcosm-of-the-interactions-in-open-source-projects/

by transpute • 2 years ago

526 339 2 years ago

gladiatr72• 2 years ago

Funny. I was just saying the same thing to one of my partners just 4 hours ago.

Culpability also must be laid at RedHat's feet for sanctioning the practice of side loading libraries into such a critical service's address space. Their drive to cellularize systems management has overtaken their common sense.

The idea that they could not be bothered to answer the call of the sole maintainer of a library used in such a critical path in his time of need is, well, astonishing.

dralley2• 2 years ago

1) Debian includes this downstream patch, also.

2) A potential explanation for "why now" is that systemd DID prevent these dependencies from loading automatically in a patch one month ago [0], and the patches to lzma enabling the backdoor merged a few days later, followed by (as we know) an immediate and somewhat heavy push to get distros to upgrade driven by sockpuppets. It could be a total coincidence, or it could be that the attacker jumped to pull the trigger before the window of vulnerability started closing on them

[0] https://github.com/systemd/systemd/pull/31550#issuecomment-1...

I think it's a bit cheap to blame systemd here, and systemd does not equate directly to Red Hat either.

1oooqooq• 2 years ago

I believe they were talking about sideloading a binary tarball instead of building and repackaging.

Choosing a distro is nothing but choosing where you place your trust.

I can understand debian cutting corners here and there. But RH have little excuses with the money they make. Yet, even a superficial analysis, show them to be less trustworthy than the anime-avatars maintaining gentoo or arch.

fredamaral2• 2 years ago

awkward, but heard very recently that open source is not "vc backable, go away". maybe it will change now, after the infrastructure pillars of the modern world ruins in front of those many saas/ai/web3/cloud/whatever investors

creshal• 2 years ago

Why should VCs back oss infrastructure directly? It doesn't help the VCs and only creates perverse incentives for the oss projects.

The SaaS/cloud/etc. companies themselves should fund the projects they depend on. They actually know what those are and don't have to force their own monetization/growth models onto the projects.

331c8c71• 2 years ago

The SaaS/cloud/etc. companies themselves should fund the projects they depend on.

There is a bit of a free rider problem it seems.

fredamaral2• 2 years ago

definitely. not talking about the vc specifically, but the investment chain. a typical free rider, directly or indirectly.

supriyo-biswas6• 2 years ago

The SaaS/cloud/etc. companies themselves should fund the projects they depend on.

If anything, the same "valuable" "community" members, a lot of which includes such SaaS businesses and other freeloaders, seems to shout at projects trying to ensure their survival through formal ownership structures and license changes.

noobermin4• 2 years ago

Sure, may be VCs specifically shouldn't be where the money comes from but this labour NEEDS to be paid otherwise shit like this will keep happening. Dear god, I looked it up just now, but next week will be the 10th anniversary of Heartbleed being introduced to the world. A decade, and really little has changed.

forgotpwd16• 2 years ago

Our no-longer-reasonable requestor also offers a suggestions. Notice there is no offer to actually help.

Help in maintainship how? Patches had already been made and were awaiting to be reviewed and merged. This was up to the maintainer to do and requestor couldn't help with it.

ribasushi• 2 years ago

Help in maintainship how?

Pay.

amszmidt• 2 years ago

That assumes the maintainer wants to be paid. There are plenty of us who maintain FLOSS projects that do it for other reasons and any monetary exchange would burden us, since it might pressure one that this is now a job and you have to execute on tasks - there are enough headaches handling other things as it is.

yunohn5• 2 years ago

Huh? So anyone who asks/says something in OSS, should just throw money into the ring to have an opinion?

lostpwagain• 2 years ago

To me it would make more sense to add more eyeballs looking at what gets committed. For example, in this case who would you pay? The new (co-)maintainer was compromised and it would not help to pay him. Thus, in order for payments to help one would need to have some assurance that the person getting paid is not compromised. The easiest way to have some level of such assurance seems to be to pay ones own employees. This is of course not bulletproof, but certainly adds another layer to pulling something like this off.

At the same time, this attempt nicely illustrated that the chain is only as strong as the weakest link since, as I understand it, no part of the backdoor was committed to the git repository in cleartext. Instead, the part of the backdoor that was at least somewhat identifiable was only included in the tarballs that would be downloaded and used by Debian/Fedora when building the packages for these distributions, thus giving a very nice trade-off between the chance of someone detecting what was going on and the potential impact of the backdoor.

rhdunn• 2 years ago

Pay is not the only thing regarding maintainership.

Time is another factor. It takes time to maintain software, improve the codebase, add features, etc. Then there are the other tasks such as answering questions, reviewing PRs, triaging bugs and feature requests, etc.

So getting more contributors, people to assist with bugs and bug investigations, etc. is arguably more important. Especially projects developed by a single person, or a small number of people. That's the avenue that opened up this attack.

It is easy to get burned out implementing features that end up being more complex than expected, interacting with users that want different things from a project, and having a growing list of issues and PRs. That's the scenario that happened with xz, and is common with popular software that is maintained by a solo developer.

The other aspect to this is the direction the maintainer wants to take the project in. If another maintainer has a different direction in mind, that's going to cause tension.

move-on-by• 2 years ago

Collin gives his thoughts on both PRs and essentially says they just need more cross-arch testing with a focus on performance. The requester could absolutely help by doing some of those tests and providing back the results. Seems very straightforward to me what Collin is needing help with.

SuperHeavy256• 2 years ago

So the first step of this huge mess was: a social engineering attack. Attacking a tired, burnt-out open source project developer and peer pressuring him into giving more control of the repo to the attacker.

aborsy• 2 years ago

Enabled by customers who don’t pay or donate.

grumpyprole• 2 years ago

Are these customers or additional attackers who have never posted before and will never post again?

rini17• 2 years ago

On the other hand, accepting significant amounts of money causes overhead (accounting, taxation). Plus it reinforces the psychological obligation and it's not fun anymore. Thus many maintainers avoid it.

makapuf• 2 years ago

Well before that: letting a central lib and project to be maintained by a single tires burned out project dev (see xkcd2347)

fulafel2• 2 years ago

The tiredness or other problems of the developer seems an easy narrative, but did anything actually happen that wouldn't in any understaffed open source project? A contributor shows up and does work for 2 years, I feel most projects would have given the person full project developer status by then.

orbital-decay• 2 years ago

It's not like organizations writing proprietary software are magically immune to sleeper agents either. Social engineering is not a software or tech problem in general. Trust is required to get anything done, and can also be abused to hell and back by a sufficiently motivated actor.

But important software needs to be identified and proportionally more scrutinized by multiple independent parties, that's the lesson. Identification is the hard part. You can't easily determine that half of the world relies on this particular piece of software, or that it enables access to desirable targets.

jrochkind14• 2 years ago

The attacker really did useful work for 2 years, before taking over the project and injecting malware? I wasn't clear on the timeline.

That is pretty hard to defend against. I wonder if they were planning this all along, or if they wound up with a greedy plan later, or what.

resource_waste2• 2 years ago

I often debate if I should go into the hacking world, best case I get bug bounties, worst case I get rich and I contribute immoral actions.

I think its far easier to make $3,000,000 as a hacker than a worker/entrepreneur.

Its way easier to find flaws/bugs than to do the entire Capitalism thing correctly.

Then I see that half of these major attacks required social engineering.... Maybe being a hacker is significantly easier. I only need to fool 1 person, there are a lot of people, and merit isnt exactly how everyone got to their position.

Anyway point being: People are amazed by hacking, they shouldn't be, its relatively easy if you are a mere 10 year programmer. Most of us pick relatively moral work, so the number of attacks are small. It is also why we really need to treat security on computers like its no stronger than your home's front door lock. There are too many attack vectors to be perfectly safe.

AndyMcConachie• 2 years ago

People are amazed by hacking, they shouldn't be, its relatively easy if you are a mere 10 year programmer.

Between roughly 1999 and 2013 I was primarily a test engineer for networking switches/routers/telephony products. I found bugs for a living and wrote them up, and in the process I found plenty of security vulnerabilities and wrote them up. Security bugs aren't really that different from other bugs. Yet for some reason we lionize people who find security bugs.

Most security issues are simply quality issues. But by calling them security issues we shift the focus away from the software producer creating shit code to an attacker doing something bad.

The case with xz is a little different, because we have someone who intentionally added bad code and tried to hide it. But for unintentional bugs that rise to security vulnerabilities it's 99% a QA problem.

saulpw2• 2 years ago

I think your worst case is a bit off. Worst case you go to prison for a very long time?

And if you do make that $3m as a hacker, now you have to spend your time worrying about whether someone will eventually catch you because of that one time you logged into a service without using enough VPNs.

saagarjha9• 2 years ago

Security engineer here. Your viewpoint is exceptionally skewed to the point of being wildly inaccurate. Being good at security work is like any other skill and has a payout distribution more akin to being a star performing artist than a 9-to-5 job. That is, everyone hears about the people that make millions, but they’re actually missing that most people make far less than that, even when they’re good at their jobs. And in your case the people who are not good at their jobs also end up in jail. And just like learning to play music, you can get really good at it if you practice for 10 years–but most people won’t, and even then if you’re in the top 1% but not the top 0.1% the difference in pay might be several orders of magnitude. Just being a software engineer is a far more reliable way to be well off.

arp24210• 2 years ago

In the end you're only pressured as much as you allow yourself to be pressured.

"I don't feel like it, if it's important to you then feel free to fork".

That's really all that's needed. "I don't feel like it" is all the justification you need.

Some guy just made a compression tool, because some people like doing that kind of thing, or because it was useful for him. He didn't ask to be made "critical infrastructure" or to be responsible for the security of sshd or to have some business depend on it, or anything like that. No one even asked him.

gmerc• 2 years ago

When it was created, it was a different time. There was a sense of community around open source, much more tightly nit. And the more socially minded you are, the more vulnerable you are to these kind of attacks.

stavros8• 2 years ago

This ignores the very fact that peer pressure works and puts the entire blame on the victim. No, people react differently when pressured vs when not pressured. That's the entire reason why peer pressure works.

em-bee3• 2 years ago

"I don't feel like it, if it's important to you then feel free to fork".

That's really all that's needed.

that's much harder than it sounds.

having someone fork your project can give you the feeling of loosing control over the project as potentially all your users might go with the fork. that fear is often strong enough to push yourself to do things that will avoid a fork.

it's a desire for harmony and a fear of conflict

cqqxo4zV46cp5• 2 years ago

This is incredibly naive. Anyone that thinks that pressure doesn’t work is exactly who I’d personally put top of my list to try to social engineer. Everyone is human. Nobody has infinite strength against persistent pestering. Everyone is capable of finding oneself in a scenario where they feel unsolicited responsibility. All you’re saying here is that you haven’t personally experienced it.

tonnydourado• 2 years ago

The maintainer did mention mental health issues. It can be hard to say no in the best of times, let alone when your own mind is trying to screw with you.

EasyMark4• 2 years ago

This is harsh but tree, but it's not simple. People are different. Some shrug off things naturally, others try to be empathetic almost to a fault. I've learned to be the former, but when I was younger was much more the latter. I try not to act like it's easy because it took me a while to have that skill. And it is a skill and not built in to me, I still feel the urge but built up the skill of saying "no"

2OEH8eoCRo0• 2 years ago

You don't even have to say, "I don't feel like it." You don't have to say anything.

kqr4• 2 years ago

I'm curious: do you maintain a popular open source project?

The reason I ask is maybe the people who do are (in general) self-selected from the subgroup of people who don't just say "sod off" when someone is rude or inconsiderate.

If that was their stance in life, they would likely not put up with maintaining a popular open source project for very long.

WesolyKubeczek5• 2 years ago

Here’s the rub: peer pressure counts if done by peers. Users who subscribed to a mailing list only to campaign for a governance change are not peers. Longtime contributors who have earned street cred, can be.

(I guess in the case of xz, Jia Tan has earned the cred before going rogue; the one-off “maintainer needs to be replaced” campaigners, however, haven’t.)

wood_spirit2• 2 years ago

Occams razor says that those commentators were not in on it; that pressure is common on all projects, there is no need to think that they were part of an attack.

In fact, Occam’s razor says that the malicious code was injected by a compromised account and not a malicious actor who spent years steadily getting into position to attack?

acdha2• 2 years ago

I don’t think that’s a good application of Occam. Does it really seem parsimonious to think that someone who shows up with no prior history or subsequent activity is really just a random open source user who cares deeply about new maintainers for a low-level library they’re otherwise silent about?

AtlasBarfed2• 2 years ago

Here's the deliciousness:

Let's take on face value that it was the Chinese, and that China is communist. I mean, "Kumar" and "Tan"? Maybe it wasn't, but it doesn't matter for my purposes:

They took an overworked peon of the capitalist enemy that provides a ...

... do I even need to expound? well it's fun ...

... collectively and idealistically produced common operating system "for the people"

... that is exploited and neglected by the rich and powerful, to a degree that society, not just computers, overall society operates on this operating system, and the profits from that are hoovered up by the powerful.

The communists attacked the exploited proletariat to get to the enemy. And they are forcing the enemy capitalists to either pay the proletariat properly (they won't) or continue to be vulnerable to the growing communist power in the far east.

That's what this distills so well, within a political/ideological conflict that has now spanned 100 years: capitalism vs communism.

To wit, a proper functioning capitalist system to reward market value for produced value would properly pass compensation to this poor soul, and incentivize others to help him. Barring that, a proper functioning government would recognize the public good of this and provide support to the core infrastructure software to enable other private enterprise to produce tax revenue.

THOSE AREN'T HAPPENING, so the Communists can attack this with impunity and in perpetuity.

Here's the thing folks, we've been living in, as they used to say "uninteresting times". Post-WWII Pax Americana, even the Cold War was basically peace, has been cranking for 80 years now.

People, that is coming to an end:

- Russia / China are destabilizing demographically and simultaneously becoming militant and totalitarian

- Global Warming will ramp up the pressure on populations, food shortages, production

- The US will likely retreat to a more regional focus as production is onshored, regionalized (or at least centralized to our hemisphere)

There's going to be more state conflict, the Ukraine war is just the beginning. The world stakes are rising.

saagarjha9• 2 years ago

Let's take on face value that it was the Chinese, and that China is communist.

Let’s not.

feverzsj2• 2 years ago

Considering how kind(naive) our open source projects maintainers are, backdoors are probably already everywhere.

lionkor• 2 years ago

theyre much harder to add if people use a buildsystem from the last, idk, 20 years

feverzsj2• 2 years ago

Didn't Jia Tan add a single '.' in cmake script to disable the landlock?

redox99• 2 years ago

I don't think this kind of backdoor is that widespread. But intentional vulnerabilities for sure are.

WesolyKubeczek5• 2 years ago

“Community desires more”

At which point if “the community” consists of one guy doing everything and a couple whiners making demands, fuck it.

resource_waste2• 2 years ago

As someone who has a community who follows my stuff and demands more....

I somewhat get it.

I became the central spot for this particular product. I get occasional emails from people saying the did this same thing, but on a smaller scale. I have scale, so I can do things much bigger than anyone else.

I have a million people using this, vs a startup who has ~100.

(Btw, I'm barely profitable, I was selling to frugal people...)

kijin3• 2 years ago

Well, in this case it's even worse. The "community" here in all likelihood was overrun with sock puppets of a malicious state-level actor.

WesolyKubeczek5• 2 years ago

I also don’t really understand the notion of a community around a fucking data compression program. “Bro, do you even compress?”

Is there a community of ½" plumbing pipe enthusiasts?

keybored11• 2 years ago

I by default distrust any statement that claims to speak on behalf of some community.

noobermin4• 2 years ago

This line in light of what we know now makes me think that while OP has a point about open source, this thread and others like it definitely were sockpuppets trying to goad Collin into transferring maintainership quickly.

sumanthvepa• 2 years ago

The Jigar Kumar account should be treated with extreme suspicion. It seems like it was part of a social engineering attack.

armada6512• 2 years ago

Given how all participants in the discussion were very focused on transferring ownership to the attacker I'd treat everyone in the entire e-mail thread with suspicion. I would not be surprised if Dennis Ens, the one who started the thread in the first place, was also a sock puppet of Jia Tan.

WesolyKubeczek5• 2 years ago

The word that springs to mind is “campaigning”. If persons campaigning are nobodies as their contribution is concerned, whatever they are pushing for should be considered noise and put straight into the bin.

WhyNotHugo• 2 years ago

I don’t agree with speculating like this, but I also can help wonder how many participants in this thread are the same person.

kimixa• 2 years ago

I do sometimes wonder if by trying to be "nice" to users and try to see the best intentions of commenters, many developers waste huge amounts of mental energy.

For context, I've really only worked on "fun" side projects, namely emulators and game remakes, where I've explicitly avoided any mention of donations or similar. Both as it's intended to be a distraction from my job, not become part of it. And generally avoid as many issues as possible around said money distribution within the project, and possible copyright problems that such things often skirt.

And those sort of projects tend to be extremely limited in progress not by total interest, but skilled interest with the ability to contribute. I wish as many as 1% of the regular discussion participants in the community could contribute to that level.

There's a natural belief that unless someone is egregiously out of line, all discussion around a project from users is allowed, even encouraged.

But many people just can't seem to help themselves making "suggestions" (or, another reading, "demands"), and that can have a significant impact on motivation of volunteers. I believe the vast majority are from based in good intentions, wanting a "better" result, but arguably are mostly self-defeating in that I find the discussion around much of these things to be draining and demotivating. I want to have "fun" coding game remakes, not defusing drama in discord threads.

But it's expected to have a "community" for all such projects, and not exclude non-direct contributors. Even writing it down here I'm worried I sound like I'm trying to curate a closed community of ego-boosting yes men.

But I sometimes wonder if allowing this sort of open community is better in the long run.

TeMPOraL• 2 years ago

Even writing it down here I'm worried I sound like I'm trying to curate a closed community of ego-boosting yes men.

There's this weird trend that's been happening for some time now that tries to make non-fully-open groups look wrong, but honestly, has anything ever actually been done by such open groups? As far as I can tell, "closed community" is a necessary (but not sufficient) condition for any kind of quality creative output (this includes individual projects as single-person closed group). Having a core of creators surrounded by distinct group of fans and secondary contributors is a natural organization that forms spontaneously.

lukan3• 2 years ago

I think the linux kernel developement is quite open. But Linus is famous for telling people directly in not so nice words, that they are not helping.

I don't think insults are the solution, but giving a clear no, is a skill many people struggle with. And if some cannot cope with that, blocking individuals also works.

aumerle2• 2 years ago

You dont need to wonder. Any long term maintainer of even semi popular open source projects will tell you that engaging with the peanut gallery is completely counter productive.

Engage with people that have earned it in your eyes, whether by contributing to your project via code, assets, bug triage, writing a good and effortful bug report, whatever. Just ignore what the larger internet has to say about you and your creations.

graemep2• 2 years ago

I think this is a wider social problem of the internet.

Most people want to be helpful to others. This works fine when it is with a limited no of in real life interactions with mostly reasonable people. It can get wrecked by one or two unreasonable people, but the more fundamental problem is that it does not scale well, and the internet takes it to a far greater scale.

It is one reason so many of us essentially work for free for FB and the like: we do things to help others (advice and information), and the side effect is creating content and data for FB.

ninkendo2• 2 years ago

What form should this ignoring take, in your eyes? I’ve never maintained an open source project but I’d imagine this is the hard part, how to politely decline the peanut gallery’s feedback. Do you disable GH issues? Leave them open and ignore them? Decline with some boilerplate language? How do you stop people from being mad that you’re ignoring them? (IME these types of people are likely to take things personally and start harassing you or complaining loudly in other forums…) These are the kind of things that stress me out just thinking about it, if a project of mine actually got popular.

supriyo-biswas6• 2 years ago

The points you make aren't unreasonable.

It is necessary to establish clear boundaries of what can and can't be provided by the maintainers. If not done at an earlier stage of the project, the support burden becomes too much to bear at which point the maintainer transfers ownership, and the project suffers from catastrophic consequences such as the xz backdoor we're talking about here, or other cases where the project mostly stalls and serves as an ego-boosting platform for the new maintainer, as was the case with PhantomJS[6] before it was shut down.

This can also happen in your life, where a "friend" sees that you possess a certain skill, and then gradually tries to push an inordinate amount of their personal work related to this field onto you.

Personally, I think it's best to use an approach with extremely clear communication as to what the maintainer can and cannot provide. This can be seen, for example, in yt-dlp[1], where the consumer is clearly informed upfront that not providing detailed information as requested will lead them to block said consumer; or sqlite where their position regarding contributed patches[2] and support[3] is similarly made clear.

Having a shouty BDFL like Torvalds can also help improve code quality[4] and questionable contributions[5], though it is better that the shouty BDFL makes statements that are professional and do not show as much aggression; so for example, "Mauro, shut the fuck up"[7] would become "Mauro, your response is completely unbecoming for a Linux kernel maintainer, and is not in line with the promise of not breaking userspace."

[1] https://github.com/yt-dlp/yt-dlp/issues/new?assignees=&label...

[2] https://www.sqlite.org/copyright.html

[3] https://www.sqlite.org/support.html

[4] https://www.theregister.com/2024/01/29/linux_6_8_rc2/

[5] https://cse.umn.edu/cs/linux-incident

[6] https://github.com/ariya/phantomjs/issues/14541

[7] https://lkml.org/lkml/2012/12/23/75

apantel5• 2 years ago

"Mauro, your response is completely unbecoming for a Linux kernel maintainer, and is not in line with the promise of not breaking userspace."

This is just PC corporate speak. Linus is an actual human being who says what he really means.

matheusmoreira3• 2 years ago

That LKML episode always makes me feel pretty bad. Mauro Carvalho Chehab maintains the ZBar project now. I briefly worked with him when I sent some patches in for a feature I wanted. There were things I had to address that I hadn't even considered and he worked with me all the way, ended up learning quite a bit about DBus. He's an awesome maintainer and set the bar for me for what qualifies as a great experience in contributing to free and open source projects.

traverseda• 2 years ago

There's a natural belief that unless someone is egregiously out of line, all discussion around a project from users is allowed, even encouraged.

I mean sure, but how do you intend to stop people? You can't stop them from setting up a forum somewhere.

This is why github discussions is useful though. A feature request in a ticket is something that needs to be handled. A feature request in a discussion is just some users talking about stuff they think would be cool.

c_joly• 2 years ago

Thanks for writing this! Your comment and the article have prompted me to explicit the social contract for my Open-Source project: https://github.com/cljoly/.github/pull/4/files#diff-eca12c0a...

PaulRobinson• 2 years ago

I've been thinking about contributing to some open source projects, and starting a couple of things of my own, and the community aspect is a significant downside for me, and I say that as somebody who was quite active in several open source projects and things like the IETF years ago.

I'm not saying the community shouldn't be there, it's just that the barrier to entry is so low, that people can "contribute" with very little effort, that's become a significant problem as Internet usage (and acceptance of shit-posting trolls as just being a cost of anything online), so the signal/noise ratio is awful and we end up in an easy slide to controversy and spite and the good actors wanting to walk away.

I think this is why I don't do social media any more. I think it's why I'm not on the mailing lists I once was, or have interest in discord or a great many open community sites. It's just exhausting. HN, slashdot and some group chats in Signal with people I have known for many years is the closest I get to anything like that any more.

I've thought about doing some blogging without comments and a means to contact me directly another way privately. I've thought about making contributions to other projects with my only conversations being about my own commits, but that doesn't work for leading my own projects if I want to be "nice".

I think leading a project for me might involve _not_ letting others contribute to my version without clearing some arbitrary bar. No mailing list, no discord, no developer community. That doesn't feel optimal on some metrics, but it might be optimal for the metrics I care about.

rrr_oh_man4• 2 years ago

I think this is why I don't do social media any more

Guess what you’re doing here, buddy.

bregma• 2 years ago

Thank, you for participating in the open source community by sharing your opinion. Your contribution is valuable but not consistent with the direction we are currently going in at this time. Please don't let this stop you for further participation.

apantel5• 2 years ago

There are no stupid contributions, just stupid people.

figassis3• 2 years ago

Usually, I try to use projects in languages I work with so if the issue is not important enough for me to make a PR, then it’s not important enough. One thing I wish were possible is to sponsor an issue. I don’t usually sponsor open source projects, but I would likely sponsor a lot of issues just to speed up resolution. I think this could move a lot of projects in the right direction. Depending on difficulty, the maintainer could define a budget to have it in the roadmap with a reasonable timeline defined by them, maybe measured in quarters.

withinboredom• 2 years ago

As long as that money is held in escrow. Otherwise vaporware would make a killing.

johannes1234321• 2 years ago

An issue is that the maintainer may have other priorities. By sponsoring an item you "force" them to look at that sooner.

cqqxo4zV46cp5• 2 years ago

My layperson’s opinion is that anything to do with gaming is particularly riddled with people that interact poorly with maintainers. Again, purely my opinion: gamer culture invites a particular sort of Dunning-Kruger-prone ‘power user’ type. Every gamer community carries with it a corpus of baseless, fictitious, technical information. “The developers didn’t do this because x”, “it’s ridiculous that they didn’t just y”, “They just aren’t listening to users!”

I’m not much of a gamer at all. But the once in a blue moon where I dip my toes in, it’s not that long before I’m met with a forum post or reddit comment making some technical assertion that sounds incredibly presumptuous if not downright incredibly unlikely.

supriyo-biswas6• 2 years ago

You need to look no further than what was happening on the xz commit threads post disclosure; there were multiple people preaching as to how to do "real" software development.

anotherhue• 2 years ago

The gamers seem to have particularly short fuses. It's like they skipped the socialisation phase of childhood development.

TillE• 2 years ago

My favorite know-nothing gamer meme these days is "bad optimization". What they mean is that the performance is worse than what they would expect (?) on a given hardware configuration and game settings.

They haven't poked around with RenderDoc, they don't have the debug symbols to profile the CPU code, they have zero idea what kind of work the software needs to do, they simply know the lazy devs have failed to "optimize" their game.

fireattack• 2 years ago

I think what you said is true but the major reason is simpler: there are way more gamers than developers.

So if your project's audience is gamers, of course it's going to attract more people which also means lower quality community almost automatically.

tb_technical• 2 years ago

Games are meant to be accessible to the public (in order to make the most money) - as in, easy to obtain, install, configure, and use. When you interact with the public, you're going to get all kinds of people. Even the crazy awful ones.

As for reddit and forum posts? The people willing to make an account to just to complain online are the most motivated of the negative people.

jasoneckert• 2 years ago

I'm always surprised at how much developers and maintainers will willingly put up with. For example, if you visit the Matrix feed for the Asahi Linux project, you'll see hoards of trolls and time-wasters that are regularly engaged by Asahi team members who are giving them the benefit of the doubt, when they really should be removing those posts without comment or acknowledgement.

I believe this masochistic behaviour stems from a combination of both general optimism and dedication to a particular cause. The more that developers are passionate about what they are doing, the more they are willing to engage with the peanut gallery.

kabes• 2 years ago

I recently discovered a bug in the react-hooks-form library.

The author there does the absolute minimum of interaction.

Issues not following exactly the template or lack a repro gets closed immediately without any comment, actual bugs also see very limited comment or openness to discussion.

It felt a bit weird at first, but I do get the author and the library is succesful, maybe exactly because of this approach.

It made me reconsider some of the effort I spend in my open source projects with useless issues / comments.

eternityforest• 2 years ago

For me it's easy to get very excited about any attention or interest at all as meaning "This project is finally worth something and all this effort wasn't wasted!" and start thinking "the customer is always right!"

forgotmyinfo4• 2 years ago

Boomer hat: it's just toxic positivity, and the frustrating trend lately of assuming that everyone is equally skilled when writing software. Everyone's input is valid, or else you're just being negative and overly critical.

I'm not saying everyone need to be Linus Torvalds circa 2012, but I do think more people need to be a bit less precious and sensitive, especially when receiving direct communication about their abilities.

You don't need ego-boosting yes men, you just need to work with more experienced folks, and that's okay. But the problem is that there is an army of people online who will take offense to that, and I don't know what the solution is. Best of luck.

lamontcg2• 2 years ago

Another old person opinion: the issue tracker of an open source project needs to be firmly anchored into the needs of the maintainers and not the users. The argument that bugs/issues need to be kept open forever unless they're fixed and the rejection of WONTFIX as a reason for closing them needs to die in a goddamn fucking fire. When the maintainer decides to close an issue because they're just not getting to it in this lifetime the users need to suck it up.

AtlasBarfed2• 2 years ago

Apologies in advance for the swear words, but this deserves swear words.

WHO FUCKING GRANDSTANDS ENDLESSLY ABOUT CYBERTHREATS AND THE NEED FOR BETTER SECURITY?

WHO FUCKING SITS ON BILLIONS IF NOT TRILLIONS OF DOLLARS IN FUNDING?

That would be the FUCKING GOVERNMENTS OF THE FIRST WORLD. Whose modern economies increasingly rest on open source to a degree they possibly don't understand.

Yes I understand part of this comes from funding from another part of these governments. Oh well, cat and mouse. Once we wrung our hands over these governments having power over these. Well, these days we are facing far more totalitarian state threats and totalitarian wannabes in the private sector.

Government save me, as the less totalitarian option!

Linux and BSD the operating systems should UNQUESTIONABLY be funded to the degree of multiple billions per year by, I'll just put it, "NATO". Here's the true value of open source in this model: it is the perfect public record, unlike untold other aspects of government output.

You must produce public vetted code in Linux/BSD.

This person should not have been "a person". This should have been 5 people, funded likely at least 10-50k/year for their roles.

The described "hit" by the intel services of course is frighteningly easy. But what is most horrid is that this poor person is being subject to very common abuse that comes from non-state-actor manipulation. The article says it:

"this is where our software comes from" -- this poor dude that is trying to help and being abused by state actors on one hand, and ridden thanklessly and for no monetary or fame benefit by massively deep pocketed corporations, governments, and billionaires.

For all of Linus's famous swearing abilities, he hasn't used it to address this publicly. He should be dressing down all corporations and governments at this point that need Linux. There isn't a "going back" from Linux at this point to some closed source option.

Linus and Linux have massive sway here. They could threaten to stop work on the kernels.

I guess fundamentally, this makes me angry because this is basically bullying the nerd in high school to get his homework/answers. The nerds need to realize their power here.

Other vulnerabilities snuck through the commit chain? I get that, usual spy stuff. They actually kind of messed up here. Usually it is within the "cordiality" of things, just quietly ride the overworked unpaid nerd to benefit. Here they inadvertently exposed the real truth of everything: the hidden contempt we treat these people with as a society.

InvaderFizz• 2 years ago

With a personal repo I maintain, I once had someone look me up on linked in, and then file a support ticket with my employer demanding I reply to his issue on github. He was asking for handholding following a clearly documented process and ignored all the warnings.

Needless to say, I did not assist him in any way and closed the issue as wontfix.

I've also had people demand that I expand the narrow scope of my project to cover a whole class of devices with completely different APIs, instead of the sole focus on the series from the same manufacturer of devices I own and use. Closed as wontfix as well with a clear statement that I will not be expanding the scope, but am more than happy to link to their project covering it.

atoav• 2 years ago

This is fundamentally a problem of perspective. As the guy using software X you believe you are doing open source maintainers a favour by going there and telling them about a thing that is wrong or missing in your opinion.

But we all should realize that any interaction also comes with a cost. If you report a problem and don't manage to describe it well it doesn't help the maintainers one tiny bit, in fact it might just create work and a bad feeling.

So my approach is to "just" write stellar bug reports including ways of bisecting where the problem might come from.

account427• 2 years ago

But it's expected to have a "community" for all such projects, and not exclude non-direct contributors.

By whom? You don't have to accept such expecteations - plenty of projects that just throw code over the wall and do fine.

slavik812• 2 years ago

I'm starting to feel that one of the lessons here is that individuals invited into trusted positions should be identifiable. Jia Tan is not a real person. We don't know who they are, so there is no way to hold them accountable.

armada6512• 2 years ago

As much as I support the privacy of maintainers I do feel like this is necessary to raise the bar for these types of social engineering attacks.

However that would've likely done little to impede this attack if it is backed by organized crime or by a state as is being speculated. It is trivially easy for those types of actors to simply use a stolen identity or create an entirely plausible one out of thin air.

sedatk• 2 years ago

Such identity checks would also hurt honest contributors who want to hide their identity for other, legitimate reasons.

avsteele• 2 years ago

It is trivially easy for those types of actors to simply use a stolen identity or create an entirely plausible one out of thin air.

Can you provide a historical example of when something like that has happened?

It sounds like hyperbole to say that it is 'easy' (even for a state) to impersonate a real-life identity or create one for a professional developer with a multi-year work history.

fulafel2• 2 years ago

Reputation works with pseudonymous identities too, and those have security upsides (eg can't be as easily pressed into service of others by extortion or rubber hose). And of course privacy is a value in itself.

dist-epoch• 2 years ago

You can build multiple pseudonymous identities in parallel. If one is burned, it doesn't matter, you still have 9 other.

solatic• 2 years ago

Think of it like politics: politicians are public figures who, by virtue of becoming a public figure, give up many of their rights to privacy. Why? Because they have power, and the public has a right to know who exercises that power.

For security-hardened distributions, I can easily imagine a security control where contributor identity must be public and publicly verifiable, and to reject code which cannot be reliably attributed. Don't like it? Contribute to software with less impact instead.

teddyh• 2 years ago

No. <https://geekfeminism.fandom.com/wiki/Who_is_harmed_by_a_%22R...>

slavik812• 2 years ago

I despise real name policies on social networks, but it's a very different thing to be anonymous while in a position of public trust.

arp24210• 2 years ago

While all of this is of course quite serious, we also need to remember that these types of things are actually fairly rare. Last major one was that JS event-stream thing, and that was in 2018 (5 and a half years ago).

I don't think this is really a structural problem requiring these kind of sweeping changes; it's just an occasional rare incident.

secondcoming2• 2 years ago

No, the problem is that we don’t know how common it is.

We only know about the ones found.

qsi• 2 years ago

As far as we know, they're rare...

AtNightWeCode• 2 years ago

Humans can always be compromised. This specific case can only be solved by improved tooling, workflows, visibility, runtime environments and so on.

MASNeo• 2 years ago

Sadly, this is how it works in many large social systems, left unchecked. The “reasonable consumers” drain the kind contributors until they cannot do this anymore. The curse of kindness strikes in software and other industries alike.

goku12• 2 years ago

This is all due to the ongoing attempt to convert FOSS into an obligation. The original idea behind FOSS was to code to scratch your itch and then leave it out there for anyone to find, modify and use. Instead we now treat FOSS code like commercial projects with performance goals and expectations.

Why? My guess is that this new culture is promoted to extract free labor from hobby developers. Take for example, Github. It has a bot to close stale issues after a given interval of inactivity. But what if the issue was valid and unresolved? What's the problem in leaving it open for someone interested to take it up? But what GH wants to promote is a culture where open issues are considered as bad performance on the maintainer's side. It indirectly prompts the maintainers to take open issues too seriously.

kqr4• 2 years ago

Why?

Well, also a lot of people get into FOSS for altruistic reasons. They want to pay it forward and make the world a better place.

For these people, applying good operations principles allows them to more efficiently do good -- just like it allows a commercial company to deliver more end-user value for lower cost.

Running an efficient operation in FOSS isn't necessarily bad. Sometimes it takes an obligation viewpoint to get there. It can be unhealthy, but it can also give us crazy levels of neat software, as we see proof of daily.

keybored11• 2 years ago

I agree that the maintainer doesn’t really owe anything to anyone. Including the so-called consumers on the issue tracker. Furthermore the consumers have no real power over the maintainer; since they aren’t doing anything for them they in turn have nothing to withhold from the maintainer. And they can’t bother the maintainer unless they decide to turn truly vile and evil by doxing and harassing the maintainer. Thus the complaints of these consumers can be completely tuned out.

It follows then that the maintainer can just log off. Because either you believe that the maintainer has some minimal obligation to the community (like a quarterly update about maintainership status or making a notice in the readme if you decide to abandon the project) or you don’t.

But sometimes in these discussions things get muddied because people claim that

1. The maintainer doesn’t owe anything to anyone whatever

2. But she’s a nice gal which means that she will triage some bugs and fix some bugs and explain to a few of the issue reporters that they haven’t encountered a bug they just haven’t installed the program so that PowerShell can find it and

What’s the groans moral angle, here? We’ve already established (1). But what ought the maintainer do for herself? That’s also a moral problem. Are you willing to say that the maintainer ought to log off if they are experiencing burnout? (Again: this is about the moral obligations that the maintainer has to herself only.) Or does that trample on her rights?

And if you are unwilling to say that the maintainer ought to log off, what does that make the maintainer? Are you going to claim that they have (a) started an altruistic hobby by themselves (b) got too caught up in it and (c) are now a victim of circumstance/outside forces because they have no moral obligation to log off? Considering how much power the maintainer has (and how the consumers have none), how is the maintainer a victim of circumstance when the whole enterprise was created by them and can be terminated at will by them?

lifeisstillgood5• 2 years ago

>> Software developers are not fungible cogs that you can swap in and out at will.

I am thinking a lot about this. One of the issues is scale and proof. I suspect that I am interested in introducing gated ability to comment / participate in a community

Say for example github introduces “gates”. The first might be add a test to the test suite that generates a haha of the version number and test output. Then adding that number to your profile means github trusts you have at least downloaded and run make test. It shows some level of commitment. (I suppose the zero level is logging in to github and commenting on some maintainers mental health)

Kwpolska• 2 years ago

What would that magic number achieve? Having basic coding ability can be checked by looking at the user's repos and contributions. But the primary attacker had pretty good skills, and the other commenters (the attacker's sockpuppets) could also demonstrate some skills.

klabb32• 2 years ago

Are you suggesting to ask a state actor to demonstrate commitment before participating in the discussion? :)

I mean, the idea of commitment -> privilege isn’t bad. But it’s got nothing to offer in the story of xz.

dwb6• 2 years ago

I'm under no illusions that this is a totally new thought, but for me first with cryptocurrencies, then "AI", and now this, the fundamental issue that the biggest problems come back to is one of trust. Cryptocurrencies try to code around it, LLM boosters try to dazzle you into it, and the attacker here half-succeeded in laundering it. The most consequential (rightly or wrongly) technologists of our time are failing to properly think about trust. In this case, the failure was 100% understandable – I have all the sympathy for burnt-out and (almost always) unpaid open source developers. In each case, capital either lures people away from thinking about trust, or neglects and exploits people to the point that they're broken down.

zvmaz• 2 years ago

The most consequential (rightly or wrongly) technologists of our time are failing to properly think about trust.

It is implicit in your statement that you know some "proper" way to think about trust. If it is the case, please do share with us your thoughts on trust.

cassianoleal4• 2 years ago

I didn't read it that way. I read it as in "actually giving it thought" as opposed to just barely acknowledging it or even at all.

So, not so much a judgement of the quality of the thought but rather of its existence in the first place.

o11c5• 2 years ago

Trust is a lot easier if real-life identities and real-life consequences are involved.

Unfortunately the Open Source community has a long history of ties to anonymity culture, which is fundamentally untrustworthy.

lifeisstillgood5• 2 years ago

So I see software as a form of literacy. And we treat some literacy as “fun” (most books) yet privilege some literacy (law, a DMV policy manual).

Laws are highly regulated. The DMV policy manual is, like most government day to day policy, run through “managers” who report to a elected official (ie it can be very well written and considered or made up on the good after a bad newspaper headline)

Anyway I think I am saying there is likely to be a consolidation of FOSS - where we spend a fortune trying to bring it under some “approved and controlled” process.

We can call that trust.

I am not sure.

concordDance• 2 years ago

The most consequential (rightly or wrongly) technologists of our time are failing to properly think about trust.

Because that's an insanely hard problem that's outside our area of expertise.

Consider how much money governments spend on all the red tape they add to increase trust. If there was naturally perfect objective alignment and trust I bet any infrastructure project would cost about 10% of what it does.

dwb6• 2 years ago

It is hard, but I think corporate open source users and entrepreneurial software developers have a moral responsibility to do more than I see being done in the areas that I listed. To be crystal clear, in this case where I don't think Lasse is at all culpable, the responsibility would be with the companies that profit from the work done on xz-utils.

someguydave4• 2 years ago

A few minutes reflection on “who can I trust” would lead one to discard the extremely egalitarian ethos built up around open source software. But casting out meritless ankle-biters will immediately lead to being attacked as an unreconstructed egotist who needs to be hung from the neareat Code Of Conduct

dwb6• 2 years ago

I have never seen a code of conduct that would prevent a polite rejection of whatever new suggestion or help to a project.

account427• 2 years ago

I think this is an interesting line of thought but think we should take it further. These codes of conduct that focus of the feelings of third parties over the good of the project (the code) can themselves be seen as an attack on the project. They also often are imposed on projects via peer pressure and other social engineering methods. Perhaps we can use these codes of conduct and the people that push them as open examples of how projects can be infiltrated by those with different agendas and learn from them how to better defend against such attacks.

twelfthnight• 2 years ago

I think the biggest problems come from single points of failure. This wouldn't be as problematic if _everyone_ didn't use the same libraries. I guess you could argue that's similar to trust.

In any case, I agree technologist aren't thinking about it because the only way to extract massive amount of wealth is by centralization/monopoly. Hard problem though, the appropriate amount of redundancy is hard to know.

lifeisstillgood5• 2 years ago

Some random thoughts:

- every Fortune 500 company tracks exactly which FOSS code it includes in its ecosystem (usually code scanning and fingerprinting - can’t remember the usual Provider of such)

- this is essentially the software BOM that Biden signed a while back.

- this (made public) would give a real time map of the dependancies of all organisations - and linking that to things like the above thread (“cry for help”) would be an interesting place for “intervention” - anything from plain old cash to “here are three interns doing two years in gov.uk. They will help for the next 2 years

(It’s not a great idea that last one but we need to start somewhere- no way can government “pick winners” but also no way can society just sit back and hope. And commercial incentives break this horribly.

Essentially we are going to find at some point we treat some developers like lawyers - this stuff is our societies laws, rules, processes

kzrdude• 2 years ago

“here are three interns doing two years in gov.uk. They will help for the next 2 years

Having temp workers come and go into all kinds of various open source projects.. does that help? :)

lifeisstillgood5• 2 years ago

Does two years count as temp? What if one in five stays on for “life”

I am not claiming I have perfect solutions, just that we need to be more mindful of our supply chains. And when we are we start demanding things of those supply chains - and if we are sensible we support those suppliers

klabb32• 2 years ago

Agreed it’s a decent way to create a “map” of vulnerable projects. But it’s over-fitting the MO of adversaries - plugging an arbitrary hole.

Why not play into the strengths of OSS instead of trying to replicate frictionous trust models from elsewhere?

Like, listen to ourselves. We are admitting here that open source software is not feasible to audit? Like did we just accept that an adversary can basically smuggle exploit payloads in plain sight?

Can we at least try to simplify and reduce the messiness of checked in generated code, irreproducible builds, and magical binary blobs, conditional compilation, “ifuncs”? Wtf? Granted I’m not a domain expert in this area, but I’m horrified by all the complexity for what I understand to be a compression library.

I would much rather play whack-a-mole with obscurity, which as a side effect is great for software quality overall.

lifeisstillgood5• 2 years ago

I have to admit that is a good point - myself I think there is some other goal here - I am trying to build reproducible workstations and servers (nix os). And until right now I have not even considered can I put all the source code I am sitting on in one place.

God knows if I could. It that has to be the first place to start

xenocratus• 2 years ago

As a maintainer of a security-oriented open source library, the paranoia of "is this person trying to help or to exploit?" weighs down on me every time I have to read a PR (EDIT: even though the libraries are by no means as widely used), regardless of whether it's from a long-time contributor or someone new. I think accepting a slower pace of development is the only viable solution (as I'm not interested in making these libraries my full-time job), but that comes with the same feelings reflected in the article listed.

If there was a simple way of advertising the need for help to a community of experts that I/we could trust, I'd take that any day.

barcist3• 2 years ago

As a maintainer of a security-oriented open source library, the paranoia of "is this person trying to help or to exploit?"

That's an excellent mind set when reviewing code, no matter security or not. But especially for security. How could this be wrong? What are the corner cases? How could anyboy break this? What do we need to test? That kind of scrutiny is crucial for keeping the quality of your code base high, no matter who posts the PR.

saulpw2• 2 years ago

Yes but that's a huge amount of work, and is not the 'fun' part of coding. It's one thing to have to spend those cycles looking for errors or incompetence, and hopefully helping the contributor improve their skills so they can in turn help the project more in the future. It's quite another to spend even more energy being suspicious and looking for subterfuge from someone that has been one of your most helpful contributors.

zb3• 2 years ago

Could we somehow search for such takeovers done in the past?

tkiolp4• 2 years ago

If I were a chinese hacker trying to do something evil, why on earth would I use a chinese handler/username? Wouldn’t it be better to use an English/European name to gain (even more) trust from open source maintainers?

On the other hand, if I were a non-chinese hacker trying to do evil, then using a chinese handler does make more sense (China is evil, blah blah blah)

yunohn5• 2 years ago

100% this. I don’t know why this obvious angle is being ignored. The other puppet accounts had Indian and German names - it’s very clear they were trying to obscure their real origins.

michaellee88• 2 years ago

The hacker's name "Jia Cheong Tan" actually includes phonics from both variants of Chinese from China, Singpore and Hong Kong region, hence no way it is a name of a real person, so the name probably don't indicate anything. However, since the Hong Kong and Singapore variants is rarely used, the group behind the hacker probably has good knowledge of Chinese linguistics. Either way, the fact that the hacker's account shows GMT+8 and that the hacker's activity is mostly in Southeast Asia working hours (read GMT+8) would only make sense if the hacker is originated from Asia countries. I doubt that group of people behind the hack can consistently stay late night for 2+ years just to fake the timezone.

damagednoob• 2 years ago

Something I've observed as an English speaking immigrant is that in general, the US/UK is very forgiving when it comes to foreigners trying to speak English. Where certain types of phrasing and harsh words would not be tolerated with first language English speakers, the benefit of the doubt is given to 2nd/3rd language speakers because they

1. Might not have the vocabulary to express themselves correctly

2. Not understand the nuance and connotations that their word choices imply

I've seen my colleagues get frustrated in meetings while simulatenously having to assume good intent.

Choosing names like the bad actor(s) did gives them an advantage in this scenario.

secondcoming2• 2 years ago

The English used in the commit messages I’ve seen was pretty much perfect, but unfortunately the repo has been suspended now.

cqqxo4zV46cp5• 2 years ago

At the end of the day this is such low hanging fruit that I don’t think that it’s even worth paying attention to. This was almost certainly a state actor and this detail would be blindingly obvious to all state actors capable of pulling this off. You could choose to try to dissect the 10 dimensional game of chess involved in picking fake names, or you could focus your attention on more meaningful heuristics.

graemep2• 2 years ago

According to a comment on another thread it is not a mainland Chinese name, so that might still be chosen to mislead in a more subtle way. https://news.ycombinator.com/item?id=39869047

Western and Chinese are not the only possibilities, it could be a Russian, or Indian or pretty much anywhere else that engages in intelligence gathering operation.

AnonymousPlanet2• 2 years ago

Since we're reading tea leaves:

They also used a sock puppet with a seemingly German name (Hans Jansen). However "Hans" has not been a popular baby name in German speaking countries for many decades. You'd expect any "Hans" to be over 70 or 80 by now. Unless they are American, like Hans Niemann.

This could be a cultural oversight on the attacker's side, which points towards any culture that has a wide spread belief that typical German names are Hans or Fritz. From my experience, that is especially true for English speaking countries, mainly because of stereotypes displayed in Hollywood WWII movies. I wouldn't be surprised if cultural perception of Germany was different in China.

Edit: some clarifications

ginko• 2 years ago

They also used a sock puppet with a seemingly German name (Hans Jansen). However "Hans" has not been a popular baby name in German speaking countries for many decades. You'd expect any "Hans" to be over 70 or 80 by now. Unless they are American, like Hans Niemann.

Jansen is a Danish/Norwegian surname and Hans is still fairly popular in Denmark.

https://www.dst.dk/en/Statistik/emner/borgere/navne/navne-i-...

xtracto• 2 years ago

What? I know at least 2 Hans who are native german, and are no older than 40. And I lived in Germany only for 4 years.

silverquiet• 2 years ago

Seeing an Asian (any part of Asia) name wouldn’t give me any pause in the slightest. I don’t want to start anything here, but aren’t they disproportionately represented in the American software industry as compared to their percentage of the population?

What would raise my hackles would be someone with a name that codes western who had grammatical idiosyncrasies common to ESL Asians.

noobermin4• 2 years ago

So, a couple of thoughts, this likely be chosen in the event that they wouldn't be caught, not that they were. Thus, them choosing a Chinese sounding name to frame China or to go along with the narrative of China being the evil communist country (etc) doesn't really make sense. Also, even if they did that, only westerners would be fooled as others have pointed out "Tan" is a dialect surname (hokkien may be), pointing to someone in Southeast Asia, not mainland China.

A better assumption imo is they chose their identity first and foremost with the intent to garner trust first with Lasse Collin.

kqr4• 2 years ago

If I were a chinese hacker trying to do something evil, why on earth would I use a chinese handler/username?

1) Why not?

2) You'd likely want a relatively common-sounding name, I suppose. Chinese names are relatively common-sounding.

nuc1e0n2• 2 years ago

My take away from this is that people are still far too blasé about introducing hard dependencies and complexity, even after the left-pad incident as a warning. OpenSSH is a massive wall of code. Such complex systems are inherently untrustworthy to me, no matter what language they might be written in. Even with ernest devs there are still more opportunities for mistakes.

lrvick6• 2 years ago

Agreed, which is why IMO ssh in production is a terrible idea that reveals even worse problems.

A server with ssh generally implies there is a shell, and cli tools, and an administration model that involves humans connecting to servers to manually update them in place like pets.

Having a full workstation-optimized distro like ubuntu or debian with hundreds of packages constantly shifting and updating as a critical production server is wildly high risk in terms of both reliability and security.

I understand this is how most sysadmins were taught but it is a 70s unix mainframe mindset from a time before security was a thing and everyone on the internet was a good actor.

Only a workstation or dev server should have tools for humans like ssh installed.

Production servers should be hardened immutable appliance kernels with read only root filesystems that verify and run signed containers, or run a tiny shim init system in a couple hundred lines that spawns a single application specific binary you trust.

No production systems I launch today even have xz installed, or a package manager, or a shell.

ris2• 2 years ago

Production servers should be hardened immutable appliance kernels with read only root filesystems that verify and run signed containers, or run a tiny shim init system in a couple hundred lines that spawns a single application specific binary you trust.

And then you need to debug something. What do?

noobermin4• 2 years ago

This wasn't a vulnerability in openssh though, this was from systemd, where I think your point in fact is stronger.

ninkendo2• 2 years ago

It’s not a vulnerability in systemd either, it’s a vulnerability in xz, which libsystemd depends on, and libsystemd is only integrated into OpenSSH in certain distributions, and only to avoid writing ~5 lines of C code to write the string “READY=1” to a socket specified in an environment variable.

Systemd itself doesn’t even use lzma/xz… libsystemd does, and that’s a library meant to help other software integrate with systemd. It’s not really the same thing.

rafpin2• 2 years ago

Would ssh servers with port knocking set up be safe from this backdoor?

I'm not sure I got it correctly, but seems the RCE can only be performed after connecting to the ssh server, but if the port is hidden behind a reasonable sequence of tcp/udp knocks, then it won't happen?

I've been using port knocking on ssh servers, and it definitely does not replace proper ssh configuration, but so far seems like a cheap extra layer of defense that might either prevent or give extra time to respond when these ssh vulnerabilities appear.

lrvick6• 2 years ago

I do not use or recommend security through obscurity.

Such things can always be automatically discovered.

https://github.com/eliemoutran/KnockIt

Any effort spent on setups like this is likely better spent removing the need for having ssh at all by moving to immutable appliance distros be they specialized like homeassistant or general purpose like TalosOS.

est• 2 years ago

security through obscurity

This is not obscurity, as someone said on HN earlier, good security is good armor, good port knocking is camouflage. They are both important.

rafpin2• 2 years ago

Such things can always be automatically discovered.

Wouldn't fail2ban + reasonably sized (10+ ports using a mix of udp and tcp) knock sequence prevent brute force attacks like these?

nebulous1• 2 years ago

Well, it appears that it would be as safe as any other vulnerable server behind port knocking. Although there's still the possibility that the backdoor is doing something we aren't aware of yet. So yeah, it's as "safe" as you consider port knocking to be.

That said, just make sure you aren't using the backdoored versions. I don't see how you can reasonably skip this step.

jmclnx• 2 years ago

This is another interesting link about the issue:

https://lcamtuf.substack.com/p/technologist-vs-spy-the-xz-ba...

It mentions something not being reported, the issue is with a Linux specific patch to OpenSSH. If the patch did not exist there would be no issue.

This is not an OpenSSH issue but a Linux issue.

iforgotpassword• 2 years ago

It mentions something not being reported

This has been reported and discussed from the start, even in the initial report. Also it's not a Linux specific patch or issue, rather a patch used by some Linux distros for tighter systemd integration.

jpz• 2 years ago

I think the idea this was HUMINT operation by a state sponsored intelligence service is more likely.

The twitter thread here was interesting.

https://x.com/thegrugq/status/1774392858101039419

Raging about these being inconsiderate people, when they were likely fictional personalities that were part of a long con seems to be a bit foolish to me.

mustache_kimono2• 2 years ago

I think the idea this was HUMINT operation by a state sponsored intelligence service is more likely.

It's not an either/or proposition. I definitely think it was state sponsored, AND one method used was social engineering a burned out maintainer.

bloak• 2 years ago

It seems to me that people are very much exaggerating how "professional" this attack was. Yes, it doesn't look like the actions of a single bored teenager but I don't think the government of a country like the USA or China would deliberately permit their employees to get involved with crap like this. Any backdoor they try to insert would look exactly like an innocent bug. So my (uninformed) guess would be that this is done by criminals, something like a ransomware gang branching out a bit. Though North Korea sometimes sponsors activities that are indistinguishable from those of a criminal gang so it could come from there.

I'm just speculating, of course. I don't know anything really.

paulmd• 2 years ago

this is going to be impossible to prove or disprove, but given the state-sponsored nature of the attack (which seems fairly likely at this point)... I also wonder if maybe there wasn't some tips pushing Andreas Freund down the path of discovering it too.

like let's say you're the NSA and you know Russia (/china/etc) is trying to do this backdoor. maybe you send Freund an email through one of your cutouts and say hey, I've been looking at the ubuntu RCs and we noticed some performance regression in the postgres tests, etc... do it from some corpo email from a "friend" at some bigtech company that legitimately uses postgres/ubuntu and it's completely 100% deniable and innocuous.

it'd be interesting to see correspondence to/from Freund on his mailing lists too, see if there was anyone that (in retrospect) might have been tipping him down the path of discovery too.

(which is not to diminish in any way what he did... chasing a tiny perf regression in core library functionality back to root-cause is no mean feat. especially when it's code that is actively trying to evade detection - watching for debuggers, etc. Although that heisenbug nature might have also made it more compelling to these sorts of people ;)

EasyMark4• 2 years ago

Saying that it's a microcosm for "open source" is a bit hyperbolic. What would be more accurate is saying that it's a microcosm of "there is a certain set of users to be ignored" when writing open source software. One has to learn to develop a thick skin and guilt free ignoring of those who don't contribute anything to a project such as constructive suggestions, bug reports, or actual code.

higeorge13• 2 years ago

Serious question, but what is your expectations for linux and all these libraries/dependencies after 20-40+ years when most of old school or current devs are retired? How can anyone guarantee reliable continuity by good actors in all these dependencies?

apantel5• 2 years ago

Steve Jobs said that someone needs to be the keeper and maintainer of the vision. I think that’s the key to the making of anything great over the long run: there needs to be a Benevolent Dictator For Life who has the vision, competence, energy, passion, and caring to consistently iterate the thing well.

The best case scenario for a project that loses its BDFL(s) is to hold on maybe as decently as Apple has under Tim Cook. They haven’t done anything truly innovative since Jobs died in my opinion — they just figured out more ways to apply the combination of a multitouch display, compact computer, and sensors; but they’ve iterated well on everything they produce and have held on to their lead in terms of hardware, user experience, and ecosystem cohesion.

In short: someone very competent has to care a lot about the project for the right reasons, and continue driving it forward. If the old guard is rotating out, someone has to step up.

If no one steps up, then leeches of various types will attach themselves to the project and simultaneously milk it for value and kill it.

mnau• 2 years ago

44% of active open source projects have only one maintainer. Out of all maintainers, 60% would describe themselves as unpaid hobbyist.

They already are BDFL for themselves or one other person. The problem is a chronic deficiency of maintainers, which is at the root of this attack.

Your comment implies there is a wide pool of peoplecwho want to do that, along with mythical "someone".

Here is a little story:

There was an important job to be done and Everybody was sure that Somebody would do it.

Anybody could have done it, but Nobody did it.

Somebody got angry about that because it was Everybody’s job.

Everybody thought that Anybody could do it, but Nobody realized that Everybody wouldn’t do it.

It ended up that Everybody blamed Somebody when Nobody did what Anybody could have done.

- Charles R. Swindoll

matheusmoreira3• 2 years ago

Someone has to step up. Someone's got to care. Someone this. Someone that.

Who if not you? When if not now? Be someone.

rrr_oh_man4• 2 years ago

This is the way it works. It needs to change.

Well, it’s on you to set your boundaries. "Feel free to submit a PR" works wonders.

matheusmoreira3• 2 years ago

Be careful what you wish for. Someone will eventually take you up on that challenge. The responsibility will lie with you when they do make that pull request. Is that something you actually want? I gotta ask because it seems there's lots of people out there who are saying that but don't actually want it to happen.

Look I get it, it's a great line, it instantly weeds out anyone who can't code, it even filters out those who can code but don't actually care enough about your project. If you choose that route though, you better be prepared for the day when the fabled PR finally arrives in your inbox. You told them to do it themselves if they wanted it so bad, they went and did it. Even though you know your code base and could probably have implemented it 10 times faster than they did, they labored over it to spare you the trouble instead of begging for the feature in your issue tracker. You asked to see the code, you got the code. Now the buck stops with you.

Seriously, if you say this, be ready to engage with whomever does show up with a patch. Unless they send you literal irredeemable garbage, the least they are entitled to is your sincere peer review. Nothing's worse than putting in effort into learning somebody else's project and writing the patch, only to not be given the time of day, or worse. I'd rather have a Linus Torvalds type take a big shit on the patch in public than be ignored like that. Someone who does that is at least treating you like a peer.

rrr_oh_man4• 2 years ago

I like (and upvoted) how you replied, but disagree with you.

What's the problem, honestly? Telling someone that you don't like / want their code / direction / feature?

It just takes a modicum of emotional maturity to do that.

gavinhoward2• 2 years ago

And this is why, for my build system coming out in two days, you need to pay to have me respond to bug reports.

sethammons• 2 years ago

That would stop a nation state or other nefarious actor because? This isn't spam that works in volume

gavinhoward2• 2 years ago

It means I can ignore loud but non-paying users.

I also don't take outside contributions.

fl0ki• 2 years ago

You just reinvented the "it's open source but we charge for support" model which has been so well known for so long that it was a line in the movie Antitrust (2001).

krupan2• 2 years ago

This should make us all reconsider any criticisms we've levied towards Linus Torvalds, Richard Stallman, or any other gruff open source project lead

account427• 2 years ago

It should also make us reconsider the motivations of those who push such criticisms.

jrochkind14• 2 years ago

The multi-entity (sock puppet or not hard to say) social engineering attack that was apparently two years in the making (?), is to me a much more salient story than "users are often mean." Although it's true that the expectation of mean users let the attack fit right in, I guess.

mwcampbell• 2 years ago

The "users are mean" story is something that we can all do something about, and, honestly, I prefer stories with morals that most of us can actually put into practice.

jrochkind14• 2 years ago

Ah, well, I guess my thought is that we need to figure out what to do about the "take over from a burned out maintainer and then inject malware" attack.

It's not obvious what to do about it to me either. Which is why I'm concerned to talk about it.

That this attack was run by someone who had been participating in the project for possibly years before making the attack -- is not what i would have expected, and makes it even much harder to defend against. Before I was thinking it was about awareness of "don't just turn over the thing to someone new who just showed up, they might be an attacker." But not that easy in this case.

I suppose "try to get users to be less mean, by doing our part by being less mean individually" is arguably one piece of strenghtening defenses to this kind of attack, I guess, ok.

lamontcg2• 2 years ago

”Progress will not happen until there is new maintainer. … The current maintainer lost interest or doesn’t care to maintain anymore. It is sad to see for a repo like this.”

While this might be a sockpuppet deliberately manipulating the maintainer, the tone of this post is all too common, and the real thing that pisses me off is actually that last sentence.

The use of the words "sad", "shame" and "pity" have been weaponized by a generation of people to dump negative emotions onto to overworked/understaffed people in issue trackers.

If you're using those words to express an opinion in an issue tracker, you're probably being a fucking asshole.

anthk• 2 years ago

Not an issue of OpenSSH, but some GNU/Linux distros patching and linking OpenSSH against xz.

bshanks• 2 years ago

Even if other users (or possibly, sock puppets of the attacker) had not been complaining, the original maintainer probably would have began to trust the attacker after enough real contributions.

shp0ngle3• 2 years ago

Stop making software open source, start making it closed source and charge for licenses.

That works! Unfortunately it's bad for open source.

cryptonector• 2 years ago

This thread is a microcosm of the interactions in Open Source projects. Consumers make demands (some polite, some not-so-polite) of one maintainer (rarely two) that does everything.
Make no mistake. This is the way it works.
It needs to change.

How?

pizzafeelsright2• 2 years ago

Deny by default anything from a State that's hostile to American interests.

fluoridation3• 2 years ago

How would that help with the problem of people being rude to unpaid OSS maintainers?

shp0ngle3• 2 years ago

let's add passport (and visa) checks to github

semiquaver• 2 years ago

At first I thought it was paranoid to suggest that people on the linked thread might be complicit in the attack but then I read this message:

  > You ignore the many patches bit rotting away on this mailing list. Right now you choke your repo. Why wait until 5.4.0 to change maintainer? Why delay what your repo needs?

It genuinely looks like we’re seeing a demonstration of supply chain psyops in retrospect. Amazing how sophisticated and patient this attack was.

jijijijij• 2 years ago

It genuinely looks like we’re seeing a demonstration of supply chain psyops in retrospect.

Worthwhile noting: It happened in the open, archived for the world to see at any time! The attacker needed to be extra careful not to raise suspicion, both acutely and for accumulated evidence in history. Of course, easy to say "Hindsight is 20/20", but we can probably agree in actual hindsight, we easily see a lot of suspicious acts around the issue. An individual incident may be chance, but as a whole things become clear more easily.

So, I think there is a teachable moment here: When something seems just a tiny bit fishy, do a background check, investigate. You may not catch all something, but you probably won't miss everything.

Now, looking at the level of sophistication and risk sneaking people and code into open infrastructure, imagine chances of getting caught infiltrating a large company writing closed source binary blobs for drivers, firmware, ...

shp0ngle3• 2 years ago

There are not that many suspicious acts tbh. Randos complaining about unmaintained repos and unclosed issues is a constant in FOSS world.

Sometimes it's even true, some projects really die and stop actually addressing real issues.

But that's just inherent downside of the "bazaar" model. I don't think how we can "treat maintainers better" without going full corporate/without going full "cathedral".

kuschkufan• 2 years ago

Also the authors of these messages never interacted on that mailing list before or after. They only turned up to put pressure on the original maintainer.

asddubs• 2 years ago

this was two years ago though. why rush someone when you're planning to play the long game anyway?

jrochkind14• 2 years ago

Because successfully creating a sense of urgency will help the maintainer make a hurried unwise decision. It's part of the whole pattern.

archi42• 2 years ago

User Hakkin made an interesting observation: The "consumers" are likely sock puppets controlled by the attacker.

I did not independently verify the claim, but it seems reasonable to me.

Post with Hakkin's reply: https://news.ycombinator.com/item?id=39869703

(But yeah, I've seen some entitled idiots trying to push their demands on FOSS maintainers and being irritated by the harsh reaction...)

Xz: A microcosm of the interactions in open source projects

Related Stories