The RubyGems "Security Incident"

https://andre.arko.net/2025/10/09/the-rubygems-security-incident/

by semiquaver • 10 months ago

177 65 10 months ago

andre.arko.net

ChrisArchitect• 10 months ago

Rubygems.org AWS Root Access Event – September 2025

https://news.ycombinator.com/item?id=45530832

mbStavola4• 10 months ago

One of the primary justifications given for the takeover was to secure the gems service and offer trustworthy stewardship. Reading this, I don't really get the sense that the new maintainers are really prepared to deliver on either.

That said, I really don't like the hand waving of the HTTP log thing in this post. Yeah sure, company names aren't as sensitive/radioactive as an SSN or an email, but selling usage data isn't exactly a noble endeavor.

I don't think anyone comes out of this looking good. Some are worse than others, sure, but this is just a mess from top to bottom.

plorkyeran• 10 months ago

My primary takeaway from all of this is that I do not want to be depending on infrastructure run by Ruby Central. Maybe it’ll turn out that the previous status quo was even worse and we just got incredibly lucky that it never exploded, but the people now running things have consistently failed to inspire confidence.

adamors• 10 months ago

That is my takeaway as well, this whole saga is a comedy of errors and the butt of the joke is the new RC.

darkwater2• 10 months ago

That said, I really don't like the hand waving of the HTTP log thing in this post

What "hand waving"? André explicitly mentioned he did not have any log or information.

mbStavola4• 10 months ago

No but he was seeking it, from the email in the RubyCentral article and directly from TFA:

I have no interest in any PII, commercially or otherwise. As my private email published by Ruby Central demonstrates, my entire proposal was based solely on company-level information, with no information about individuals included in any way.

Here Andre is downplaying his ask of the logs. Even if Andre didn't get them, the logs were desired. Had Ruby Central acquiesced the logs would've been parsed and sold. Might not be an issue for you but I am frankly not interested in having any data shared or sold like this.

bigiain3• 10 months ago

They were all spitballing ideas about how to recover from the DHH-driven dropping of corporate sponsorship dollars, and how too keep the support lights on.

I think an offer of covering all the 2nd level support costs in return for the right - that Ruby Central's own T&Cs grant - to monetise company usage stats, is a reasonable offer.

The "other side's" alternative was to steal ownership and control of a whole bunch of volunteer gem authors work at the behest of a different corporate sponsor who was clearly demonstrating they wanted to be able to not only throw their weight around and force policies and priorities on RubyGems/RubyCentral, but also to make it personal by explicitly calling for long term contributors to be removed entirely on a whim.

phoronixrly5• 10 months ago

I blame DHH for all of this. He needs to step up, walk his words back and mend the damage to the Ruby community he has done. Including chipping in with the funding he cost Rubygems.

ksec• 9 months ago

This is interesting, because I would have thought after all the information revealed, at least both sides could be blamed and usage stats is a no - no.

We all do see things very differently.

prescriptivist• 9 months ago

This is such a strange take. Ruby Central, for better or worse, is the steward of Rubygems/Bundler. If Mike Perham wants to withdraw his funding because he thinks DHH is a white supremacist, then that's fine. But DHH didn't do that, Perham did.

Arko is not a completely innocent, non-self-interested character here. He has announced a project to end-run the existing rubygems, bundler, etc infrastructure before all this, in the name of "better tooling", but his tooling is solely owned by him and a handful of people that really, really don't like DHH. Controlling this aspect of the ruby toolchain ecosystem is in their own self-interest and overlaps with their deep disdain for the politics and corporate nature of the existing stewards of the ruby toolchain ecosystem. Maybe their approach and stewardship of this fork of the toolchain is more just, secure and equitable, but make no mistake -- they are fighting the same war that DHH and Shopify are, which is who controls the keys to the toolchain. Do you think if Arko, Perham, et. al. had control they would somehow be completely neutral, apolitical stewards of the ecosystem? No! They have made it clear with their money and machinations that they do not want to operate in the same ecosystem as DHH and their politics and ethics are intertwined with their relationship to the ruby community. They are no different than him.

Meanwhile those of us who just want stability are stuck between two factions who claim righteousness and ownership. I wish they all could be deposed and some more mature non-individual foundation could take over.

tetha• 10 months ago

Mh, one of our security admins recently said something that's very fitting to the discussion: If you are removing an employee from a company, and you have to rely on their personal integrity instead of technical controls to avoid problems, you are doing very basic access control wrong. And if you're doing absolute fundamentals like that wrong, how much is your entire information security worth then?

And reading this, and the other disclosure from Ruby Central, they seem to be handling this maintainer/employee offboarding woefully incompetently at really, really basic levels. Obtaining control to secret management and doing a general secret rotation of management secrets isn't an obscure first step.

psadauskas2• 10 months ago

Plus, its not a good look for RubyCentral for trying to smear Andre for it, when it is perfectly acceptable within their own Privacy Policy[1]:

We may share aggregate or de-identified information with third parties for research, marketing, analytics, and other purposes, provided such information does not identify a particular individual.

[1] https://rubycentral.org/privacy-notice/

anon848736284• 10 months ago

In a comment under the submission for Ruby Central's post, I said Arko changing the AWS password was an inexcusable ethical violation.

This context does slightly soften my view, especially the part about multiple 1Password accounts being in play. However there is a big thing still missing to me... Why would Arko not immediately notify RC that he had changed the password due to these concerns?

If it was really a noble good faith action by the assigned on-call, giving a heads up to the remaining stakeholders would be the obligatory next step, no?

According to RC's timeline, the password reset happened on September 19, but Arko did not disclose the issue to RC until September 30. From what I can tell, he has not refuted that timeline or explained the gap.

bigiain3• 10 months ago

To me it looks like he wasn't sure who was trustworthy at the time.

"The erratic and contradictory communication supplied by Marty Haught, and the complete silence from Shan and the board, made it impossible to tell exactly who had been authorized to take what actions. As this situation occurred, I was the primary on-call. My contractual, paid responsibility to Ruby Central was to defend the RubyGems.org service against potential threats. "

and

"Given Marty’s claims, the sudden permission deletions made no sense. Worried about the possibility of hacked accounts or some sort of social engineering, I took action as the primary on-call engineer to lock down the AWS account and prevent any actions by possible attackers."

and

"Within a couple of days, Ruby Central made an (unsigned) public statement, and various board members agreed to talk directly to maintainers. At that point, I realized that what I thought might have been a malicious takeover was both legitimate and deliberate, and Marty would never “fix the permissions structure”, or “follow up more” as he said. Once I understood the situation, I backed off to let Ruby Central take care of their “security audit”. I left all accounts in a state where they could recover access."

According to RC's timeline, the password reset happened on September 19, but Arko did not disclose the issue to RC until September 30.

The password reset happened on September 19, and "within a few days" he realised it was an intentional/malicious takeover, and he walked away knowing they had the means to recover their own access - no longer his monkeys, no longer his circus. The 30 Sep date was when he was asked by someone if he still had access, and he discovered he did, and let them know immediately.

That all seems way more likely to be true and feels more plausible than anything Ruby Central has published over the last month or so...

anon848736284• 10 months ago

I realize I don't live in their world and none of this whole mishap makes sense if you're looking through the lens of a nominally functional organization. But this explanation still strains belief.

1) If you're actually concerned about phishing or some sort of hack, all the more reason to pick up the phone and hash things out. Arko knew his AWS access was revoked on the 18. You either believe that is an error and get to the bottom of it ASAP, or you step back and accept it for what it is. Resetting the root password silently the next day is like an inconsistent half measure.

2) As you quote, he had a "contractual, paid" relationship with Ruby Central. In contrast to the GitHib repos, there was never ambiguity that RC owned the AWS infrastructure. Therefore I don't buy any confusion about who the legit authority is... You talk to the person signing the checks. I understand this implication about some sort of internal coup, but the actions do not seem consistent with that being a serious belief at the time either.

dismalaf5• 10 months ago

To me it looks like he wasn't sure who was trustworthy at the time.

It wasn't his decision to make. His consulting was terminated, he needed to hand over whatever credentials he had and wash his hands of it.

rys• 10 months ago

He does explain it in his blog post. He changed it after the erratic communication and actions of RC leadership, then after realising what they were really doing, left them to complete their “security audit”, assuming they’d discover it themselves and take appropriate action as part of that. That never happened (which is wild), so he let them know.

They still don’t seem to be in complete control or understanding of the infrastructure they forcefully took control of.

skywhopper4• 10 months ago

Also seems pretty obvious that there was no clear chain of command for the operators. The board themselves certainly aren’t deeply involved given the statement by the one board member about how they couldn’t be bothered to communicate with the community about what was happening because they are so busy in their day jobs.

So who should Arko contact? The guy who’s his “boss” just suspended a bunch of access, twice, and emailed contradictory things. Given how sloppy the overall security situation clearly was and continues to be, I’m guessing no one really understands how AWS security works except for Andre anyway.

james_marks• 10 months ago

From Arko’s post I get the sense he actually cares about security.

Seeing that he still has root, which means others may, changing root is the most benevolent thing he can do.

It immediately means he has the only unauthorized access instead an unknown many, and that they’ll now cycle keys like they should have in the first place.

mikeg8• 10 months ago

Ruby central looks so incompetent it’s not even funny. Bummer all around.

wgjordan• 10 months ago

I think the biggest missing piece in the opposing accounts of this incident is how exactly the production-access removal was communicated. There's a huge gap between how the two posts are framing the clarity of the communications that happened on Sept 18:

September 18 2025 18:40 UTC: Ruby Central notifies Mr. Arko, via email, of the board’s decision to remove his RubyGems.org production access, and the termination of his on-call services.
Marty Haught sent an email to the team within minutes, at 12:47pm PDT [19:47 UTC?], saying he was (direct quote) “terribly sorry” and “I messed up”. [...] the complete silence from Shan and the board, made it impossible to tell exactly who had been authorized to take what actions. As this situation occurred, I was the primary on-call.

André also mentioned that he disclosed further remaining production access a few days ago, on Oct 5. Looking forward to Ruby Central's followup post-incident review for this subsequent incident, which they failed to address or mention at all in their initial publication.

emmelaich• 10 months ago

So weird that Marty is using corporate speak to someone who I presume he's been working with for up to ten years.

All of them really, not just Marty H.

skywhopper4• 10 months ago

Yeah, given that RC was willing to publish an email from Arko about an unrelated topic in their “security incident review”, it’s unfortunate they aren’t publishing how the access suspension was actually communicated to folks. Sounds like it was sudden enough and weird enough that Arko’s actions in response of locking down the AWS account were totally justified.

ethagnawl• 10 months ago

While I can't imagine how sad, stressful and confusing this all is for the people directly involved, it's also been hard to watch from the outside. For the past few years (decade, really) the community has been Ruby's biggest asset and seeing it torn apart like this is tragic.

lgleason• 10 months ago

Reading the tea leaves, I think this incident may have more to do with politics than it being a real incident per se.

For the past 10 years the Ruby community had been co-opted by political activists. Things like COC's and the Contributor Covenant etc. started in the Ruby community. The activists went after many top contributors in the community because of personal political beliefs etc, instead of behavior in the community itself. Some even called for ejecting DHH, the creator of rails, and Matz, the creator of the language, from the community.

When the Overton window finally stopped shifting to the left and started to move right, a lot of people who had remained quiet due to real threats of loss of business, work etc. finally started to speak up. DHH was one of them and has been very outspoken with his beliefs that open source software should be a-political and open to all instead of the political purity tests the activists were pushing.

From what I observed, when I was in involved in the Ruby community, Arko appeared to be a political activist. While there may have been an actual security concern here, my guess is that this had more to do with a desire to not have someone who may have been involved in trying to eject the top creators in the community being a point of failure for key infrastructure for the Ruby ecosystem.

watwut• 10 months ago

DHH is political activist, hardcore radical one. Always was. It is completely absurd how radical right wing gets labeled "apolitical" in these takes.

grim_io2• 10 months ago

woah, got any references to back this up?

I've seen a lot of DHH content, and I'd never describe it as radical right wing.

jjgreen• 10 months ago

Cheerleading the fascist "Tommy Robinson" is not apolitical

noelwelsh3• 10 months ago

Politics is unavoidable when groups of people get together, as politics is defined as how groups make decisions [1].

Therefore, "open source software should be a-political and open to all" is by definition both impossible (you cannot have a group without politics) and a political statement (as it is suggesting a decision making process.) Furthermore, don't mistake a conservative position (e.g. everything should stay the same) for an apolitical one.

[1]: For example:

politics: “who gets what, where, when, and how”—the process for resolving disputes and allocating scarce resources"

https://openstax.org/books/introduction-political-science/pa...

skywhopper4• 10 months ago

Open source “should be open to all” is extremely political. DHH himself regularly posts rants making it clear he doesn’t agree with this. Railing against codes of conduct that are meant to making open source more welcoming to folks who have historically been excluded is itself a political statement that “not everyone is welcome”.

mikl• 10 months ago

I took action as the primary on-call engineer to lock down the AWS account and prevent any actions by possible attackers.

So he suspected an attack, but did not contact his employer about it or other team members. No action taken to mitigate the attack or to identify what was going on. Just changed the AWS root account password and nothing else.

Even assuming the very best intentions, I don’t think it unreasonable that Ruby Central found that a little bit suspicious.

anon848736284• 10 months ago

This is the part that strains my belief as well. If you're really the concerned responsible professional working for the greater good of the community, then pick up the (metaphorical) phone ASAP and sort it out, regardless of how pissed off and insulted you are by the boss's incompetence.

dismalaf5• 10 months ago

Andre Arko seems to be trying to reframe this by concern trolling and appealing to people's sense of community.

Here's the thing: when a corporation terminates you, no matter the situation, you delete all your credentials, apps, everything, wash your hands of everything and never attempt access again. It's nice to say that the corporation should be better at rotating passwords but legally, you need to simply delete everything and move on.

Hence the letter from RC's lawyer to Arko. And a good chance he'll be prosecuted.

shivenigma• 10 months ago

If you consider the timing of this, there were supply chain attacks happened in other ecosystems and changing the root password seems to be the right approach and it feels justified to me.

The RubyGems "Security Incident"

Related Stories