Railway knows better than you

https://anukari.com/blog/devlog/railway-knows-better-than-you

by humbledrone • 7 months ago

4 2 7 months ago

half0wl• 7 months ago

I appreciate this blogpost. I'm the one at Railway responsible for this decision, so I wanted to share some context from our side.

First off, sorry you got nailed by this. I genuinely empathize because _we_ got nailed too - the Railway.com frontend is hosted on Railway, and we had references to these vuln versions buried in old packages that weren't used in live code. We couldn't deploy for a bit until we sorted it out. It sucked.

That said, I believe this was the right call for a few reasons:

1. We have to think about our entire userbase. Our DX makes deploying easy, which attracts a lot of non-technical folks such as PMs, vibe cobers, newbies, etc. A significant chunk of them would either have no idea this was happening, no idea what an RCE even is, or no clue how to fix it.

2. We're trying to break the "I'll fix it later cycle" because that mindset is how security debt piles up. Yes, it's a heavy-handed approach. It shifts the action item left in the SDLC by blocking vuln deploys outright. We _could_ just alert people, and we did, but we've learned the hard way that people don't read emails. This was the only intervention that actually worked. Other platforms like Vercel also took the same approach.

3. This disproportionately impacted users who weren't using Next.js. We had to scramble when attackers leveraging this exploit started causing degradation across <10% of our fleet [0].

Your feedback on container and resource isolation is valid; there's stuff we could do better, and we're working on it. As a platform, it's a hard dance between "you got pwn'd for ignoring shit" and "why didn't you protect us from this?"

We made this call to protect the majority, and I recognize it's not going to make everyone happy. Despite this, I would still have made the call. I wished the majority of our userbase knew better than us, but the reality is they don't. My only regret is not making this call earlier when we were first notified. The sad thing here is people like you who _do_ know better than us doesn't have an escape hatch out of this - and I would argue that this isn't an escape hatch we should be providing.

(And for the record, we aren't actively killing live running workloads on vuln versions unless our scanner picks up that they're compromised using heuristics we've developed for known cryptominers, etc.)

[0] https://blog.railway.com/p/incident-report-december-16-2025

edit: typos and minor phrasing tweaks

justjake• 7 months ago

Jake from Railway here

And my hosting provider is saying, "you are not allowed to push out your urgent fix, because we see that your app contains a far less urgent problem." There is no button that says "I understand, proceed anyway." Railway knows best.

We rolled this out quickly because of the React/NextJS CVE. I think this is actually a really good suggestion and we can look into it! Thank you for the thoughtful blogpost, and I'm sorry we let you down. We will work hard to re-earn your trust.

Railway knows better than you

Related Stories