HN user

half0wl

130 karma

meow@ray.cat

Posts7
Comments49
View on HN

I appreciate this blogpost. I'm the one at Railway responsible for this decision, so I wanted to share some context from our side.

First off, sorry you got nailed by this. I genuinely empathize because _we_ got nailed too - the Railway.com frontend is hosted on Railway, and we had references to these vuln versions buried in old packages that weren't used in live code. We couldn't deploy for a bit until we sorted it out. It sucked.

That said, I believe this was the right call for a few reasons:

1. We have to think about our entire userbase. Our DX makes deploying easy, which attracts a lot of non-technical folks such as PMs, vibe cobers, newbies, etc. A significant chunk of them would either have no idea this was happening, no idea what an RCE even is, or no clue how to fix it.

2. We're trying to break the "I'll fix it later cycle" because that mindset is how security debt piles up. Yes, it's a heavy-handed approach. It shifts the action item left in the SDLC by blocking vuln deploys outright. We _could_ just alert people, and we did, but we've learned the hard way that people don't read emails. This was the only intervention that actually worked. Other platforms like Vercel also took the same approach.

3. This disproportionately impacted users who weren't using Next.js. We had to scramble when attackers leveraging this exploit started causing degradation across <10% of our fleet [0].

Your feedback on container and resource isolation is valid; there's stuff we could do better, and we're working on it. As a platform, it's a hard dance between "you got pwn'd for ignoring shit" and "why didn't you protect us from this?"

We made this call to protect the majority, and I recognize it's not going to make everyone happy. Despite this, I would still have made the call. I wished the majority of our userbase knew better than us, but the reality is they don't. My only regret is not making this call earlier when we were first notified. The sad thing here is people like you who _do_ know better than us doesn't have an escape hatch out of this - and I would argue that this isn't an escape hatch we should be providing.

(And for the record, we aren't actively killing live running workloads on vuln versions unless our scanner picks up that they're compromised using heuristics we've developed for known cryptominers, etc.)

[0] https://blog.railway.com/p/incident-report-december-16-2025

edit: typos and minor phrasing tweaks

I've been here. Taking a vacation never helped, because I'd be stressed during the vacation thinking about "work" in an active manner.

What really helped me was fully disconnecting and shifting my mindset to think about work passively: I don't forbid myself from thinking about my work, but I do not action on it. Instead, I keep a physical scratchpad and write ideas and thoughts down continuously. This helps me refactor my perspective and shift it from "active building" to "passive expansion".

I've come to realize burnout is a function of expectation mismatch, and I think of it from thermodynamics perspective. Burnout operates like a pressure differential system where the gap between internal expectations and external reality creates unsustainable energy expenditure. In a closed system, energy imbalance leads to heat loss or system failure. Similarly, when your mental model of "what should be happening" constantly fights against "what is happening," you're burning cognitive energy just to maintain equilibrium.

The passive approach allows me to transform that pressure into potential energy. Instead of forcing immediate resolution (active building), you're allowing ideas to exist in a low-energy state (passive expansion). This mirrors how heat dissipates naturally rather than through forced cooling - it becomes a capacitor, storing energy for later use rather than demanding immediate conversion.

So the answer, for me, is "disconnect from doing while staying connected to thinking". This helps me recover much more efficiently, while keeping myself sharp and free of expectations of doing anything.

I highly recommend Fastmail [0] for this purpose. I have been with them for 5 years, and they're rock solid. I have ~10 domains on it, they support wildcards (send/recv at anything@you.com), the composer has sane defaults for multiple domains (e.g. replies are sent from the address it was mailed to) that can be overridden when you need, and you can easily create masked emails using your own domain.

$50/yr for all that is a steal. They don't charge you per-domain or have any funky tiered pricing for the stuff that matters.

Disclaimer: I am not related to them, just a happy customer :-)

[0] https://www.fastmail.com/

How are you "training" them? It's important to drill down into what they're struggling with before you throw resources at them.

I've found that code reviews are the best way to mentor and transfer knowledge. And it goes both ways: you review their code, they review your code. When I'm explicitly training someone, I get them to review my code and I occasionally throw in some minor curveballs that I expect them to pick up on in the first few rounds (e.g. obviously incorrect conditionals, some convoluted loop, leaky interfaces, etc.) If they stamp the PR without picking up on those items, I highlight it using a "do you think this is correct/is there a better way for us to do this" line of questioning. 5-10 or so PRs later, they become the first to spot those issues :-)

I've been having a blast writing plain old HTML templates for the past few weeks. It's a traditional Django app. I also use plain old CSS instead of frameworks like Tailwind (well, I use Bootstrap, but I don't use its utility classes).

It's practical, pragmatic, and allows me to ship faster.

Anything else worth looking into?

I've been looking into using ChatGPT as a writing assistant for myself. If you write a lot, it's worth a shot.

What AI tools should I evaluate to improve my office?

Start with the problem - what do you need to improve? Talk to your office, ask them what pain points they have, what's slow and inefficient, what they think could be improved, what keeps them up at night, etc. Then figure out an "AI way" to solve it for them :-)

I'm in the process of standing up my own hardware (1U server/some simple compute box + FreeNAS) for doing this.

There's a variable I don't understand with self-hosted cloud/storage: what guarantees do you have that they're not peeking at what you store? Why do you trust them over $CORP? Do you encrypt your data at-rest (dm-crypt, fscrypt, etc.), and do block storage providers support this?

edit: fix typo

I'm currently doing so. I'm pretty early in the process, and I've had a great experience thus far: quick replies, nice challenge (it was actually a challenge for me because I haven't touched the relevant language/framework in many years), and I generally appreciate these sort of interviews (I know I'm an outlier; unpaid work isn't a huge deal for me).

Out of curiosity, what did they send you, and what do you find weird/off about the process? It's definitely not the norm [0] but isn't it better than spending a bunch of time working through back-and-forths in "normal" hiring pipelines?

[0] https://fly.io/docs/hiring/hiring/

Generally, I think having an open handbook is the trending culture. Building in the open is becoming more popular - GitLab really moved the needle on this, so IMO from an outsider's perspective, Airbyte [0] and Posthog [1] has the best (publicly available & documented) processes, apart from GitLab of course (I didn't like that they removed stuff like their compensation calc, but I understand the motivation).

[0] https://handbook.airbyte.com/

[1] https://posthog.com/handbook

edit: addendums

GabeN hit the nail on the head. If you make paying for a service/content impossible or hard, people who want it will find other ways to get it. Regional DRM/content licensing is one annoying culprit.. Consumers want something instant, and if you instantly tell them "this is not available in your region," they're obviously not going to go out of their way to figure out where, whom, and how to pay for it.

The train is the longest part of the journey, and seemingly one of the calmest (compared to subway and that short tram ride). Are you able to do any deep work (reading, learning, brushing up on CS, etc.) during this commute on laptop or phone?

I don't think anyone can bear through that commute for extended periods of time, and trying to bear through it is a fool's errand. The quickest way out of your current situation for the medium- and long-term is to get a job in tech that allows full WFH/remote. In the short-term, it would be looking for new work or new accomodation, but I think that's pretty hard given what you said.

So, I would look for reasonably low-hanging fruits in tech such as webdev, specifically front-end - it's a great starting point and you can make some very tangible side projects for your portfolio to showcase. If you want any advice on getting started or some mentorship/code reviews/resource recommendations, I'm more than happy to help (for free of course) - feel free to shoot me an e-mail (in profile) anytime!

Generally, yes. Learning how to navigate will help you in any *nix systems.

since there are quite some comments highlighting the effectiveness of a good VIM setup and muscle memory of 10+ years, I would like to know your opinion on starting to get into it now (end of 2022)

I think this is not a good mindset to have. Knowing what's effective and building that muscle memory will take a long* time. If you approach thinking that "this is going to make my productivity fly because other people said so," the learning curve will flatten you and make you sad.

IMO, do it for curiosity's sake - having a hacker's mindset in the beginning will smooth that curve out!

As a testimonial mirroring other comments in this thread: you really can't go wrong with it once you get the hang of it. My experience with vim has been always positive & I've never had to change the core of my config for years. 99% of every LoC I've written in the past 10 years has been in vim. Every time I had to use/try a new editor, vim keybinding is the first extension I install (or setting I enable). It has helped my RSI a bunch (huge QoL improvement) and this is especially crucial with laptops: no more trackpad fiddling in the middle of typing!

I think this is very much a double-edged sword, and no one is truly a night owl.

If you are a night owl can you unlock your full potential when the sun is up or does your brain only start to fully function when it's dark outside?

I have an interesting personal story to share about this.

I'm fully remote, and my main interfaces are in vastly different timezones (PST/EST). I'm in Singapore (UTC+8), so I'm always 12-15 hours ahead of most of my team.

Which means that:

- Most of my meetings/syncs etc. are in PST (early morning for them, late night for me; sometimes 3am for me, and sometimes early morning for me, evening/night for them)

- I typically follow a graveyard schedule so I'm available in their timezones

- I am usually most productive during night hours

- (see my username)

I have been doing this for 5+ years, and yes, anecdotally - I feel like I'm "myself" when there is no sunlight.

However, I've come to realize that this is unhealthy for me. I barely have any energy to run daytime errands and appointments: literally nobody in my social circle is awake after 10pm, no doctor's clinic is open in the middle of the night, I can't head out and buy a pair of shoes, etc., but at least I have my basic amenities available (10-15mins walk away from two 24/7 supermarket, 7-11's, etc.)

The bit about social circle affects me the most: I've given up a lot in terms of networking opportunities and/or dinner & drinks with buddies just so I could attend meetings at 10pm, 1am, ...

So as much of a night owl as I am - I feel that being a night owl is great for my brain juices to flow, with the side effect of neglecting pretty much everything else in my life.

Not sure where I'm going with this; it may not make any sense, but your question just struck a chord with me :-)

Create a call stack map and write it down

(personal anecdote) I strongly agree with this. Writing it down is very important. Whenever I trace through a call stack, I tend to construct a visualization of it in my mind (in the form of a DAG + each root starting with the caller, if you will). But I could never hold that image especially when it gets too complex. Forcing myself to write it down helped me a lot - it doesn't matter if it's messy writings or drawings, somehow the _act of writing_ just helped me so much in reconciling concepts and connecting the dots.