justjake

Appreciate the feedback. We got some feedback previously that things were "too technical" and not acknowledging it from the what users saw.

I've gone ahead and re-added the surrogate keys statement to the press release. Thank you for the feedback and if there's other things that you believe can be better please let me know!

We indeed run tests as well as stage releases. For this issue, when rubber met road in production, we saw cases which weren't visible in staging.

We've rolled out some changes (detailed in the blogpost) which should avoid this in the future. Deepest apologies

Honestly for a production-grade _platform_ company, that also does compliance (SOC2/3, HIPAA etc.), not having a staged release is negligent, and how you guys are handling this is a huge red flag. I've done such changes myself in production envs, for deployments that don't have the stakes you guys have. I'm normally more sympathetic on incidents, but the lack of transparency thus far from railway leaves me doubting more than anything.

We do indeed have a staging environment as mentioned previously. The issue arose in the rollout to production as mentioned previously.

The blog post reads like PR compared to the initial incident status report, and the resolved timestamp does not match which is sloppy.

I've gone ahead and added the surrogate key mention into the post mortem. We initially got in trouble for having it be too technical centric and not enough on the user impact. It's a delicate balance; apologies. As I mention, we are open to critical feedback here.

Emailing only affected users isn't working out, because affected people aren't yet emailed (I know one personally). Just check the post on your own forum (https://station.railway.com/questions/data-getting-cached-or... did you actually read it?) and see the list of people affected still not emailed, and left on read.

We have people working directly in that thread. For anybody who believes they were affected but not reached out to, we're working directly with them. We do take this very seriously. If you know someone here, please have them reach out either there or directly to me at jake@railway.com

Again, it's not an excuse if you're a _platform_ company that customers pay a lot of money to be reliable. You can't just keep saying you're open to feedback and being transparent as vanity.

In the directly linked tweet I've mentioned that we're focusing on scaling the current system vs adding new features. We absolutely do need to do better on reliability, and my point is "Is there a specific poor engineering practice you're seeing here, or is it just based on reliability". Either is a fine crit we just want to make sure all our basis are covered

Did you read the thread? Yes, only _one_ employee commented 5 hours after my HN comment. Still almost everyone left of read, unanswered questions etc.

Indeed I've read the thread, and we have people working it (you can see as of 8 hours ago).

Railway founder here, providing some color

Why were they making CDN changes in prod? With their 100M funding recently they could afford a separate env to test CDN changes. Did their engineering team even properly understand surrogate keys to feel confident to roll out a change in prod? I don't think they're beating the AI allegations to figure out CDN configs, a human would not be this confident to test surrogate keys in prod.

We went deep on them, tested them prior, and then when rubber met road in production we ran into cases we didn't see in testing. The large issue, and mentioned in the blogpost, is that we didn't have a mechanism to to a staged release.

During and post-incident, the comms has been terrible. Initial blog post buried the lede (and didn't even have Incident Report in the title). They only updated this after negative feedback from their customers. I still get the impression they're trying to minimise this, it's pretty dodgy. As other comments mentioned, the post is vague.

Our initial post definitely could have been more clear, and we revised it the moment we got customer feedback to do so.

They didn't immediately notify customers about the security incident (people learned from their users). The apparently have emailed affected customers only, many hours after. Some people that were affected that still haven't been emailed, and they seem to be radio silent lately.

We notified customers even before we did a wide release, as is process for anything security related. You create space for as much disclosure area as possible, and then follow up with a public disclosure

Their founder on twitter keeps using their growth as an excuse for their shoddy engineering, especially lately. Their uptime for what's supposed to be a serious production platform is abysmal, they've clearly prioritised pushing features over reliability https://status.railway.com/ and the issues I've outlined here have little to do with growth, and more to do with company culture.

Do you have any specifics here? We're scaling the system at 100x YoY growth right now, working 24/7 to scale the entire thing. Again, all ears on if you have specific crits as we're always open to receiving feedback on how we can do things better!

Their forum is also getting heated, customers have lost revenue, had medical data leaked etc., with no proper followup from the railway team

There are team members in that thread linked, are you certain you linked the right thread? Happy to have a look at anything you believe we're missing!

Would you mind pointing out these claims? Happy to address them personally

Fixed the typo in that second paragraph and aligned the section on the Set-Cookie stuff. Anything else that can be made more clear?

Totally! People who see the impact will likely see more impacted than say, 3% of their services. Not all disruption created equal.

We rolled out a change to update our fraud model, and that uses workload fingerprinting

Since, in all likelyhood, your projects are similarly structured, there will be more impacted workloads if the shape of your workloads was in the "false positive" set

Will have more information soon but very valid (and astute) feelings!

Railway founder here. <3%.

That said, we treat this exigently seriously!

Any downtime is unacceptable and we'll have a post mortem up in the next couple hours

Hello! Railway founder here

We'll have a post mortem for this one as we always write post mortems for anything that affects users

Our initial investigation reveals this affects <3% of instances

Apologies from myself + the Team. Any amount of downtime is completely unacceptable

You may monitor this incident here: https://status.railway.com/cmli5y9xt056zsdts5ngslbmp

Jake from Railway here

And my hosting provider is saying, "you are not allowed to push out your urgent fix, because we see that your app contains a far less urgent problem." There is no button that says "I understand, proceed anyway." Railway knows best.

We rolled this out quickly because of the React/NextJS CVE. I think this is actually a really good suggestion and we can look into it! Thank you for the thoughtful blogpost, and I'm sorry we let you down. We will work hard to re-earn your trust.

Bingo. Nix doesn't give you a generalizable-across-languages-and-ecosystems way of specifying specific versions without blowing up your package size, unless you hand Nix to your users (which we didn't want to do)

Maybe we were holding it wrong, but, we ultimately made the call to move away for that reason (and more)

Hey y'all! 3 years ago we built Nixpacks. However, we ran into some pretty large pains using Nix for dependency resolution

So today we're rolling out Railpack, the successor. It results in:

- Up to 75% smaller images - Up to 5x faster builds - Higher cache hit ratio

The goal is to provide a seamless alternative for the Dockerfile frontend. Railpack will automatically find your dependencies, you can add any additional ones, and it'll auto construct the build pipelines for you

We're actually profitable on this round of funding. We've venture backed, but we also wanna make sure people know we're here for the long haul.

Author here. We've talked previously about our datacenter buildout (https://blog.railway.com/p/data-center-build-part-one)

This is the "next step". We're using our baremetal presence to now undercut the "big guys", which means 50% cheaper storage and networking for our users

This is the first week of our launch week so, expect more stuff all this week

We use ~~arch~~ nix BTW

I think, least for us, this is tablestakes stuff. You should be able to get the repo going day 1 and get a commit in

But there's just a fuckload of context beyond the code. You can get it from the RFCs we write internally, but there's still a lot to absorb

Happy to answer any questions. This all seems very obvious in retrospect, but I wanted to share for any other early stage companies in the hopes people can avoid some scaling pains!

That $0.10 per GB is direct pass along for the cloud ingress fees

We can lower that once we’re fully on metal

FWIW this is the advantage of being able to run in the cloud and on perm

If we have to we can “burst” into the cloud

We're blessed with some kickass investors. They gave us just the right level of scrutiny. We were super clear about why we wanted to do this, we did it, and then they invested more money shortly after the first workloads starting running on metal

If you're looking for great partners, who actually have the gal to back innovation, you'd be hard pressed to do better than Redpoint (Shoutout Erica and Jordan!)

Not ATM. We use it in a lot of our stack, so we will likely pull it in in the future

Thank you! Anything you think we can do better?

FWIW, we just pass the costs on from the current cloud providers. Doing this work will let us lower those egress prices!

"Booo who let this guy cook?"

Fair tbh

We will indeed write more on this so this is great feedback for next time!

Yup. Bingo. We've had to pass the cloud egress costs onto our customers, which sucks.

With this, it'll mean we can slash that in half, lower storage costs, remove "per seat" pricing, etc

Super exciting

Winner winner chicken dinner!

I think that's my fault BTW (Railway Founder here). I asked Charith to cut down a bit on the details to make sure it was approachable to a wider audience (And most people have only done Cloud)

I wanted to start off with the 101 content to see if people found it approachable/interesting. He's got like reams and reams of 201, 301, 401

Next time I'll stay out of the writing room!

Thanks dude <3. We are indeed doing the thing :D

Kinda. It's like if you had everything from an infra stack but didn't need to manage it (Kubernetes for resilience, Argo for rollouts, Terraform for safely evolving infrastructure, DataDog for observability)

If you've heard of serverless, this is one step farther; infraless

Give us your code, we will spin it up, keep it up, automate rollouts service discovery, cluster scaling, monitoring, etc

The requirements end up being pretty specific, based on workloads/power draw/supply chain

So, while we could have bought something off the shelf, that would have been suboptimal from a specs perspective. Plus then we'd have to source supply chain etc.

By owning not just the servers but the whole supply chain, we have redundancy at every layer, from the machine, to the parts on site (for failures), to the supply chain (refilling those spare parts/expanding capacity/etc)

Everything is dual redundancy. We run RAID so if a drive fails it's fine; alerting will page oncall which will trigger remote hands onsite, where we have spares for everything in each datacenter