ndneighbor

I see a lot of analysis here that this is good for Ant, but I beg to differ, it's a very bad place to be as a company serving enterprises when deployment risk is now present. This might delay Ant's financial goals in their ability to monetize Fable and other Mythos class models.

huh- I guess there are two HN submissions with meaningful replies...

I said this in the other thread, we got access to our account back, but even with a Account Rep. and a CSM on our account- it still took them a while to figure out what was going on.

I'm sure it could have been worse if we didn't have a rep on our account.

Actually, when I made the TOS check, I put that in Redis. That + the feature flags got reset.

They deleted our GCP proj. sans warning. Still working the details, but that's how this whole thing began.

decisions are easy to criticize in hindsight

I mean, the pain we have caused our customer ultimately proves you correct. That said, we made our decisions with the information and constraints that we knew in that moment in time. Railway has hosts in AWS/GCP/and co-los, so coordinating those workloads in a fully distributed manner would be ideal but end of the day, we didn't forsee that would just have our project get deleted just like that.

(Even if we did get assurances from them in 2024, that it wouldn't happen again, although we just got auto-rate limited the last time.)

We have a CSM, Head of Customer Support contact, and further contacts with GCP. Despite that, we still had this issue.

Yea, I mean, that's the whole MO of our platform and we failed at that. So yea, that's disappointing and more so for our customers.

I can provide an explanation about the GCP dependency. Yes, we have host workloads off GCP, and we have been able to build a good business by performing a cloud exit. However, we were worried that we would have a circular dependency on our own cloud. I don't think we expected to get auto-modded out of our own account, hence we left our DB on CloudSQL.

It was never our intent to deceive people that we didn't own our own destiny with our business. The last GCP issue, we were assured that this scenario wouldn't happen (when we got auto-ratelimited, which was bad, but survivable) - but it seems like we have further work to do. Apologies.

confirmed

(I help host nycsystems w/ Phil- we don't mind, just an easier way to know who is who other than email)

I think the intent of Mike Judge's joke was less so an outright promotion of eugenics and more so mocking the upper crust of American society's approach to family planning. (That of which Judge was intimately familiar with during his time in SV when he worked for a graphics card company.)

A lot of his work with KotH analyzed the same dynamics of educated and uneducated America and the interplay and I think Idiocracy is essentially the terminus of the observations he would make where if the idiots got their way. (A semi-common plot point with Hank in KotH where he would be pit against rediculous circumstances.)

We have more info coming soon but I think the best way to frame this is actually working backwards and then explain how it impacted yours and other services.

So Railway (and other cloud providers) deal with fraud near constantly. The internet is a bad and scary place and we spend maybe a third to half of our total engineering cycles just on fraud/up-time related work. I don't wanna give any credit to script kiddies to the hostile nation states but we (and others) are under near and constant bombardment from crap workloads in the form of traffic, or not great CPU cycles, or sometimes more benignly, movie pirating.

Most cloud providers understandably don't like talking about it because ironically, the more they talk about it- the bad actors do indeed get a kick from seeing the chaos that they cause work. Begin the vicious cycle...

This hopefully answers:

If 3% of your services were affected, does that match your expected fraud rate? That is an awful lot of customers to take down in one go, and you'd want to be very accurate in your modeling. I can't see how you'd plan to kill that many without false positives and negative media.

In our 5 year history, this is the third abuse related major outage. One being a Nation State DDoS, one being coordinated denial. This is the first one where it was a false positive taking down services automatically. We tune it constantly so its not really an issue except when it is.

So- with that background, we tune our boxes of lets say "performance" rules constantly. When we see bad workloads, or bad traffic, we have automated systems that "discourage" that use entirely.

When we updated those rules because we detected a new pattern, and then rolling it out, that's when we nailed the legit users, since this used the abuse pattern, it didn't show on your dash, hence the immediate gaslighting.

Which leads to the other question:

How or why were customers not notified? I have used services before where if something seemed dodgy they would proactively reach out and say 'tell us if it's legit or in 24 hours it will be shut down' or for something truly bad, eg massive CPU usage affecting other services, they'd kill it right away but would _tell you_.

We don't want to tell fraudulent customers if they are effective or not. For this instance, it was a straight up logic bug on the heuristics match. But we have done this for our existence like black holing illegitimate traffic for example, then ban. We did this because some coordinated actors will deploy, get banned with: "reason" and then they would have backup accounts after they found that whatever they were doing was working. If you knew where to look, sometimes they will brag on their IRCs/Discords.

Candidly, we don't want to be transparent about this, but any user impact like this is the least we can do. Zooming out, macro wise, this is why Discord and other services are leaning towards ID verification. ...and it's hard for people on the non service provider side to appreciate the level of garbage out there in the internet. That said, that is an excuse- and we shovel that so that you can do your job and if we stop you, then thats on us which we own and hopefully do better about.

That said, you and others are understandably miffed (understatement) all we can do is work through our actions to rebuild trust.

Hey there Dave, Angelo from Railway here-

First off, super duper sorry. It's sometimes a good/bad thing if I can remember someones handle. ...and I specifically remember the support thread where we did have an outage before your demo :| - the number one goal for us is to deliver a great product. Number two is that we should never embarrass a user, outages do exactly that.

We just wrapped up the post mortem and that'll be published soon where it explains why the dashboard was reporting the state of the application incorrectly and would be more than happy to credit you for the impact to keep your business. That said, totally understand if two is way too much impact for your services.

(Angelo from Railway here)

Heard. Being transparent, usually the delay on ack is us trying to determine and correlate the issue. We have a post mortem going out but we note that first report was in our system 10 minutes before it was acked, to which the platform team was trying to see which layer the impact was at.

That said, this is maybe concern #1 of the support team. Where we want the delta between report and customer outage detected to be as small as possible. The way it usually works is that we have the platform alarms and pages go first, and then the platform engineer usually will page a support eng. to run communications.

Usually the priority is to have the platform engineer focus on triaging the issue and then offload the workload to our support team so that we can accurately state what is going on. We have a new comms clustering system that rolling out so that if we get 5 reports with the similar content, it pages up to the support team as well. (We will roll this out after we communicated with affected customers first.)

Angelo from Railway here, Railway runs our own metal for the sheer reason to preserve margins so we can run for perpetuity.

We're nuts for studying failure at the company and Heroku's margins was one of the things we considered to be one of the many nails in that coffin. (RIP)

(my rant here: https://blog.railway.com/p/heroku-walked-railway-run)

I think this acquisition makes a lot of sense and it's good business. Finding good MacOS developers who know the system level APIs more so than the docs is a tough go. It would make a lot of sense that OpenAI would just go ahead and hire out this expertise as they try to get their Mac app and their iOS app to get closer and closer to the system.

The unfortunate irony is not lost on me that Windbourne's H1 is "record breaking Weather Balloons".

I don't think any company would want this record. I am very glad the pilot and the souls on board are safe.

A man can dream ;-;

Author here, not my intent! My deepest apologies. English is my first language but people do joke that they say I write English like I learned it as a second language.

I have fixed the sentence fragment and connected the two thoughts together. Thank you for keeping me honest.

I am more than happy to add color here, I am sorry, I try my best to write everything but my editor cuts as much as I add. We also tend to hire really autonomous engineers who tend to like just going off on their own to try to solve the issue.

There have been a few times where we would commit to the problem, assign a DRI, and then find out midway that... no we have to hire/consult our way out of the issue. I think that's okay, we then look back at the retro to see what we missed.

If interested, I think we can blog about what happens when a problem gets converted to an RFC and then we have more engineering discussions with the stakeholders but the piece was pushing a 10 min read time as it was...

Hello there! Author there, and surprised/delighted with the response. I don't think we had the issue with the cadence, the quarter is arbitrary, but we think it gives us the ability to just go heads down to focus.

With that said, one thing we did and I don't why we did it was that we would "re-justify" why we would want to work on something every three months which isn't great. There is a world where if we had more eng. resources we could have more people than problems and we could take stuff on board as it arrives, but for us deciding on what to work on is a hard decision.

I also agree that market fit is a key factor. I think Railway was lucky that we didn't have to pivot the product 3 to 5 times to get some latch.

What would be the post-quarterty planning process that you would like to see?

Slight tangent brought up by the article, but usually the greats aren't good at just one thing but have a combination of eccentricities that form the person. I find it heartening that Cormac had these in spades.

It brings me no joy to say that now that I am on the maintainer side of some OSS, Hacktoberfest is a fear inducing event where people would just submit junk PRs just so they could get a T-Shirt...

We're so far from the original spirit of the event.

Yea, I found the article to be overly reductive. I work on a shared ownership codebase, which at times, I am not going to be able to pull the original author at all times when I work on my branch.

At least having a partner explain some aspects is a huge unlock for me. Maybe the OP shadowboxing a world with no humans, but no humans is already a status quo issue that my team and I face sometimes.

My day job includes moderation. I think I am more empathetic to the issues that moderators deal with on a day to day basis given that they are underpaid, under-appreciated, and overworked.

It's very difficult to scale a community to be both welcoming and productive. New users don't have the same context as existing ones. You find that norms and manners aren't transferred from one group to the next. So although that I noticed that SO started getting more strict from 2014 onward, I wouldn't know immediately what to do about the content quality issue.

My take is that, like most things, the medium of the old will be appreciated the way it wasn't in 2014. As the Brian Eno quote goes: "Whatever you now find weird, ugly, uncomfortable and nasty about a new medium will surely become its signature." People will yearn for the human forums the same way they did years past when people tire of the LLM slop. (If they do.)

The number one issue I have with Android is that while this looks cool, because of the fragmentation of the OS delivery between vendors- I have no idea which phone or timeframe when I could see the rollout of Material 3 Expressive.

More than 10 years later, shopping for an Android phone with the latest OS is a nightmare. Android leadership keeps on getting shuffled around, Google changes priorities every 6 months it seems. Despite Apple flubbing the ball on AI, at least I know that the phone will be supported for at least 4 years.

They will need to improve on their ecosystem commitments if they'd like people like me to switch back.

I see a lot of love for Elixir/Phoenix floating around but oft posted is that Ruby is just an easier and more approachable language for many people.

I think that aside from Rails, that makes it one of the easiest languages to pick up and have fun with. I do miss my Rails days for that reason.

Knowing Oracle, they will take it to court if they can. To paraphrase Cantrill, it's a company that behaves very much like a lawnmower.

I have been following Matrix development for sometime and I do like the direction where they are going at.

However, a lot of my time is trying to reach companies where they are and the majority of them are in Slack. I think there is a case to be made where more companies use Matrix over Discord for their community operations but I think we are seeing a trend of users willing to forego freedoms (as in speech) in exchange for services that are free (as in beer).

I think the challenge for protocol developers is to make sure that their client experience is top notch to win and keep those users, something that Mastodon faced vs. the now popular rise of Bluesky. The policy our company practices on whether to support a channel is: we build where our users are.

With that said, it's looking like a Teams integration is more likely so I guess we will be learning how to hook into there...

(Rip Spectrum chat)

Ooooo- thanks for sharing. Is this the bot that you are referring to?

https://slack.com/marketplace/A044USEAHRV-unblocked

I will try it out and see how it performs.

I can qualify this- when we began, we had a tonne of fraud and a lot of students looking to host a VNC to circumvent their school restrictions. Our goal is to be the last cloud a developer has to think about. (More code repo integration vs. give me a machine.) So- when we wrote our fair use policy, we blanket banned VNC use cases.

It genuinely sucks that to stop 90 percent of suspicious use cases, you do get legitimate use in that net. For that I apologize.

Now since Railway has a better handle on fraud, we feel that we can loosen up the restrictions, so if a customer clarifies their use case, we don't mind. With that said, we don't offer raw VMs (yet) so you will find the platform a bit limiting a bit after our complete migration to self-hosted metal.

Forgive me! Whenever I write internally or externally, it's usually a super long rambling piece and then we cut it down. The hardest part about writing for me making sure that everyone can have something to take away from it.

I'll zoom out- Railway got its first users from a Twitter post from the founder asking people to try the product. The founder then sat in a Discord voice channel during the winter break just answering mundane questions from people. Railway (like many DevTools at the time in 2021) was significantly more free than what it was today.

When I joined, we made the determination that Support Engineering, for the sake of our customers (staying sharp) and mental stimulation (we're not just help desk) - build our own automations to help users. (Blog post here: https://blog.railway.com/p/support-engineering-is-engineerin...)

(Railway also, intentionally, keeps our team small. Not out of elitism but out of alignment. The hardest part of working remote is making sure that people are on the same page. The tradeoff is that we are busier and need to be mindful about what work we take on. I, however, take it all on!)

Around 18 months ago or so- I started getting pulled into more revenue conversations as we moved upmarket. Railway moved into it's first proper plans. A lot of the prevailing advice we were getting was to start gating the product and start booking demos. So we did that for a bit much to the annoyance to developers. Then we started emailing them to see if they wanted to chat- I can't speak for you, but a lot of people in our book didn't reply. (Blog post: https://blog.railway.com/p/scale-not-sales) In Railway's eyes- we think that Support/Sales is just the same thing just at different points of the developer's journey. A lot of engagements begin with a question.

The channel that did work was Slack Connect. After answering a question or two with companies on there- their usage would grow a lot. So we scaled that out!

Again, sorry if it read like an ad- not my intent.