Incident report for February 21st, 2024

https://resend.com/blog/incident-report-for-february-21-2024

by RyeCombinator • 2 years ago

73 62 2 years ago

yatish27• 2 years ago

"While building a feature, we performed a database migration command locally, but it incorrectly pointed to the production environment instead, which dropped all tables in production."

This was scary.

Rygian• 2 years ago

Separation between office and production should be the norm.

hn_throwaway_99• 2 years ago

Extremely so in my opinion. In this day and age, viewing something like this means this company fundamentally doesn't understand security. It should simply be impossible to "incorrectly point to the production environment", because whoever ran this shouldn't even have access to those credentials in the first place.

ttul• 2 years ago

This implies that their production environment is mutable. As in, a command can run and change the production environment. That’s a no no.

But I give them a pass because they are a young company. My company was similarly reckless early on, but as we scaled, we had to tighten things up and turning to an immutable deployment approach has saved our asses so many times.

syslog• 2 years ago

How does immutable work in regard to databases? As in „we need to add a column“?

ZephyrBlu• 2 years ago

Is being a young company really an excuse when any half-decent engineer knows these things are bad?

Being a young company doesn't mean you ignore all the mistakes other people have made and figure them out for yourself.

I really surprised someone has access to the prod DB, and that it's possible for them to connect to it in dev (Meaning they have a copy of the credentials???).

disillusioned• 2 years ago

That single sentence contains multitudes:

* Production should be immutable

* No one doing dev in a dev environment should have such trivial access to prod

* Are there still good reasons for a migration to drop all tables? I guess it's for the dev environment to etch-a-sketch to a known state?

Yikes.

anonzzzies3• 2 years ago

No one doing dev in a dev environment should have such trivial access to prod

It’s the new and hip ‘cloud’! Probably using planetscale or something like that, which (last I checked, maybe it changed but wasn’t on), doesn’t even have ip protections outside the mysql user settings (while bad, would’ve protected them).

Are there still good reasons for a migration to drop all tables?

We haven’t found any.

tianzhou2• 2 years ago

We have been working on bytebase (https://github.com/bytebase/bytebase) for 3+ years to address this. With a change review workflow, environment propagations, and try not to disturb the dev flow if possible.

briffle• 2 years ago

In addition, Every centralized storage, and every cloud provider lets you take a snapshot of a database's disks.

We can restore a 10TB disk in about 12 minutes. its much faster to snapshot, do migration, then if necessary, drop disk and remake from snapshot. (and then replay the replay any other WAL changes up to the exact second you want with a tool like barman, wall-e, pg_backreset, etc.

Postgresql backups are critical for disaster recovery, but the restores are so very, very slow, they should be a last resort.

sergiotapia• 2 years ago

I did this once early in my career, the ice cold feeling in the pit of my stomach permanently etched this lesson into my soul lol.

JackFr3• 2 years ago

I did too, but it was 25 years ago and quite honestly there was a different attitude about developers accessing production.

Today if a developer can bring down the operation accidentally, that’s a problem with the org more than the developer.

(On the other hand if a developer screws up the shared dev environment, it is his or her fault and they deserve the wrath of their coworkers.)

JackFr3• 2 years ago

At a company with a modern, mature process this simply should not be possible.

anonzzzies3• 2 years ago

This sounds like one of those horrible tools like prisma which drop everything if something is not in sync on dev. We removed this type of stupid in favour of our own which, you know, fixes this actually instead of lazily dropping everything when they cannot resolve some trivial thing, for instance, a new required field without default when there are already rows and other crap which they call ‘opinionated’. No idea why we ever used that stuff as it caused so much grief (never on prod though); after prisma/drizzle and some other ‘modern’ horrors, we mistrust everything that’s ’hip and new’ so ‘everyone uses it’. One of those hip and new things I mistrusted was Resend even though ‘suddenly everyone uses it’. I’ll wait a few years before even contemplating it.

satvikpendem3• 2 years ago

I like that Prisma drops the database. Developer databases should be idempotent and easily reseedable with sample data. It's the same concept as cattle, not pets, from the devops world but now applied to databases. There should be nothing special about a database in development that cannot be dropped and recreated.

anonzzzies3• 2 years ago

I don’t care about dev databases; I care about why it does it; there is no need for it, at all.

I like that Prisma drops the database.

‘Like’ ; so if it didn’t but still worked perfectly, you would complain that it didn’t?

kgeist4• 2 years ago

We have a script which recreates the dev DB, but developers must run the command manually so that there was an understanding what's happening.

janpio• 2 years ago

Quick note that Prisma (and most of the other modern tools) never "drops everything" for normal migrations. A database reset is a solution suggested when your migration history and the actual database are not in sync, which makes it impossible for Prisma to "calculate" the migration SQL to get the database in the desired state (which is defined in Prisma schema).

The commands that suggest a reset are explicitly designed for use with development databases, not with production. The "deploy migrations to production database" command on the other hand only applies already existing migration files (that you have reviewed before) and does not suggest a reset, ever.

walrus013• 2 years ago

From the company's homepage: "deliver marketing emails at scale"

Maybe this company doesn't need to exist, and shouldn't.

vmfunction• 2 years ago

It is actually very open of them to post that! I'm sure things like that happens in larger companies, but I'm sure we won't get a incident report like that from them. I would rather work with that kind of company. Then we can also see if they learned from their lesson, and this doesn't repeat again.

heyarviind2• 2 years ago

Many companies are already solving the same problem, and then resend comes into the picture with the support of YC. They are doing everything except sending the mails.

walrus013• 2 years ago

They do, or do not operate MX and deliver SMTP traffic to other mail servers?

kgeist4• 2 years ago

Reminds of this classic thread: https://www.reddit.com/r/cscareerquestions/s/cqrama0L1z

accidentally destroyed production database on first day of job

Same symptoms: while developing a feature locally, they accidentally pointed to the production DB, which was destroyed when running tests.

doctor_eval3• 2 years ago

If they’re small then you can see it happening where someone was logged into prod using some environment variables to sort out some issue - probably didn’t even update production, just a few queries - and then went back to work.

Hours later they run some script that does DROP DATABASE from that same shell they used to troubleshoot, which takes a little longer than usual…

Anyway I can totally see it happening to me in my little one man shop, but now i think I might look at removing some privileges from my prod account :)

pistoriusp6• 2 years ago

Or just use https://www.snaplet.dev

doctor_eval3• 2 years ago

Unclear how that would have helped.

laborcontract• 2 years ago

Man, I use Resend, I really want to like them and yeah, it is really simple to get started which is great but man, this is, I think they really need to slow down a bit and maybe try to figure out how to put some processes in place to maybe intentionally slow things down.

This is the second incident that can be characterized by a very hazy delineation between development and production environments. The first incident had to do with an attacker gaining access to private credentials due to devs leaving keys set in NEXT_PUBLIC environmental variables on their site.

dbbk2• 2 years ago

I don't say this to be rude, but honestly it's because they are an amateur company. When you're dealing with critical infrastructure like email, you should stick to the battle-tested services like SES, Postmark, Sendgrid etc.

hw• 2 years ago

Production database access must always be locked down from external traffic, and only allow traffic from the production application or within the production environment.

Aside from mitigating local dev accidentally pointing to the prod db, if you have the db accessible externally means it’s susceptible to network attacks and password attacks

globular-toast• 2 years ago

Sometimes I think the infra at the small company I work at isn't great, but we've not had direct dev access to prod DBs from day one. They're locked down at the IP level. You'd have to go through some serious hoops to accidentally connect to prod even if you had the keys.

I remember as a child thinking adults had everything under control. That they know what they're doing. I guess I assumed a day would come when I too would know. That day never came. It's easy to think when you look at shiny websites and the first paragraph of this comment that other adults do know. But I'm often reminded of the truth: nobody knows. Everyone is always operating at least slightly outside their comfort zone or, in the case of the article, wildly.

nbittich• 2 years ago

incident report / post mortems could be the best way to promote a dev tool. Didn't know what resend was before someone deleted the prod database. Now I wonder if I need it / if I want it.

wrftaylor• 2 years ago

I love what Resend are doing and am a customer. I can also absolutely empathise as our lead engineer did exactly the same thing at a startup I was running a decade ago. It's a horrible situation.

But yeah, both the incident and the report are really tough to read. It would be great if they can do a follow-up with further actions they're taking.

There's a neo-bank called Revolut that allegedly at one point had just two teams: "go fast" and "don't screw it up". I feel like an infrastructure play needs some dedicated hires in camp 2.

pistoriusp6• 2 years ago

Unfortunately these sort of mistakes are seen as a "right of passage" for many developers. I ran "`DELETE FROM users;` without a WHERE clause against production in my first year on the job. I felt absolutely terrible. I thought I was connected to a development machine.

Fortunately we had backups available.

Often this isn't a problem with the individual developer itself, but points to a problem with the organization. Frankly most developers shouldn't have access to a production database, let alone mutable access.

One major concern is loss of data, but another is privacy.

It's so frustrating to see this happening when there are tools that solve this like Snaplet (I'm a founder), and Replibyte that allow you to generate or obfuscate data for usage in dev-environments, and Neon that allows you to branch your database.

tianzhou2• 2 years ago

We are also building Bytebase, which enforces the change review process for such operations

pistoriusp6• 2 years ago

It looks like we’re getting downvoted, lol. Not sure why that’s a thing. Bytebase is awesome. We should write about each other in our docs. You do the migration and we’ll do the data.

kgeist4• 2 years ago

I think DB engines should simply have some sort of a default option in the interactive mode where if you write "delete from table", it asks "Are you sure? You're going to wipe out the entire table! Y/N". Would've probably solved 99% problems.

namaria• 2 years ago

I think you mean 'rite of passage'

pistoriusp6• 2 years ago

Indeed. Thanks for the heads up.

rnts08• 2 years ago

This will keep happening as long as people are unable to learn from the past. Yes it's expensive to have a good experienced infrastructure engineer on the team, but at least you know there's someone testing your backups and procedures for when your eager dev team screw up.

tedchs• 2 years ago

I did this, around 2005, but I dropped ALL the prod tables. I was using a SQL GUI called Toad (awesome) and had separate windows open, for both "dev" and "prod". I was trying to reset the dev database, and used the wrong window. Thankfully, the "real" DBA at the time had 15-minute backups, and was able to restore it, and then I replayed a few transactions from logs. Lesson learned!

While building a feature, we performed a database migration command locally, but it incorrectly pointed to the production environment instead, which dropped all tables in production.

JackFr3• 2 years ago

I deleted a years worth of trades on a production database for a bond trading system in 1997. They were restored within an hour, but it made a lasting impression.

Different color scheme on prod consoles.

Always use “BEGIN TRAN”.

tetha• 2 years ago

This is why I've conditioned myself: Once I start running unsafe commands in a development-ish pattern (DDL changes, reboots, service starts/stops and such), I make sure to close any shell to productive systems and databases first. They are color-coded differently, normal users don't have these permissions and so on and so on, sure.

But it's better not to have these shells open and available to accidentally choose them once my brain switches from ops-mode to dev-mode.

dpcx• 2 years ago

I did something like this nearly 15 years ago at a job. I was trying to use something like MySQL Workbench to export the database and generate a map of connections through foreign keys. Apparently I selected some option backwards and it deleted out the entire production database.

Luckily for me I'd been working on some other things related to it and had taken a backup not long prior, but it was pretty nerve wracking to hear the CTO/CEO nearly running through the halls to find out what had happened.

Pretty sure they never implemented more stringent access controls at that company, either.

PeterZaitsev• 2 years ago

You should always be ready for your database to get trashed - application bugs, operator error, hacker intervention...

What strikes me in the incident report they focus on failed migration, where the real issue is not planning or not testing for recovery if migration goes very wrong.

Even if backup recovery would take just 6 hours would it be acceptable ?

_andrei_• 2 years ago

I hate it that some companies focus more now on how their product's website looks than on the product itself, its quality and stability. I also hate it that users are so used to having issues, errors, and their data leaked, that a startup can cut all the corners and do things completely the wrong way, and still have success. I hate that because I couldn't do it. Last month Resend leaked a database API key [0] from their env, how is that even possible? Now a developer tried to run a migration locally and did it on production, again, WHY is that possible? How can you talk about enterprise plans, SLAs, ACL features, when you can't do the basics right? What's it gonna be next month?

[0] https://resend.com/blog/incident-report-for-january-10-2024

Incident report for February 21st, 2024

Related Stories