How I Fired Myself

http://edu.mkrecny.com/thoughts/how-i-fired-myself

by mkrecny • 13 years ago

620 411 13 years ago

edu.mkrecny.com

malux85• 13 years ago

Whoever cancelled the backups was equally responsible

dsschnau• 13 years ago

Literally every engineer who didn't think to have a dev database can also be responsible. This is just a case of 'shit happens' since nobody had the sense to not work on, not back up, the live db.

vineet• 13 years ago

Perhaps more responsible was the person who decided that development work happened on the production database.

lmm• 13 years ago

More responsible, I would say. You expect a junior to make mistakes; the company should be structured to handle that happening.

Though I would look askance at whoever hired a philosophy grad as well, to be perfectly honest. The author admits he didn't have the experience to spot bad practice at the time.

islon2• 13 years ago

You don't expect a junior to make mistakes, you expect everyone AND everything to make mistakes/fail. Backups and separated environments is the least you could expect from a company earning millions.

eloisant2• 13 years ago

Actually even senior developers or architects make mistakes. Philosophy grad or not, it doesn't matter. That's to be expected.

What's more questionable is:

* Developers have access to the production database from their machine, while it should only be accessible to the front machines within the datacenter.

* Junior developers don't need an access to production machine, only sysops and maybe the technical PM.

* No backup of the production database. WTF???

If they had a hardware failure they would have been in the same shit.

ErikAugust• 13 years ago

Well - to be fair - if the company's practices include developing on the production database and not doing daily/weekly backups, then hiring an inexperienced Philosophy major is the least of their problems.

Also, someone who actually had the development experience and knowledge of better-practice would not have taken that position.

typicalrunt• 13 years ago

This. Times 1000. I would even go as far as to fire the CTO because if the data is your bread and butter, the you protect the data.

Protecting the data is called a disaster recovery (DR) plan in those big outdated companies that people like to make fun of.

The reason that these companies have a DR plan is to tell the CEO that 'when' (not 'if') all of the data goes away, (a) how long will it take to get it back and (b) how out-of-sync will this data be (2 minutes from freshness? 6 hours?).

cedsav• 13 years ago

Whoever was your boss should have taken responsibility. Someone gave you access to the production database instead of setting up a proper development and testing environment. For a company doing "millions" in revenues, it's odd that they wouldn't think of getting someone with a tiny bit of experience to manage the development team.

wyqueshocec• 13 years ago

This. Because of this and the lack of backups.

tseabrooks• 13 years ago

We sell middleware to a number of customers with millions of dollars in revenue who don't have backups, don't have testbeds for rolling out to "dev" before pushing to "prod" and don't have someone with any expertise in managing their IT / infrastructure needs.

My experience is that this is the norm, not the exception.

alttab• 13 years ago

Normal = OK.

Piskvorrr5• 13 years ago

Then perhaps "insane" is the word we're looking for, not "exceptional." Lack of sanity can be the norm, but that still doesn't make it desirable.

jzelinskie• 13 years ago

As a college student, this fucking horrifies me. Is there anyway I can guarantee I don't end up at someplace as unprofessional as this? I want to learn my first job not teach/lead.

lucian1900• 13 years ago

Even worse, they didn't have a backup.

hcarvalhoalves• 13 years ago

You should fire the company for not having a staging environment nor up-to-date backups.

pja3• 13 years ago

I doubt the problems this company had started when they employed the author of this blog post!

kyllo• 13 years ago

Sounds like no one else at that company had any more of a clue what they were doing than you did. The whole scenario is horrifying.

jiggy20113• 13 years ago

Wow, that's quite a story. If your company is ever 1 button press away from destruction. Know that this will eventually happen.

I'm quite surprised stuff like this hadn't happened earlier. When I am doing development with a database I will quite often trash the data when writing code.

cmbaus• 13 years ago

Any manager who doesn't take responsibility for this isn't a manager you'd want to work for. The manager should be fired.

hakaaaaak• 13 years ago

Agreed, but if the manager took responsibility for this he or she probably would be fired. Still, it is the only way to be; otherwise, you're not a real leader.

eric_bullington• 13 years ago

Wow, I'm sorry you had to experience that. I'm sure it was traumatic -- or perhaps you took it better than I would have. It must be of some comfort to look back now and realize that you only bore a small part of the blame, and that ultimately a large potion of the responsibility lies on the shoulders of whomever set up the dev environment like that, as well as whomever cancelled backups.

gearoidoc• 13 years ago

This post just made me feel 10x smarter (not that I blame the author - the blame here lies at the feet of the "Senior" Devs).

xentronium• 13 years ago

This is certainly a monumental fuckup, but these things inevitably happen even with better development practices, this is why you need backups, preferably daily, and as much separation of concerns and responsibilities as humanly possible.

Anecdote:

I am working for a company that does some data analysis for marketers aggregated from a vast number of sources. There was a giant legacy MyISAM (this becomes important later) table with lots of imported data. One day, I made some trivial looking migration (added a flag column to that table). I tested it locally, rolled it out to staging server. Everything seemed A-OK until we started migration on the production server. Suddenly, everything broke. By everything, I mean EVERYTHING, our web application showed massive 500-s, total DEFCON1 across the whole company. It turned out we ran out of disk space, since apparently myisam tables are altered the following way: first the table is created with updated schema, then it is populated with data from the old table. MyISAM ran out of disk space and somehow corrupted the existing tables, mysql server would start with blank tables, with all data lost.

I can confirm this very feeling: "The implications of what I'd just done didn't immediately hit me. I first had a truly out-of-body experience, seeming to hover above the darkened room of hackers, each hunched over glowing terminals." Also, I distinctly remember how I shivered and my hands shook. It felt like my body temperature fell by several degrees.

Fortunately for me, there was a daily backup routine in place. Still, several hour long outage and lots of apologies to angry clients.

"There are two types of people in this world, those who have lost data, and those who are going to lose data"

MBCook4• 13 years ago

Ug. MyISAM.

More than once (over the last few years) I've been doing some important update. I tend to do it the same way.

  START TRANSACTION;
  --run SQL--
  --check results--
  COMMIT; or ABORT TRANSACTION;

Of course, if you happen to run into the 1 or 2 MyISAM tables that no one knew were MyISAM, abort doesn't do anything. You've screwed up the data and need a backup.

So you always have to make a backup and check that the tables are defined they way they should be. Nothing is quite as much fun as the first time you delete a bunch of critical data off a production system. Luckily the table was small, basically static, and we had backups so it was only a problem for ~5 minutes.

mst2• 13 years ago

Any CREATE, ALTER or DROP on mysql does an implicit, silent COMMIT (not rollback) before running it.

PostgreSQL can do DDL inside a transaction though.

perlgeek• 13 years ago

Reading those stories makes me realize how well thought-out the process at my work is:

We have dev databases (one of which was recently empty, nobody knows why; but that's another matter), then a staging environment, and finally production. And the database in the staging environment runs on a weaker machine than the prod database. So before any schema change goes into production, we do a time measurement in the staging environment to have a rough upper bound for how long it will take, how much disc space it uses etc.

And we have a monthly sync from prod to staging, so the staging db isn't much smaller than prod db.

And the small team of developers occasionally decides to do a restore of the prod db in the development environment.

The downside is that we can't easily keep sensitive production data to find its way into the development environment.

wpietri4• 13 years ago

When moving data from prod to other environments, consider a scrambler. E.g., replace all customer names with names generated from census data.

I try to keep data having the same form (e.g., length, number of records, similar relationships, looks like production data). But it's random enough so that if the data ever leaks, we don't have to apologize to everybody.

Since your handle is perlgeek, you're already well equipped to do a streaming transformation of your SQL dump. :)

hef198982• 13 years ago

Remebers me of adifferent story from back the day when I worked around a pretty huge SAP system (as some kind of super user whatever). In one, seemingly trivial update (trivial compared to the complete system upgrade from one version to the higher one that worked without problems) cleansed the database including all purchase orders from the last two days company wide. Ah, and the back-up becamo "unusable", too.

But as far as I know, nobody was fired for this. Because, yes, things like this just can happen. An eventually it got fixed anyway.

danielna• 13 years ago

I still remember the all-consuming dread I felt as an intern when I ran an UPDATE and forgot the WHERE clause. I consider it part of the rite of passage in the web developer world. Kind of like using an image or text in a dev environment that you never expect a client to see.

Luckily the company I was at (like any rational company) backed up their db and worked in different environments, so it was more of a thing my coworkers teased me for than an apocalyptic event.

islon2• 13 years ago

I know how you felt. Many years ago when I was a junior working in a casual game company, I were to add a bunch of credits to a poker player (fake money). I forget the where in the SQL clause and added credits to every player in our database. Lucky me it was an add and not a set and I could revert it. Another time I was going to shutdown my pc (a debian box) using "shutdown -h now" and totally forgot that I was in a ssh session to our main game server. I had to call the tech support overseas and tell him to physically turn on the server...

hollerith• 13 years ago

To avoid mistakes like that is why I put the hostname and only the hostname plus one character in my shell prompt.

(The other character is a # or $ depending on whether the user is root or not.)

protomyth4• 13 years ago

At one job I went with this scheme for terminal background color: green screen for development, blue for testing, yellow for stage / system test, and red for production. This saved a lot of problems because I knew to be very careful when typing in the red.

Nick_C2• 13 years ago

I also change the prompt colour according to server. I can tell at a glance which machine I'm on (plus the hostname of course).

It means having a slightly difference .bashrc for each machine, but it's trivial.

Piskvorrr5• 13 years ago

That's why the very first package installed after ssh-server should be molly-guard. Saved my butt a few times...

sethammons• 13 years ago

molly-guard -- looks great. I've always aliased my commands so if I accidentally type 'po' while in an ssh session on a production server then it complains. On my local dev box, 'po' is just quicker for me to power off the box. It has saved me at least once.

stevekemp2• 13 years ago

Turning off the wrong-server is a thing that bit me before I installed molly-guard. These days that, and similar, is a tweak I apply to all hosts I control.

(molly-guard makes you type in the hostname before a halt/shutdown/reboot command.)

mappu• 13 years ago

Steve Kemp? Slightly unrelated, but i just want to say i'm (still) a big fan of your old 2004 program window.exe. Very handy for unhiding the odd broken program.

Have a wonderful day! (and i'll definitely look at installing molly-guard on my production debian servers)

caseysoftware• 13 years ago

One of the things I like asking candidates is "Tell me about a time you screwed up so royally that you were sure you were getting fired."

Let's be honest, we all have one or two.. and if you don't, then your one or two are coming. It's what you learned to do differently that I care about.

And if you don't have one, you're either a) incredibly lucky, b) too new to the industry, or c) lying.

mootothemax4• 13 years ago

One of the things I like asking candidates is "Tell me about a time you screwed up so royally that you were sure you were getting fired."

I asked similar, and agree that it's a really useful question.

I think it's an especially great one for startups, as successful candidates are more likely to come into contact with production systems.

For these positions you not only want people capable of recovering accidents, but also people who have screwed up because, conversely, they've been trusted not to screw up. Those who've never been trusted enough to not damage a system are unlikely to be of much use.

blowski• 13 years ago

This. We all mess up, but only the best ones will deal with it professionally and learn from it. Sounds like the OP is in that group. He didn't try to hide it, blame it on anyone else, or make excuses. He just did what he could to fix his mistake.

When people say "making mistakes is unacceptable - imagine if doctors made mistakes" they ignore three facts:

1. Doctors do make mistakes. Lots of them. All the time.

2. Even an average doctor is paid an awful lot more than me.

3. Doctors have other people analysing where things can go wrong, and recommending fixes.

If you want fewer development mistakes, as a company you have to accept it will cost money and take more time. It's for a manager to decide where the optimal tradeoff exists.

jiggy20113• 13 years ago

> If you want fewer development mistakes, as a company you have to accept it will cost money and take more time. It's for a manager to decide where the optimal tradeoff exists.

This is absolutely it, of course it is possible to become so risk averse that you never actually succeed in getting anything done and there are certainly organisations that suffer from that (usually larger ones).

However some people seem to take the view that it is impossible to protect oneself from all risks therefor it is pointless protecting from any of them.

The good news is that usually protecting against risks tends to get exponentially more expensive as you add "nines" therefor having a 99% guarantee against data loss is a lot cheaper than a 99.999% guarantee.

Having a cronjob that does a mysqldump of an entire database, emails some administrator and then does rsync to some other location (even just a dropbox folder) is something that is probably only a couple of hours work.

mcritz• 13 years ago

> We all mess up, but only the best ones will deal with it professionally and learn from it.

This. I don't regret my life's many failures. I regret the times I've flamed out, blamed others, or ran away.

Doing things means making mistakes. You can spot a professional by how they deal with mistakes.

Nick_C2• 13 years ago

The aviation industry is the same. Most aviation authorities around the world have a no-fault reporting system so that fixes can get implemented without pilots worrying about losing their job.

ismarc2• 13 years ago

Part of our culture/personality/team fit questions is like this. We also have one that's something like "tell me about a time you failed a commitment (deadline, etc) and how you handled it". Numerous people with over a decade claim perfect records, frequently blaming all those around them as having failed. It's been a real easy way to eliminate candidates, particularly because almost every team member we have gave an example that was within like a month of when they interviewed.

jiggy20113• 13 years ago

Interesting, though perhaps they misinterpreted the question. Some companies seem to have a culture where you never admit failure and perhaps they assumed that was the answer you were looking for?

pjungwir• 13 years ago

In my first year as an engineer, I shut off the production application server. It was an internally-developed app server built in & for an internally-developed language, with its own internally-developed persistence mechanism, and it was my first day working with it. I had several terminals open, and in one I'd logged on to production to check how the data was formatted, so I could get something comparable for my dev environment. Unfortunately I forgot to log off, and at some point I issued the `stop server` command, thinking I was doing it on dev. A few minutes later the project leader, one of the most senior engineers, came over and started yelling at me. Fortunately not much was lost (other than a few users' sessions--not a high-traffic site). But I was really appalled at my mistake and I certainly wondered if I was about to be fired. Since then I've taken steps to (1) keep myself off production as much as possible (2) make a production terminal visually distinct from a non-production one. As a sometime project leader myself now, I also make sure newcomers can easily obtain an adequate development environment, and I don't give out the production login to every junior dev. :-)

benjaminwootton2• 13 years ago

I disagree with this.

In most companies there's enough process to protect you from the big screw ups.

bashinator• 13 years ago

Or d) too scared/timid to be an effective employee and learn new skills.

imsofuture• 13 years ago

Wow, that's terrible. Mistakes happen, and for the notion of 'blame' to surface requires some monumentally incompetent management... the exact kind that would have their junior programmers developing against an un-backed-up production database.

The immediate takeaway from a disaster should always be 'How can we make sure this doesn't happen again?' not 'Man, I can't believe Fred did that, what an idiot.'

columbo• 13 years ago

News flash,

If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?"

If you are a CTO you should be asking this question: "How quickly can we recover from a perfect storm?"

They didn't ask those questions, they couldn't take responsibility, they blamed the junior developer. I think I know who the real fuckups are.

As an aside: Way back in time I caused about ten thousand companies to have to refile some pretty important government documents because I was doubling xml decoding (& became &amp;). My boss actually laughed and was like "we should have caught this a long time ago"... by we he actually meant himself and support.

seivan2• 13 years ago

Well said!

lsc3• 13 years ago

>If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?"

This is a question that the person in charge of backups needs to think about, too. I mean, rephrase it as "Is there any one person who can write to both production and backup copies of critical data?" but it means the same thing as what you said.

(and if the CTO, or whoever is in charge of backups screws up this question? the 'perfect storm' means "all your data is gone" - dono about you, but my plan for that involves bankruptcy court, and a whole lot of personal shame. Someone coming in and stealing all the hardware? not nearly as big of a deal, as long as I've still got the data. My own 'backup' house is not in order, well, for lots of reasons, mostly having to do with performance, so I live with this low-level fear every day.)

Seriously, think, for a moment. There's at least one kid with root on production /and/ access to the backups, right? At most small companies, that is all your 'root-level' sysadmins.

That's bad. What if his (or her) account credentials get compromised? (or what if they go rogue? it happens. Not often, and usually when it does it's "But this is really best for the company" It's pretty rare that a SysAdmin actively and directly attempts to destroy a company.)

(SysAdmins going fully rogue is pretty rare, but I think it's still a good thought experiment. If there is no way for the user to destroy something when they are actively hostile, you /know/ they can't destroy it by accident. It's the only way to be sure.)

The point of backups, primarily, is to cover your ass when someone screws up, primarily. (RAID, on the other hand, is primarily to cover your ass when hardware fails) - RAID is not Backup and Backup is not RAID. You need to keep this in mind when designing your backup, and when designing your RAID.

(Yes, backup is also nice when the hardware failure gets so bad that RAID can't save you; but you know what? that's pretty goddamn rare, compared to 'someone fucked up.')

I mean, the worst case backup system would be a system that remotely writes all local data off site, without keeping snapshots or some way of reverting. That's not a backup at all; that's a RAID.

The best case backup is some sort of remote backup where you physically can't overwrite the goddamn thing for X days. Traditionally, this is done with off-site tape. I (or rather, your junior sysadmin monkey) writes the backup to tape, then tests the tape, then gives the tape to the iron mountain truck to stick in a safe. (if your company has money; if not, the safe is under the owner's bed.)

I think that with modern snapshots, it would be interesting to create a 'cloud backup' service where you have a 'do not allow overwrite before date X' parameter, and it wouldn't be that hard to implement, but I don't know of anyone that does it. The hard part about doing it in house is that the person who managed the backup server couldn't have root on production and vis-a-vis, or you defeat the point, so this is one case where outsourcing is very likely to be better than anything you could do yourself.

Piskvorrr5• 13 years ago

> If there is no way for the user to destroy something when they are actively hostile, you /know/ they can't destroy it by accident.

Which also means they can't fix something in case of a catastrophic event. "Recover a file deleted from ext3? Fix a borked NTFS partition? Salvage a crashed MySQL table? Sorry boss, no can do - my admin powers have been neutered so that I don't break something 'by accident, wink wink nudge nudge'." This is, ultimately, an issue of trust, not of artificial technical limitations.

> one case where outsourcing is very likely to be better than anything you could do yourself.

Hm. Your idea that "cloud is actually pixie dust magically solving all problems" seems to fail your very own test. Is there a way to prevent the outsourced admins from, um, destroying something when they are actively hostile? Nope, you've only added a layer of indirection.

(also, "rouge" is "#993366", not "sabotage")

stephengillie• 13 years ago

Does Amazon Glacier offer this?

johngalt2• 13 years ago

From a sysadmin:

Redundancy = Reduce the number of component failures that can lead to system failure (RAID, live replication, hot standby).

Backup = Recover from an obvious failure or overwite (Weekly full backups, daily differentials).

Archival = Recover from a non-obvious failure and/or malicious activity (WORM tapes, offsite backup).

As a failsafe against malicious sysadmins is to split up the responsibilities. The guy handling backups isn't handling archival etc...

joecurry• 13 years ago

>> I think that with modern snapshots, it would be interesting to create a 'cloud backup' service where you have a 'do not allow overwrite before date X' parameter, and it wouldn't be that hard to implement, but I don't know of anyone that does it.

I test backups for F500 companies on a daily basis (IT Risk Consulting) - this would be missing the point really, the business process around this problem is really moving towards live mirrored replication. This allows much faster recall time, and also mitigates many risks with the conventional 'snapshot' method through either tapes, cloud, etc.

pja3• 13 years ago

I've said it elsewhere, but it bears repeating: RAID is about availability first and foremost. The fact that it happens to preserve your data in the case of one form of hardware failure is a side effect of its primary goal.

ritchiea2• 13 years ago

This is so true. I had to leave my first junior dev position for similar reasons as the OP, though nothing as monumental.

I was handed a legacy codebase with zero tests. I left a few small bugs in production, and got absolutely chewed out for it. It was never an issue with our processes, it was obviously an issue with the guy they hired who had 1 intro CS class and 1 rails hobby project on his resume. The lead dev never really cared that we didn't have tests, or a careful deploy process. He just got angry when things went wrong. And even gave one more dev with even less experience than I had access to the code base.

It was a mess and the only thing I was "learning" was "don't touch any code you don't have to because who knows what you might break" which is obviously a terrible way to learn as a junior dev (forget "move fast and break things" we "move slowly and work in fear!"). So I quit and moved on, it was one of the better decisions I've ever made.

paganel• 13 years ago

As a programmer I consider myself very lucky that one of the first advices I got when I was a junior was from one of my senior colleagues (and a very smart guy): "one of the most valuable qualities of a good programmer is courage".

Seven and a half years later I make sure that I pass that knowledge on to my junior colleagues. I'm proud to say that just in the past 2 weeks I've said this twice to one of my younger team-mates, a recent hire, "don't be afraid to break things!"

smrtinsert• 13 years ago

slowly and work in fear businesses usually collapse. the technical problem eventually becomes a business problem and no one realizes it can be solved at the source level.

it first builds inefficieny at the technical level then at the business level, finally it causes issues at the cultural level, and thats when the smart people start leaving.

hga6• 13 years ago

If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?"

In high tech this can get really messy, these are frequently inherently more fragile companies. My favorite example is from Robert X. Cringley in this great book: http://www.amazon.com/Accidental-Empires-Silicon-Millions-Co... ; from memory:

One day Intel's yields suddenly went to hell (that's the ratio of working die on a wafer to non-working, and is a key to profitability). And no matter how hard they tried, they could only narrow it down to the wafers being contaminated, but the wafer supplier swore up and down they were shipping good stuff, and they were. So eventually they tasked a guy to follow packages from the supplier all the way to the fab lines, and he found the problem in Intel's receiving department. Where a clerk was breaking open the sealed packages and counting out the wafers on his desk to make damned sure Intel was getting its money worth....

His point is that you can have a Fortune 500 company, normally thought to be stable companies that won't go "poof" without ample warning, in which there are many more people than in previous kinds of companies who can very quickly kill it dead.

gefh• 13 years ago

I physically cringed at that. Even the mail clerk should have noticed there was a 'big deal' about clean rooms and had some idea what the company he worked for did...

geoka92• 13 years ago

Where a clerk was breaking open the sealed packages and counting out the wafers on his desk to make damned sure Intel was getting its money worth....

I find this hard to believe. At some point a person in a space suit was introducing them into a clean room; she should have noticed that the packages were not sealed.

deservingend• 13 years ago

Seems like sabotage.

novax81• 13 years ago

The difference between working with bad bosses and good bosses really shows itself when there's a disaster going on.

A mere few months into my current job, I ran an SQL query that updated some rows without properly checking one of the subqueries. Long story short - I fucked up an entire attribute table on a production machine (client insisted they didn't need a dev clone). I literally just broke down and started panicking, sure that I'd just lost my new job and wouldn't even have a reference to run on for a new one.

After a few minutes of me freaking out, my boss just says: "You fucked up. I've fucked up before, probably worse than this. Let's go fix it." And that was it. We fixed it (made it even better than it was, as we fixed the issue I was working on in the process), and instead of feeling like an outcast who didn't belong, I learned something and became a better dev for it.

zdgman• 13 years ago

I really wish this was at the top. Everyone will fuck up at some point (even your best engineer). Whether you learn anything from fucking up really determines how bad your mistake was.

pmelendez2• 13 years ago

There are several reasons why you should not feel guilty. The company was asking for trouble and you just happen to be the trigger. These are the three top things that could prevented that incident.

1) A cron job for the manually task you were doing.

2) Not working directly on production.

3) Having daily backups

And this could happened to anybody. After midnight any of us are at junior level and very prone to do this kind of mistakes.

tlrobinson• 13 years ago

Hopefully prioritized in reverse order

pmelendez2• 13 years ago

Touché! I didn't intended to prioritize my list but you are right :)

vinceguidry• 13 years ago

I used to be a web freelance web developer/tech guy with one client, a designer. What made me quit was an incident where his client's Wordpress site hadn't been moved properly to the new hosting. (not by me)

The DB needed to be searched and replaced to remove all the old urls. After doing so, the wp_options cell on the production site holding much of the customizations kicked back to the defaults for the theme, the serialized data format being used was sensitive to brute DB-level changes.

I had talked to my client before about putting together a decent process including dev databases, scheduled backups, everything needed to prevent just such a screwup, but he waffled. Then blamed me when things went wrong.

I'd had enough and told him to do his own tech work, leaving him to fix his client's website himself. Being that I didn't build it, I didn't know which settings to flip back. I left freelance work and never looked back.

People and companies do this all the time, refuse to spend the time and money ensuring their systems won't break when you need them the most, then scapegoat the poor little technician when it does.

I'd like to say the answer is "don't work in such environments," but there's really no saying that it won't be this way at the next job you work, either.

I certainly wouldn't internalize any guilt being handed down, ultimately it's the founders' jobs to make sure that the proper systems are in place, after all, they have much more on the line than you do. Count it a blessing that you can just walk away and find another job.

laumars• 13 years ago

I really feel sorry for this guy. Accidents happen, which is why development happens in a sandboxed copy of the live system and why back ups are essential. It simple shouldn't be possible (or at least, that easy) for human error to put an entire company in jeopardy.

Take my own company, I've accidentally deleted /dev on development servers (not that major of an issue thanks to udev, but the timing of the mistake was lousy), a co-worker recently dropped a critical table on dev database and we've had other engineers break Solaris by carelessly punching in chmod -R / as root (we've since revised engineers permissions so this is no longer possible). As much as those errors are stupid and as much as engineers of our calibre should know better, it can only takes a minor lack of concentration at the wrong moment to make a major fsck up. Which is doubly scary when you consider how many interruptions the average engineer gets a day.

So I think the real guilt belongs to the entire technical staff as this is a cascade of minor fcsk ups that lead to something catastrophic.

taurussai• 13 years ago

Bold on your part to own up and offer a resignation. (he "higher ups" should have recognized that and not accepted it). From the movie, "Social Network" http://www.youtube.com/watch?v=6bahX2rrT1I

daemonfire300• 13 years ago

This could've happened anyone. It's a huge shame for those in charge not for you. Any business letting such operations happen without having backups or proper user-right-management should consider why they still exists, if they really make huge amounts of many as you mentioned.

hawleyal• 13 years ago

TIFU

lallouz• 13 years ago

I would love to see some reflection on this story from OP. What do you think you learned from this experience? Do you think your response was appropriate? What would you have done differently? Are you forever afraid of Prod env?

Many, many , many of us have been in this situation before, whether as 'monumental' or not. So it is interesting to hear how others handle it.

mkrecny• 13 years ago

OP here.

I realize that the dev environment was a recipe for disaster, and I was simply the one to step on the mine .. but I believe my guilt about leaving the company is 'quite right'. Thankfully I'm not forever afraid of Prod env - I still do a lot of risky stuff .. but I always have nightly backups, and other 'recreate the data' strategies in place.

Kesty3• 13 years ago

Everyone makes, has made and will make mistakes. Junior/Senior is not important.

You also could set up 20 layers of dev environments and it still doesn't matter, mistakes can still reach the outer layer.

You need to have the ability to recover from any problem quickly and with the data as updated as you need it to be.

Morendil2• 13 years ago

See this: http://www.slideshare.net/danmil30/how-to-run-a-5-whys-with-...

Guilt is a moral concept; when it comes to a run-of-the-mill operations mistake like yours, it does not belong in analysis of its consequences to the business. You are not a robot. You have made and will make mistakes this bad and worse.

Only consequentialist thinking should be the order of the day here; "what do we know that can prevent a similar mistake from hurting our bottom line". In this case backups are the standard, reasonable, well-known practice. Nothing will be improved by a firing or a resignation, by blaming or by shaming.

Insofar as the real root cause of the problem was not addressed, it's a reasonable prediction that any such company eventually joins the deadpool due to similar oversights.

mootothemax4• 13 years ago

The CEO leaned across the table, got in my face, and said, "this, is a monumental fuck up. You're gonna cost us millions in revenue".

No, the CEO was at fault, as was whoever let you develop against the production database.

If the CEO had any sense, he should have put you in charge of fixing the issue and then making sure it could never happen again. Taking things further, they could have asked you to find other worrying areas, and come up with fixes for those before something else bad happens.

I have no doubt that you would have taken the task extremely seriously, and the company would have ended up in a better place.

Instead, they're down an employee, and the remaining employees know that if they make a mistake, they'll be out of the door.

And they still have an empty users table.

j_baker4• 13 years ago

To be fair, if the CEO were willing to take those steps, the company would probably not have a deleted USERS table.

cantlin• 13 years ago

Last year I worked at a start-up that had manually created accounts for a few celebrities when they launched, in a gutsy and legally grey bid to improve their proposition†. While refactoring the code that handled email opt-out lists I missed a && at the end of a long conditional and failed to notice a second, otherwise unused opt-out system that dealt specifically with these users. It was there to ensure they really, really never got emailed. The result?

http://krugman.blogs.nytimes.com/2011/08/11/academia-nuts/

What a screw up!

These mistakes are almost without fail a healthy mix of individual incompetence and organisational failure. Many things - mostly my paying better attention to functionality I rewrite, but also the company not having multiple undocumented systems for one task, or code review, or automated testing - might have saved the day.

[†] They've long been removed.

niggler• 13 years ago

If you want to see monumental screw-up, look at knight capital group (they accumulated a multi billion dollar position in the span of minutes, losing upwards of $440M, and ended up having to accept a bailout and sell itself to GETCO):

http://dealbook.nytimes.com/2012/08/03/trading-program-ran-a...

newishuser2• 13 years ago

You did them more good than harm.

1) Not having backups is an excuse-less monumental fuckup.

2) Giving anyone delete access to your production db, especially a junior dev through a GUI tool, is an excuse-less monumental fuckup.

Hopefully they rectified these two problems and are now a stronger company for it.

generalpf• 13 years ago

I think it's a bit extreme to say he did more good than harm. He might have done some long-term good by having the company re-examine permissions and environments, but he probably did a lot of long-term harm by alienating current and future customers.

newishuser2• 13 years ago

Better that it happened 2 months after backups were canceled than 6 months or later. If you're going to cancel your backups you're begging for disaster.

stanleydrew• 13 years ago

There's an argument to be made that the company is doing harm to customers just by existing in such a precarious state. Anything that forces the technical leadership of the company to do the right thing or fail completely is actually better for customers in the long term.

VexXtreme• 13 years ago

In the author's defense, it wasn't all his fault. Whoever thought it was a good idea to:

1. Work directly on the production database 2. Not have daily backups 3. Not have data migrations in place for these kinds of situations

needs to be fired immediately. My guess is it was one of the 'senior' engineers and that the author only worked with what they gave him.

I've worked with all kinds of bozos but I've never seen this kind of incompetence. Ridiculous.

kordless• 13 years ago

I'm getting internal server error.

namityadav• 13 years ago

Me too. Here's the cached version: http://webcache.googleusercontent.com/search?q=cache:http://...

navid_dichols• 13 years ago

If your senior management/devs are worth anything, they were already aware that this was a possibility. There is no excuse for what ostensibly appears to be a total lack of a fully functioning development & staging environment--not to mention any semblance of a disaster recovery plan.

My feeling is that whatever post-incident anger you got from them was a manifestation of the stress that comes from actively taking money from customers with full knowledge that Armageddon was a few keystrokes away. You were just Shaggy pulling-off their monster mask at the end of that week's episode of Scooby Doo.

anton-107• 13 years ago

It's unlikely that the database had no foreign keys related to users table. And if so DBMS should have prevented deleting all users from the table.

Perhaps the Database Designer also failed his job. As well as the guys who cancelled backups and set up dev environment.

beering• 13 years ago

Well, the foreign key could have been set up to cascade deletes, in which case they would have been extra-screwed.

hackoder2• 13 years ago

I was in a situation very similar to yours. Also a game dev company, also lots of user data etc etc. We did have test/backup databases for testing, but some data was just on live and there was no way for me to build those reports other than to query the live database when the load was lower.

In any case, I did a few things to make sure I never ended up destroying any data. Creating temporary tables and then manipulating those.. reading over my scripts for hours.. dumping table backups before executing any scripts.. not executing scripts in the middle/end of the day, only mornings when I was fresh etc etc.

I didn't mess up, but I remember how incredibly nerve wracking that was, and I can relate to the massive amount of responsibility it places on a "junior" programmer. It just should never be done. Like others have said, you should never have been in that position. Yes, it was your fault, but this kind of responsibility should never have been placed on you (or anyone, really). Backing up all critical data (what kind of company doesn't backup its users table?! What if there had been hard disk corruption?), and being able to restore in minimum time should have been dealt with by someone above your pay grade.

arethuza3• 13 years ago

Out of interest, why not create a database user account that is read only and use that?

hackoder2• 13 years ago

It was a bunch of different tasks. For some, we did use a read only account. Other tasks (updating top 10 scores, updating the users table with their geo-ip based location etc) required write access.

Just to add some more thoughts based on other comments.. yes a lot of companies do stuff like this, particularly startups. The upside in these situations is that you end up learning things extremely quickly which wouldn't be possible in a more controlled environment. However not having backup and restore working is just ridiculous and I keep shaking my head at how they blamed the OP for this mistake. Unbelievable.

petriw• 13 years ago

Just remember to always verify it's still read only.

Or a coworker will find the login in your scripts, repurpose it, then notice they need more rights and "fix" the account for you.

leothekim• 13 years ago

Mistakes happen, and there should have been better safeguards -- backups, locking down production, management oversight.

But, I actually applaud how he tried to take responsibility for his actions and apologized. Both "junior" AND "senior" people have a hard time doing this. I've seen experienced people shrug and unapologetically go home at 6pm after doing something equivalent to this.

The unfortunate thing here seems to be that he took his own actions so personally. He made an honest mistake, and certainly there were devastating consequences, but it's important to separate the behavior from the person. I hope he realizes this in time and forgives himself.

coldclimate• 13 years ago

Did you click the wrong button - yes. Was this your fault - no. So many things wrong here.

I hope he came out ok in the long run, it's a hell of a story.

Kesty3• 13 years ago

I've did a very similar thing after one year working at my company instead of clearing the whole user table I replaced every user information with my account information.

I forgot to copy the WHERE part of the query .....

The only difference is that it was policy to manually do a backup before doing anything on production and the problem was restored in less than 10 minutes. Even if I had forgotten to make a backup manually we had a daily complete backup and an incremental one every couple of hours.

noonespecial• 13 years ago

This was a "sword of Damocles" situation. No backups, no recovery plan, and now clue how important any of these things were.

A thousand things can make an SQL table unreadable. "What do we do when this happens" is what managers are for, not finding someone to blame for it.

seivan2• 13 years ago

I find it disgusting that the "game designers" are the so called overlords. Fuck them. If you're a developer and a gamer then you're practically a game designer. What ever "education" they had is bullshit. You can go from imagination to reality with just you alone. And perhaps an artists to do the drawing. All those "idea" fuckers a.k.a game designers are just bullshitters.

And yeah this wasn't your fault. It was the CTO's fault. He holds responsibility.

"They didn't ask those questions, they couldn't take responsibility, they blamed the junior developer. I think I know who the real fuckups are."

mootothemax4• 13 years ago

If you ever notice that your employer or client isn't backing up important data, take a tip from me: do a backup, today, in your free time, and if possible, and again in your free time, create the most basic regular backup system you can.

When the time comes, and someone screws up, you will seem like a god when you deliver your backup, whether it's a 3-month-old one-off, or from your crappy daily backup system.

danielh2• 13 years ago

That is good advice, just make sure that it doesn't look like you are stealing your customers/clients data.

mootothemax4• 13 years ago

That is good advice, just make sure that it doesn't look like you are stealing your customers/clients data.

Excellent point! Any tips on how to avoid that, other than not taking the data home / copying to personal Dropbox-type things?

rocky11382• 13 years ago

This.

Even if it's old, when you're facing no data or old data, old data looks like heaven.

engtech• 13 years ago

I'm guessing the game was one of these? Likely Age of Champions.

http://www.klicknation.com/games/

Aardwolf3• 13 years ago

In the article it says your coworkers looked differently at YOU. Did anyone look differently at their database without back ups though?

systematical• 13 years ago

Your CEO was correct. He should have also said the same thing to the guy who cancelled backups as well...and the guy who never put in place and periodically tested a disaster recovery plan. So much fail in this story, but mistakes happen and I've had my share as well.

I once (nah twice) left a piece of credit card processing code in "dev mode" and it wasn't caught until a day later costing the company over 60k initially. Though they were able to recover some of the money getting the loss down to 20k. Sheesh.

andyhmltn• 13 years ago

Stuff like this happens. The best thing to prevent something like this is to completely sever the line between production and development. I've worked with companies that work directly on the production database. It's horrible. How can the person in charge of managing the workflow expect something like this not to happen eventually?

Yhippa• 13 years ago

"The implications of what I'd just done didn't immediately hit me. I first had a truly out-of-body experience, seeming to hover above the darkened room of hackers, each hunched over glowing terminals."

Holy crap. I know that _exact_ same feeling. I had to laugh. I know that out-of-body feeling all too well.

scottschulthess• 13 years ago

In my opinion, it's the fault of the whole organization, or at least the engineering team, for making it so likely that something like this would happen.

Database backups would've solved the problem

elomarns• 13 years ago

Although the author of the post obviously did a huge mistake, he is far from being the actual responsible for the problem that follows his mistake. It's the job of the CTO to make sure no one can harm the company main product this way, accidentaly or not.

He could never write code against the production database when developing new features. And if he was doing it, it wasn't his fault, considering he was a junior developer.

And who the hell is so stupid to don't have any recent backup for the database used by a piece of software that provides millions of revenue?

In the end, when you do such a shity job protecting your main product, shit will eventually happens. The author of the post was merely a agent of destiny.

ry0ohki• 13 years ago

I think all developers have to do something like this at some point to get the compulsion I have which is backups to the extreme. I can never have enough backups. Any DROP/ALTER type change I make a backup. (And also learned to pretty much never work on a production db directly, and in the event I need to, doing a complete test of a script in staging first...)

danielweber• 13 years ago

I feel tremendous empathy for this guy.

Not because I've done this. But because there but for the grace of God go I. It wouldn't take much changing in the universe for me to be this guy.

I'm very glad he's posting it, and I hope everyone reads it, so you can learn from his very painful mistake instead of committing it yourself.

S_A_P• 13 years ago

Sounds to me like this operation was second rate and not run professionally. If this sort of incident is even able to happen, you're doing it wrong. Maybe it's just my experience with highly bureaucratic oil and gas companies, but the customer database has no backup for 2 months?!?!?!?!?!?!

That is asinine. What would they have done if they couldn't pin it on a junior engineer? A disk failure would have blown them out of the water. I think he did them a favor, and hopefully they learned from that.

msdet11• 13 years ago

Things like this fall on the shoulders of the team as a whole. Certainly a tough pill to swallow for a junior engineer, but a good more senior developer or PM should've also realized you were working on prod and tried to remedy that situation. Humans are notoriously prone to fat fingering stuff. Minimize risk where ever you can!

bambax• 13 years ago

Once, a long time ago, I spent the best part of a night writing a report for college, on an Amstrad PPC640 (http://en.wikipedia.org/wiki/PPC_512).

Once I was finished, I saved the document -- "Save" took around two minutes (which is why I rarely saved).

I had an external monitor that was sitting next to the PC; while the saving operation was under way, I decided I should move the monitor.

The power switch was on top of the machine (unusual design). While moving the monitor I inadvertently touched this switch and turned the PC of... while it was writing the file.

The file was gone, there was no backup, no previous version, nothing.

I had moved the monitor in order to go to bed, but I didn't go to bed that night. I moved the monitor back to where it was, and spent the rest of the night recreating the report, doing frequent backups on floppy disks, with incremental version names.

This was in 1989. I've never lost a file since.

hga6• 13 years ago

Yeah; I was lucky that my first experience where I could lose data like that (before it was on punched cards) was a nice UNIX(TM) V6 system on a PDP-11/70 that had user accessible DECTAPEs. Because I found the concept interesting, I bought one tape, played around with it including backing up all my files ... and then I learned the -rf flags to rm ^_^.

That was back in the summer of 1978; today I have an LTO-4 tape drive driven by Bacula and backup the most critical stuff to rsync.net, the latter of which saved my email archive when the Joplin tornado roared next to my apartment complex and mostly took out a system I had next to my balcony sliding glass doors and the disks in another room with my BackupPC backups.

As long as we're talking about screwups, my ... favorite was typing kill % 1, not kill %1, as root, on the main system the EECS department was transitioning to (that kills the initializer "init", from which all child processes are forked). Fortunately it wasn't under really serious heavy use yet, but it was embarrassing.

clavalle2• 13 years ago

Wow. So many mistakes.

Working in production database? Bad.

No backups of mission critical data? Super bad.

Using a relational database as a flat data store? Super bad.

Honestly...I think this company deserved what they got. Good thing the author got out of there. Hopefully in their new position they will learn better practices.

jpalacios2• 13 years ago

Yah this story made me cringe. What exactly do you mean by:

"Using a relational database as a flat data store? Super bad."

Are you referring to the users table? I am not too accustomed to using flat files, so I am curious.

clavalle2• 13 years ago

Users is a bit of a core table in most applications. If they were using the relational database as it should be used there would be references to the user table elsewhere in the database.

If you tried to delete the table, it would fail stating that a deletion would violate the constraints assuming you didn't have deletions cascade automatically (which would be equally bad).

On the other hand (and it probably happened here) there will be one table with all sorts of data bolted on.

So say you want a user to have multiple pieces of armor (following the spirit of this post). You should have an armor table and a user to armor many to many table. But instead you just add an Armor column to the user record and create a new user record (with a the same username for example but with a different unique artificial key) with the new piece of armor in the armor column. Then to retrieve it you just select armor where username = whatever and iterate through the list. Adds and deletions are just as easy. So, why not? Well, duplication of data, for one thing. And no referential integrity protection for another. Delete a username and everything is deleted. Forget a where clause and you are sunk.

bguthrie2• 13 years ago

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.

liquidise2• 13 years ago

Exactly what you said. Rigor is truly the right word to use here. Cancelling your db backups is basically asking for a disaster. I'm not sure i have ever been at job that didnt require a db backup for some reason at some time.

mooreds• 13 years ago

To say nothing of running development code against a production database.

If there were only two junior folks, what were the senior folks doing?

benjaminwootton2• 13 years ago

I came here to type that almost exactly word for word.

This is a truly amazing story if this system was really supporting milions in revenue.

potatolicious3• 13 years ago

More than that, it's telling that the company threw him under the bus when it happened. I've been through major fuckups before, and in all cases the team presents a united front - the company fucked up, not an individual.

Which is, if you think about it, true, given that the series of events leading up to the disaster (the lack of a testing environment, working with prod databases, lack of safeties in the tools used to connect to database, etc...).

The correct way to respond to disasters like this is "we fucked up", not "someone fucked up".

coverband• 13 years ago

Exactly this.

oseibonsu• 13 years ago

I believe the company may be http://www.klicknation.com/games/ Can anyone confirm?

damm2• 13 years ago

When things like this happen, you have to realize there is more than one 'truth'.

There is the truth that here is someone who truncated the users table and because of that it caused the company great harm.

Here's another 'truth'.

1. The company lacked backups 2. The junior developer was on a production database.

Note: I'm from the oldschool of sysops who feel that you don't give every employee the keys to your kingdom. Not every employee needs to access every server, nor do they need the passwords for such.

3. Was there a process change? I doubt it, likely they made the employee feel like a failure every day and remind him of how he can be fired at any moment. So he did the only thing he could do to escape: Quit!

Horrible and wrong, if there was a good ethics lawyer around he would say it smells ripe of a lawsuit.

... That said, that lawyer and lawsuit won't fix anything.

j_baker4• 13 years ago

I've been through major fuckups before, and in all cases the team presents a united front - the company fucked up, not an individual

You should consider yourself very lucky. Or very savvy at knowing which companies to avoid.

trotsky• 13 years ago

When you think about it it's almost a logical certainty that he'd take the fall. Any company collectively able to understand that the actual failure was inadequate safeguards would have been able to see it coming and presumably would have prevented it from happening. If you're so inexperienced that you expect no one will ever make a mistake you'll obviously assume that the only problem was that someone made one.

It's somewhat fascinating to hear that a company that damaged managed to build a product that actually had users. And I thought my impression of the social gaming segment couldn't go any lower.

franze• 13 years ago

    The correct way to respond to disasters like this is "we fucked up", not "someone fucked up".

can't agree more! a company that starts playing the blame game - and even communicates this to the outside a) looks unprofessional b) is unprofessional c) poisons the corporate culture

its a fail on every possible level, the technical part beeing only minor.

memracom• 13 years ago

Just think, the company spent millions of dollars training this guy in protecting production data, and then instead of treating him like gold, and putting him in charge of fixing all the weaknesses that allowed this to happen, they pushed him out of the company. Stupidity beyond measure.

Not surprised that this was a gaming company because lack of teamwork seems to be endemic in that industry.

pavel_lishin• 13 years ago

> It's noble of the author to quit

It's not just noble. Considering how everyone treated him, and the company's attitude in general, he had no future there, and neither should anyone else.

x3ord• 13 years ago

Exactly. I'm ashamed that my first reaction reading this was to blame OP. But in the 2 min it took to read the post I had come full circle to wondering what kind of terribly run company would allow this to happen--I guess the type that hires philosophy majors straight out of college without vetting their engineering skills.

pnathan• 13 years ago

> the type that hires philosophy majors straight out of college without vetting their engineering skills.

Or hires them and then adequately educating them in software development.

jcromartie• 13 years ago

Indeed. If it had been a sporadic hardware failure, they would have been exactly as screwed. The fact that they gave an overworked junior dev direct read/write access to the production database is astounding.

gte910h2• 13 years ago

It's not that they gave him r/w at all that's so criminally stupid. It's that the required him to clear a table manually, using generic full access tools, over and over.

In reality, this should have been re-factored to the dev db.

If it couldn't be, the junior dev should have been given access to the raids table alone for writes.

Lastly the developer who didn't back up this table is the MOST to blame. Money was paid for the state in that table. That means people TRUST you to keep it safe.

I count tons of people to blame here. I don't really see the junior dev as one of them.

indrax• 13 years ago

Amazon is at the other end of the spectrum: randomly breaking things so that everything has to be fault tolerant.

http://www.codinghorror.com/blog/2011/04/working-with-the-ch...

This only happened because nobody even asked "What happens if I press this button?"

ajross4• 13 years ago

This really needs to be more of a standard thing. I've been near (but as an engineer, never responsible for) production systems my whole career. None of these systems were as terribly maintained as the one in the linked article. Production data was isolated. Backups were done regularly. Systems were provisioned with fault tolerance in mind.

Not once have I seen a full backup restore tested. Not once have I seen a network failure simulated (though I've seen several system failures due to "kicking out a cable" that sort of acts as a proxy for that technique). On multiple occasions I've seen systems taken down by single points of failure[1] that weren't forseen, but probably could have been.

[1] My favorite: the whole closet went down once because everything was plugged into a single, very expensive, giant UPS that went poof. $40/system for consumer batteries from Office Depot would have been a much better bet. And the best part? Once the customer service engineer replaced whatever doodad failed and brought the thing back up? They plugged everything right back into it.

Plutor• 13 years ago

Minor correction: the Chaos Monkey was Netflix's innovation. It just happened to be implemented on Amazon's cloud. It would have been just as useful if they had their own colocated servers or used a different cloud computing provider.

r00fus• 13 years ago

Apple did this before Amazon or Netflix in this regard [1], but the point needs to be made that a system needs to be tested and not just in a controlled aseptic way, because the real world isn't.

Another story supporting Chaos Monkey is what the Obama team did for their Narwhal infrastructure - they staged outages and random failures to prepare for their big day, meanwhile Romney's team who outspent the Obama team at least an order of magnitude, had their system fail on e-day.

[1] http://folklore.org/StoryView.py?story=Monkey_Lives.txt

j_baker4• 13 years ago

"Nobody cares about technical infrastructure. Our customers don't pay us for engineering rigor. We need to just ship!"

Of course the person saying that is likely to care about technical infrastructure when it costs them money and/or customers due to being hacked-together.

chris_mahan3• 13 years ago

Nobody cares about the diameter of the cylinders in the engine, all they care about is going from point A to point B safely, affordably, in style.

downandout• 13 years ago

Agreed. How could a company with "millions in revenue" not backup critical databases? Not only were they exposed to the threat of human error, but hardware failures, hackers, etc. When he submitted his resignation, the company should have encouraged him to stay. Instead, anyone at the company that had anything to do with the failure to implement regular database backups and the use of redundant databases should have been fired.

I've had more than my fair share of failures in the start up world. It always drives me crazy to see internet companies that have absolutely no technical ability succeed, while competing services that are far superior technically never get any traction.

happycube• 13 years ago

And worse, they had a backup service, and then dropped it to save money.

rodgerd• 13 years ago

It's the difference between understanding the incident (programmer makes mistake) and the root cause (failing at data integrity 101). Not just no backups, but no audit log of what's happened in the game - I'd expect an MMO to have an append-only event history quite apart from their state information.

sybhn• 13 years ago

Absolutely. Your immediate technical management sucked, and you were made the scape goat for your management's failure. Welcome to the real world. Don't get me wrong, you should feel bad, very bad, bad enough so that never again you do that. But you shouldn't feel guilty nor rethink your career.

hga6• 13 years ago

A little bit of feeling guilty is in order; the author "didn't know that he didn't know", and I'm sure this motivated him to learn a lot more about proper engineering processes ... something I'll note aren't particularly a focus in CS degrees. Especially since I haven't come across anyone who's really dedicated to them who hadn't first gotten burned in one way or another.

There's a big difference between being told "Do X, don't do Y" and that sinking feeling you get when you realize a big problem exists, regardless of the eventual outcome.

fitandfunction• 13 years ago

Exactly. Problems with production databases are inevitable. It's just a matter of time.

The guy who should be falling on the sword, if anyone, is the person in charge of backups.

Better yet, the CEO or CTO should have made this a learning opportunity and taken the blame for the oversight + praised the team for banding together and coming up with a solution + a private chat with the OP.

gawker• 13 years ago

Agree with the point that humans are fallible. We should always have back ups. A company with 1000s of paying customers should at least take steps to protect itself from this sort of catastrophe.

jalayr• 13 years ago

Completely agreed.

mncolinlee• 13 years ago

I cannot even begin to fathom how they functioned without a working development environment for testing, let alone let their backups lapse.

The kind of table manipulations he mentions would be unspeakable in most companies. Someone changing the wrong table would be inevitable. If I were an auditor, I would rake them all over the coals.

coopdog• 13 years ago

That's the Agile way, people not processes.

When the people screw up, there's no processes to hold them back, they can really, really screw it up.

Once you hit millions in revenue it's probably wise to put a couple of fall back processes in place, reliability becomes as important as agility.

cybernomad99• 13 years ago

To the credit of the management, they did not fire him. He resigned. But the coworkers felt he was responsible personally. That makes a uneasy work environment.

ftay• 13 years ago

That's not much to give credit for. They could certainly have done more to help him recover from this.

gtirloni• 13 years ago

When I was a junior analyst, I once deleted the main table that contained all 70k+ users, passwords, etc. The problem was fixed in 15min after the DBA was engaged to copy all data back from the QA environment that was synchronized every X minutes. Or we could have restored a back from a few hours ago.

The whole company fucked this one up pretty badly. NO excuses.

jliptzin• 13 years ago

Unbelievable. Even at my 3 person startup, back in 2010, with thousands in revenue, not millions, we had development environments with test databases and automated daily database snapshotting. Sure I've accidentally truncated a few tables in my time, but luckily I wasn't dumb enough to be developing on a production server.

SteveGerencser• 13 years ago

I worked for a startup a few years back where the CEO deleted the entire 1+TB database/website when the server was hacked and being used as a spam server because he 1. didn't know who to disable the site short of deleting and 2. could reach anyone that did know how.

The next morning he told us to restore the site from backups and fix the security hole. That's when we reminded him, again, that he had refused to pay for backup services for a site of that size.

We all ended up looking fro new jobs withing a couple days.

richo• 13 years ago

You're forgetting the guy who didn't speak up to say "Hey, maybe we shouldn't do this in prod?".

I feel for him, but at the same time there's a point at which you have to if testing guns by shooting them near (but not specifically at) your coworkers is actually a good idea.

readme2• 13 years ago

I wish I could upvote this more. Production database? Seriously, they were one copy away from avoiding this whole outcome.

CEO sounds incompetent as hell.

mehwoot• 13 years ago

Hindsight is 20/20.

jami• 13 years ago

Awesomely honest and painful story.

This happened somewhat in reverse to someone I worked with. He was restoring from a backup. He didn't notice the "drop tables" aspect, assuming, as one might, that a backup would simply supplement new stuff rather than wipe it clean and go back in time to a few weeks ago.

He is (still) well-liked, and we all felt sick for him for a few days. Our boss had enough of a soul to admit that we should have had more frequent backups.

friendly_chap• 13 years ago

I think it is entirely clear from the writing that the author is a humble being. I feel sorry for him, from the writing it seems he is a much better person and engineer than most of the others at that company, pointing fingers at him.

The guy may be absentminded, but that is a trait of some of the brighest people on earth.

aidos• 13 years ago

Ah ha ha ah yeah.... I've done that.

Something similar anyway (was deleting rows from production and hadn't selected the where clause of my query before I ran it).

It was on my VERY FIRST DAY of a new job.

Fortunately they were able to restore a several hours old copy from a sync to dev but the wasn't a real plan in place for dealing with such a situation. There could have just as easily not been a recent backup.

This was in a company with 1,000 employees (dev team of 50) and millions in turnover. I've worked other places that are in such a precarious position too.

At least my boss at the time took responsibility for it - new dev (junior), first day, production db = bad idea.

shocks2• 13 years ago

I'm getting a 500 error. Anyone else? Anyone got a mirror or able to paste the content?

shocks2• 13 years ago

Finally got through. Mirrored here in case other people get 500 errors. http://pastie.org/6348763

superflit• 13 years ago

I would fire the CTO for canceling the backups.

NEVER.. NEVER go production without backup.

Backup is not only to 'recover' but to have a 'historical' data for audits, check for intrusion, etc.

And the 'other' guy on Skype?

'You are lucky to be here..'

Seriously? You are lucky to still talk using skype because I am sure skype has some kind of backup at their user table..

cmos2• 13 years ago

When I was 18 I took out half my towns power for 30 minutes with a bad scada command. It was my summer job before college and I went from cleaning the warehouse to programming the main SCADA control system in a couple weeks.

Alarms went off, people came running in freaking out, trucks started rolling out to survey the damage, hospitals started calling about people on life support and how the backup generators were not operational, old people started calling about how they require AC to stay alive and should they take their loved ones on machines to the hospital soon.

My boss was pretty chill about it. "Now you know not to do that" were his words of wisdom, and I continued programming the system for the next 4 summers with no real mistakes.

gnarbarian2• 13 years ago

I'm interested in knowing some more details regarding the architectural setup and organizational structure that would allow something like this to happen.

sophacles2• 13 years ago

Honestly, you don't. The IT engineering in power and other SCADA systems is downright scary.

cmos2• 13 years ago

It was the small New England town I grew up in outside of Boston. Typically they manage their own power distribution, and purchase power from a utility at a rate fixed by the highest peak usage of the year, which tends to be 1:00 sometime in early August.

I was setting up a system to detect the peak usage and alert the attendant to send a fax (1992) to our local college generator plant to switch over some of their power to us, thus reducing our yearly power bill by a lot.

There was a title grabber in charge of the department, and a highly competent engineer running everything and keeping the crews going. That was my boss. Each summer I was doing new and crazy things, from house calls for bad TV signals from grounding to mapping out the entire grid. Oddly enough there was no documentation as to what house was on what circuit.. it was in the heads of the old linesman. Most of the time when they got a call about power outages they drove around looking at what houses were dark and what were not.

Sometimes the older linesmen would call me on the radio to meet them somewhere, and we'd end up in their back yards taking a break having some beers. I learned a lot from those old curmudgeonly linesmen. They made fun of me non stop, but always with a wink and roaring laughter. Especially when I cut power to half the town.

grey-area4• 13 years ago

Tens of thousands of paying customers and no backups?

No staging environment (from which ad-hoc backups could have been restored)!?!?

No regular testing of backups to ensure they work?

No local backups on dev machines?!?

Using a GUI tool for db management on the live db?!?!?

No migrations!?!?!

Junior devs (or any devs) testing changes on the live db and wiping tables?!?!?!

What an astonishing failure of process. The higher ups are definitely far more responsible than some junior developer for this, he shouldn't have been allowed near the live database in the first place until he was ready to take changes live, and then only on to a staging environment using migrations of some kind which could then be replayed on live.

They need one of these to start with, then some process:

http://www.bnj.com/cowboy-coding-pink-sombrero/

juan_juarez• 13 years ago

Hell - this could have been avoided if they weren't using graphical tools or had a database that used transactions.

krallja2• 13 years ago

Neither of those things would have fixed this problem.

1. "Oops, I wrote TRUNCATE TABLE User instead of TRUNCATE TABLE Raids"

2. Transaction complete. ... ... "oops!"

ptaipale3• 13 years ago

Not really. You obviously haven't (yet) done anything like update t1 set status=0; where status=4; when you wanted to release (set status to 0) objects that are stuck in state 4, and let all other objects keep their existing statuses.

This is an easy mistake to make on command line. I hate GUIs too but not having one doesn't really help when your fundamental operating model is wrong.

jakejake2• 13 years ago

My hypothesis is that it's a game company and all of the focus was on the game code. The lowly job of maintaining the state server was punted off to the "junior dev" just out of school. Nobody was paying attention. It was something that just ran.

They paid the price of ignoring what was actually the most critical part of their business.

grey-area4• 13 years ago

Sounds convincing. I guess now they'll realise their mistake though as the servers are critical to their business.

ptaipale3• 13 years ago

I disagree slightly. If you're a game company, your most critical part of the business is the game.

Even if you have a rock-solid database management, backup, auditing etc process, if your game is not playable, you won't have any data that you could lose by having a DB admin mis-click.

Still, not handling your next-most-critical data properly is monumentally stupid and a collective failure of everyone who should have known.

narrator• 13 years ago

The development environment should not be able to make a direct connection to production. GitHub temporarily deleted their whole prod database because of a config screwup because the dev server could talk to the production db. https://github.com/blog/744-today-s-outage

grey-area4• 13 years ago

I wasn't suggesting it should, ideally they'd be on completely isolated machines, and there's no reason it has to connect to production. Just because you use production backups to set up your dev environment, doesn't mean the dev environment should be able to talk to production servers, quite the opposite.

What I'd normally do is have a production server, which has daily backups, copies of which are used for dev on local machines, and then pushed to a dev server with separate dev db which is wiped periodically with that production data (a useful test of restoring backups), and has no connection with the production server or db.

Can't work out why they would possibly be doing development on a live db like this, that's insanity.

Kiro• 13 years ago

What do you mean by "no migrations"?

beilabs• 13 years ago

Think of your database in terms of code version control. You want to track the changes that are made to it, adding a column could be a migration, renaming a column could be another.

A migration enables you to track the changes you made and possibly rollback to previous migrations (database states) if ever required.

Domenic_S2• 13 years ago

> Using a GUI tool for db management on the live db?!?!?

I still use the mysql CLI and have for 10 years plus-or-minus, but I actually use Sequel Pro a lot. If I'm perusing tables with millions of rows, or I want to quickly see a schema, or whatever, it's been a net gain in productivity.

http://www.sequelpro.com/

hugoc• 13 years ago

I worked for the largest cellphone carrier in my country. I had write permission for the production db (not to all views, though) from my second week there onwards, the first week I used the credentials of the guy training me. The guy training knew the whole thing was wrong, mainly because once he ran a query that froze the db for half day. I was not a developer I was working in the help desk.

Spooky23• 13 years ago

If I were the author, I would rewrite this and reflect on what was actually wrong here. At the end of the day, you resigned out of shame for a serious incident that you triggered.

But the fact that the organization allowed you to get to that point is the issue. Forget about the engineering issues and general organizational incompetence... the human side is the most incredibly, amazingly ridiculous.

I respect your restraint. If I was singled out with physical intimidation by some asshat boss while getting browbeaten by some other asshat via Skype, I probably would have taken a swing at the guy.

Competent leadership would start with a 5-why's exercise. Find out why it happened, why even the simplest controls were not implemented. I've worked in places running on a shoestring, but the people in charge were well aware of what that meant.

desireco422• 13 years ago

I was working two years ago in very successful, billion dollar startup. All developers had production access, but then, if you didn't know what you were doing, you would not be working there. Also, we didn't routinely access production and when we did, mostly for support issues on which we rotated, we did through 'rails console' environment that enforced business rules. In theory you could delete all data, but only in theory, and even then, we could restore it with minimal downtime.

I think it is obvious that CEO/CTO are the one to be held responsible here.

desireco422• 13 years ago

To add to this. I work again in billion dollar company (I think, they are really big0, I don't work on their main property, I have production db access. This is something senior developers definitely should have access to, but also with great privilege comes appropriate responsibility.

I routinely run reports, and sometimes I would wipe spammer out that passed our filters etc.

cafard• 13 years ago

He did well to leave a company that a) had such practices in place and b) would hang out an inexperienced employee to dry like that.

Morendil2• 13 years ago

So the person who made a split-second mistake while doing his all for the business was pressured into resigning - basically, got fired.

What I want to know is what happened to whoever decided that backups were a dispensable luxury? In 2010?

There's a rule that appears in Jerry Weinberg's writings - the person responsible for a X million dollar mistake (and who should be fired over such a mistake) is whoever has controlling authority over X million dollars' worth of the company's activities.

A company-killing mistake should result in the firing of the CEO, not in that of the low-level employee who committed the mistake. That's what C-level responsibility means.

(I had the same thing happen to me in the late 1990's, got fired over it. Sued my employer, who opted to settle out of court for a good sum of money to me. They knew full well they had no leg to stand on.)

tetsuseus• 13 years ago

I once fired everyone at a nonprofit foster care company with a careless query.

I cried to the sysops guy, and he gave me a full backup from 12 hours before, and before any cronjobs ran I had the database back in order.

Backups are free. It was their fault for not securing a critical asset to their business model.

greghinch• 13 years ago

Whoever a) gave production db access to a "junior" engineer and b) disabled backups of said database is at fault. I hope the author takes this more as a learning experience of how to (not) run a tech department than any personal fault.

Someone who has to use a GUI to manage a db at a company of that scale shouldn't have access to prod

developingJim• 13 years ago

Know a lot of others have said it, but no production backups? Blame a junior dev for a mistake that almost 100% of the people I've worked with have made at some point or another (including me)? I feel horrible for the author, it's sickening the way he was treated. Now they'll just move on, hire another junior, never mention this, and guess what? The next guy will do the same thing and there probably still aren't any backups. Didn't learn anything, well, other than how easy it is to blame one person for everyone's failure.

stretchwithme2• 13 years ago

While working for a large computer company in the late 90s, I joined a team that ran the company store on the web. The store used the company's own e-commerce system, which it was also selling.

The very first day, at home in the evening, I went to the production site to see if I could log in as root using the default password. Not a problem.

Anyone with any experience with the product could easily have deleted the entire database. I immediately changed the password and emailed the whole team.

No one ever responded.

stretchwithme2• 13 years ago

This is handing a heart for transplant to the Post Office and hoping for the best.

meshko• 13 years ago

I really hope this is some kind of hoax and no real company was operating like that in 2010.

JohnBooty• 13 years ago

This happened to me once on a much smaller scale. Forgot the "where" clause on a DELETE statement. My screwup, obviously.

We actually had a continuous internal backup plan, but when I requested a restore, the IT guy told me they were backing up everything but the databases, since "they were always in use."

(Let that sink in for a second. The IT team actually thought that was an acceptable state of affairs: "Uh, yeah! We're backing up! Yeah! Well, some things. Most things. The files that don't get like, used and stuff.")

That day was one of the lowest feelings I ever had, and that screwup "only" cost us a few thousand dollars as opposed to the millions of dollars the blog post author's mistake cost the company. I literally can't imagine how he felt.

anywherenotes• 13 years ago

That is pretty hilarious. I guess you can save a lot of money on tapes, if you do incremental backups only on files that never change.

Personally I felt bad when I deleted some files, that were recovered within the hour, and learned from that experience. But when you create a monumental setback as the OP by simple mistake, that's an issue with people at higher ranks.

rhizome2• 13 years ago

To be sure, when someone is saying your fuckup is "costing them millions of dollars," they're usually lying and just being a dick.

banachtarski• 13 years ago

What company that makes millions in revenue doesn't replicate their database or at least have snapshots?

What engineer uses any GUI to administrate MySQL?

This story feels totally unreal to me (unreal as in just crazy, not disbelief).

anovikov• 13 years ago

LOL a gaming startup i worked for in 2010 had the same fuckup! But nobody was fired or quit, there was just a total anger around the place for a few days, and almost all data was eventually recovered. The startup still flopped in about one year after that with ever falling user retention rates - the marketplace was more and more flooded with those more and more similar games.

nighthawk• 13 years ago

the monumental fuck up was cancelling mysql backup and having all engineers work directly with the production database, what you did was INEVITABLE..

mydpy• 13 years ago

My name is Myles. I read this and felt like I was looking into a crystal ball. Fortunately, my work doesn't require I interact with the production database (yet). Gulp.

blisterpeanuts• 13 years ago

Good lord, that's unbelievable! If millions of dollars are riding on a database, they should have spent a few thousand to replicate the database, run daily backups and maintain large enough rollback buffers to reverse an accidental DROP or DELETE.

We've all screwed up at various times (sometimes well beyond junior phase), but not to have backups.... That's the senior management's fault.

doktrin2• 13 years ago

I found myself doing very much this my very first day on the job working for a software startup.

We had a Grails app that acted as a front end for a number of common DB interactions, which were selected via a drop down. One of these (in fact, the default) action was titled "init DB". Of course, this would drop any existing database and initialize a new one.

When running through the operational workflow with our COO on the largest production database we had, I found myself sleepily clicking through the menu options without changing the default value. I vividly remember the out of body experience the OP describes, and in fact offered to fire myself on the spot shortly thereafter.

It's fun to laugh about in hindsight, but utterly terrifying in the moment - to say nothing of the highly destructive impact it had on my self confidence.

roryokane• 13 years ago

How did the company deal with the loss of that database? Did they actually have backups, and just restored the data? Did they reconstruct the data from other sources?

doktrin2• 13 years ago

In our case we had periodic backups, and together with filesystem logs were able to restore most of the data. However, we were hosting highly sensitive data and the work being done was time critical. The downtime was therefore not popular with our clients, who were losing ~$15k per hour offline.

gte910h2• 13 years ago

He worked at a company stupid enough to test on the prod databases without tools to safely clear them. The former is stupid, the later is REALLY stupid.

This is a multi-layer failure and almost none of the blame falls on him. Stupid compounded stupid, and this guy did nothing more than trip over the server cord several people who knew better stupidly ran past his cube exit.

johngalt2• 13 years ago

"How I was setup to fail."

Who asks junior engineer to develop directly on live systems with write access and no backup? Are you kidding me?

Edit: No one ever builds a business thinking about this stuff, until something like this happens. There are people who have learned about operations practices the hard way, and those who are about to. They hung the author out to dry for a collective failure and it shows that this shop is going to be taught another expensive lesson.

kibwen• 13 years ago

"I found myself on the phone to Rackspace, leaning on a desk for support, listening to their engineer patiently explain that backups for this MySQL instance had been cancelled over 2 months ago."

Here's something I don't get: didn't Rackspace have their own daily backups of the production server, e.g. in case their primary facility was annihilated by a meteor (or some more mundane reason, like hard drive corruption)?

Regardless, here's a thought experiment: suppose that Rackspace did keep daily backups of every MySQL instance in their care, even if you're not paying for the backup service. Now suppose they get a frantic call from a client who's not paying for backups, asking if they have any. How much of a ridiculous markup would Rackspace need to charge to give the client access this unpaid-for backup, in order to make the back-up-every-database policy profitable? I'm guessing this depends on 1) the frequency of frantic phone calls, 2) the average size of a database that they aren't being paid to back up, and 3) the importance and irreplacebility of the data that they're handling (and 4) the irresponsibility of their major clients).

pbhjpbhj• 13 years ago

Nope, not going to happen. At least one good reason and that is that if Rackspace leak your data via a backup they're going down to the tune of millions.

Yes it would be nice if Rackspace could speculatively create a backup but they'd be dancing on ice doing so.

zulfishah• 13 years ago

Every engineer's worst nightmare. I've worked at a one of the biggest software companies in the world, and I'm working on my own self-funded one-person startup: the panic before doing anything remotely involving production user data is still always nerve-wracking to me. But agree with everyone's assessments here of the failure of the whole company to prevent this. A hardware failure could have just as likely have wiped out all their data. If you're going to cut corners with backing up user data, then you should be prepared to suffer the consequences.

Thanks for sharing this. Took real guts to put it out there.

nekitamo• 13 years ago

Using LinkedIn, you can easily figure out the name of the company and the name of the game. Using CrunchBase you can figure out the name of the CEO.

munificent• 13 years ago

> The CEO leaned across the table, got in my face, and said, "this, is a monumental fuck up. You're gonna cost us millions in revenue".

Yes, it is a monumental fuck-up. You put a button in front of a junior developer that can cost the company millions if he accidentally clicks it and doesn't even have undo.

jtchang• 13 years ago

I'm with everyone else in this thread: you screwed up but in reality it is EXPECTED.

Do you know why I have backups? Because I'm not perfect and I know one day I will screw up and somehow drop the production database. Or mess up a migration. Or someone else will. This is stuff that happens ALL THE TIME.

Your CEO/CTO should have been fired instead. It is up to the leadership to ensure that proper safeguards are in place to avoid these difficult conversations.

madao• 13 years ago

What I want to know is, why didnt the guy who canceled the database backups get fired also?

alexrson• 13 years ago

If your data is not backed up it may as well not exist.

rhizome2• 13 years ago

The CEO leaned across the table, got in my face, and said, "this, is a monumental fuck up. You're gonna cost us millions in revenue". His co-founder (remotely present via Skype) chimed in "you're lucky to still be here".

This is when you should have left. That's no way to manage a crisis.

ferrouswheel2• 13 years ago

Ah, I remember being called away from my new year holiday when an engineer dropped our entire database.

This happened because they didn't realise they were connected to the production database (rather than their local dev instance). We were a business intelligence company, so that data was vital. Luckily we had a analysis cluster we could restore from, but afterwards I ensured that backups were happening... never again.

(Why were the backups not already set up? Because they were not trivial due to the size of the cluster and having only been CTO for a few months there was a long list of things that were urgently needed)

greyboy• 13 years ago

This brings to mind on of my common responses: if it's not important enough to back up - it's not important!

It may be expensive, either in complexity, costs of storage/services, etc, but it's a necessity.

I'm curious about many of the comments in this thread - why are people logging in as table owners? It's not too difficult (for talented data-driven companies) to create roles or accounts that, while powerful, still make it difficult to drop a table and such.

ferrouswheel2• 13 years ago

One answer... MongoDB.

jinfiesto• 13 years ago

This has been stated by others, but it's not the author's fault. It's totally idiotic for a database like that not to have been regularly backed up. At worst, this should have been only a couple hours of down time while the database was restored.

alan_cx• 13 years ago

I cannot believe that people still don't have reliable back up in place.

My feeling is this: If you are in any way responsible for data that is not backed up, you should be fired or resign right now. You should never work in IT, in anyway, ever again. If you are the CEO of a company in a similar state, again, fire your self right now. Vow to never ever run a business again. This is 2013. And guess what? You still can't buy your unique data back from PCWorld. Your data is "the precious".

As for the treatment of this guy, IMHO, his employers were the worst kind of spineless cowards. This was 100% the fault of the management, and you know what? They know it. To not have backups is negligent, and should result in a high up firings. Yet these limp cowards sought to blame this kid. Pure corporate filth of the lowest order. Even the fact he was junior is irrelevant, any one could have done that, more likely a cocky senior taking some short cut. Let me tell you now, I have made a similar cock up, and I think I know it all. But I had backups, and lucky for me, it was out of business hours. Quick restore, and the users never knew. I did fess up to my team since I thought it had direct value as a cautionary tail.

Frankly, I am utterly amazed and gutted that such a thing can still happen. The corporate cowardice is sadly expected, but to not have backups is literally unforgivable negligence.

Yeah, Im quite fundamentalist about data and backups. I'd almost refer to my self as a backup jihadist.

praptak• 13 years ago

They should reward him. Seriously, anyone who exposed such a huge weakness deserves a reward. He limited the damage to only 10k users' data loss. With such abysmally crappy practices the damage would happen anyway only perhaps with 30k users and who knows what else instead of a mere 10k.

SonicSoul• 13 years ago

this is insanity! it was already pointed out in comments but i still can't believe a company that mature (actually having 1000's of users and millions in revenue!!!) would omit such a basic security precaution. Giving [junior?!] developers free reign in production database and no backups???? seriously, the CTO should have been fired on the spot instead of putting blame on developer.

no matter how careful you are (i'm extremely careful) when working with data, if you're working in dev/qa/uat/prd sooner or later someone on dev team will execute on wrong environment.

elicash• 13 years ago

I think it's admirable that you stayed long enough to help fix everything before quitting, despite it being rough -- even though, as others have said, others screwed up even bigger than you did.

lexilewtan• 13 years ago

So many structures in life are based around 'not fucking up.' We protect our assets & our dignity as if they mean anything; and yet at the end of the day nobody knows what the fuck is going on.

really simple, revealing story. kudos.

Yare• 13 years ago

If it helps explain things, the only experience the CEO had before this social game shop was running a literal one-man yogurt shop.

This happened a week before I started as a Senior Software Engineer. I remember getting pulled into a meeting where several managers who knew nothing about technology were desperately trying to place blame, figure out how to avoid this in the future, and so on.

"There should have been automated backups. That's really the only thing inexcusable here.", I said.

The "producer" (no experience, is now a director of operations, I think?) running the meeting said that was all well and good, but what else could we do to ensure that nobody makes this mistake again? "People are going to make mistakes", I said, "what you need to focus on is how to prevent it from sinking the company. All you need for that is backups. It's not the engineer's fault.". I was largely ignored (which eventually proved to be a pattern) and so went on about my business.

And business was dumb. I had to fix an awful lot of technical things in my time there.

When I started, only half of the client code was in version control. And it wasn't even the most recent shipped version. Where was the most recent version? On a Mac Mini that floated around the office somewhere. People did their AS3 programming in notepad or directly on the timeline. There were no automated builds, and builds were pushed from peoples' local machines -often contaminated by other stuff they were working on. Art content live on our CDN may have had source (PSD/FLA) distributed among a dozen artist machines, or else the source for it was completely lost.

That was just the technical side. The business/management side was and is actually more hilarious. I have enough stories from that place to fill a hundred posts, but you can probably get a pretty good idea by imagining a yogurt-salesman-cum-CEO, his disbarred ebay art fraudster partner, and other friends directing the efforts of senior software engineers, artists, and other game developers. It was a god damn sitcom every day. Not to mention all of the labor law violations. Post-acquisition is a whole 'nother anthology of tales of hilarious incompetence. I should write a book.

I recall having lunch with the author when he asked me "What should I do?". I told him that he should leave. In hindsight, it might have been the best advice I ever gave.

KenL• 13 years ago

I agree with the comments here that spread the blame past this author.

I manage a large number of people at a news .com site and know that screw-ups are always a combination of two factors: people & systems.

People are human and will make mistakes. We as upper management have to understand that and create systems, of various tolerance, that deal with those mistakes.

If you're running a system allowing a low-level kid to erase your data, that was your fault.

I'd never fire someone for making a stupid mistake unless it was a pattern.

jdmaresco• 13 years ago

I had a similar situation when collaborating with a team on a video project during a high school internship. Somehow I managed to delete the entire timeline accounting for hours of editing work that my boss had put in. To this day I don't know how it happened, I just looked down and all the clips were gone from the timeline. In the end, I think we found some semblance of a backup, and at least we didn't lose the raw data/video content, but I can relate to the out-of-body experience that hits you when you realize you just royally screwed up your team's progress and there's nothing you can do about it.

KVFinn• 13 years ago

>I was 22 and working at a Social Gaming startup in California.

>Part of my naive testing process involved manually clearing the RAIDS table, to then recreate it programatically.

>Listening to their engineer patiently explain that backups for this MySQL instance had been cancelled over 2 months ago.

"The CEO leaned across the table, got in my face, and said, "this, is a monumental fuck up. You're gonna cost us millions in revenue".

What. The. Fuck.

The LAST person I would blame is the brand new programmer. They don't backup up their production database? If it wasn't this particular incident it would have been someone else, or a hardware failure.

TheTechBox• 13 years ago

A lot of people have said it before on here but really?! The company is blaming on person, whilst yes it was technically his fault, why in the first place was he allowed on the production database and why was the company keeping very regular backups of all this mission critical data.

If the company saw that the data contained in this live database was so critical you would have thought that would not have given the keys to everyone and that if they did, they would at least make sure that they can recover from this, and fast.

rorrr• 13 years ago

I don't see how it's your fault, other than making a slight error of clicking on the wrong table name

1) Senior developers / CTO letting anybody mess with the prod DB should be grounds for their firing. It's so incompetent, it's insane.

2) No backups. How is this even possible. You even had paying customers.

fotoblur• 13 years ago

I think that everyone does this at some point in their career. Don't let this single event define you. The most important thing to ask yourself is what was the lesson learned...not only from your standpoint but also from the business'.

In addition, to heal your pain its best to hear that you're not the only one who has ever done this. Trust me, all engineers I know have a story like this. (Please share yours HN - Here I even started a thread for it: http://news.ycombinator.com/item?id=5295262)

Here is mine: When I worked for a financial institution my manager gave me a production level username and password to help me get through the mounds of red tape which usually prevented any real work from getting done. We were idealists at the time. Well I ended up typed that password wrong, more than 3 times...shit, I locked the account. Apparently half of production's apps were using this same account to access various parts of the network. Essentially, I brought down half our infrastructure in one afternoon.

Lesson learned: Don't use the same account for half your production apps. Not really my fault :).

Shorel2• 13 years ago

Awesome tale.

However, I think the CTO was the one who deserved to be fired.

Not having at the very least develop and production environments is a higher ups fault.

Where I work, developers can't even touch production systems, there's a separate team responsible for that.

I even have a solr, nginx, php, mysql, etc separate install of almost everything in my workstation, so I only touch test servers when doing testing.

logn• 13 years ago

Your response should have been: "With all due respect sirs, I agree that I am still lucky to be here, that the company is still here being that it's so poorly managed, that they cancelled their only backups with rackspace, that they had no contingency plans, and that you were one click from losing millions of dollars--in your estimate. It makes me wonder what other bills aren't being paid and what other procedures are woefully lacking. I will agree to help you though this mess and then we should all analyze every point of failure from all departments, and go from there."

ArenaSource• 13 years ago

I can't believe this history, but if it's true, don't worry, you just did what they deserve

samstave• 13 years ago

This was clearly a lack of oversight and sound engineering practices.

Who cancelled the backups? Why were they cancelled? Was it for non-payment of that service?

---

I worked for an absolutely terrible company as Director of IT. The CEO and CTO were clueless douchebags when it came to running a sound production operation.

The CTO would make patches to the production system on a REGULAR basis and break everything, with the statement "that's funny... that shouldn't have happened"

I had been pushing for dev|test|prod instances for a long time - and at first they appeared to be on-board.

When I put the budget and plan together, they scoffed at the cost, and reiterated how we needed to maintain better up-time metrics. Yet they refused to remove Dave's access to the production systems.

After a few more outages, and my very loud complaining to them that they were farking up the system by their inability to control access - I saw that they had been hunting for my replacement.

They were trying to blame me for the outages and ignoring their own operational faults.

I found another job and left - they offered me $5,000s to not disparage them after I left. I refused the money and told them to fark off. I was not going to lose professional credibility to their idiocy.

Worst company I have ever worked for.

hkmurakami• 13 years ago

I'm. a little worried for OP because he obviously took the time to keep the characters in this article anonymous, but we now know who this CEO with ridiculous behavior must have been, since we know the name of OP's former company from his profile. Not sure what said former CEO of Noe acquired company can do, but this is the kind of thing I fear happeneing toe when/if I write something negative about a past employer, being a blogger myself.

elhowell• 13 years ago

Wow dude, that's quite a story. That must have been an awful feeling, I hope you're doing better now

capex• 13 years ago

The senior engineers got to own the mistakes of their juniors. That's how teams are built. This clearly didn't happen in this case.

chris_mahan3• 13 years ago

Let me make it really simple: Anything that happens in a company is always, always management's fault. The investors hire the management ream to turn a pile of money into a bigger pile of money, and if management fails, it is management's fault, because it can do whatever it needs to do (within the law) to make that happen. That they failed to hire, train, motivate, fire, promote, follow the law, develop the right products, market them well, ensure reliability, ensure business sustainability, ensure reinvestment in research and development, and ultimately satisfy investor, is their fault, and they further demonstrate their failure by not taking responsibility for their own failure and blaming others.

ballstothewalls• 13 years ago

This is the most ridiculous thing ever. Why weren't there backups? Sure, the author was the one who "pulled the trigger" but the management "loaded the gun" by not making sure there were back-ups.

zensavona• 13 years ago

How is it that a company with 'millions in revenue' is directing a junior developer to develop on a production database with no backups?

outside1234• 13 years ago

this is a great example of why you run the "five whys?" after a failure like this.

The CEO/CTO should have fired himself as the answer of one of those.

joeblau• 13 years ago

On the bright side, now you know not to test on the production database :).

lkrubner• 13 years ago

Klicknation is hiring. Of themselves, they say:

"We make astonishingly fun, ferociously addictive games that run on social networks. ...KlickNation boasts a team of extremely smart, interesting people who have, between them, built several startups (successful and otherwise); written a novel; directed music videos; run game fan sites; illustrated for Marvel Comics and Dynamite Entertainment with franchises like Xmen, Punisher, and Red Sonja; worked on hit games like Tony Hawk and X-Men games; performed in rock bands; worked for independent and major record lables; attended universities like Harvard, Stanford, Dartmouth, UC Berkeley; received a PhD and other fancy degrees; and built a fully-functional MAME arcade machine."

And this is hilarious: their "careers" page gives me a 404:

http://www.klicknation.com/careers/

That link to "careers" is from this page:

http://www.klicknation.com/contact/

I am tempted to apply simply to be able to ask them about this. It would be interesting to hear if they have a different version of this story, if it is all true.

tn13• 13 years ago

I dont think this is author's fault. These kind of human mistakes are more than common. It is said that the top management actually assigned the blame to this young man. This was an engineering failure.

I can understand what this person must have gone through.

coolSCV• 13 years ago

This is why you have backups.

snambi• 13 years ago

This company sucks. You are out of college and doing the first job. Are these stupid enough to give you direct access to production database? if they are making millions in revenue where was there DBAs? Obviously the management got what they deserved. Its unfortunate that it happened through you.

Nikolas0• 13 years ago

To be honest as a CEO I would fire myself for letting someone in the team work that wrong (I mean in the production server)

Plus there is no excuse for not having backups...

monkeyonahill• 13 years ago

That wasn't your failure per se. But the failure of pretty much everyone above you. That they treated you like that after the fact is pretty shitty. In hindsight I'd say that you are much better off by not being there, where you would learn bad practices.

No Stage Environment. Proactively Cancelled Backups on a Business Critical System. Arbitarily implementing features 'because they have it' rather than it having some purpose in the business model. No Test Drills of disaster scenarios. The list goes on. As I say, and you probably realise now, that you are lucky to no longer be there.

misleading_name• 13 years ago

It's not your fault. it's the fault of the person who cancelled backups, the person who didn't check that backups are being created, the "senior" people who let you work on the production database.... etc.

ommunist• 13 years ago

You are accidental hero and should be proud! You freed thousands of souls from one of the worse digital addictions.

bwb• 13 years ago

It was a mistake, but not huge. You should never have not have had backups, and that wasn't your responsibility. + their should have been a dev instances and a proper coding environment.

So don't blame yourself there!

jgeerts• 13 years ago

This article sounds so incredible to me, I think I might have been holding my breath reading it. These are two major mistakes that the company is responsible for, not the author. Why would they let anyone in on the production password and do direct queries onto that database instead of working on a different environment, it's laughable that they sent this to their customers admitting their amateurism. Secondly, no backups? At my previous project, a similar thing happened to our scrum master, he accidently dropped the whole production database in some kind of the same situation. The database was back up in less than 10 minutes with an earlier version. It's still a mistake that should not be possible to make, but when it happens you should have a backup.

bart42_0• 13 years ago

You can't make an omelette without breaking eggs.

Clicking on 'delete' with the user table selected was not very wise. The software maybe even asked 'Are you sure?' and of course you reply 'yes'.

But operating your company with proper recovery tools is a bit like climbing Mount everest without a rope.

If something goes wrong you are in deep sh.t.

kunil• 13 years ago

But why did you cleared users table at the first place? I don't get it.

flayman• 13 years ago

This is not your fault. Not really. And it's galling that the company blamed the incident on the workings of a 'junior engineer'. There was NO DATABASE BACKUP! For Christ's sake. This is live commercial production data. No disaster recovery plan at all. Zilch. And to make matters worse, you were expected to work with a production database when doing development work. This company has not done nearly enough to mitigate serious risks. I don't blame you for quitting. I would. I hope you have found or manage to find a good replacement role.

ThePhysicist• 13 years ago

The only people that should have gotten fired for this are:

* The person responsible for the database backup (no backup plan for your production DB!? wtf)

* The person having designed the SQL admin tool (not putting an irreversible DELETE operation behind a confirmation dialogue!? wtf)

* The person giving full write access to the company's production database to a junior developer (data security!? wtf)

Sure, the employee made a mistake, but most of the failure here is due to the bad management and bad organizational design.

deciob• 13 years ago

This is the most incredible story I have read in a long time. To have such a business relying all on one database and no backups... unbelievable!

alyrik• 13 years ago

Oh dear... I once logged into the postgresql database of a very busy hosted service in order to manually reset a user's password. So I started to write the query:

UPDATE principals SET password='

Then I went and did all the stuff required to work out the correctly hashed and salted password format, then finally triumphantly pasted it in, followed by '; and newline.

FORGOT THE WHERE CLAUSE.

(Luckily, we had nightly backups as pg_dump files so I could just find the section full of "INSERT INTO principals..." lines and paste in a rename of the old table, the CREATE TABLE from the dump, and all the INSERT INTOs, and it was back in under a minute - short enough that anybody who got a login failure tried again and then it worked, as we didn't get any phonecalls). It was a most upsetting experience for me, however...

Tichy• 13 years ago

Just wondering, when consulting I usually take care that there are appropriate clauses in the contract to make me not liable. But what is the rule for employees, are they automatically insured?

In Germany there is the concept of "Fahrlässig" (negligence) and "severe negligence". Per law you are already liable if you are just negligent, but it is possible to lower it to severe negligence in the contract. That is my understanding anyway (not a lawyer). Usually I also try to kind of weasel out of it by saying the client is responsible for appropriate testing and stuff like that... Overall it is a huge problem, though, especially if the client has a law department. Getting insurance is quite expensive because it's easy to create millions of dollars in damages in IT.

Before court "standard best practices" can become an issue, too. This worries me because I don't agree with all the latest fads in software development. It seems possible that in the future x% test coverage could be required by law, for example. Or even today a client could argue that I didn't adhere to standard best practices if I don't have at least 80% test coverage (or whatever, not sure what a reasonable number would be).

zaidf• 13 years ago

It seems insane that you still worked 3 days in a row after a gigantic mistake that can be attributed in good part to being overworked.

Once the damage was done, I would have sent you home instead of overworking you further.

mac1175• 13 years ago

Wow. This reminds me of a time in which I used to work for a consulting agency. It was back in 2003 and I was working on a some database development for one of the company's biggest clients. One day, I noticed the msdb database had a strange icon telling me it was corrupted. I went onto MSDN and followed some instructions to fix it and, BAM, the database I was working for months on was gone (I was running SQL Server 2000 locally where this all happened and I was very junior as a SQL developer). I was silently freaking out knowing this could cost me my job. I got up from my desk and took a walk. On that walk, I contemplated my resignation. When I got back from my walk, a thought occurred to me that maybe the database file is still there (I had zero clue at the time that msdb's main purpose was just cataloguing the existing databases among other things). I did a file search in the MSSQL folders and found a file named with my database's name. So, that day I learned to attach a database, what msdb's role is, and to make sure to take precautions before making a fix! However, OP's post shows that this company had no processes in place control levels of access or disaster recovery. Show the company's faults more than OP's.

ownagefool• 13 years ago

I worked at a small web hosting company that did probably £2m in revenue a year in my first programming job. They had me spending part of my time as support and the other part on projects.

After about 3 or so months they took me took me out of support and literally placed my desk next to the only full time programmer that company had.

They made all changes direct on live servers and I'd already raised this as a concern and now that became my full time job it was agreed that I'd be allowed to create a dev environment.

Long story short, I exported the structure of our MySQL database and imported it into dev. Some variable was wrong so it didn't all import, so I changed the variable, dropped the schema and back to redo.

Yeah that was the live database I just dropped. After a horrible feeling that I can't really explain I fessed up. I dropped it during lunch so it took about two hours to get a restore.

The owner went mad but most other people were sympathetic, telling me their big mistakes and telling me thats what backups were for.

The owner was going crazy about losing money or something and the COO pulled me into a room. I thought I was getting fired but he just asked me what happened and said "yeah we all make mistakes, thats fair enough, just try not to do it again".

I was then told to get on with it and it must have took me a day to finish what would have taken me an hour but I done it and now we had a process and a simple dev environment. I lasted another two years there. I left over money.

smallegan• 13 years ago

This reads like a PSA for backups and RI.

geldedus• 13 years ago

it is the fault of those responsible with creating a regular backup procedure and/or a hotswap database server

and developping on the production database speaks volumes about the incompetency level of that company and of the "developer" in particular, afterall

shaurz2• 13 years ago

It's an organisational failure if a junior employee can bring down the company in a few clicks. No backups, testing on the production database, this is no way to run a company. Feel sorry for the guy who made a simple mistake.

toomuchcoffee• 13 years ago

The CEO leaned across the table, got in my face, and said, "this is a MONUMENTAL fuck up."

It certainly was -- on multiple levels, but ultimately up at the C-level. Blaming a single person (let a lone a junior engineer) for it just perpetuates the inbred culture of clusterfuckitude and cover-ass-ery which no doubt was the true root cause of the fuck-up in the first place.

bfrenchak• 13 years ago

This was not a junior engineer's fault, but the DBA's fault. Any company should be backing up their database regularly, and then testing the restores regularly. Also don't give people access to drop table's etc. This was a very poor setup on the part of the company/DBA not the engineer.

fayyazkl• 13 years ago

While i fully agree with the position of author not being responsible entirely, i find it hard to believe it happened the way it appears to be.

It could be but there are bunch of loopholes. - I can believe that he was lousy enough to click no "delete" on users table. I can believe that he when the dialog box asked "are you sure you want to drop this table" he clicked yes. I can believe that after deleting he "committed" the transaction. But what i can't believe that the database let him delete a table which was base for every other table implemented by a Foriegn key constraint ? It could be argued that due to efficiency they hadn't put constraints on the table but it's hard to digest.

Probably the story is some what tailored to fit to a post.

lucb1e• 13 years ago

Let me get this straight.

- Tens of thousands of paying customers

- No backups

- Working in a production database

- Having the permissions to empty that table

- Even having read access to that table with customer info...

You are hardly responsible. Yeah you fucked up badly, but everyone makes mistakes. This was a big impact one and it sucks, but the effect it had was in now way your fault. The worst-case scenario should have been two hours of downtime and 1-day-old data being put back in the table, and even that could have been prevented easily with decent management.

jblotus• 13 years ago

just awesome

How I Fired Myself

Related Stories