Fired employee wipes out whole season of TV show

http://www.msnbc.msn.com/id/42366946/ns/technology_and_science-tech_and_gadgets/?gt1=43001

by briandear • 15 years ago

65 91 15 years ago

msnbc.msn.com

Splines• 15 years ago

Man, that sucks. Hopefully it's a lesson to everyone that you don't leave the keys to the kingdom to one guy, especially if they've just been fired.

r00fus• 15 years ago

No backups? 300GB doesn't sound like much. I just processed and shuttled 2.3TB today (sneakernet backup).

famousactress• 15 years ago

I was just thinking the same thing.. 300gb, mostly of presumably high-res art and video.. and that's 2 years work? Sounds really small.

robin_reala• 15 years ago

Two man-years maybe?

dagw• 15 years ago

2D Animation doesn't take up much space. Beyond a few painted high res background, most files are small and a lot of things are procedurally generated and take up almost no space at all. I worked on a 2D animated movie 10 years ago, and the data needed to render the final movie weighed in at less than 100 GB (and that was 2-3 years of work by 30-40 people). Also I understood from the story that they didn't loose everything, just enough to make it impossible to recreate the episodes without basically starting from scratch.

MichaelApproved• 15 years ago

the lost dada is said to include fragments of shows

It wasn't all their data but enough to make what was left unusable.

robryan2• 15 years ago

This is surprising, it would be like sitting my web apps code/ database on one live server without a local copy, and possibly multiple backups elsewhere. How can a TV production company justify this? I mean it would cost just about nothing relative to production costs to put 300gig on S3.

mbreese3• 15 years ago

You're assuming that the production company can afford to have someone around who knows how to put data on S3. My guess is that they outsourced this to the company who had the fired employee. Still, they should still have some kind of original files on some other medium somewhere... but who knows, maybe their production workflow revolved around this data-host. In which case they are really borked.

detst• 15 years ago

We have tons of decades-old video alive and well today. They may not have expertise with S3 but if they don't have the expertise to ensure robust backups, they're doing it wrong.

alexqgb7• 15 years ago

In other news, soon-to-be-fired producer learns the true meaning of 'backup'. Also, 'redundancy'.

maushu• 15 years ago

Isn't the data hosting company job to do backups and redundancy? Unless they clearly state that they don't do that.

gm4• 15 years ago

I'd say the opposite: Unless they clearly state they _do_ backups, they are not (legally) responsible for doing so.

alexqgb7• 15 years ago

Redundancy - by definition - cannot be supplied by a single company.

Yes, your vendor may have their own redundant systems in place so that their internal problems don't become your problems. And that's nice. They should. But you always have a Plan B. And you always keep a local copy.

mbreese3• 15 years ago

The thing that really sucks for the TV show is that it wasn't even their own employee. It was an employee for their data hosting company. You know... the guys who you're supposed to outsource to to make sure that this stuff doesn't happen.

d2• 15 years ago

You mean The Cloud has disgruntled employees?

derrida• 15 years ago

Idea: Outsource disgruntled employees to the cloud. Hit the cloud and money can come out of it.... sort of like Mario.

jrockway6• 15 years ago

You know... the guys who you're supposed to outsource to to make sure that this stuff doesn't happen.

Actually, outsourcing is designed to be cheap, not to lower risk. Trusting the lowest bidder to an important task usually results in disappointment.

pero2• 15 years ago

Actually, big corporations outsource to transfer risk and not because it's cheap. To compliance/legal departments, paying exorbitant fees to qualified vendors is saving money in their risk analyses.

gm4• 15 years ago

Unless "back up our data" is part of the hosting contract, the host is off the hook.

There's up to two fools here:

- If the hosting company does not have a disclaimer in their contract that the data backup onus is not on them, then they deserve to be sued for all they got.

- The TV producer should keep their own backup. Trusting a third party blindly with your most precious data is simply stupid, and unworthy of anyone involved with IT in this day and age. I would back up data even if the host did have a contractual obligation to back up my data. Shit happens, it's not a matter of "if", but of "when".

dhughes• 15 years ago

Can you hire him to wipe Jersey Shore?

dschobel• 15 years ago

Don't watch the show then. Or do you object to other people watching content you don't personally approve of?

Because there are many terms for that particular personality type, none of them flattering.

michaelcampbell2• 15 years ago

LiteralNet is in full force today, I see.

daemin• 15 years ago

Shouldn't the company have off-site backups of the data its hosting, like, say, on tape or something?

jrockway6• 15 years ago

That employee may be going to jail, but it's not his fault that this happened. If your fired employees have access to your servers, you fucked up the firing horribly. Similarly, if one data center going offline can take out your entire season of work, perhaps you need to learn what redundancy is.

uptown• 15 years ago

What you said about redundancy is true, but it's still his fault this happened.

jemfinch• 15 years ago

I would not be surprised if their "data hosting service" claimed redundancy as a feature.

ericd• 15 years ago

How is this not his fault?

jrockway6• 15 years ago

What I'm saying is that relying on crazy people to not go crazy is not a good data loss prevention strategy.

(In the end, putting someone in a situation where they'll do something bad is worse than being the person that does something bad. A good example is the Colgan Air crash from a few years ago. Anybody knows that when your plane is stalling you're supposed to push the control column forward and apply full engine power. The plane does the pushing the control column forward part automatically! But for some reason, the trained airline captain did the exact opposite, stalling the plane and killing everyone on board. Why? Because he got paid so little that he couldn't afford a hotel room to sleep in before his work that day, and had to commute across the country and then work a full shift. This is the airline's fault for impairing someone to the extent that he killed 50+ people by doing the exact opposite of what any trained pilot would do. $300 for a hotel room and everyone on that flight would still be alive today.

Similarly, when you fire someone that works with critical data, you need to make sure the guy's access is revoked. It's a precaution that needs to be taken, as this case shows. When necessary precautions are omitted, bad things happen. Planes crash, and important data goes away forever.)

random42• 15 years ago

Because when the company has to man up, and tell the reason of the Fuck up to the stakeholders, the should not be pointing fingers to a person who was not even on their payroll.

It was someone's job to avoid this from happening, or they do not have people covering such cases... either way, someone other (person or the company as a whole) than the fired employee messed up for the business/customers and they are liable/answerable.

Because people fail... all the time. Good, Smart, Sincere people fail. (not in this particular case, though)

This is just how things are. We all have heard numerous stories of rm -r / or drop database or something similar. Hell, the whole widely accept idea of "bugs" in software industry is based on the fact that people WILL do something wrong. Not because they are bad people, want to do bad things, but to err is human.

So your process should not be designed around the idea of people doing the right things always.

This is what I think he means (and I concur)

firemanx2• 15 years ago

I really don't get this philosophy. I've seen this a lot recently in various discussions on HN, and I'm honestly dumbfounded. How do you justify this line of thought, that negative action against someone, done deliberately, is not the perpetrator's fault?

I mean this completely academically, so please don't take it as an insult (I'm genuinely curious) - but was this perspective something you were raised with or did this develop out of experience / general personal exploration of the world?

palehose2• 15 years ago

A co-worker of mine once wiped out the entire CVS repository at the company we worked for because a symbolic link was placed in his home directory and he thought it was ok to 'rm -Rf' its contents. We were able to build a new repo based off of production and everyone who had changes not in production had to re-introduce those changes, but it was not the developer's fault for wiping out CVS. It was because the sysadmin had set the symbolic links for the primary repository. If you enable developers to do something bad, it isn't always their fault when they do it.

Having intent to destroy something on purpose is a different matter though, and I'm not sure that there is a good argument for it.

alexqgb7• 15 years ago

Personal experience, 100%

As a producer, I can safely say that your entire job is based around one rule: the show must go on. That means you have backups. And your backups have backups. And your backup's backup's have backups.

Is it your fault when bad things happen? If you're good at what you do, probably not - especially when you consider the AMAZING number of things that can go wrong on any given day. However, it IS your fault if the bad things that inevitably happen cause the show to fail because you had no plan B.

Maybe it's a disgruntled employee with a set of passwords he really shouldn't have. Or maybe it's an outbreak of anthrax (yeah, I was on a job where that was suddenly an issue). Perhaps you're shooting on a beach and a dead body randomly washes up, suddenly turning your carefully chosen location into a crime scene crawling with cops and coroners (another true story / very rough day.) God forbid it's something really awful, like a bike messenger carrying an important batch of media getting hit by some asshole who was answering his phone instead of paying attention to the road, landing the kid in hospital, and turning your package into collateral damage (horrific moments like this, by the way, make the job seem suddenly unimportant).

Regardless, the show must go on. And if you have a good producer, it will. But losing TWO YEARS worth of work because one data center (effectively) went offline? That's seriously embarrassing. Also, very bad for credibility. And yes, reason enough to loose your stripes.

swift• 15 years ago

I wouldn't say that it's not the employee's fault, but if you are suggesting that it's other peoples' fault as well, I'd definitely agree. In an incident like this a lot of people have to have been very negligent.

Confusion• 15 years ago

You judge without knowing enough details to do so. For all we know, the contract with the data center may have included regular backups, including off-site backups. In such a case, as a company, you've done everything you could and should have and the data center is entirely to blame, if the employee managed to destroy the backups as well.

SeoxyS• 15 years ago

Somewhat different situation, but after having quit one a Silicon Valley startup after its acquisition, I still have access to a lot of its network. I'm sure it's a common occurrence. Shutting people completely out of a network is really hard, where there are so many different accounts to close. (Unix access on all boxes, mysql, cms, email, bug-tracking, apple dev accts, irc, support forums, etc. etc.) Especially doing so without deleting data (which you might need in the future, to check on something that the employee worked on, or an old email containing documentation, etc…)

Rather, I think the issue is the lack of a good backup strategy. Off-site, automated backups should have been in place. I've seen someone `rm -rf /var/mysql` and, after a heart attack, recover it. Your system should be rm-proof!

davidandgoliath2• 15 years ago

You may want to find a way of notifying them without alerting them it's you. Otherwise if something goes awry down the road and one of your accounts is involved (say an old username that gets brute-forced), you don't get something pinned on you.

pyre• 15 years ago

A couple of those things are incredibly easy:

1. Don't use per-machine unix accounts. Use something like NIS or LDAP. Now you can easily disable someone's account on all boxes. THe bonus is that their home directories still stick around (though you can't use aliases like ~username to access them).

2. Use an email system where you can disable a user without deleting all of their emails.

ajb• 15 years ago

"it's not his fault that this happened"

This is wrong; and is a class of error which I see frequently, especially in political comment; both from the left and the right.

The underlying incorrect assumption is that responsibility is mutually exclusive.

In many cases, responsibilities are divided cleanly between people, but this is an organisational convenience, not a moral law. There are cases where it is inescapable that more than one person has the same responsibility. Consider an army sargeant: he may assign a task to a soldier, but still has the responsibility for seeing that it is done. This is true for all leadership positions.

Much political discourse revolves around who has responsibility, and frequently one sees it argued that A is not responsible because B is, often in cases where this is incorrect. Eg, employer versus employee, individual versus government, etc etc.

The error is made in both directions, depending on the political beliefs of the arguer: an individual is not responsibly because the organisation is, or the organisation is not responsible because the individual is..

A particularly dumb one, in my opinion, is the argument that government is not responsible for something, because individuals are. In my view, all the valid responsibilities of government are derived from responsibilities of the individual: the responsibility to defend the nation is derived from the individuals responsibility to protect him/herself and family, etc.

astrofinch• 15 years ago

Responsibility is one of those concepts that we humans made up and spend a lot of time uselessly arguing over. Other examples are whether an action was "justified", whether someone "deserves" something, who gets the credit for stuff, etc.

Examples I'm less sure about: Who "owns" something, whether something is "clean" or "dirty", whether two things are the "same", whether something is "normal" or not. I collect these; if anyone has more to share I'd love to hear them.

JonnieCache2• 15 years ago

This is called Fundamental Attribution Error. As the name suggests it is a foundational part of human social psychology and it has been studied for years.

Personally I'm convinced it is one of the cognitive flaws that are needed to stop us from going teeth-gnashingly insane after ten minutes exposure to reality.

https://secure.wikimedia.org/wikipedia/en/wiki/Fundamental_a...

See also: "She should have known what would happen to her, going out at night dressed like that."

No.

hartror• 15 years ago

I assume this is them: http://www.cyberlynk.net/ with the late 90s "new" graphic against off-site data backup. I hope the show producers sue someone, either Cyberlynk if they claimed they did redundant backup or their IT guy/company for setting them up with such a bum deal.

trafficlight2• 15 years ago

I just don't understand how there could have only been a single copy of those videos.

Retric2• 15 years ago

Who says he only deleted a single file? It's not hard to delete stuff from online backup systems.

Udo2• 15 years ago

No, I think what trafficlight meant was: how can there be only a single copy of this material? What about the media they actually shot the episodes on? Surely there must be some kind of copy on the cutting room computers? It looks as if they're mainly out to collect damages from the company.

dzorz• 15 years ago

April 1st?

ctdonath• 15 years ago

Must be. I thought the link would be about Charlie Sheen.

sabalaba• 15 years ago

(Warning anecdotal evidence) Having worked in China on a number of startups and worked alongside people who have had to fire employees, it's often common practice to just lock fired employees out of their systems. I've had this happen to people I know and had to do it once when letting a programmer go (which is nearly impossible to do after a probationary period due to the labor laws in the country, it's very very difficult to fire somebody, the company has to provide reason, pay social benefits, or face the employee in a court--forget it if you're a foreign company).

I have heard stories of letting people go to find out that all of the work that employee did for the last month or two deleted, source missing, machines just wiped clean.

I read that title and kind of laughed because, well, it is a seemingly commonplace response in some parts of the world.

yardie2• 15 years ago

In my case I gave a months notice I was resigning and was locked out immediately. Imagine how much it sucks to come into work everyday with nothing to do except go to lunch. When I wasn't offering the other developers advice on the code I had worked on or APIs I was familiar with I was just shooting the breeze.

This was the hole in the policy. It was designed to keep disgruntled employees from damaging the system but the ones leaving on good terms were treated exactly the same.

michaelcampbell2• 15 years ago

This is more/less common in certain industries, and perhaps partially based on regulatory issues too, although I'm guessing there.

I have worked on financial software the majority of my career; at one point in a Wall St. investment banking firm. The very explicit protocol there was if you were going to leave the 2 week notice was customary, but you would probably be locked out of everything instantly, and would be around for hands-off consulting at the whim of your manager for the duration. It was all handled very professionally and friendly, at least for me, and they let me keep working for the 2 weeks since anything I touched had a LOT of layers to go through before it ever saw production data.

monological• 15 years ago

Is the guys name really Micheal Scott? April fools!

tlrobinson• 15 years ago

For once I agree with the editorializing of a headline on Hacker News.

Gorm-Casper• 15 years ago

Am I the only one who found the idea with tiny bubbles at the scrollbar more interesting than the story itself? Front-end developers' curse :P

nvictor• 15 years ago

april fools?

nopassrecover• 15 years ago

Surely the data company has redundancy or backups or data recovery software available? Surely they considered the risk of a staff member deleting copies / a hacker / a fire.

Surely the producers have at least working copies somewhere (I can't see people live-editing on a cloud server). Surely they considered the risk of the company going bust / data loss / their own staff deleting the copies.

Would assume this was an April Fools thing it's that bad (except for the fact it quite obviously isn't due to the quite obvious reputation damage that will effectively destroy this company).

scythe• 15 years ago

Kind of ironic that this would happen on World Backup Day, perhaps?

http://worldbackupday.com/

pinwale• 15 years ago

The real site is http://worldbackupday.net

shadowpwner• 15 years ago

Interesting how this happens on the same day of World Backup day. (http://blog.reddit.com/2011/03/better-have-backup-plan-world...)

ThomPete• 15 years ago

Aprils fool?

motters• 15 years ago

This is one thing which can happen when you put all your data into "the cloud" and hope for the best. One risk which any business has to take into account is the possibility of a rogue employee or ex-employee running amok. Putting all your eggs (or data) into one basket, especially when that basket is administrated by people who don't particularly care either way what happens to your business, may not be the wisest strategy.

pbhjpbhj2• 15 years ago

>"According to a lawsuit that was filed last week in Hawaii District Court, a man named Michael Scott Jewson was terminated from CyberLynk. From his parents' residence, he allegedly accessed CyberLynk's data and intentionally wiped it out. Jewson is alleged to have been charged in February with a federal computer crime violation and admitted his guilt in a plea agreement.

The data breach allegedly knocked out 6,480 WER1 electronic files, or 300 gigabytes of data, comprising two years of work from hundreds of contributors globally, including animation artwork and live action video production."

These sort of reports often over do it with the use of "allegedly" as an attempt to mitigate any libel claim.

"Jewson is alleged to have been charged"

Surely it's a matter of record whether he was charged. If the reporter can't even confirm this then they don't have a story.

"The data breach allegedly knocked out 6,480 WER1 electronic files"

Again they don't need that "allegedly", it's presumably a reported fact, the allegation they have to be wary off is the one that says the person responsible.

This really annoys me, allegedly.

ubuntuftw2• 15 years ago

Former TV news person here.

While I agree with you on your first point (that the reporter shouldn't have to use the word "allegedly" if he/she can confirm the man was charged), personally, I think using allegedly in the sentence about the files being knocked out is warranted. "The data breach" is referring to the data breach the man *allegedly carried out.

TV news reporters != attorneys. So when in doubt, attribution is your best friend.

pbhjpbhj2• 15 years ago

The data breach he allegedly carried out is reported as a known quantity. The only allegation that needs mitigating is that it was this person that carried it out. If one is unsure of the details then it would be "reportedly" or an "unconfirmed source stated" or similar, no?

Thanks for your viewpoint though.

Fired employee wipes out whole season of TV show

Related Stories