“A Windows 7 deployment image was accidently sent to all Windows machines”

http://it.emory.edu/windows7-incident/

by slyall • 12 years ago

337 138 12 years ago

it.emory.edu

facorreia• 12 years ago

A Windows 7 deployment image was accidently sent to all Windows machines, including laptops, desktops, and even servers. This image started with a repartition / reformat set of tasks.

Wow. That is very unfortunate, to say the least...

spiantino• 12 years ago

As soon as the accident was discovered, the SCCM server was powered off – however, by that time, the SCCM server itself had been repartitioned and reformatted.

malkia• 12 years ago

I feel bad, but I laughed at this for few seconds...

rasz_pl• 12 years ago

We apologize again for the fault in the subtitles. Those responsible for sacking the people who have just been sacked have been sacked

filmgirlcw2• 12 years ago

I've never been prouder of my alma mater. /s

gt1337h• 12 years ago

You should be. Emory is the 2nd best school in Georgia! :)

miles• 12 years ago

Snark and sarcasm aside, I am impressed with the level of detail that the IT department is sharing; it is refreshing to see such a disaster being discussed so openly and honestly, while at the same time treating customers like adults.

song• 12 years ago

Yes, shit happens. It's how you solve it and make sure that it doesn't happen again that's important. With the way they're communicating, they seem to be on the top of their game

harrystone• 12 years ago

It is impressive. I can't imagine dealing with that kind of nightmare.

I once worked as an admin on Solaris boxes at a big pharma company. There were ~77,000 users in their LDAP directory. I was very careful.

cfreeman• 12 years ago

That sounds terrifying.

keithpeter• 12 years ago

At one place I worked, in the days of XP, the 'index server'(?) had a problem and un-installed all the application programs. The basic OS was there but MS Office, MSIE, all the doodads just got removed as each machine logged in.

This was a small college, so the IT guys just went round explaining and told us to log out and log in again. Applications re-installed. No data loss and so no shouting.

Stuff happens. We did 'assignment action planning' that morning: mind maps, essay plans and research ideas. Results better than normal anyway

filmgirlcw2• 12 years ago

100% agree. My initial response was to laugh, but dude, I so feel for the team that has to clean the mess up.

swasheck• 12 years ago

It's not a standard across the board, but I've found academic institutions to be more honest about their technological mistakes (outside of large-scale breaches) than the private sector.

SoftwareMaven• 12 years ago

I worked at a company for several years that provided software to the IT groups in the higher ed vertical. What I found is there are two types of people in higher-ed IT (and they often congregate at different campuses):

First are the people who believe in the mission. They are really good, and are willing to take a cut in pay for some combination of social good, great working environment, etc. These kinds of people tend to be forthright about problems.

The second group are people who would struggle with the demands of the normal corporate world. They are getting paid less in higher ed and are worth what they are getting paid.

TheRealWatson• 12 years ago

Yup. There's reason for some initial amusement but much respect for openly taking care of the problem. This will win them appreciation and confidence in the aftermath of this disaster.

magpi3• 12 years ago

I imagine this is what the sysadmins conversations at Emory are like right now: https://www.youtube.com/watch?v=eOkAyUmyQko

nissehulth2• 12 years ago

Indeed. Though the step described as "We are developing a more automated approach" is a bit scary. :)

arde• 12 years ago

Snark and sarcasm somewhat aside, it's not like they could tell the users about the remediation progress through their intranet, so they probably didn't have many options besides posting it on the Internet for all to see.

smegel• 12 years ago

Automation can also mean automated disaster.

batmansbelt• 12 years ago

...Or automated the-opposite-of-disaster.

Already__Taken• 12 years ago

Bigger tools have bigger sharper edges. It's important the message is stressed as the developers to operators worlds merge in the middle.

ameoba• 12 years ago

Why did the Roman Empire collapse?

What is the Latin for office automation?

perlgeek• 12 years ago

https://twitter.com/DEVOPS_BORAT/status/41587168870797312

"To make error is human. To propagate error to all server in automatic way is #devops."

Frankly, I'm surprised things like this don't happen more often. Kudos for the incident management. Also a big plus for having working backups, it seems.

toomuchtodo• 12 years ago

Devops Borat: This not mistake. This how we test recovery operations in production.

ibisum• 12 years ago

This one goes on the first page of my playbook as of today.

timClicks• 12 years ago

Propogation of a software mistake is what appears to have caused the Gmail outage of 2011 http://youtu.be/eNliOm9NtCM?t=28m49s

mey• 12 years ago

I wish that account was still updating.

reitanqild• 12 years ago

For anyone who hasn't read it yet the last tweet on that account is gold.

misnome2• 12 years ago

I found an alternative in https://twitter.com/BigDataBorat, not quite as insightful but still occasionally funny

ar7hur• 12 years ago

"In startup we are practice Outage Driven Infrastructure."

wiml• 12 years ago

I'm not sure that's so unlike Netflix's Chaos Monkey.

laumars• 12 years ago

I think it does happen often but isn't as well reported. I certainly know of more than one place that's suffered from this kind of accident (thankfully not places I personally work so I've not had to deal with the fallout. These are places I have friends or family who work there)

wiredfool• 12 years ago

The most frightening movie ever for sysadmins -- The Sorcerer's Apprentice.

IgorPartola2• 12 years ago

@DEVOPS_BORAT is actually very insightful in about 1/5 tweets. Snide for sure, but there are quite a few good points in there if you read carefully:

"In devops we have best minds of generation are deal with flaky VPN client."

"Single point of failure in private cloud is of usually Unix guy with neckbeard."

These are gold.

Edit: based on the above advice I once grew out a neckbeard while going through a multi-month rollout of a large product. It itched like crazy, but I did work much faster to get rid of it.

Havoc5• 12 years ago

best minds of generation are deal with flaky VPN client

So true. I'm on the receiving side of this..."No you can't work on that multi million deadline project of yours...the only way to fix the VPN is to re-image the machine back at head office [an international flight away]". Me..."Could you repeat that?" And thats a Cisco Enterprise VPN...(turns out IT was right...re-image & avoid conflicting software is the only solution). So much for Cisco...

joezydeco• 12 years ago

"In devops is turtle all way down but at bottom is perl script."

tetha• 12 years ago

Same for TheCodelessCode. A lot of these are cryptic and weird, but some are pure gold. Especially since no one understands the koans until they fall flat on their face just like the student does and a huge floodlight turns on.

brokenparser• 12 years ago

These are gold.

But not English, I can't make sense of them.

appplemac• 12 years ago

In startup we gamify operations: 3 infrastructure failures, and ops team are out.

jerf• 12 years ago

My favorite personal variant: "To err is human. To screw up a million times per second, you need a computer."

shill• 12 years ago

Worst mistake is always happen in batch. --Devops Borat

https://twitter.com/DEVOPS_BORAT/status/251885078794366976

antmldr• 12 years ago

Frankly, I'm surprised things like this don't happen more often.

They do. This happened to the largest bank in Australia mid 2012[1]. Very similar circumstances. I've been told that SCCM's UI doesn't help here- something about the default action when nothing is selected to apply it to all devices managed by SCCM. Someone more familiar with SCCM may want to correct me here.

[1] http://delimiter.com.au/2012/07/30/disastrous-patch-cripples...

beloch2• 12 years ago

This reminds me of my undergrad CPSC days. The CPSC department had their own *nix-based mainframe system that was separate from the rest of the University. The sysadmin was a pretty smart guy who was making less than a third of what he could get in industry. Eventually he got fed up and left. About a week or two later the servers had a whole cascade of failures that resulted in everyone losing every last bit of work they'd done over the weekend (This was a weekend near the end of the semester when everyone was in crunch mode).

Long story short, the sysadmin was hired back and paid more than most of the profs. Academia may tend to skimp on salaries for certain positions, but sysadmins probably shouldn't be one of them.

gertef• 12 years ago

The BOFH sabotaged his systems before departing? Didn't build a backup plan? And they hired him back?

SDGT• 12 years ago

You know what? Fuck them. Fuck higher education completely. Undervalued and underpaid is the name of the game for any important IT roles in that shit hole of an industry.

Disclaimer: dev at a university.

Fuzzwah2• 12 years ago

I've just been hired to run a project using SCCM to upgrade ~5000 PCs from XP to Win7.

This was amazing reading. Reading such a detailed wrap up of an IT team going through my worst possible nightmare was enlightening.

sswaner• 12 years ago

It must be reassuring that your new employer is so proactive in getting that XP upgrade project running.

Sanddancer• 12 years ago

Sometimes these products get bogged down because of three important problems outside of the control of IT. First off, you need to get in touch with all the upstream vendors to get updates to any sort of custom software that has compatibility problems with newer versions of windows; Vista/7 got a lot more strict about giving admin access, for example, which may cause problems in the updates. Second, you've got to keep in mind the training costs. There are a lot of users who may be brilliant financial minds that can make numbers dance and bow to their whims, but get terribly locked up if an icon changes. Doesn't make them horrible people by any means, but you've got to keep it in consideration when planning a rollout. Finally, you have to keep in mind the petty turf wars. If Joe in Accounting gets the upgrade to 7 before Bob in Legal, Bob in Legal may feel slighted and start raising a holy shitstorm, even if he's scheduled to be upgraded a week later. Upgrades are ugly, no matter when they happen. Sometimes that proactive upgrade project takes many years just because of all the moving parts involved.

asdfaoeu• 12 years ago

Looks like it will be pretty easy these guys did it by accident.

rfolstad• 12 years ago

On the bright side they are no longer running XP!

sorennielsen• 12 years ago

This happened at one a former workplace too. Only the Solaris and Linux servers was untouched.

It "mildly" amused the *nix operations guys to see all the "point and click" colleagues panic.

sergiotapia• 12 years ago

Isn't this more the fault of the system architect than the guy who accidentally fired the bad deploys?

It's similar to a database firehose: If you accidentally start deleting all data you should have a quick working backup ready to quickly bring the dead box up to production.

fred_durst• 12 years ago

I don't know. This could very well be a case of not much more than a bad drag and drop in SCCM. Its not quite that simple, but I'm not sure this is some custom process they setup.

not_kurt_godel• 12 years ago

Any tool that allows you to easily perform the antithesis of its function without making it abundantly clear what clicking the OK button will do is fundamentally broken.

Fuzzwah2• 12 years ago

I was just watching the "What’s New with OS Deployment in Configuration Manager and the Microsoft Deployment Toolkit" session from TechED and hit the section on "check readiness" option which MS have added to SCCM 2012 in R2. It sounds like having this in part of the task sequence at Emory would have (at the very least) stopped this OS push from at least hosing all the servers.

http://channel9.msdn.com/Events/TechEd/NorthAmerica/2014/PCI...

stark3• 12 years ago

There was a similar catastrophe at Jewel osco stores many years ago. Nightly, items added to the store pos were merged back with the main item file at each store location. The format of the merged data was exactly the same as loading a new file, except the first statement would be /EDIT instead of /LOAD.

One of the programmers decided to eliminate some code by combining the two functions, with a switch to control whether /LOAD or /EDIT was used for the first statement.

There was a bug in the program, and the edits were sent down as loads.

A guy I knew, Barry, was the main operator that night. He started getting calls from the stores after around 10 of them had been reloaded with 5 or 6 items.

Barry said it was the first time he got to meet the president of the company that day.

pierlux• 12 years ago

Failure should never be the only time you get to meet upper management :(

ceejayoz• 12 years ago

A night IT operator for an organization with 176 locations is pretty unlikely to ever meet the company president.

jonmrodriguez• 12 years ago

Forgive my beginner question:

Since a reformat was done to the affected machines, does this mean that researchers' datasets, drafts of papers, and other IP were lost? Or were researchers' machines not affected?

InclinedPlane• 12 years ago

Deletion and formatting doesn't necessarily destroy data, it just destroys the pointers to the data. If they're lucky the data can be recovered via software utilities (undelete) with backfill from backups. If they're unlucky then important and un-backedup data has been written over, and those people are going to be SoL.

johnchristopher2• 12 years ago

Are there actual backup procedures out there that foresee and automate the restoration of wiped drives and partitions ? I might be wrong bu I doubt it should even be considered though.

wtallis2• 12 years ago

In my experience with campus networks, home directories are never stored locally on any remotely-administered machines. Any specially-configured researcher's machine that stored data locally would not have been subscribed to get the automatically deployed OS images.

pbhjpbhj• 12 years ago

"As soon as the accident was discovered, the SCCM server was powered off – however, by that time, the SCCM server itself had been repartitioned and reformatted." //

If the SCCM server was pushed the "update" then there doesn't seem much hope for other machines? Surely no rule should be able to format the server running the ruleset; seems like a failsafe failure there at least.

pling• 12 years ago

Not quite as disastrous but when I was at university the resident administrators configured the entire site's tftp server (everything was netbooted Suns) to boot from the network. This was fine until there was a site-wide power blip and it was shut down. When it came back it couldn't tftp to itself to boot because it wasn't booted yet (feel the paradox!). Cue 300 angry workstation users descend on the computer centre with pitchforks and torches because their workstations couldn't boot either...

Bad stuff doesn't just happen to Windows networks.

grumblepeet• 12 years ago

I _very_ nearly did this whilst working for a University back in the early noughties. Luckily I managed to get to the server before the "advert" activated and wiped out everything. It was so easy to do I am surprised that it is stil possible. I feel for their pain, but it does sound like they are doing a good job of mopping up. I did allow myself a snort of laughter when I read the bit about the server being re imaged as well. That is pretty darn impressive carpet bombing the entire campus.

leccine3• 12 years ago

We accidentally re-imaged all of the Windows servers with Linux the other day. Nobody noticed though...

jacalata2• 12 years ago

That probably means nobody is using the servers you maintain and your department should be eliminated completely.

leccine3• 12 years ago

Or somebody was just joking... Who knows...

zacharycohn• 12 years ago

I thought this "accident" may have been on purpose... until they mentioned the servers.

In my days of university tech support.

Fomite• 12 years ago

Reminds me of some emails that went out at my old university during a cluster outage, and got progressively more informal as the night went on, detailing people leaving dinners with extended families, a growing sense of desperation, etc. The last email might as well have ended with "Tell my wife I love her."

It was both direct and funny enough that I was only mildly annoyed that the cluster was down.

tbyehl• 12 years ago

I've built a few systems for deploying Windows... and the last thing that every one of them did before writing a new partition table and laying down an image was to check for existing partitions and require manual intervention if any were found.

mantrax52• 12 years ago

You know how in movies you need at least two people to bring their special secret keys, plug them in, and turn them at once to enable a self-destruct sequence?

That is a real principle in interface design - if something would be really, really bad to activate unintentionally, make it really, really hard to activate.

If you design a nuclear missile facility, you don't put the "launch nukes" button right next to "check email" and "open facebook".

Same way it shouldn't be easy for users to delete or corrupt their data by accident due to some omnipotent action innocently shoved right in between other trivial actions.

I wouldn't blame the person who triggered this re-imaging process. I'd blame those who designed the re-imaging interface, to allow it to happen so easily by accident.

kalleboo2• 12 years ago

In my experience, the key is that the UI makes it clear exactly what you're doing. What I mean is, instead of a button that says "Start Imaging", it should be "Start Imaging of the 12,600 computers this rule applies to". Of course, that's a lot more work for the programmer, so it's never done.

lstamour2• 12 years ago

But then that leads to people automatically clicking "Yes" to the "Are you sure..." dialog. Though even I would pause at "Are you sure you want to reformat 12,500 machines including this one?" ;-)

mantrax52• 12 years ago

Event that is hardly enough. There should be a physical (well virtual physical) obstacle towards launching a high stakes command.

The system should be able to assess the scope of a task, and ask you to confirm 10 times if it has to, in blinking red dialogs, to make sure you really want to do what you are doing.

Of course, it's crucial that "clicking 10 times" is not the default behavior for any trivial action. Or boredom and the subsequently formed mechanical 10-click habit of the operator will kill the effectiveness of this approach...

7952• 12 years ago

It also helps to have sensible conventions for naming hosts and groups. If you need to select a subset of machines then you are sometimes going to make a mistake as simple is getting a wildcard pattern wrong. Instead have groups with explicit and obvious names that require no memory to understand.

randlet• 12 years ago

Reading that just made me feel sick to my stomach and my heart goes out to the poor gal/guy that pushed "Go" on that one. Shit happens, but a screw up that big can be devastating to ones psyche.

CamperBob22• 12 years ago

stark3, you seem to be hellbanned.

taspeotis• 12 years ago

What gives you that idea?

https://news.ycombinator.com/item?id=7758203 is a dupe of https://news.ycombinator.com/item?id=7758193

CamperBob22• 12 years ago

(Sorry, didn't see the dupe.)

svec• 12 years ago

With great power comes great responsibility.

8ig82• 12 years ago

Mistakes are made. In related news...

Lawn care error kills most of Ohio college's grass

http://www.wral.com/lawn-care-error-kills-most-of-ohio-colle...

unclesaamm• 12 years ago

Haha this is great. How did you find this?

Just to clarify for the passersby, it isn't the University of Ohio, rather, Findlay University

8ig82• 12 years ago

Coincidently on the local news website.

Findlay University did not release the name of the company that made the mistake but is working with the business' insurance company to pay for it to be fixed.

Findlay apparently doesn't value transparency as much as Emory.

The university says grass was killed on as many as 54 of the campus' 72 acres.

dredmorbius• 12 years ago

Makes you wonder what chemicals are going into lawns generally.

One of the lessons from my college days (informally acquired, take with appropriate quantities of salt) was that walking barefoot was as much a risk for chemical exposure as puncture wounds.

frogpelt• 12 years ago

I can understand the Embry problem a lot more simply because the distance between the decision and implementation of the decision is not that far.

How on earth do you "accidentally" load up enough weed killer to treat 54 acres of grass and never realize its the wrong stuff?

dfc• 12 years ago

How do you click/not-click the button in SCCM...? How are these two events any different?

rfrey• 12 years ago

My nomination of the top bullet point of 2014:

* As soon as the accident was discovered, the SCCM server was powered off – however, by that time, the SCCM server itself had been repartitioned and reformatted.

mehrdada• 12 years ago

As soon as the accident was discovered, the SCCM server was powered off – however, by that time, the SCCM server itself had been repartitioned and reformatted.

I guess that's how robot apocalypse is gonna look like.

ww520• 12 years ago

Disasters as well as mistakes are unavoidable, such is life. A hallmark of a competent organization is how they handle the situation and recover from disasters or mistakes.

So far all the signs have indicated they are doing great in recovering. I just hope there won't be onerous processes and restriction afterward due to desire on "make sure it won't happen again" stance.

lucio• 12 years ago

reads like a short dystopian novel

k_sze• 12 years ago

Funny how they mention iTunes as one of the "key components" that are restored first, whereas Visio, Project, Adobe application are relegated to a second round.

krexit• 12 years ago

It's possible they're delivering academic materials through iTunes, I suppose.

lstamour2• 12 years ago

Not hard to imagine. The others likely have specialty licenses and so aren't as easily distributed to everyone. In addition, Adobe software itself wasn't working earlier ;-)

zachlipton• 12 years ago

Presumably iTunes is part of their base system image for all workstations, along with Office, Firefox, Adobe Reader, and the like. In other words, a basic set of software to handle standard officework tasks. iTunes is free and IT would probably rather distribute it everywhere than have people trying to install it themselves (or calling the helpdesk to get someone with administrator rights to do it). They then offer additional applications on an as-needed basis to individuals and departments with specific tasks. So the designers who do print publications and the faculty who teach digital art might get the Adobe suite, while people in Facilities who plan construction will get Project. This keeps licensing costs down and simplifies systems according to their uses.

commandar• 12 years ago

Generally you're going to keep your base images in SCCM limited to software that's only infrequently updated. Otherwise somebody has to update the entire image every time an update gets pushed out. Instead, you package them up and deploy the apps on top of the base image at install time. It takes a little longer to deploy, but it takes less admin time to manage since the actual installs are automated anyway.

keehun• 12 years ago

I asked my friend attending Emory right now, and he didn't even realize anything was going on. He says that the Emory IT department has a notorious distinction on campus as being regularly terrible, mostly with an unreliable internet connection.

However, it looks like they handled this accident the best they could! Perhaps this accident would not have happened at a more reliable IT department.

gojomo• 12 years ago

"... to the cloud!"

http://www.youtube.com/watch?v=jR6xbulUmsg

"Yay, cloud!"

chromaton• 12 years ago

Reminds me of The Website Is Down, episode 4: https://www.youtube.com/watch?v=v0mwT3DkG4w

imgur• 12 years ago

  > As soon as the accident was discovered, the SCCM server was powered off – however, by that time, the SCCM server itself had been repartitioned and reformatted.

That made me laugh. Poor SCCM server :)

deckar01• 12 years ago

"As soon as the accident was discovered, the SCCM server was powered off – however, by that time, the SCCM server itself had been repartitioned and reformatted."

Unicast fail.

durkie• 12 years ago

hah! delighted to see this here.

my roommate works at the emory library and has had a fun slow week there of coming home early many days because no one could do work. they were apparently also given laptops as an interim solution, but those somehow also wiped themselves eventually (?).

poor IT people...just as they're starting to get a handle on the actual sitation it starts blowing up on the internet.

nissehulth2• 12 years ago

Next time I'm about to complain about a bad day at the office, I will read this story again.

“A Windows 7 deployment image was accidently sent to all Windows machines”

Related Stories