Knightmare: A DevOps Cautionary Tale (2014)

http://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/

by strzalek • 11 years ago

207 85 11 years ago

Mandatum2• 11 years ago

I remember reading a summary of this when it occurred in 2012. It's obvious to everyone here what SHOULD have been done, and I find this pretty surprising in the finance sector..

Also your submission should probably have (2014) in the title.

johansch• 11 years ago

The art of rehashing old stories for exposure. Without mentioning where you got your material of course.

Mandatum2• 11 years ago

https://news.ycombinator.com/item?id=4333089

I believe it was this, as it was a Guardian article - which has since been subject to the digital graveyard.

bsdpython2• 11 years ago

"I find this pretty surprising in the finance sector.."

...not to anyone who has worked in financial software

recursive• 11 years ago

"Power Peg"? More like powder keg.

solarmist5• 11 years ago

Why would they repurpose an old flag at all? That seems crazy to me unless it was something hardware bound.

vertex-four• 11 years ago

To keep the messages as short as possible, to reduce the time-costs of transmitting and processing them. It's HFT, they do things like that.

jemfinch• 11 years ago

I assume "flag" in this context means something akin to a command-line flag.

mason240• 11 years ago

Which is why I don't feel bad at all. Live by the sword, die by the sword.

solarmist5• 11 years ago

Okay, I can see that if they're binary messages and a single bit is no longer being used.

solarmist5• 11 years ago

Okay, yeah, if the flag is in the message not just an internal flag that makes MUCH more sense.

bsdpython2• 11 years ago

I've seen it done on a regular basis at a large financial institution. Like, so often that almost half the fields had a different meaning now or have dual purpose on some old systems.

ratsbane• 11 years ago

I can't read something like this without feeling really bad for everyone involved and taking a quick mental inventory of things I've screwed up in the past or potentially might in the future. Pressing the enter key on anything that affects a big-dollar production system is (and should be) slightly terrifying.

click1703• 11 years ago

I have the same fear, but I wonder if that fear stems from lack of training and or documentation and or time.

When I ask myself why I am afraid of deploying to production servers its always because I don't fully understand what the deployment process does. If I had to deploy manually I would be lost at sea.

Is it just nievete thinking that enough documentation and training makes that fear go away?

vinceguidry2• 11 years ago

It is, ultimately, a management problem.

Computer systems are rarely useful on their own. They need to be attached to some other process in order to derive any value from them. Sometimes it's a network, somethings it's a factory, sometimes it's just process run by humans.

In most domains that you attach computers to, it's possible for one person to understand both the computer side of it and the other side of it. Web development is like this, you can easily understand both the web and computer systems, as one is really just another version of the other.

If you're attaching the computer system to another very hard-to-understand system, like, say, a surgical robot, then the person that understands both domains well enough to avoid problems like this is a unicorn, there might well be only one person in the world that's built up expertise in both fields.

To get around this you need careful, effective management of the two pools of labor, and robust dialogue between the two teams. The second this starts breaking down, is the second you start marching down the road to disaster.

In the aforementioned example, everyone would be well-aware of the failure modes. So it's a bit easier to manage. In finance, failure modes can be so subtle, particularly as the system grows more complex, that they can escape detection by both teams unless they're both checking each other's work and keeping each other honest.

Institutional knowledge transfer has to constantly be happening, areas of ignorance on both sides have to constantly be appraised and plans undertaken to reduce said ignorance. The more everyone knows about the entire system, the more likely it will be that critical defects like this can be discovered before they strike.

This kind of effective interaction of those at the bottom can only be organized and directed at the top. It's very much a "captains winning the war"-type situation, but captains can't lead without support from the generals.

NhanH• 11 years ago

Everytime I'm reading the story, there is one question that I've never understood: why can't the just shutdown the servers itself? There ought to be some mechanism to do that. I mean, $400 millions is a lot of money to not just bash the server with a hammer. It seems like they realized the issue early on and was debugging for at least part of the 45 minutes. I know they might not have physical access to the server, but wouldn't there be any way to do a hard reboot?

jpatokal• 11 years ago

Violently shutting down trading also exposes the bank to significant risk, eg. massively leveraged trades on stocks that were meant to be held for 3 milliseconds suddenly hanging on until the system's back up.

In hindsight, this would still have been preferable to losing $400 million, but quite obviously nobody at the time realized just how catastrophic this was going be.

xyzzy123• 11 years ago

With the benefit of hindsight I think company wide realtime dashboard would be high priority. I guess actual deployment procedures would be higher though.

jacques_chester• 11 years ago

In the story he points out that there was no kill switch.

And, as has been found in other disasters in other industries, kill switches are hard to test.

derefr2• 11 years ago

It feels more and more like the only responsible way to engineer systems is with a built-in always-on-in-production chaos monkey, to always be killing various parts of them. Normally this is done to ensure that random component failure results in no visible service interuption, but in this situation, you'd also be able to reuse the same "apoptosis" signal the chaos monkey sends to just kill everything at once.

vacri• 11 years ago

The story also points out that there were no emergency procedures. While not as instantaneous as a kill switch, known good procedures could have significantly reduced the final effect.

beat2• 11 years ago

Long ago in a previous life, I worked in a factory that made PVC products, including plastic house siding. One of my co-workers got his arm caught in a pinch roller while trying to start a siding line by himself. There was a kill switch on the pinch roller - six feet away and to his left, when his left arm was the one that was caught. Broke every bone in his arm, right up to his collarbone.

He screamed for help, but no one could hear him over the other noisy machinery. Welcome to the land of kill switches.

Bluestrike2• 11 years ago

Given the types of transactions Knight was involved in, it's unlikely they had physical access as the servers would be in lower Manhattan to keep latency down. Couple that with the lack of any established procedures to kill their systems, and it's one hell of a nightmare. If the idea of just pulling the plug even came to them, I can't imagine how well that phone call would be received by the datacenter techs even if they believed it wasn't a hoax. And that's assuming that pulling the plug wouldn't cause other problems.

But really, the 45 minutes probably flew by faster than you or I could really imagine. You're in a crisis situation, you tell yourself you just need another five minutes to fix something. Five becomes ten, becomes twenty, and before you know it, your company is looking at a $400M nightmare.

masonium• 11 years ago

As a designated market maker, there are probably regulatory requirements that force them to be in the market. Granted, if they'd known the full extent of the damage, they almost certainly would've pulled the plug. But, I'm guessing that was a factor.

easytiger2• 11 years ago

Hedging.

serve_yay3• 11 years ago

If you fill the basement with oily rags for ten years, when the building goes up in flames, is it the fault of the guy who lit a cigarette?

wingerlang• 11 years ago

Did the article specify that some one person was at fault?

serve_yay3• 11 years ago

This is a story about the importance of DevOps, i.e., this company went bankrupt because of poor DevOps practices.

dredmorbius3• 11 years ago

In the case of wildfires, why yes, yes indeed we do.

serve_yay3• 11 years ago

The conditions for wildfire do not arise out of neglect and laziness.

hcarvalhoalves2• 11 years ago

As usual in catastrophic failures, a series of bad decisions had to occur:

- They had dead code in the system

- They repurposed a flag for a previous functionality

- They (apparently) didn't had code reviews

- They didn't had a staging environment

- They didn't had a tested deployment process

- They didn't had a contingency plan to revert the deploy

It could be minimized or avoided altogether by fixing just one of the points. Incredible.

carrotleads• 11 years ago

and if you do it all well, you are paid the avg dev salary.

The value of a good dev is a realised only when someone screws up.

UnoriginalGuy• 11 years ago

They (apparently) didn't had code reviews

I don't get that. There was no code issue. The old and new code both worked as intended, it was a deployment and deployment-verification problem.

They didn't had a staging environment

Yes they did. They staged the new code and tested it. They did a slow deployment also.

They didn't had a contingency plan to revert the deploy

They did revert the deploy within the 45 minutes. It made it worse.

I think you need to re-read the article. Your assessment is strange given the event.

paulrademacher• 11 years ago

Code review could have been another set of eyes to predict the problem of reusing a flag.

hcarvalhoalves2• 11 years ago

I don't get that. There was no code issue. The old and new code both worked as intended, it was a deployment and deployment-verification problem.

A code review could raise the issue of repurposing a flag in case they had to revert the deploy. Changing the semantics of a flag is a big no-no anyway, and there are ways to guard against that.

Yes they did. They staged the new code and tested it. They did a slow deployment also.

But they didn't had a staging environment that matched their live environment, apparently. You want a staging environment that is 1:1.

They did revert the deploy within the 45 minutes. It made it worse.

If you think reverting a deploy by simply pushing an older version is the same as a contingency plan, think again.

aosmith• 11 years ago

Wasn't Knight in trouble for some other things as well?

bevacqua• 11 years ago

Ah yes, this story is legendary. I discuss it in my JavaScript Application Design book[1]. Chaos-monkey server-ball-wrecking sounds like a reasonable way to mitigate this kind of issues (and sane development/deployment processes, obviously)

[1] http://bevacqua.io/bf

danbruc• 11 years ago

What really looks broken to me in this story is the financial system. It has become an completely artificial and lunatic system that has almost nothing to do with the real - goods and services producing - economy.

raincom• 11 years ago

Trickle down economics = financialization of capital. Those who influence the policy are beneficiaries of the financialization of capital: think of grads from elite law schools and b shools!! If there is no financialization of capital, these grads from elite law/b schools wont get fat bonus.

narrator2• 11 years ago

Somebody was on the other side of all those trades and they made a lot of money that day. That's finance. Nobody loses money, no physical damage gets done and somebody on the other side of the poker table gets all the money somebody else lost.

noonespecial• 11 years ago

Its probably weirder than that. As soon as their system "lost containment" all of that money probably just diffused into the market like smoke amid hundreds of thousands of trades. There's no "big winner" on the other side of the table grumble at.

HFT systems aren't like a bunch of poker players slyly eying one another across a table, they're more like electro-chemical gold miners concentrating gold from the rivers of cyanide sludge pumping up from the financial system mines. Heaven help them if they depolarize.

SixSigma• 11 years ago

Also - if they had accidentally made a $450m profit, none of this would ever be public.

otakucode2• 11 years ago

While articles like this are very interesting for explaining the technical side of things, I am always left wondering about the organizational/managerial side of things. Had anyone at Knight Capital Group argued for the need of an automated and verifiable deployment process? If so, why were their concerns ignored? Was it seen as a worthless expenditure of resources? Given how common automated deployment is, I think it would be unlikely that none of the engineers involved ever recommended moving to a more automated system.

I encountered something like this about a year ago at work. We were deploying an extremely large new system to replace a legacy one. The portion of the system which I work on required a great deal of DBA involvement for deployment. We, of course, practiced the deployment. We ran it more than 20 times against multiple different non-production environments. Not once in any of those attempts was the DBA portion of the deployment completed without error. There were around 130 steps involved and some of them would always get skipped. We also had the issue that the production environment contained some significant differences from the non-production environments (over the past decade we had, for example, delivered software fixes/enhancements which required database columns to be dropped... this was done on the non-production systems, but was not done on the production environment because dropping the columns would take a great deal of time). Myself and others tried to raise concerns about this, but in the end we were left to simply expect to do cleanup after problems were encountered. Luckily we were able to do the cleanup and the errors (of which there were a few) were able to be fixed in a timely manner. We also benefitted from other portions of the system having more severe issues, giving us some cover while we fixed up the new system. The result, however, could have been very bad. And since it wasn't, management is growing increasingly enamored with the idea of by-the-seat-of-your-pants development, hotfixes, etc. When it eventually bites us as I expect it will, I fear that no one will realize it was these practices that put us in danger.

bd_at_rivenhill• 11 years ago

You should read Charles Perrow's "Normal Accidents" and all will be revealed. This is hardly a new problem.

http://www.amazon.com/Normal-Accidents-Living-High-Risk-Tech...

__abc• 11 years ago

This must be an old wives tale. I live in Chicago and a trading form on the floor beneath us went bankrupt, in roughly the same time, with a similar "repurposed bit" story.

Maybe it's the same one .....

fnordfnordfnord2• 11 years ago

Knight Capital. It was a big deal. Not an old wive's tale. http://www.nanex.net/aqck2/3522.html

easytiger2• 11 years ago

HE was joking.

ooOOoo• 11 years ago

The post is quite poor and suffer a lot from hindsight bias. Following article is so much better: http://www.kitchensoap.com/2013/10/29/counterfactuals-knight...

jackgavigan• 11 years ago

Great link. Thanks for posting.

rgj• 11 years ago

Repurposing a flag should be spread over two deployments. First remove the code using the old flag, then verify, then introduce code reusing the flag.

Even if the deployment was done correctly, during the deployment there would be old and new code in the system.

beat2• 11 years ago

It's nice to see a more detailed technical explanation of this. I've used the story of Knight Capital is part of my pitching for my own startup, which addresses (among other things) consistency between server configurations.

This isn't just a deployment problem. It's a monitoring problem. What mechanism did they have to tell if the servers were out of sync? Manual review is the recommended approach. Seriously? You're going to trust human eyeballs for the thousands of different configuration parameters?

Have computers do what computers do well - like compare complex system configurations to find things that are out of sync. Have humans do what humans do well - deciding what to do when things don't look right.

gunnark01• 11 years ago

I used to work in the HFT, and I don´t understand is why there was no risk controls. They way we did it was to have explicit shutdown/pause rules (pause meaning that the strategy will only try to get flat).

The rules where things like: - Too many trades in one direction (AKA. big pos) - P/L down by X over Y - P/L up by X over Y - Orders way off the current price

When ever there was a shutdown/pause a human/trader would need to assess the situation and decide to continue or not.

neomantra• 11 years ago

To add insult to injury, Knight got fined for not having appropriate risk controls after this incident.

vijucat3• 11 years ago

I once shut down an algorithmic trading server by hastily typing (in bash):

- Ctrl-r for reverse-search through history

- typing 'ps' to find the process status utility (of course)

- pressing Enter,....and realizing that Ctrl-r actually found 'stopserver.sh' in history instead. (There's a ps inside stoPServer.sh)

I got a call from the head Sales Trader within 5 seconds asking why his GUI showed that all Orders were paused. Luckily, our recovery code was robust and I could restart the server and resume trading in half a minute or so.

That's $250 million to $400 million of orders on pause for half a minute. Not to mention my heartbeat.

Renamed stopserver.sh to stop_server.sh after that incident :|

P.S. typing speed is not merely overrated, but dangerous in some contexts. Haste makes waste.

Beltiras• 11 years ago

Repercussions?

vijucat3• 11 years ago

Luckily, and just that one time, there was nothing but disapproving glares.

phaemon2• 11 years ago

Might be better having something like the following at the top of your script:

    echo "This will STOP THE SERVER. Are you sure you want to  do this?"
    echo "Type 'yes' to continue:"
    read response

    if [[ $response != "yes" ]]
    then
      echo "You must type 'yes' to continue. Aborting."
      exit 1
    fi

    echo "Stopping server ..."

It barely takes any time to type 'yes' but it makes you stop and think.

marcosdumay2• 11 years ago

That'll make it impossible to use in scripts. This remember me the time I tried to fix my "rm -r * .o" with a CLI trash system, instead of doing proper backups.

Might be better to start reviewing EVERY command one sends to important servers, and testing them if viable. What probably is the line that vijucat took... that's the line that everybody ends up taking, the only thing that changes is the number of accidents needed.

vijucat3• 11 years ago

Yes, this solution would have worked for our situation.

brazzledazzle2• 11 years ago

Am I wrong to assume that you weren't using an init script of some sort because this stopped multiple services, perhaps in a specific order?

Knightmare: A DevOps Cautionary Tale (2014)

Related Stories