Trading Program Ran Amok, With No ‘Off’ Switch

http://dealbook.nytimes.com/2012/08/03/trading-program-ran-amok-with-no-off-switch/

by mindblink • 14 years ago

143 92 14 years ago

dealbook.nytimes.com

nmcfarl• 14 years ago

They lost $440 million (and amount greater than their market cap), and possibly the company, on what the world knows to be incompetence.

At some point if I couldn’t stop it - I’d be tempted to just kill the power to the server rooms, all of them. There just has to be a way to cut your losses.

jonhendry• 14 years ago

The server rooms might not be nearby. They might be located closer to the big exchanges' servers.

jonah7• 14 years ago

Very likely. Speed of light matters in these situations.

slurgfest• 14 years ago

Agreed, the incompetence of pushing changes like that to production (with such high stakes) is pretty bad... but not even having a switch to flip?

rorrr• 14 years ago

It happened very quickly, after years of operations.

Technically, it can take seconds to lose that much.

What they did wrong is do all their trading with the same algorithm. Way to put all eggs in one basket.

patdennis• 14 years ago

According to the article, it took them a half hour to shut it down. That's more than a few seconds.

But as a torrent of faulty trades spewed Wednesday morning from a Knight Capital Group trading program, no one at the firm managed to stop it for more than a half-hour.

lostlogin• 14 years ago

It wasn't after years of operation. The article states that the system was brand new.

sp3324• 14 years ago

I'm pretty sure they don't do all their trading with one algorithm. What makes you think they do?

forgotusername3• 14 years ago

I'd love to know what qualifies you to throw a word like incompetence around here. My best guess is the reason it took 45 minutes to shut it off was due to a judgement call: burn through free cash, or take out all their customers too. Bear in mind some of the largest retail brokerages in the world hang off Knight.

Their primary functions are acting as an order destination and a market-maker, for efficiency's sake an obvious conclusion would be that both functions are combined in the same software (in a market where microseconds matter). So given the choice of taking a cash hit (a potentially short term affair), or a reputation hit (a much longer term and most likely fatal affair), it's entirely possible Knight knowingly made the right decision.

It's worth note that the eventual deficit amounts to somewhere in the region of one year's net income, hardly insurmountable (and how many investment opportunities promise close to 100% return in a single year?).

Listening to the CEO on Bloomberg, it was clear that minimizing damage to customers was their primary goal (he made this point several times in the 5 minute interview), and that he appeared comfortable with the outcome.

rhizome• 14 years ago

I'm sure they would have tried pushing code in that time at least a few times.

fauigerzigerk• 14 years ago

$440 million is four times their 2011 net income. I doubt the CEO is comfortable with the outcome of being on the brink of bankruptcy. He is trying to arrange a fire sale of the company as we speak.

But I think you are right that they tried to avoid an outage. The incompetence, if any, is that they apparently did not know how much money they were losing and still kept the system going. It wasn't a caclulated risk but rather an incaculable one.

I imagine it's not easy to know how much you're losing at any moment in time. They certainly knew they were building huge positions, but knowing how much they were going to lose on those positions requires an estimate of the price at which the positions can be closed (or a hedge).

What I cannot imagine is that it is common practice to leave this kind of decision to an individual's judgement call. There have to be rules for a situation like this. And there's only one sensible rule for a rogue algo racking up unknowable losses. Kill it and deal with the consequences later. Anything else is negligent.

jussij• 14 years ago

> I'd love to know what qualifies you to throw a word like incompetence around here.

I'm not sure what other word you could use. Total stuff maybe?

> So given the choice of taking a cash hit (a potentially short term affair), or a reputation hit (a much longer term and most likely fatal affair), it's entirely possible Knight knowingly made the right decision.

Except thanks to their competence they have taken a massive cash hit and their reputation lies in tatters.

I suspect the damage to their reputation is so bad they will be lucky to survive.

wpietri3• 14 years ago

I could buy that they competently made the best of a bad situation. But I have a hard time believing that getting into the situation was the result of perfect competence.

My time writing trading software was never on the automated end of things, so I'm only modestly qualified to comment. But if I were doing the post-mortem on this one, the first thing I'd look for is middle management time pressure forcing a large release without adequate testing. And my standard for "adequate testing" would be pretty high.

If you're going to release something that can take down the company, it's worth making sure it works. In this case, they lost circa 400x the lifetime median income of a US worker. It's hard to imagine the upside that would have justified that kind of risk.

Zenst• 14 years ago

Your right, there are so many details we dont see beyond the attention grabbing headline were everybody will naturaly go and instantly think `but all systems have a off switch, you pull the ruddy plug out` and with such headlines I'll bet most people were thinking that before even reading the article. This and how loved people in any form of finance, with HFT being one of the most loved area's. But with the blame culture in many area's of life, there will be investigations and somebody will either chuck themselfs onto a sword or somebody will be blamed and made a scapegoat. It is all down to confidence at the company from now and nomatter if they did the right things or the wrong things it is that alone that will dicate the fallout. Sad in many ways. RIM being a classic case were the media have controled the stock price, which has controled the media that control the consumers that just ends up in a deadly spiral of self forfilling doom and gloom, nomatter how well or badly they actualy do. I'm sure there are better examples but as you said, its down to company image. Nothing to do with anything else and that alone is the most important thing as a years income is not a problem if your around for more years, if it looks like you wont then it starts to spiral badly.

iskander3• 14 years ago

One nitpick: You can't "kill the power" on a market-making algorithm co-located with the exchange somewhere in New Jersey.

sanxiyn• 14 years ago

Pure speculation. Maybe there was an off switch, which used to work, but not regularly tested, and silently broken? Wouldn't surprise me.

CaveTech• 14 years ago

Highly doubt it.

alexchamberlain• 14 years ago

Sounds highly likely!

jonah7• 14 years ago

"Knight is also working with Goldman Sachs to help unwind the trades behind its extensive loss, according to people briefed on the matter.

"Goldman has agreed to buy, at a discount, the shares that the trading firm had accumulated. Such a move would help Knight by taking the portfolio off its hands and freeing up capital."

What does this mean? Why would GS do this? Why would Knight do this? Couldn't they just sell them on the open market at a better price instead?

jkimmel• 14 years ago

Yes and no. Given the kind of volume Knight purchased during the faulty trades, it could be difficult to offload that many shares on the open market in a timely manner. Maybe those stocks are hot Monday, maybe they're not. Knight needs capital yesterday to keep floating, so they're likely looking to sell everything in one basket.

As for GS's motivation, they're buying at a discount. Due to the time sensitive nature of Knight's predicament, they're probably trading the portfolio to Goldman at a reduced rate. Unlike Knight, Goldman has the cash to sit on it for a while and sell the shares directly out into the open market, even if it takes a few days. Given the discount they bought the shares at, they're likely selling with a decent margin.

jonah7• 14 years ago

So GS is primarily the go-to since they're willing to leisurely unload stock they bought at a discount and they've got the cash in hand to do so. Makes sense.

robryan• 14 years ago

Would be interesting if Goldman also made a stack off the original trades.

fsckin• 14 years ago

I would rarely suggest this, but if something is so incredibly broken that you're loosing money at a rate of 800 million dollars per hour, screw the customers.

Turn it off at any cost. If you are forthcoming and transparent, customers will understand.

pheon• 14 years ago

Point is you dont know what the loss is.

1) They bought too much stock (incorrectly)

2) realized WTF, stopped everything

2a) more likely their clients said WTF is wrong first

3) had to sell the stock for the rest of the day.

Its only after they sold everything did the $440MM price tag surface. Hopefully they sold most of their positions to goldman (instead free market) so one of their investors made a boatload of cash.. giving them favorable terms for a line of credit.

sp3324• 14 years ago

This is wrong. The algorithm was buying and selling constantly, sometimes losing small amounts of money (usually about $15) each time, sometimes as often as 20-40 times per second for each of about 150 symbols.

wtracy2• 14 years ago

Another thread here has a link to a blog post with (admittedly speculative) evidence that, at two different times, they tried rebooting the system only to have it come back up and start making random trades again.

svdad• 14 years ago

What I wonder, following this story this week, is how the software quality controls at a place like Knight compare with those for life-critical systems like those in, e.g., aviation.

On one hand, you'd think the QA in finance would be pretty solid, considering that the survival of the company could be at stake (witness Knight). On the other hand, I have a feeling that even there, people just don't take it that seriously.

Would love to hear from anyone with more experience writing software for these industries.

izaidi• 14 years ago

Unlike high-frequency trading, aviation is highly regulated. In the United States the FAA specifies pretty detailed development standards for avionics software (e.g., DO-178B: http://en.wikipedia.org/wiki/DO-178B). We're unlikely to see similarly strict requirements for financial software anytime soon.

wpietri3• 14 years ago

Having talked with people who write life-critical code, the regulation isn't really what makes it safe. Safety comes from good engineering.

The regulation just makes it much harder to bring an unsafe product to market, and makes it clearer who to blame when people die.

jimmytucson• 14 years ago

I don't blame anybody who thinks it's not as regulated as it should be but you might be surprised at how extensive the current requirements are. Storing ~200 million daily transactions in order to fulfill arbitrarily complex reporting specifications from SEC, FINRA, internal audit (rogue trading detection, etc.), not to mention analytics for other trading systems... Oracle/IBM/whatever enterprise big data provider gets the contract must be sending one gigantic fruit basket around Christmas time.

EDIT: Here's a little light reading for you http://finra.complinet.com/en/display/display_main.html?rbid... (or you could just take sandpaper and rub it against your brain)

AncientPC• 14 years ago

I can't find the links at the moment, but some HFT guys have responded to relevant Slashdot articles over the years.

From what I remember from those accounts, traders often like to tweak algorithms at the last second and there is little to no QA before changes gets pushed live.

It's sort of a byproduct of the necessity for speed. Algorithms are quite complex, and even running a basic test suite that takes a few minutes may be deemed unnecessary.

thibaut_barrere• 14 years ago

I can tell first-hand that it depends on the places: some have fairly clean tests including fuzzy testing on trading automation, while other have, well, "fix it when it fails in production".

ewood2• 14 years ago

I have experience in HFT, there are similarities to market making and I have plenty of colleagues who've worked in market making. Just like any company the culture is largely dependent on those in charge. Founders of these companies fall into three buckets - traders, techies, and mathematicians/physicists - and quality control will generally be a function of the founder mix. Mostly techies: strong software culture, unit testing, realtime monitoring. Mostly scientists: strong algos, software that works, monitoring varies. Mostly traders: risk v reward is the driver, software quality is unimportant unless it affects short-term profit, monitoring is unimportant - dollars and cents are sufficient.

Obviously every firm's goals are driven by the goals of those in control. In the case of Knight they are largely a trader driven firm that has arrived late to the algo party. They were looking to get ahead by being one of the first market makers on the NYSE's new retail order matching system and probably cut some corners to get there. From a risk v reward perspective it probably looked like a good bet - with no major competitors customers would flood in and any bugs could be ironed out in live. Unfortunately the 'fat tail' (http://en.wikipedia.org/wiki/Fat_tail) struck and it may have sunk their company.

For a closer look at what went wrong see http://www.zerohedge.com/news/what-happens-when-hft-algo-goe...

jriley2• 14 years ago

Thanks for link. Given your experience, do you have any thoughts about a small circuit-breaker on every security that trips for, say any 3-standard deviation event?

In a quick Google search, it seems circuit breakers do not kick in for first 15 minutes of trading: http://www.nytimes.com/2012/08/02/business/unusual-volume-ro...

njharman• 14 years ago

> considering that the survival of the company could be at stake

It's at stake either way.

QA is not risk free. Time spent in QA is opportunity cost lost. Many things are very time to market sensitive. One must balance "perfect" against "shipped".

This is even more true in (electronic) financial industry.

elmarks• 14 years ago

Doesn't this mean that others made a killing, taking advantage of all the mispriced orders?

jonah7• 14 years ago

Most likely. c.f. the recent JPMorgan losses[1].

[1] http://www.pbs.org/newshour/businessdesk/2012/06/who-benefit...

sahilz79• 14 years ago

This was apparently an infrastructure problem of some sort: http://www.bloomberg.com/video/tom-joyce-knight-is-open-for-...

Infrastructure changes can be notoriously difficult to back out by simply using an "off" switch, particularly if this was some type of a firmware upgrade that impacted all of their production servers. Backing it out at a minimum would require some type of a reboot, which would cause problems with an active trades. It could very well be that they were running an Active-Active environment, they had to go Active-Passive, back out the changes from the passive environment, reboot, and surgically cut over to the passive environment. This could easily take 30 minutes.

teekarja• 14 years ago

Strange article. Lots of text but missing the main thing I was looking for. What kind "erroneus trades"? where did the money go? If you buy stock at the market you did not intend to buy, why not just sell them the next day?

jonah7• 14 years ago

Seems like maybe they couldn't hold on to the stock for long enough to unload the enormous volume they were dealing with. It sounded like at one point they were doing AS MUCH VOLUME AS EVERYONE ELSE on the exchange combined.

http://news.ycombinator.com/item?id=4337750

Devilboy2• 14 years ago

Since there's 2 parties to every trade doesn't that make 50% the limit?

iskander3• 14 years ago

You can get more technical details from nanex: http://www.nanex.net/aqck2/3522.html

Essentially, they were buying high and selling low. Many times a second.

noselasd• 14 years ago

Because the program sold the stock it bought immediately, at a loss. Leaving you with nothing to sell the next day.

SagelyGuru2• 14 years ago

'Prediction is difficult, especially with regard to the future'.

It looks like they were using some new algorithm, which should have made them a lot of money, had the market gone up after their massive purchases. In that case, they would have pocketed fat bonuses and would not be on the news.

However, it has not happened, so the crying and the search for a scapegoat is on. It sounds like the case of the banking business as usual: 'heads I win, tails you lose'.

Ultimately, there is a really serious problem with the concept of limited personal liability for companies engaging in speculation. It is an assymetric arrangement, whereby the directors are entitled to the profits but are never personally responsible for the losses. With such rules of the game, it is advantageous to take crazy risks. Expect to see a lot more of this and many more taxpayer funded bailouts.

justincormack• 14 years ago

They were not taking big positions expecting the market to go up, they appear to largely have been burning money buying and selling fast.

There is no government money involved.

SagelyGuru2• 14 years ago

Of course, buying high and selling low is always the 'reason' for making a loss. In this case, I think the program caught itself out by manipulating the market, which it perhaps naively assumed to be non-manipulable.

In other words, it was creating so much volume that, when buying (or selling), it made the market go up (or down). It was then reading the price as going up (or down) and jumping on its own bandwagon. This, of itself, would create growing oscillations in the market and growing losses.

For this to work for you, you need to first create a trend and then sit back and let the suckers pile in on it and take the losses. You then return only when you want to reverse the trend again, at a profitable level (for you). I suspect the program was just too fast for its own good and not a match for the human Masters of this art.

beagle3• 14 years ago

That's not what happened, according to nanex.

It just kept making markets in reverse (instead of joining the bid and the offer, it bid on the offer and offered on the bid).

Nanex speculates that Knight ran their tester software on the real market (the tester losing money on purpose to the main algorithm). Alternatively, it could simply be a bug that sends a bid instead of an offer and vice versa. One bit flip at the wrong place could cause that.

ams6110• 14 years ago

Guess their developers never watched "Trading Places"....

"Wilson!!! Get back in there and SELLLLLLLL"

iskander3• 14 years ago

They weren't taking long-term speculative positions, their short-term market making had a major bug in which instead of buying low and selling high it was doing the opposite (thus losing a small amount on each pair of trades). For more details read Nanex's blog (http://www.nanex.net/aqck2/3522.html).

So, they launched a busted market making algorithm, lost a ton of money and no one is going to bail them out.

jonah7• 14 years ago

And the nanex guess at why it happened: http://www.nanex.net/aqck2/3525.html

yummyfajitas3• 14 years ago

Huh? Knight Capital lost a bunch of money, and will likely go bankrupt. Essentially, Knight's software bug transferred a bunch of money from Knight to everyone else. This poses minimal systematic risk to anyone else, and they will almost certainly get no bailout.

The markets have already recovered. The S&P was down a little bit on thurs and recovered by friday. Knight is down 60%.

http://www.google.com/finance?q=INDEXSP%3A.INX%2C+NYSE%3AKCG

This is ultimately a situation of the market being a robust and stable dynamical system.

gkuan4• 14 years ago

Yes, no risk to others unless you happen to be one of their newly acquired futures broker unit's customers with $411 million in deposits. http://www.reuters.com/article/2012/08/02/knightcapital-regu...

white_devil• 14 years ago

> This is ultimately a situation of the market being a robust and stable dynamical system.

Not while it's being shaped by algorithms competing against each other, which you're a part of.

bagosm• 14 years ago

So, some of the owners were looking for a way out, and magically this thing broke loose and started giving away (basically) free money to undisclosed receipients. In the meantime all the technicians were fast asleep and couldnt kick the machines down or something, while they were losing milions of dollars per minute. This article is a completely honest recap by completely honest people, about completely honest traders/bankers (bankers are not people).

Edit: on a COMPLETELY unrelated note, trading firms/banks are known to actively pursue the extraction of money from their clients with bogus trades/advice http://www.nytimes.com/2012/03/14/opinion/why-i-am-leaving-g...

0x02• 14 years ago

Why isn't automated high-frequency trading banned already?

Does it not go directly against the spirit and purpose of having a stock market with proper investors?

retube3• 14 years ago

this is issue nothing to do with speculative trading. Knight is a broker. They provide an interface to the market for retail brokers, spread betting outfits and so on. Their algos execute orders placed by their clients. Their new algo had a bug. That's it.

0x02• 14 years ago

I don't know the details of retail brokers vs HFT, but this writeup http://www.nanex.net/aqck2/3522.html has a lot of charts showing trades with 25 millisecond intervals.

Just looking at things in a big perspective, the fact that the system is designed for allowing trades at such frequencies makes it seem like markets these days no longer exist for the benefit of the listed companies.

Then again, maybe I don't know wtf I'm talking about :-S I guess I don't understand how real value can be created from such a system.

retube3• 14 years ago

There seems to be a lot of confusion around market making, brokering, execution algorithms and HFT in this thread.

sgt101• 14 years ago

I wonder if the lack of a kill switch was linked to the power black outs in India - what if they set it going, got cut (power) and then only came back online 45 mins later? Could be that just one link - say a local exchange or power for an FTTP line failed.

brokenparser• 14 years ago

Can anyone provide some context on this matter? What happened wednesday and where?

davvid• 14 years ago

http://blogs.wsj.com/marketbeat/2012/08/02/knight-capital-tr...

Basically, they deployed a new HFT algo and it started buying high and selling low. oops!

reddiric• 14 years ago

Can someone smarter than me please explain why a system where trades got matched / executed at a granularity of once per second or once per several seconds wouldn't work? What would be the problem with exchanges accumulating and keeping secret buy and sell orders and executing them at a reasonable interval?

SeanDav• 14 years ago

This is an extreme example of what what can happen when what should be a software company thinks it is some other sort of company. I am sure they thought they were a trading company and software development was the necessary evil required to get things done.

Well 400 million odd dollars in the red later I doubt they still feel that.

I do feel sorry for them and they probably didn't deserve this huge loss. Hopefully valuable lessons can be learned.

tlogan• 14 years ago

I don't believe the problem was caused by "bug" - this reminds me of "rouge trader" stories.

It is kinda weird that all problems in Wall Street are caused by "bugs in software" or "rouge traders": while executives are never hold accountable.

teyc2• 14 years ago

I read the nanex article. Regardless whether it is true or not, the general trend is towards development of more sophisticated load testing programs.

The most benign ones were developed for use in IT systems. E.g. Apache bench. While these can cause disruptions if aimed at production services, this does not necessarily threaten the health of an entire enterprise.

However, the trend is that all software sectors are starting to adopt this particular technique of testing software with not sufficient regard to what happens if it is released into live systems.

For example, we have chaos monkey, from Netflix, which randomly shutdown services in a cloud based system.

What would happen if software which simulated meltdown at a nuclear facility was accidentally bundled into the build system by a tired operator? Or some one does the same with flight software?

The main software running trading platforms would presumably be supervised by another program to ensure that bad algorithms do not lose e company too much money. However there was no such tool for the component that generated the test data.

To me, it sounds like the supervision should be done at a higher level, e.g. A wrapper around existing APIs. All software running against live systems must call into the wrapper.

Secondly, test software should conduct some kind of verification. E.g. Check for evidence that it is testing against a Test system. This might be the presence of a nonexistent company, et c.

I am more than happy to compile any other ideas you may have so that the IT industry is able to build more fail safes into software.

We are starting to see some of these fail safes in practice. E.g. When you try to send out an email to everyone in the organization, email software may warn you if you are sure you want to do that. The problem is we haven't thought enough about these scenarios that we don't adequately address them.

Incidentally, over in Australia, the Commonwealth Bank suffered a major downtime when it's outsourcer HP accidentally pushed out system wide updates instead of doing this to select machines as originally intended.

Trading Program Ran Amok, With No ‘Off’ Switch

Related Stories