How the Robots Lost: High-Frequency Trading's Rise and Fall

http://www.businessweek.com/articles/2013-06-06/how-the-robots-lost-high-frequency-tradings-rise-and-fall

by nohuck13 • 13 years ago

131 86 13 years ago

businessweek.com

yummyfajitas• 13 years ago

tl;dr; Competition between HFTs has reduced margins on market making to near zero.

Wonderful news for consumers of liquidity, bad news for sellers of it.

eksith• 13 years ago

More than that though, the robots only win when they're fighting panicked human traders who try to maximize and fire blind (or at least a few seconds slower). Speaking of intelligent programs, they failed to achieve equilibrium with their natural environment, as Agent Smith would say.

The takeaway:

  HFT also lacks the two things it needs the most: trading 
  volume and price volatility. Compared with the deep,
  choppy waters of 2009 and 2010, the stock market is now a
  shallow, placid pool.

nissimk2• 13 years ago

Yes, that's the integral section of the article, but what it's really saying is that at least one component of this is the current cycle low in volatility and volume. This is a classic wall st shakeout when the best capitalized firms that make it through the down part of the cycle come back much stronger when their market returns because their competition closed up shop. They haven't scared everybody away, they're just suffering a market downturn in volatility and volume right now.

The robots win when people trade more and when prices move around more. When those conditions return, the robot party will be back on.

ott22• 13 years ago

The second main point: with declining volatility, there is less space for HFT strategies to profit.

What I didn't see in the article was an explicit discussion of transaction costs. Transaction tax would increase those costs, but was part of the decline because transaction costs have already gone up?

crusso2• 13 years ago

That and this sad kicking in the groin of a system that's working through technology issues as it should:

Democrats in Congress would go further. Iowa Senator Tom Harkin and Oregon Representative Peter DeFazio want a .03 percent tax on nearly every trade in nearly every market in the U.S

Politicians just have to get in and start pecking for some extra revenue and attempt to give the illusion that they're going to help.

ihsw• 13 years ago

Interestingly Sweden tried a 0.5% tax in 1984:

> During the first week of the tax, the volume of bond trading fell by 85%, even though the tax rate on five-year bonds was only 0.003%. The volume of futures trading fell by 98% and the options trading market disappeared.

Needless to say, 'tax avoidance' became the norm (especially since it was exceedingly easy to circumvent) and it was later repealed in 1991.

[1] http://en.wikipedia.org/wiki/Swedish_financial_transaction_t...

whazzmaster• 13 years ago

In today's deficit boogeyman environment I can't say that extra revenue isn't a consideration, but really the tax was to attempt to stem a runaway financial system that has become the blood-sucking vampire squid[1] that's taking more and more of the wealth while providing little of value to the country.

[1] http://www.guardian.co.uk/business/andrew-clark-on-america/2...

clavalle• 13 years ago

Yep. They are currently scrambling into every market on the planet to try and eke out profits there but the pace is accelerated.

Before too long I think we will see HFTs merging until there are only a handful of players globally.

IOW, HFT is transitioning into a commodity.

macavity23• 13 years ago

Nice anecdote about Knight Capital:

"When the market opened on Aug. 1, a new piece of trading software that Knight had just installed went haywire and started aggressively buying shares... By the end of Aug. 2, Knight had spent $440 million unwinding its trades"

I'd love to know more about that particular launch. A hole in somebody's test suite!

themckman2• 13 years ago

I think the story goes that something not ready for production made it into production. I obviously can't say anything about their test suite, but it sounds more like a mismanaged merge/deployment process.

nissimk2• 13 years ago

There are some interesting links here:

http://spectrum.ieee.org/riskfactor/computing/it/new-insight...

I had initially heard that the test program was released to production causing the system to take the wrong side of every trade, but I don't think it was ever confirmed.

joezydeco• 13 years ago

Nanex has lots of fun data on the Knight incident:

"We believe Knight accidentally released the test software they used to verify that their new market making software functioned properly, into NYSE's live system."

http://www.nanex.net/aqck2/3525.html

"We can fully appreciate the nightmare their team must have experienced that morning."

And a whole boatload of amazing charts and data here:

http://www.nanex.net/aqck2/3522.html

themckman2• 13 years ago

Nanex is a company that intrigues me immensely. I follow Scott (their founder, I think) on twitter and I wish I understood even a small percentage of what they produce on a daily basis. The stock market in general is something I wish I understood more of. Can anyone recommend some reading material?

chollida12• 13 years ago

One of the prevailing theories from http://www.nanex.net/ is that when Knight went live they also put their order simulator live as well.

The thinking behind this theory was that a simulator would keep adding liquidity to a fake market for testing a market making algo.

During the day that Knight lost their pants nanex saw that while their algo was taking liquidity someone else was constantly providing it, ie the bid ask spread didn't really open up that much and the price held relatively flat.

This meant that someone else was constantly adding liquidity that Knight was taking away. The thinking was that Knights market simulator was the one sending out the live orders to add liquidity.

TL/DR Knight crossed themselves many many times in a few minutes.

See http://www.nanex.net/aqck2/3525.html for the acutal details

mcphilip• 13 years ago

As an example of why Nanex is worth listening to, here is an article [1] describing how Nanex claimed Monday's ISM manufacturing data was leaked 15 milliseconds early, leading ultimately to an admission by Reuters that it caused the leak. Eric Hunsander, a software developer at Nanex, went so far as to hint to Reuters that they should check their NTP settings since a drift of 15 milliseconds is allowed by default [2].

Eric is very interesting to follow on Twitter [3]. He highlights mysterious HFT related events on a daily basis. My only complaint is that his articles and charts, in particular, are very difficult to interpret. As an outsider to the HFT industry, I don't know where to find the resources to learn enough to interpret most of Eric's insights.

[1] http://www.cnbc.com/id/100792260

[2] http://www.nanex.net/aqck2/4306.html

[3] https://twitter.com/nanexllc

minimax5• 13 years ago

Some tweaks to your explanation:

The thinking behind this theory was that a simulator would keep adding liquidity to a fake market for testing a market making algo.

If you're testing a market making algo (i.e. to participate in NYSE's RLP program), you're simulator needs to remove liquidity, not add it. The idea is that the simulator (the tester) accidentally got deployed, and then went around crossing the spread (removing liquidity) and building up large positions in various instruments.

TL/DR Knight crossed themselves many many times in a few minutes.

A self trade doesn't affect your profit. Any self-trades would have been problematic because it's illegal to paint the tape, but that's not what caused the losses.

beachstartup• 13 years ago

applying occam's razor, i'm not sure which would be the simpler explanation:

1. a test suite got deployed accidentally, that happened to exactly match the buying predisposition of the production algorithms and fronted a bunch of their own trades @ a loss, or

2. someone got greedy and spilled the beans, and a competitor knew which algorithm would be unleashed on day X and buried it with their own

anonu2• 13 years ago

apparently there was a test component that took every order the model generated and copied the message 5 or 10 times - supposedly to stress test the system. Somehow this component was in production. Real orders were going out to the market and executions flowed back to an error account that was not being monitored. I imagine if you were monitoring the PnL you would have seen massive profits in the first few moments that the system went haywire - only to realize that you were simply just driving up the prices of your own positions...

anonu2• 13 years ago

also - the changes were made on the same day NYSE's RLP program went live. so the guess is that these changes were made to handle RLP flow.

fmstephe• 13 years ago

From what I read about this crash the most important lesson was to have a kill switch. The biggest problem Knight Capital had was that they could not switch the system off quickly. It was roughly 45 minutes (according most accounts) between Knight realising something was wrong to ceasing trading.

That there was a bug in some code isn't surprising. That it was so hard to stop their trading is really surprising.

chollida12• 13 years ago

In Knights defense they aren't just a high frequency hedge fund that trades their own book to make a profit.

They have legitimate customers that they put orders out on behalf of and allow clients to trade through them.

Part of the reason why it took so long to shut the system down is that if they had just shut the system down, then they'd be in breach of contract for those clients.

So shutting down was more of a business decision that a technical one.

brazzy• 13 years ago

IIRC there were indications that they restarted the system twice in those 45 minutes, to no avail. Presumably it took that long to get someone with the authority to make a decision to switch it off entirely to decide to do so.

shrikant• 13 years ago

More details and related HN discussion here: https://news.ycombinator.com/item?id=4329101

ott22• 13 years ago

Another key point: "the SEC is sharing information with the FBI to probe manipulative trading practices by some HFT firms" and now has the infrastructure to detect anomalies.

glitchdout• 13 years ago

I get why the need to always be one step ahead pushed trading firms to automation. Still, for some reason, this makes me incredibly sad. Maybe it's because I don't fully understand how the world and the economy work. Why does it have to be this way?

woodchuck64• 13 years ago

It's the co-evolution of parasite and host. These firms are like tape worms, siphoning off nutrients in the hopes the host won't notice. But the host is continually evolving new mechanisms that make it difficult for the parasite to survive. The economy is growing, changing and becoming more efficient, thanks to the internet. If history is any indication, easy-money traders are headed for extinction.

minimax5• 13 years ago

We use technology to improve efficiencies in every industry in the world, but when we use technology in the capital markets people freak out. I don't get it.

Nursie3• 13 years ago

Huge volumes of money, a perception of it being gambling and playing with numbers rather than productive work, the possibility of screwing over basically the whole of society when it goes wrong, privileged positions for the entrenched traders (people that outsmart the algo-traders get arrested, massive costs to get the millisecond co-lo advantage on HFT etc), total divorce of algo/HFT from the idea of actual investment in companies based on what they do...

There are some reasons that people are suspicious of the entire world of finance. HFT and algorithmic trading really exemplify all of the issues some people have with the money markets in the first place.

nohuck13• 13 years ago

So, there's a correlation between HFT and volatility. Yup.

Causation is hard though.

"Trading volumes in U.S. equities are around 6 billion shares a day, roughly where they were in 2006. Volatility, a measure of the extent to which a share’s price jumps around, is about half what it was a few years ago. By seeking out price disparities across assets and exchanges, speed traders ensure that when things do get out of whack, they’re quickly brought back into harmony."

There's been academic research that HFT market making dampens intraday volatility [1]

There have also been findings in the other direction [2].

I suspect to the extent you're talking about market making, the first version is true, but you almost never see it in articles like this. Maybe it makes for poor sensationalism.

[1] http://www.wsuc3m.com/HFT_and_Volatility_Final%20Brogaard.pd...

[2] http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1691679

Mikeb85• 13 years ago

HFT basically involves an arbitrage. The more competition there is, the less profits are made. It's simply a case of the market becoming more efficient, so the arbitrage opportunity is gone.

Meanwhile, consumers of market makers (ie. normal investors) benefit from being able to make any trade in a split second, for very little cost. In the end, it still requires a human element.

programminggeek• 13 years ago

I'm sure it's been done, but I know there is a place for longer term automated trading as well. Computers have an advantage over the super short term, but even on the short, mid, long term, there is money to be made in finding holes based on human behavior over time, which still drives a big chunk of the market.

chiph• 13 years ago

Iowa Senator Tom Harkin and Oregon Representative Peter DeFazio want a .03 percent tax on nearly every trade in nearly every market in the U.S.

The camel's nose is heading towards the tent.

omni• 13 years ago

Explanation of this metaphor for people like me who had never heard it before: http://en.wikipedia.org/wiki/Camel%27s_nose

edit: fixed link, whoops

kgermino• 13 years ago

http://en.wikipedia.org/wiki/Camel%27s_nose

radikalus• 13 years ago

HFT Profit = f(Noise/Signal) - g(Tech Costs)

X4• 13 years ago

The implicit claim is that your trading engine won't beat established trading engines and because of volume.

Both can be falsified, by inventing a better trading engine and getting foreign capital into it. Yes the margin for profit isn't that high anymore, but trading engines+trading markets are one card in the stack one can fight with. There is no holistic solution to making maximum profit, except succeeding when others fail. That's why innovators are always rare goods, because capital herds dry it out.

T-hawk2• 13 years ago

"According to Rosenblatt, in 2009 the entire HFT industry made around $5 billion trading stocks."

Where does this money come from? The industry doesn't actually create any wealth. There must be a greater fool who lost this $5 billion to the HFT industry. But who is that? Who became $5 billion poorer to make the HFT guys that $5 billion richer? Is the article just counting the total profit of HFT winners and ignoring the HFT losers?

This is not rhetorical, I really want to understand this.

intrazoo• 13 years ago

The industry doesn't actually create any wealth

=> in theory, does not "investing" sorta "create" wealth? This might be the answer, otherwise I, too, am stumped.

herge2• 13 years ago

The finance industry creates wealth by making financial transactions and investment instruments cheaper and more efficient.

Same way as a rail company or Amazon creates wealth not by 'building things', but by moving things from point A to point B cheaper, more easily and for less risk.

T-hawk2• 13 years ago

But there's still a loser there: the older transportation or retail providers who can no longer sustain their high profit margins. The Pennsylvania Railroad took in profits that used to belong to Erie Canal shippers. Amazon got rich by collecting money that used to go to Barnes & Noble at brick-and-mortar retail. So from where did the profits or money come that make up the HFTs' profit?

From the other responses, it sounds like the real answer is salami slices from lots of other market participants. A large mutual fund buyer pays a slightly worse price because the HFT jumped in ahead of them to arbitrage between the selling market maker and this buyer. So the HFT's profit comes out of the mutual fund's capital. The loss would show up as tracking error for an index fund, for example.

ionforce• 13 years ago

I think it is an abstraction about how finance is as a whole. I gather you are not in finance, but neither am I.

Let's say I'm a fledgeling lemonade stand and I want $10 of investment. You give me that $10 and with it I make $15. You wanted a 10% return on your investment so you get back $11.

With this model, you have "created" $1 in investments. That $1 originally came from the $15 I made with my lemonade. And all those $15 were exchanges for convenience or sheer happiness gained from having lemonade.

So no one really "lost" in this scenario. But I think the finance people call this "creating wealth". Vs other scenarios, where if you invest in someone who is bad at selling lemonade, you only get $8 back, now you have lost $2 from your initial $10. The idea is to continually "create" wealth by making good choices, investing in good lemonade stands vs bad ones.

So when someone says x-machination (high frequency trading) has "made" $5B, there isn't necessarily a loss anywhere. That's just how much that came out on top.

etherealG• 13 years ago

in your example you left out the person that paid for the lemonade. they lost. i think in an idealistic value system, all value is exchanged not created, a lot like energy in physics. not that i'm saying that's a good model for economics, but it's a valid high level thought, and one i think the original poster might have in mind.

Symmetry• 13 years ago

HFT is a way for market makers to compete among each other. When I want to buy stock in some company or other I need someone to sell it to me, and when I want to sell a stock I need someone to buy it. I might get lucky and someone like me might be looking to sell or buy at the same time, but that's unreliable. However, there are people who have some of pretty much every stock out there, and who are always ready to sell a share of ACME to me for $11.59 or buy it from me for $11.58. Because there's a granularity to the prices of stocks the market makers can't compete on price, so exchanges end up using speed to resolve who gets the trade. Hence HFT.

oscilloscope• 13 years ago

Two ways off the top of my head, both mentioned in the article:

Arbitrage: balancing the prices between exchanges. This helps settle all exchanges on the market price, so you don't pay $5 more just because of the exchange you're on.

Market making: By placing both bid and ask orders, algorithms can tighten the bid-ask spread around the price. When the spread is too large, people are disincentivized to trade because neither buyers nor sellers can agree on a common price.

I would guess that day traders lose out, since they miss the opportunity to act on many of these same opportunities. Also I've heard index funds lose several basis points to HFTs when rebalancing.

vilhelm_s• 13 years ago

Wikipedia has this list: http://en.wikipedia.org/wiki/High-frequency_trading#High-fre...

Different trading strategies will make money in different ways. Trading ahead and ticker-tape trading will redirect some of the profits that that an index fund would have made to the HFT trader. On the other hand, market making is in effect providing a service to everyone who wants to trade in the market, and they pay for it (partly through the spread between buy and ask prices -- this is like a fee for being able to buy/sell now rather than later, and partly through explicit ECN fees charged by the exchange).

mc-lovin• 13 years ago

This is a good question. In financial models, one common approach is the "noise trader" theory where some people make random trades. These are the people who lose out to the informed traders.

As to not creating wealth, it is possible that HFT adds value by seeing where the market is going (slowly) and putting the price there right now. Also, it may be able to extract more information from the truly informed traders, so that the market as able to have more information given less informed traders.

That said, I doubt HFT adds much positive value, but as the rules of the game evolve, ultimately the will favor people who add value.

segmondy• 13 years ago

401k funds, pension funds, social security, etc.

cowsandmilk• 13 years ago

How, pray tell, do HFTs make Social Security poorer?

dustingetz• 13 years ago

if i want to buy 1000 bitcoins at $121.46 to hang on to for a while, and while i'm buying the price goes up to $121.50 because a HFT is trying to frontrun my purchases, i'm not going to stop buying, its still worth it to me at the higher price. those few pennies, in aggregate across billions of trades, can add up.

gd1• 13 years ago

If you want to buy 1000 bitcoins then cross the spread, take out the best offer, and buy 1000 bitcoins. If, on the other hand, you want to stick a bid into the market in the hope that someone else will cross and take your (lower) price instead... then yes, other participants will react to that.

fr0sty• 13 years ago

Explain in discrete steps how you imagine this works. What was the prevailing offer at the time you placed your order? How many coins did you buy before exhausting displayed liquidity at 121.46? How do you imaging HTFs front-ran your purchase? how would things look differently in the absence of HFT?

hollerith• 13 years ago

>The industry doesn't actually create any wealth.

Are you certain about that?

phyalow• 13 years ago

>Where does this money come from?

In theory those posting liquidity will reduce the bid/ask spread, those that are taking liquidity will see a reduction in frictional costs of buying and selling costs. It is the reduction in frictional costs that generates $$$. That and being able to have an edge in whatever orderbook/microstructure you are playing on.

davidmr2• 13 years ago

As in most things of this nature, I think a good deal is to be learned by who are the most vocal informed opponents. The most vocal market participants against HFT are usually large, institutional market makers who used to grow fat on $1/16 spreads and liquidity rebates (Themis, I'm looking in your direction). These guys used to not have to work particularly hard for the cash, but they're getting their lunches eaten by new market participants who can do it faster and cheaper and have built their companies around integrating new tech very rapidly.

You'll find people trotting out the same old chestnuts (brain drain, unfairness of server colocation, etc.) but those are distractions from the interesting issues.

As an aside, I once had the distinct pleasure of watching a conversation on one of the CNBC programs running in our kitchen in which a NYSE floor trader actually used the phrase "we're the institutional investor looking out for the little guy" while managing to keep a straight face. This dude probably would have curb stomped his own grandmother to pick up a $100 bill. Even the usually sympathetic hosts just laughed outright.

I'm surprised that the outrage against HFT is so strong given how comparatively little is at stake there. After the HFT's much, much, much larger financial cousins, the "big bank" market participants nearly destroyed the entire planet's financial system, it's quite a coup for them that they've shifted the outrage to the guys competing over less money than some banks pay out in bonuses.

I have no love for anyone in the financial industry (I'm a self-loather, if you haven't noticed), but the HFT guys are way, way further down on my list of people to be pissed off at.

Nimi• 13 years ago

Mark Cuban's criticism may interest you: http://blogmaverick.com/2012/09/21/what-business-is-wall-str...;

TLDR: ... their stated value add of liquidity does not compensate the US and World Economy nearly enough for the risk of collapse they introduce into the system.

Lagged2Death• 13 years ago

One of HFT’s objectives has always been to make the market more efficient. Speed traders have done such an excellent job of wringing waste out of buying and selling stocks that they’re having a hard time making money themselves.

One of its objectives?

Those philanthropic high-frequency traders just wanted to make the market more efficient for everyone? Sure they did. Sure they did.

hcal• 13 years ago

I think they mean "more efficient" like the cotton gin made the cotton industry more efficient. By reducing the labor involved in processing cotton, the companies that used cotton gins could eek more profit from every acre of production.

HFT doesn't "wring the waste" of inefficiency back into the market, but into the hands of the HF trader. HFT does benefit normal market traders, but I'm not sure to what extent.

cynicalkane• 13 years ago

You can say that about any industry that turns a profit. Those philanthropic $MEMBERS_OF_MONEY_MAKING_INDUSTRY just wanted to make $THING_INDUSTRY_DOES better for everyone? Sure they did. Sure they did.

Making money is not a character flaw. Otherwise, many prominent members of this board would be horrible people.

cma5• 13 years ago

Your caveat doesn't apply if they aren't helping markets be more efficient. HFT is basically a latency arms race ripoff scam. Whoever pays more for faster lines and closer access to the datacenter takes the cake. Thousands of PHDs are working on this instead of real stuff.

Say this is the makeup of market participants: all of them are normal merchants buying fruit for their shops but 2 are latency arbitragers. Shipments of fruit don't always come in because sometimes the sail boat crashes into something.

The latency arbitragers hire high speed skiffs to go scout for the ship before it arrives. If one of them sees that a shipment of oranges are missing, he rushs back to game the unsuspecting others in the market: orange prices are going to go up so he buys them on the cheap.

Because nothing gets better with two competing arbitragers. They simply compete with each other on hiring out more skiffs and paying for elaborate faster ones. Or signaling by latern along a chain of ships.

None of this helps the market; all other participants lose out. With competition between the two arbitragers they end up wasting most of the money buying faster ships. Ships that could be used for.. rescuing crashed orange ships? Many other things as well. The worst government bureaucracy couldn't be more wasteful.

Markets aren't efficient even in theory if there is an information asymmetry.

I hope you can see the analogy between my example and things like:

* deploying more and more servers to literally turn our energy into waste heat (they could be simulating protein fold or something)

* paying trading exchanges extra for in-house servers (this is just a complete scam)

* Digging hundreds of miles of new fiber line to save 2ms of latency over the existing line (with no plausible consumer benefit because the old lines had plenty of dark fiber and the new line is monopolized by whomever pays the most)

* Setting up microwave (lantern?) relay towers to get from New York to Chicago a millisecond faster

* Making markets more likely to have a major crash by forcing developers to use C++ to eek out a couple milliseconds over safer alternatives

You can make a little bit better case for electronic market making.

tachyonbeam• 13 years ago

What they're actually saying is that there's now too many high-frequency traders competing for there to be much profit left in this money scraping business.

peterjancelis2• 13 years ago

Awesome. Capitalism working as intended. There was an inefficiency in the market, people exploited it and low barriers to entry now make the profits disappear. Our stocks are now better priced as a result.

javert• 13 years ago

Yeah. I see this article as an amazing validation of HFT to all the naysayers.

benjamincburns• 13 years ago

http://www.businessweek.com/printer/articles/122900-how-the-... - one page version.

How the Robots Lost: High-Frequency Trading's Rise and Fall

Related Stories