U.S. stock markets are rigged, says author Michael Lewis

http://www.reuters.com/article/2014/03/31/us-markets-hft-flashboys-idUSBREA2U03D20140331

by schintan • 12 years ago

223 192 12 years ago

emmelaich5• 12 years ago

I think it's been raised here before, but why can't all trades be submitted, kept secret then only executed simultaneously at some discrete interval - say every 5 minutes?

kasey_junk11• 12 years ago

If you have 3 people buying at price x and 2 people selling at price x, which 2 of the 3 buyers get their purchase?

The rules for this are the "matching algorithm". In most markets this is determined in a fifo fashion. Adding a discrete time delay wouldn't change this, there would still be a race to be first into the 5 minute period.

emmelaich5• 12 years ago

Then randomise rather than FIFO it. Or pro-rata it -- (an option submitted with the offer)

pak• 12 years ago

Then forget the FIFO convention. How about a hardware random number generator?

However, this doesn't fix the problem with placing and canceling lots of orders to sway the bucket toward the end of the 5-minute window. I suppose if some could do this faster at the end of the window then others, they would still retain an advantage.

(Imagine somebody placing many orders, monitoring how many are likely to be fulfilled advantageously, and then cancelling the disadvantageous ones as close to the end of the window as possible.)

That could perhaps be solved with fees for cancelling bids or asks.

hapless• 12 years ago

Because it wouldn't reward insiders.

joezydeco• 12 years ago

"At an event hosted by the [Chicago Mercantile Exchange] in October, Leo Melamed, one of the pioneers of Chicago's financial futures markets, called Budish's proposal a "nonstarter." He said the markets are working pretty well and there's no need to fix something that isn't broken.

Otherwise, Melamed said jokingly, "I'm going to consider going to North Korea and show them how they can become the center of finance in about three weeks."

So yeah this has been discussed and no, the powers that be like things the way they are. Go figure. This will need stronger powers to force the hand of the traders.

http://www.chicagotribune.com/business/ct-high-speed-trading...

crdoconnor9• 12 years ago

You could hardly expect the people profiting from this to do anything except come out against it.

bloaf• 12 years ago

Its amazing how similar this line of reasoning sounds to the development of democracy mode in Twitch Plays Pokemon.

sseveran10• 12 years ago

This is available on exchanges today. It usually happens 2 - 5 times per day.

stillsut• 12 years ago

Dare I say it, but...

"To the tune of Ten of Billions" is that sooo material?

That amount's not unheard of for a single company, hedge fund, hell even individual (there was also that guy that loss 8 Billion).

If it's half of all trading volumne, but a tiny fraction of profit, then this is razor thin margins. In these areas, we've always been comfortable allowing the big boys in any industry (with their economies-of-scale) to execute efficiently, and hopefully compete each other out of ripping to rest of us off too bad.

digz• 12 years ago

The 'tens of billions' is also without context of how much money HFT save investors by basically keeping spreads at zero, and nearly always guaranteeing liquidity. Providing liquidity to the market isn't a public service... participants need to be compensated.

makomk• 12 years ago

"Nearly always" guaranteeing liquidity isn't that useful a service - HFT's liquidity evaporates when it's most needed, like when the markets crash.

snowwrestler4• 12 years ago

I don't understand how HFT can make me pay more than I want to for a stock. I set a limit order--the limit is how much I'm willing to pay. If some HFT firm pushes the price past my limit, then I don't buy.

nemasu• 12 years ago

This is true, but I believe if you don't set a limit order, then the market value will go up. Then if the price keeps going up, you'll have missed your chance to buy the rest.

wging• 12 years ago

It can make you pay more than you otherwise would, not more than your limit.

jcdavis• 12 years ago

Just because you have a limit order doesn't mean you aren't getting fleeced, as you could've bought it for less otherwise (assuming some level of fixing as is suggested)

jhonovich• 12 years ago

You still lose. In your scenario, you are blocked out of buying the stock. And without a limit order, you get the stock but pay more.

Imagine this happened when ordering food. Sure, you could put in a limit (no more than x french fries and y per fry) but your price would either be run up or you'd go hungry.

snowwrestler4• 12 years ago

But I'm not buying food, I'm buying a stock--which, unlike food is a totally optional purchase.

My limit is the price at which I thought that stock was worth buying. If the price never falls that far, then I wouldn't want to buy it anyway.

rrhyne3• 12 years ago

HFT pushes incremental buy orders, which your sell order accepts (price discovery) but are canceled before they are filled. Knowing your limit, they then fill your order at your limit.

It's illegal for anyone to issue an order they don't intend to complete, but the HFTs get a pass for some reason or other that must end with quite a few zeros.

yummyfajitas19• 12 years ago

HFT pushes incremental buy orders, which your sell order accepts (price discovery) but are canceled before they are filled.

This is incorrect. If you place a BUY@$100 and someone has a SELL@$100, the exchange instantaneously matches the orders and a trade is issued. You can't cancel an order which was matched.

The "price discovery" happens before the trade is placed. The instant the exchange receives the BUY@$100 order it is published to the world via the exchange's multicast quote stream.

"Issue an order they don't intend to complete" is an anachronism related to pit trading - in the pit, it's possible to throw the "I accept" hand signal at the other guy and then change your mind after trading hours during reconciliation. The FIX and OUCH protocols do not have a similar "oops I changed my mind now I'll be a jerk after hours" message.

mathattack2• 12 years ago

In the simplest of cases, let's say a stock you are following is at 50.50. You want to buy it at anything under $50. It drops to $49.99, and someone cuts in front of you in line, and buys it. You miss the purchase, and the stock goes back up.

Multiple this millions of times.

This isn't to say they are all completely evil. They do provide liquidity as banks have walked away from equity. But there needs to be more transparency on what is going on. (For example - should it be legal to send false buy or sell signals which will be intentionally cancelled?)

ChuckMcM2• 12 years ago

The false buy/sell signsls are a legitimate issue. Given that you can't easily make it illegal to cancel a limit order (after all people give up after a while) you could require that the order has to be valid for 60 seconds before it can be cancelled and that would put a huge crimp in the 'faking it' market.

yummyfajitas19• 12 years ago

There is no "cut in front of you" message in FIX or OUCH. Most exchanges use price/time priority - if someone "cuts in front of you" it means they offered $50.01.

Or maybe "cuts in front of you" means you wanted to buy the stock at $50, but you didn't actually place the order until the last minute because you wanted to play a speed game (and lost).

AndrewBissell8• 12 years ago

It drops to $49.99, and someone cuts in front of you in line

Um, how do they do that, exactly? The major equity exchanges are all FIFO on every tick price. If you put a limit order in for $49.99 before somebody else, and an offer crosses your bid, you'll get filled before they do, period.

ChuckMcM2• 12 years ago

It doesn't.

If you watched 60 minutes tonight, or read the book, the problem was that a large trader (the guy they profiled ran the trading desk for RBC) would make an order for a large block of stock, say 400 shares of X, and then send that out to all of the exchanges.

What would happen is it would get to the first exchange and be partly filled and when it was the HFT guys would send trades on ahead to the other exchanges faster than RBC could and buy out the stock before the RBC order got there, leaving the order only partially filled. (or filled part at one price and filled the rest at a higher price).

In the story they mention that by restructuring how they sent their trades out so that they arrived at every exchange simultaneously eliminated the possibility of the HFT guys front running them.

In order to make it possible for everyone to do that they have created their own exchange (iEX) which locks out the HFT folks (well it has them going through an extra 60km of fiber to slow down their access).

harmegido2• 12 years ago

His issue is with market fragmentation then, not HFT. If all the liquidity providers were at the same market, his large order would hit all of the liquidity at the same time. This is a valid complaint to me, but blaming the HFTs for "front-running" (if you could even call this that) is a bit like UPS complaining that FedEx is using airplanes to quickly move packages around. Keep up with the technology or stop competing.

patio112• 12 years ago

Basically, certain big buyers/sellers of stock want to be the ones to benefit from the information "Someone really wants into or out of this stock", and they really, really dislike when the HFTs democratize that information advantage faster than the larger players can gain extra returns from using it.

Trading communicates information to the market. Proprietary access to information is valuable. HFTs make certain information non-proprietary faster than the state of the art prior to HFTs. That changes the allocation of profits associated with that information. This is the entire ballgame. Retail traders swim placidly on the sea above totally ignorant of the krakens vs. mutant-sharks-with-lasers bloodbath happening 20000 leagues below.

Edmond3• 12 years ago

Actually limit orders are a perfect way for HF Traders to fleece regular investors...for instance if you set a stop loss at a certain level and a HFT causes the price of a stock to drop precipitously, you'll end up selling likely at a loss for an artificial reason.

This scenario used to play out a lot during the 2008 financial crisis. I remember loosing money because of such tricks.

patio112• 12 years ago

If you do not intend to sell a stock for $40 then it is, in general, not advisable to tell your broker "Sell this stock at $40."

hueving2• 12 years ago

A stop loss is not a limit order, at least in finance terms.

tim3332• 12 years ago

How it makes you pay more for a stock than your limit order works as follows. You see XYZ is trading for $49.99 and put in an order to buy 10,000 limit $50/share. Your trading platform replies you have bought 4000 at $50, the price has moved to $50.50 you have a 6000 share order waiting to be filled. You then say darn it I'll wait for it to go back to $50 and it goes up more and so on. As has happened to me many a time. Though as a trader you can play the game too and offer to sell shares at a high price if it looks like someone needs to buy. This stuff is one reason why paper trading gives a false sense of how easy trading is as historical prices don't actively try to figure what you're up to and screw you.

james1071• 12 years ago

I thought that what happened was that the HFT players were able to find your limit and so front run your order.

otikik2• 12 years ago

My understanding is that if the price was initially lower than your limit, they can buy it at that price before you can, and then sell it at your limit.

svedlin• 12 years ago

University of Michigan researchers proposed a "centralized call market" to mitigate fragmentation (inconsistent pricing and order handling across different exchanges) and latency disadvantages:

http://www.ns.umich.edu/new/releases/21535-high-frequency-tr...

Haim Bodek discusses how certain exchanges provide special order types that give traders priority over others:

http://online.wsj.com/news/articles/SB1000087239639044398920...

Of course HFT is just a small part of the built-in asymmetries in the markets. Firms exploiting their own clients, conflicted investment advice, corrupt fund management, routinized insider trading, retail bucket shops, hidden fees, payment for order flow, specialist/MM privileges, etc. are all ways that investors and beneficiaries can be disadvantaged.

cma5• 12 years ago

It's not just insider information that is a problem. Is the world really better off because UBS/hedge-funds hire out satellites to surveil Wal-mart parking lots in order to get an earnings prediction 1 day before Wal-mart files their public earnings disclosure?

dclowd9901• 12 years ago

Tell me you have a source for this.

gautamc• 12 years ago

There are a many of news articles and youtube videos which present HFT as a bad thing.

Since large orders are filled by different markets, anyone who has studied the mechanisms via which these orders are full-filled can theoretically arrive at a strategy that might be profitable. Overtime, more and more people will learn these things and it will no longer be a profitable strategy to implement - this is sort of like the "January Effect" described in the book named intelligent investor.

I don't understand why HFT has to be blamed for a difference in prices across multiple markets. I read somewhere that REGNMS caused market fragmentation and HFT traders spend time studying the current markets and identify inefficiencies that can be used to make profitable trades.

This Reuters article is like an advertisement for the IEX trading platform. IEX figured out a certain way to beat the other algorithms and they told Reuters about it. Now that everyone knows it, I think, pretty soon other algorithm developers will modify their implementations.

Overall, I think that HFT is basically algorithmic trading, but which needs more investment (for renting/buying server/connectivity) and more time/interest (for studying and understanding the order placement/fulfilment mechanisms involved).

dllthomas2• 12 years ago

It's not that HFT is to blame for differences between markets, but that differences between markets plus the market structure are to blame for (the socially wasteful bits of) HFT.

crdoconnor9• 12 years ago

I don't understand why HFT has to be blamed for a difference in prices across multiple markets.

Because it isn't.

Destitute• 12 years ago

To give you an idea on how HFT has blown up since 2007: http://www.technologyreview.com/sites/default/files/legacy/d...

Context: http://www.technologyreview.com/view/428756/watch-high-speed...

yummyfajitas19• 12 years ago

For those of you who want to understand this topic better (i.e., a lot of the people spinning conspiracy theories), you might want to learn the basic mechanics of HFT. I wrote a tutorial on the topic a couple of years ago, might be helpful:

http://www.chrisstucchio.com/blog/2012/hft_apology.html

http://www.chrisstucchio.com/blog/2012/hft_apology2.html

AndrewBissell8• 12 years ago

I really like your idea of allowing sub-penny ticks in the highly liquid stocks as a way of tilting algo trading away from the latency arms race. Unfortunately, it probably runs too counter to the uninformed "no one should be trading to make $0.01 per transaction!" popular perspective for the SEC & exchanges to implement it.

kasey_junk11• 12 years ago

As someone who actually makes a living in HFT, if you really wanted to get rid of me, sub-penny ticks would be the single worst thing I can think of. There is literally no transaction tax that would make trading unprofitable. 2 zero's worth of decimilazation though and I have to go find honest work.

harmegido2• 12 years ago

Nice posts. You see 0 value to the latency arms race, and I think you are probably right.

However, I think it's not hard to argue that some speed increases have value. For example, if the world's only stock market only allowed trades to happen n times a year, we would expect a pretty inefficient market and this mechanism for capital allocation would be pretty poor. And as we increase n, market efficiency would increase, and this has value.

So are we at a point right now where an increase in n has no value (say market information gets processed on the order of nanoseconds instead of the current microseconds)? What's the cutoff point?

yummyfajitas19• 12 years ago

This is a good question and one I don't have a good answer to. I don't actually know the optimal points for most of these values and I'm generally content to let the market decide.

This is why I think eliminating the subpenny rule is a good policy change - it eliminates a market distortion while not requiring you to perfectly tune a parameter (e.g., the size of a transaction tax or a minimal trade latency). All it does is allows price competition.

Of course as a real policy proposal it's dead in the water, since it doesn't provide much opportunity for moralizing or posturing.

AndrewBissell8• 12 years ago

You see 0 value to the latency arms race, and I think you are probably right.

How do you draw the line between investments in reduced latency which are socially beneficial, and those which aren't? It's not like these are cheap investments that HFT firms are making. If they're truly pointless, we should expect them to lead to overcapacity and ultimately the exit of some players (unless, of course, the arms race is being unnaturally subsidized in some way, e.g. by the lack of a sub-penny rule).

When Paul Reuter was using carrier pigeons and telegraphs to move tradeable information between stock exchanges ahead of his competitors, I wonder if people considered this an arms race of negligible social value. After all, the world had gotten by just fine without telegraphs up until then.

floatrock• 12 years ago

So are we at a point right now where an increase in n has no value (say market information gets processed on the order of nanoseconds instead of the current microseconds)? What's the cutoff point?

My intuitions want to say the point of diminishing returns is around the limits of human cognition. Stocks are ownership of some human activity, so once you start valuing them faster than human activity you're really redefining what stocks are. Now stocks become predictions on ownership of some human activity.

Since the first formalization of Sport (some might argue earlier), it's been a human tendency to go Faster, Stronger, Better. Moving from ownership of some human activity to predictions on ownership of some human activity is a manifestation of this Faster, Stronger, Better (since the prediction is itself a human activity... we're owning the human activity of predicting human activity).

All that is not to say any of this is inherently good or bad (I disagree at the loaded term 'rigged' being used by Lewis). There's no single cut-off point in additional value earned. It's more of a transition zone where you're not sure when exactly you left it, but at some point you realize on the other side is not the world you started in.

crdoconnor9• 12 years ago

This is a pretty weak defense of HFT:

Although the latency competition provides little to no value to the end consumers, all HFTs must play the game. If they don’t, a faster HFT will beat them to market and their order flow will be reduced.

So yes, it's essentially useless to the retail or wholesale investor.

Actually, worse than useless, since the HFT has a built in advantage that brings no value to the rest of the market. It's an endless zero sum game where other investors (who do the real work) are losing.

I’d also like to note that I don’t believe automated market making is useless. It isn’t. Providing liquidity and reducing the spread is valuable.

Which HFT is actually pretty poor at. Spreads were reduced significantly by having electronic trading systems (computers are cheaper than manual labor), NOT through high frequency trading systems (did spreads drop significantly in the last 5 years?).

Liquidity was already maxxed out in the equities market before HFT, too. HOUSES are illiquid -- it takes weeks to close on a sale on one of those at best. THAT market could do with additional liquidity. Would be nice to close on a sale in a day, right?

Being able to buy and sell your share a thousand times in a second is not providing useful liquidity to anybody except HFT traders.

At the end of the day they're levying a tax on the rest of the market - i.e. as the OP points out, the market IS rigged.

In the Hacker News comments on part 1 of this series, there were several comments suggesting that HFT’s somehow steal pennies from ordinary investors. I don’t agree with this claim, but I’m very interested in hearing well reasoned disagreement.

I'd be interested in hearing this well reasoned disagreement with the OP. I'm not so interested in reading the same old tired defenses of HFT, though.

It's not about liquidity or transaction costs. It's about gaining an information advantage similar to what front running or insider trading gets you.

yummyfajitas19• 12 years ago

So yes, it's essentially useless to the retail or wholesale investor.

Having liquidity available to take is not useless. You should re-read the article - it is a zero sum game between liquidity sellers, and a positive sum game between liquidity sellers and liquidity consumers.

At the end of the day they're levying a tax on the rest of the market...

It's not a tax. If you choose not to take liquidity you don't pay for it. If you do choose to take liquidity you pay less than you would under other systems (e.g., specialists).

cynicalkane• 12 years ago

You'll notice that any accurate argument against HFT can be applied to any kind of trading.

There's nothing sinister about HFT except that it's a faster version of ordinary market making. If you think market making is evil, then you'll think HFT is evil. But market making is perfectly healthy, economically speaking. Information advantages and speed have been part of this world in the 400-odd years that centralized trading has been a thing.

HFT is a threat to other market makers, but you're smart enough to know that common people shouldn't be market making and can (provably) only benefit from reduced spreads and increased liquidity. Right?

gojomo3• 12 years ago

The approach touted by the 60Minutes piece, the coop-like IEX market that aims to equalize participant lags, is one obvious countermeasure.

Another promising tactic could be the "Frequent Batch Auctions" as described in the paper referenced in this blog post:

http://johnhcochrane.blogspot.com/2014/02/budish-cramton-and...

Direct paper link: http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre...

Paper abstract:

We argue that the continuous limit order book is a flawed market design and propose that financial exchanges instead use frequent batch auctions: uniform-price sealed-bid double auctions conducted at frequent but discrete time intervals, e.g., every 1 second. Our argument has four parts. First, we use millisecond-level direct-feed data from exchanges to show that the continuous limit order book market design does not really “work” in continuous time: market correlations completely break down at high-frequency time horizons. Second, we show that this correlation breakdown creates frequent technical arbitrage opportunities, available to whomever is fastest, which in turn creates an arms race to exploit such opportunities. Third, we develop a simple new theory model motivated by these empirical facts. The model shows that the arms race is not only socially wasteful – a prisoner’s dilemma built directly into the market design – but moreover that its cost is ultimately borne by investors via wider spreads and thinner markets. Last, we show that frequent batch auctions eliminate the arms race, both because they reduce the value of tiny speed advantages and because they transform competition on speed into competition on price. Consequently, frequent batch auctions lead to narrower spreads, deeper markets, and increased social welfare.

minimax• 12 years ago

In a batch auction like that you still have a latency game. There is no reason to submit your order until the last possible moment since you will want to incorporate as much information from other markets and news sources as possible. You would just end up with a crush of orders in the last 100 uS of the batch interval.

RoboTeddy• 12 years ago

I haven't read the paper, but I would guess that the one-second-worth of orders in each batch are secret until the batch executes

thecage411• 12 years ago

What about randomizing the auction time?

dpcheng2003• 12 years ago

From the 60 Minutes piece: http://www.cbsnews.com/news/is-the-us-stock-market-rigged/

madaxe_again3• 12 years ago

Ah, a band-aid on a band-aid.

What was wrong with open outcry? Sure, you had to physically be there, but nothing has changed - traders still cluster around exchanges to keep latency low, HFT kit still lives "on the floor", and what was once done with humans and bits of paper is now done in a far less efficient fashion with computers. Less, because the marginal benefit ends up going into the pockets of those who can afford exchange co-lo and mountains of hardware to parse FIX a tick faster than the other guy, meaning that the rest of the market ends up suffering, and not having a more efficient fill mechanism, as touted.

HFT is that asshole who takes his umbrella with him into the pit, and blocks your trades.

If anything, the solution here is to abolish algorithmic trading, and decrease tick resolution substantially, which if lengthened enough, would take C out of the question and make the entire market equally accessible to all.

I mean, seriously - does the value of Brent light sweet actually change thousands of times per second? According to the markets, yes, according to reality, where people buy futures with straddles/swaps so as to have a fixed, manageable cost, no.

The entire thing is un-necessary, and doesn't add value to humans.

smoorman10242• 12 years ago

The most obvious and simplest thing wrong with open outcry and human market makers is the number of specialists that it takes to handle a full universe of stocks. A small electronic trading team can provide/take liquidity in 8000 stocks or greater. If you need specialists you are going to have to provide a living salary for every single specialist who will only be able to handle 20 stocks at most. The cost of trading for the end consumer is higher.

murbard22• 12 years ago

My favorite solution to the breakdown of correlation is to have order books for beta adjusted prices. That is, you don't bid for Google stocks, you bid for a Google stock minus an equivalent amount of S&P. The price of this pair is much more stable than the price of Google, which means spreads can be tighter.

The nuttier generalization would be to have an order book for each principal component of the covariance matrix of the market. It wouldn't work in practice for a variety of reason (least of which is that the covariance matrix is unstable, ill defined, etc) but it's a good thought experiment to understand the problem.

tdaltonc• 12 years ago

This is an interesting idea. For this to impact market correlation, would you need to insist that everyone place their orders in beta adjusted terms or would it be enough for the bid making protocol to allow for it?

d23• 12 years ago

The criticism I've heard of this is that HFT only extracts at most 2-4 billion from the economy each year. Is that the case? If so, I find the arguments against to be kind of trivial.

jamieabe5• 12 years ago

The bigger story of HFT is that it enables the big banks to manipulate the stock market. Because they know all market orders before they are executed, they are able to cancel all bids and offers before it is possible for anyone to hit or take them. They can also put up huge orders on the bid or offer to make a stock look weak or strong in the short run, knowing there is no risk of being executed. If they see another order come in they can just cancel their order. The most criminal advantage HFT gives them is that they can front run orders. Let's say for example that Fidelity calls Goldman Sachs and says they want to buy 1 million shares of Tesla. This information is priceless as Tesla stock is guaranteed to fly higher on this trade. Acting on this information is the most blatant form of insider trading that exists. However, somehow the big banks can create programs that do exactly this. Whenever they see a huge influx of buy or sell volume they can quickly jump in front and close out the transaction a few seconds later for a huge profit.

AndrewBissell8• 12 years ago

What you describe above would be highly illegal.

jamieabe5• 12 years ago

That's the whole point...Michael Lewis said he can't believe it hasn't been made illegal....5 years ago, during the financial crisis there was a good story on it in the New York Time and one of the Senators said they were outraged and he was going to change it, etc, and of course nothing ever happened.

yummyfajitas19• 12 years ago

How does the HFT know Fidelity called Goldman Sachs? Are they tapping the phones?

Whenever they see a huge influx of buy or sell volume they can quickly jump in front...

Could you tell me what FIX/OUCH command is used to do this?

jamieabe5• 12 years ago

I was drawing a real life comparison to how HFT works...it gives the big banks order information before it is executed.

As for actual commands, sorry I am not a programmer, just a former trader.

crdoconnor9• 12 years ago

Whenever they see a huge influx of buy or sell volume they can quickly jump in front... >Could you tell me what FIX/OUCH command is used to do this?

Uh, buy and sell.

Could you tell me which part of FIX/OUCH makes quote stuffing impossible?

sseveran10• 12 years ago

Can you point me to the order type or flag that allows me to jump? How can they front run? Do you even know what front running is?

jamieabe5• 12 years ago

They can front run because they receive all of the order flow decimals of a second before the orders are executed. Let's say for example you saw (or your computer told you) that X market orders for Y shares to buy a stock just entered the market...your program tells you this is a lot of volume for this stock and it should push up the stock. Before any of the orders are executed your algorithm jumps in front and buys the stock and sells it a second later for a nice profit. This is a very simplistic example.

fleitz• 12 years ago

Hmm... it's almost as if brokers break up large orders into smaller orders for exactly this reason.

jonemo• 12 years ago

The story states that IEX runs HFT trades through an additional 60km of cable. Does anyone know or have a source for how this specific number was chosen? Who decides whose orders have to go through those extra 60km and whose don't?

tim3332• 12 years ago

Interesting article - I'd only seen high speed trading discussed rather abstractly before. Katsuyama's "Essentially, our fill rates went to 100 percent. We couldn't believe it when we actually figured it out," is interesting experimental data on how it's actually working. I wish him luck with the IEX thing.

Nursie• 12 years ago

All stock markets are so far removed from actual useful endeavour that it's hard to see what their function might be.

I think we need some human-friendly reform in this area.

zaroth3• 12 years ago

The HFTs are doing something which reminds me of the mutant attack with Bitcoin. And the 'fix' is also very similar to how you can try to defeat mutants, if not by making mutation impossible in the protocol (which we are trying to do), but also making it more difficult to beat the propagation of the original transaction with mutants; by transmitting it to more nodes concurrently.

Unlike the mutant attack, in this case there are possible positive effects that can arise.

But what I don't understand is this. I place a human order to buy 1,000 shares of X for price P through a public exchange. The HFT algorithm can get those shares for (P - a) on a private order book I don't see, before my order even arrives there.

But from the perspective of the seller(s) on the private book, they had open sell orders with weighted average (P - a) for those 1,000 shares. That's open orders, a.k.a willing sellers. As long as there are real shares being bought by the HFT before the sale to the original buyer, then how can you complain?

But what I really want is to be able to place these cross-exchange orders with something like a 2 phase commit. Get a 'lock' on the public buy and the corresponding private sell(s) before having to commit. And then clear the committed transactions concurrently on both exchanges.

That's a risk-free trade, and the result you might expect is the spread is shared between buyer and seller. Maybe some of it goes to transaction fees.

And that's how I look at the HFT algos. They are basically a hidden transaction fee on a very fast routing of your order. There super-order-routers charge a transaction fee which is applied by tweaking the listed price. Over time the amount of the tweak decreases until it exactly equals your stated price, which means the super-order-router gets no fee.

If you had [paid for] access to that other order book, you could have spent less / earned more on the trade. But there's the added cost of having a direct connection to that order book. So basically you buy it yourself, or someone else pays for it and acts like an automatic passive router. In the end you get exactly the price that you asked for.

I wonder how often the HFT algorithm is selling shares that it doesn't really own. No one should have been surprised that MtGox wasn't running full reserve. Who's publishing the crypto-proof that the public exchanges are running full reserve? Haha, actually we know that they don't - they just call it naked shorting. So I guess I'm asking, how much naked shorting (even over brief time periods) do the HFT algos get away with?

sseveran10• 12 years ago

What order book are you talking about?

And any broker dealer can sell shares they don't own...Because they are a broker dealer. Call your congressman.

zaroth3• 12 years ago

So this amounts to some amount of free float for the broker dealer. How much are we talking in terms of $-days?

If they can sell and cover whenever they want, then yes it does start to sound like a 'print money' button.

HoneyDaPooh• 12 years ago

FYI, all Michael Lewis books are must-read for those interested in Finance and especially useful to understand some important Wall Street psychological features and possibly explain why some bubbles have gone bust in the past decade. These books are also technically pretty sound. In order of personal preference: The Big Short, Liar's Poker and Boomerang. Lowenstein on LTCM is also good in the same verge with a similar technical level as Flash Boy seems to be but for 1990s Arbitrage Trading.

eyeareque• 12 years ago

All time stock market highs, inflated stock prices due to HFT bidding up prices and thus siphoning off money by the billions, companies operating at a loss or minimal profits getting billion dollar valuations, it's been a while since the last major stock market drop hasn't it?.. Is it just me, or are we playing with fire here? It seems like if one thing goes wrong the whole house of cards will come crumbling down.

I'm worried about my retirement after reading more about HFT. Where should I put my 401k to avoid losing a big chunk of it as soon as the market takes its next drop?

pippy• 12 years ago

Stock markets aren't rigged, they're fundamentally flawed. HFT is only a small modern flaw that's emerged.

In theory a company can float itself in order to raise capital and improve performance. The business will invest the capital, and in turn return some of the improved profits to the share holders.

In a modern context it simply can't work this way. In a world with fiat currencies, any organisation with securities can directly invest in an stocks/bonds higher with a return higher than the Bank Rate to an extent far greater than individuals, with less risk.

This leaves individuals with little to no capacity to get a decent return without brokerage. While possible to make money, it's a roll of the die that would be better of done in a casino.

eddd• 12 years ago

I see much of propositions for solving problem with HFT. Mine? "Random delay for signal (between 100-200ms) on network devices", problem solved. Book "Dark Pools" describes how HFT works 'behind the scenes', great pice of journalism what's more: people rarly point out, how big drain on resource HFT is, bilions of dollars spended on fiber lines between continents, to gain a few miliseconds, this is madness.

prostoalex• 12 years ago

I never buy/sell at market price, always at limit.

Am I still affected by HFT in a meaningful way?

If not, perhaps the every small-time investor should be defaulted to a limit buy/sell unless they choose otherwise.

megaframe• 12 years ago

Yes if you've ever had an order not get completely filled... happens to me a lot, it's kind of annoying and usually not worth chasing the price up.

Every small investor should use limit orders. Market orders are rigged at the Market Maker levels via a process called Slippage. "Slippage occurs when a market maker changes the spread to his advantage on market orders." basically they stretch the spread out forcing you to pay a higher or get a lower price for the stock to their advantage since they fill the quantity from their own volume.

throwaway13qf85• 12 years ago

That is the entire game of market making. You are literally complaining about market makers doing their jobs.

As a market maker, I might sit out there offering to buy 1000 shares of Microsoft at $40.20 and sell 1000 shares at $40.30. Then someone comes along and buys 1000 shares from me at $40.30, so I need to put a new quote into the market. Do I put it in at $40.30 again?

Hell no! There are two main reasons why not -

1. I know that there are buyers out there, and most likely the reason there are buyers at $40.30 is because they believe the stock is worth more than that. I need to revise my prices upward to reflect that new information.

2. I'm now short 1000 shares of MSFT. If the price goes up further, I will lose out. I need to persuade someone to sell me 1000 shares to cover that short position, so I need to offer more money - therefore I need to raise my quotes.

Most likely is that I'd now put in a new offer at $40.32 and raise my bid from $40.20 to $40.22.

This is the mechanism by which market prices reflect new information. Market makers change their prices to reflect information in the order flow. This is the entire reason that liquid, efficient markets are able to exist.

bherms• 12 years ago

Bogle has a great book on how horrible hft is for our economy (well, he has several technically) called The Clash of Cultures... Reading it now and am fascinated. Could be that this is my first foray into learning about finance, but either way, I highly recommend checking it out.

chrisbennet• 12 years ago

Thanks for the recommendation. (Why did someone down vote you??)

bachback• 12 years ago

great. here is the new exchange they are talking about: http://www.iextrading.com/

next step: build an investor exchange opensource based on bitcoin. take over the world. if you're into this kind of thing let me know.

junto• 12 years ago

build an investor exchange opensource based on bitcoin

Can you elaborate?

marincounty• 12 years ago

I have always felt the markets are rigged, but by insider trading. High frequency traders just add to the problem. I thought the Internet, and websites like Stocktwits might level out the playing field, but the I've Never seen anything close to insider information on any of these sites. I truely believe we are on a bubble right now--especially tech stocks(Tesla might be the exception, even with the owner telling investors the stock is overpriced.) Facebook throwing around billions for for that disappearing picture company floored me. Oh, and Google--the minute Duckduckgo gets a little bit better--bye--bye, but I'll use your free Api's as long as they stay somewhat free. My deceased father spent years dabbling in the stock macket when he retired; the only real money he made was on a tip he got from a drunk, bolsterious father in law who was a CEO. Oh yea, this brings me to Jim Cramer. I have watched him for years. He has never timed a down turn right. He just might be the best ironic contrarian stock adviser ever? Right now, he believes the stock market will just continue to rise. I am waiting for the drop. We are in a weird time though; gold at unheard of levels. Stock market rising. The only real inflation I see is food. I have a funny feeling the economy is much worse than it seems though. Companies know we cannot go without eating--so let's raise the price. I'm a nobody, but I hate to see middle class families loose their life savings when the market takes a dive. I hope these lucky tech billionaires help the American poor when things get really bad? There are very few countries that would have allowed them to Mae there wad as easily as they had it in America. Could you imagine having a billion dollars in Mexico? You couldn't leave your home. That's the one thing they take for granted, while griping about tax rates? Sorry for rambling.

brudgers• 12 years ago

As Mark Cuban might ask, "Can you spot the sucker in the stock market?"

otikik2• 12 years ago

Group the requests by second and randomize their order. Boom.

alexeisadeski3• 12 years ago

Someone's trying to sell a book.

bayesianhorse2• 12 years ago

To me this does not sound "rigged". It sounds like one participant in the market feels entitled to the profit another market participant makes.

One issue with most criticism of financial markets is that people tend to view only the price, and may be the transaction cost. What they completely fail to see is risk management and liquidity. Both have intrinsic value, and people are selling and buying liquidity and risk management. This concept is hard to grasp. But the HFT traders are providing a value to the market.

Restricting HFT will undoubtedly lower liquidity, and maybe even increase risk. That's why regulators are so anxious about it.

The HFT traders are indeed taking on some amount of risk, which otherwise would be with the stock exchange or the other participants. The salient question is whether or not the competition among HFTs is hard enough for the Bertrand paradox to kick in.

kelvin0• 12 years ago

OK, then why would investors actually switch to doing business with iEX if HFT brings any value at all? Looks to me like HFT is a like 'free' games that 'offer' to buy in-game items which are needed to play the game properly. In this case, investing is the game and if you don`t have a billion dollar fiber optic running, you play at a net disadvantage.

bayesianhorse2• 12 years ago

"a billion dollar fiber optic" has to come from someplace. Money costs money, which is the basic tenet of finance. If an HFT company invests a few hundred million Dollars into the trading system, the profit better beat the market. That's why I think that there are probably enough competitors in the HFT sector to squeeze the profits.

The investors who "switch to iEX" are money managers who have to place orders to diversify a portfolio. Without the new strategy they essentially pay some extra money to intermediary traders. The value of intermediary traders is hard to grasp. They do carry the risk inherent in their positions, and they provide liquidity by bridging the time-gap between the intention to buy and the intention to sell.

iEX basically seems to offer their clients the chance to bridge this gap themselves by cleverly dispersing orders over different markets and obfuscating their intentions.

Finance is a fascinating technology in itself...

newblahbl4hblah• 12 years ago

WHAT????!?!!?!?!?! Oh my god...

arbuge• 12 years ago

"They are able to identify your desire to buy shares in Microsoft and buy them in front of you and sell them back to you at a higher price," Lewis, whose book is available on Monday, said on the television program "60 Minutes" on Sunday.

I'm guessing the net effect of this is that you get trades for a few cents more than they would cost you otherwise. I guess "otherwise" means a perfect world where Goldman Sachs is a charitable organization run by Mother Teresa directing a band of angels.

This might be an issue for day-traders but if you're a buy and hold investor aiming for doubling or tripling of your holdings over years-decades, this small fractional cost at inception is likely negligible. (Which is not to say that the stock market is a safe place to be - there are many other cons to be aware of if you're a buy and hold person - insider trading, bad quality earnings, etc.).

It was not that long ago that investors were stuck with paying high commissions equivalent to several percentage points to acquire a stock. Now you can get 1000 shares of anything for $5. And maybe a few dollars more to line the pockets of HFT firms. Things seem better to me now on the whole.

lee• 12 years ago

I don't think the cost is necessarily negligible.

If the system is setup in such a way that there are those who benefit from every trade, skimming fractions of a percent per trade, those add up. They are effectively shrinking the market capitalization of that stock... and over a long period of time that could be significant.

If a stock's losses to this are in the range of 3% per year, over a 25 year period the stock will lose 50% of it's market cap from this activity.

That's fairly significant.

georgehaake• 12 years ago

It's good this is getting more attention, however there were many good books before this one. "Quants" and "Dark Pools" were my favorites and a good place to start.

The Physics of Wall Street: A Brief History of Predicting the Unpredictable

Automate This

The Quants: How a New Breed of Math Whizzes Conquered Wall Street and Nearly Destroyed It

Dark Pools: High-Speed Traders, A.I. Bandits, and the Threat to the Global Financial System

Hedge Fund Market Wizards

Zenst• 12 years ago

This caught my eye "travelled along fibre optic lines and hit the closest exchange first, where high frequency traders would get a glimpse, and then use their speed advantage to beat him to the other 12 US public exchanges and 45 private trading venues."

Why not simple stagger the network dispatch to those exchanges so the order hits all exchanges at exactly the same time and with that protect you from HFT snipping tactics. Seem simple solution and somewhat obvious so what am I missing?

U.S. stock markets are rigged, says author Michael Lewis

Related Stories