Donald MacKenzie on high-frequency trading

http://www.lrb.co.uk/v36/n17/donald-mackenzie/be-grateful-for-drizzle

by joosters • 12 years ago

52 102 12 years ago

lrb.co.uk

joosters5• 12 years ago

IMO this is one of the best articles I have read on HFT. It gives a good, balanced overview of the technology and the fundamental principles behind how automated trading works, without any particular viewpoint or opinion directing the text.

The section near the end on FPGAs and programming styles was an eye-opener to me. I wonder how many other firms have taken risk checks out of their code to save some nanoseconds...

conistonwater9• 12 years ago

without any particular viewpoint or opinion directing the text

You say that like that's a good thing.

I know that a lot of rubbish has been written about HFT and algorithmic trading, but it's sad that no one seems to be able to write about its merits without starting a flame war.

Retric2• 12 years ago

It's an outgrowth of poorly designed matchmaking software at exchanges. The could completly distroy HFT without costing society anything or adding transaction fees.

mer10z• 12 years ago

I can't help but think of that article yesterday about how we're avoiding the Big Problems (https://news.ycombinator.com/item?id=8261098). It's crazy how much money and brain power people have invested in HFT which is essentially just turning time into money with no benefit to society. What could the human race have achieved if the same amount of effort was focused in other areas?

fchollet• 12 years ago

It could be that our current economic system implements a pretty terrible resource allocation algorithm, where many workers' jobs consist in undoing the efforts of employees from concurrent companies.

In trading, a quant's job is to outsmart or outspeed the other bank's team. They invest considerable brain power, effort and money into producing incremental improvements that are quickly rendered meaningless by the work of the team in the building across the street.

Another prime example is advertising, a zero-sum game where competing brands fight to undo other brands' brainwashing efforts and replace it with their own brainwashing. A waste of the time and energy of million of humans.

Some more ideas on the subject: http://www.sphere-engineering.com/blog/15-hour-work-week.htm...

Retric2• 12 years ago

Advertizing and HFT are only zero sum for competing products in an established market. For new products it can be a net benifit ex: "talk to your doctor about ED."

afafsd• 12 years ago

Don't be mad that some other people's brains are bent on stuff of no particular benefit to you. Their brains, their choice.

Just be glad that some people's brains are bent on stuff of benefit to you, and concentrate, if you like, on what of general benefit you can do with your own brain.

crdoconnor19• 12 years ago

The economy is geared towards playing such zero sum games. This is pretty much by design - policies such as austerity, ZIRP and financial deregulation shift the allocation of capital to speculation (gambling) and away from productive investment.

kasey_junk24• 12 years ago

There are a lot of misconceptions in your comment that we need to unpack.

1) there isn't that much money in HFT. It has an ever decreasing number of players working for an ever decreasing amount of profit. Other areas of finance such as hedge funds, traditional investment banks, and derivatives creation are much bigger and have just as many "smart" people working in them. Don't get me started on advertising...

2) HFT doesn't turn time into money. It buys & sells risk in the form of liquidity. One way that it does this more efficiently is by being able to get in and out of positions fast.

HN, predictably, has a huge bias towards the capital raising/allocation side of the financial markets, but this is not why most participants are involved. Hedging risk is another huge (probably bigger) reason the markets exist. HFT allow that risk to be hedged more cheaply and more predictably so they provide value to the rest of the market.

tobltobs• 12 years ago

Goldman Sachs’ Blankfein on Banking: ‘Doing God’s Work’. This kind of self-deception seems to quite prevalent with those banksters.

GFK_of_xmaspast• 12 years ago

They're totally doing god's work, it's just that the god in question is Mammon.

ord-lhr-nyc-nrt2• 12 years ago

I worked on these systems and I can categorically say this is the best summary of HFT I've ever come across.

Many of the systems, fiber networks, data centers, and software that I touched are mentioned here. I've never heard the term "bit fucking" before, but the explanation of having to understand what happens under the metal is very true. You have to have a good understanding of software, compilers, assembler, CPU micro architecture, network protocols, and operating systems to be able to operate in this space.

I don't really know what else to add, except that I and many others I worked with in this business think that batch auctions are the solution. Although the work is technically challenging and interesting, I certainly don't like the arms race aspect of it, and I wish the resources (natural and human) used by this sector could be better allocated. One of my big concerns was the added CO2 emissions of constantly spinning programs doing no useful work - all in the name of minimizing latency. I did some research into reducing the carbon footprint of our code by reducing idle spin cycles. Alas, I never really had the time to get it working well before I decided to pack up and move on.

Happy to answer any questions anyone may have.

harmegido4• 12 years ago

Also been involved in this line of work. How would you see the specifics of a batch auction working? I'd see it as not doing anything if the matches happened in a FIFO manner (it'd still pay to be first). Would a pro-rata batch auction fix the problem? Or would we have to introduce a random component to match all crossing orders in each batch?

joosters5• 12 years ago

Batch auctions also have the opposite problem, whereby it makes sense to submit your bids as late as possible, to let you utilise the very latest information prior to the auction close. This might actually balance out some of the FIFO advantages?

kasey_junk24• 12 years ago

Batch auctions already exist in electronic markets. They still allow for huge advantages for speed.

If you want to get rid of the speed element (I'm not sure it's a problem) you should explicitly allow people to pay for priority. The easiest way to do this would be to allow for extremely small fractions of price. So instead of having to pay for faster lasers I could just change my pricing structure to tack on .00001 on to the price.

yummyfajitas5• 12 years ago

This proposal, while economically sound and something I have advocated for for years, fails to allow politicians and pundits to express their strong moral disapproval of geeks making money.

Faint• 12 years ago

Why would the batch auction need to be FIFO? What about something like this:

1) When opening a "new tick", release data about trades in last tick, and start accepting new orders. Information about orders submitted to this tick is kept secret. Allow setting order price to practically arbitrary accuracy. 2) After tick closes, match orders in _random order_. That is, it should not matter at all on at which point in time inside this tick order arrived 3) repeat

Allowing arbitrary accuracy would help turn the competition from speed or stuffing the order book with huge number of orders to "spoof" the random matching, towards competition of just providing lower spread. I understand that it would still advantageous to hold submitting trades until very end of the tick, to take into account as much outside information (other markets, etc.) as possible. To counter this, maybe the exact length of the tick could be randomized also. What do you think?

kasey_junk24• 12 years ago

So the biggest argument against random matching (whether in batch auction or open market book orders) is that the easiest way to game it is to figure out your expected fill rate. Then you just raise your order volume such that your expected fill rate matches your risk profile.

For the advantage of removing the speed advantage, you get the downside of providing advantage to people that have the loosest risk profiles and you further complicate the audit burden such that it is impossible to follow.

Think of it from the perspective of a value investor that set a market order 6 months ago. Under the current system once their price becomes top of the book they have a virtual certainty of getting filled. Under a (fair) random match they have the exact same chance of getting filled as someone who literaly just put in their order.

tfinniga2• 12 years ago

I don't understand why HFT is still around, if it's considered a problem.

Naively, couldn't all the craziness about HFT be resolved by enforcing a 1-second tick on the markets?

Like, right now it appears that trades occur instantly. Instead, queue requests and have the trades occur only once a second. Or 5 seconds.

Would this work? I'm assuming that either there's no motivation to solve the problem, or there are difficulties that I'm unaware of.

---

I've been wondering this about HFT for a while and this seemed like a good place to ask. Then I went back and finished reading the article, and this solution is proposed at the end. Curious about whether it would work on not.

joosters5• 12 years ago

You should read to the end of the article, where they discuss exactly this kind of solution.

tfinniga2• 12 years ago

Yup, just got there. Oops.

harmegido4• 12 years ago

Been trying to figure this out myself. What if we have 3 offers and 2 bids crossing at the same price after the 1 second tick is up. Which two offers get filled? If the answer is, the first two, then you've solved nothing. Maybe you randomize it? That could bring up some interesting issues.

harryh8• 12 years ago

Ya, if you randomize it than I have in incentive to submit more bids than I actually want filled because I've estimated what my fill rate is going to be. Of course everyone else has that incentive too which is going to negatively effect my fill rate which seems like it would lead to a pretty unstable system.

crdoconnor19• 12 years ago

I don't understand why HFT is still around, if it's considered a problem.

Primarily because regulatory capture has, until now, overruled all attempts to reign it in. The banks and hedge funds engaged in this activity wield vast amounts of political power.

Naively, couldn't all the craziness about HFT be resolved by enforcing a 1-second tick on the markets?

Not really. That just changes the game from "who can trade a picosecond before you?" to "who can trade a picosecond before the tick?".

An almost microscopic tax on transactions would prevent it, though while having a minimal impact on all other trading activity.

joosters5• 12 years ago

Not really. That just changes the game from "who can trade a picosecond before you?" to "who can trade a picosecond before the tick?"

The lack of live market book data means that you won't need the ultra fast connections, so at least one part of the networking arms race becomes de-weaponised.

You still have the problem of it being advantageous to bid at the end of the auction, but this could be solved by pricing. Just like the exchanges give incentives to offer prices rather than to take them, they could make bidding at the start of an auction cheaper than at the end of it.

yummyfajitas5• 12 years ago

This is silly. HFT is a $10-50B/year industry, depending on who's numbers you go with. (You can ballpark the numbers by multiplying trade volume by the spread.)

For comparison, Morgan Stanley alone made $8B in profit in the last quarter [1]. HFT is tiny, a drop in the bucket compared to the rest of the financial industry. Further, many big players in the industry are against HFT since HFT reduces their information advantage over smaller players [2].

The danger in a 1-second tick is that a) a massive change like this is hard to do [3] and b) it would likely increase volatility. A stylized fact of dynamical systems is that continuous feedback reduces volatility, whereas discrete step sizes increase it - witness how many numerical ODE solvers can blow up even while the underlying ODE is stable.

[1] https://www.google.com/finance?q=NYSE%3AMS&fstype=ii&ei=bVYI...

[2] http://www.chrisstucchio.com/blog/2014/fervent_defense_of_fr...

[3] Bloomberg, Excel and other calculators contain mathematical errors (e.g., "weekends are sometimes treated as a business day for the purpose of interest accrual") which cannot be fixed because the change would be too disruptive.

rayiner3• 12 years ago

Primarily because regulatory capture has, until now, overruled all attempts to reign it in. The banks and hedge funds engaged in this activity wield vast amounts of political power.

Your invocation of the "regulatory capture" trope doesn't even make sense in this context. The big banks and hedge funds aren't the major HFT players, and many of them are opposed to HFT.

guiomie3• 12 years ago

I believe the exchanges have no interest in "solving" HFT since they make a good amount of dinero around it. IEX has made a business out of 1 second delay (700ms actually).

kasey_junk24• 12 years ago

IEX has made a business out of being a dark pool explicitly designed for buy side hedge funds to be able to avoid market forces. The 700ms delay is the least of those features & is more of a marketing gimic.

minimax• 12 years ago

The IEX order entry delay is 350 microseconds (one way). Considerably less than even a millisecond.

yummyfajitas5• 12 years ago

HFTs nearly all provide liquidity rather than take it, which means that the exchange pays them to trade. You are simply wrong - the exchange loses money on most HFTs.

https://www.nyse.com/publicdocs/nyse/markets/nyse-arca/NYSE_...

http://nasdaqtrader.com/Trader.aspx?id=PriceListTrading2

Ask yourself: why would the exchanges pay market makers to provide liquidity?

bluecalm4• 12 years ago

>Naively, couldn't all the craziness about HFT be resolved by enforcing a 1-second tick on the markets?

There are several arguments against it:

1)liquidity is good(1), more liquidity is better; why artificially limit liquidity, just let people provide as much as needed at as low price as market dictates;

2)Options are good(2), limiting liquidity increases prize of certain options (as issuer needs liquidity to offer options and the less liquidity there is the more expensive issuing those becomes, the effect might be small between 1s and 0s but it's still there)

3)Certain types of arbitrage would be less efficient. Arbitrage is good(3).

(1)-the concept of liquidity is that when you have some goods you want to sell, you don't need to invest time/effort/money to sell it; that in turns frees your time to do productive stuff; the effect is obvious in situations like retail stores buying from manufacturer (so manufacturer don't need to do advertising, building stores etc. themselves so they have more resources to actually produce goods); the effect doesn't disappear magically at 1 second scale, more liquidity (faster, cheaper) is always better for overall productivity.

(2)-options allow certain ventures which normally would be too high variance (forgive the gambling term) to pursue; they are form of insurance; reducing risk is almost always good but it costs to provide; the lower the prize, the better;

(3)-arbitrage is good because the prices of goods/stocks become close to true value; we want that, we also want to pay for that as little as possible; the more effective the arbitrage the better, why stop at 1 second, when you can have cheaper/better ?

This is not the whole story but those points are clear benefits of HFT. Liquidity is cheaper today than it ever was. Spreads are the lowest as well. Some people who used to make bank charging ridiculous spreads are now not making bank - this again is great.

The arms race argument is that resources go into something counter-productive (like stockpiling nukes) instead into something productive. In case of HFT resources (at least some of the time) go into something desirable. It is a race but it is a race to improve desirable things (liquidity, arbitrage). You rarely hear argument about "arm race" in situations like that. It's usually called progress. Usually we applaud progress in situations where service provides needs to compete to offer the best/cheapest product. In case of HFT this point is often somehow overlooked.

dalore• 12 years ago

How does 1 second ticks stop liquidity and arbitrage? It just stops the faster connections wins.

HFT aren't really adding liquidity since they only buy when they can sell straight away. So there was already a liquid purchase there.

Arbitrage, if something is cheaper elsewhere, the 1 second tick will still bring the price into parity.

conistonwater9• 12 years ago

I think none of these arguments are very good against the idea of one-auction-a-second.

1) Liquidity is good, but remember who it is for. Uninformed traders, who want possession of the asset itself, will not care about a one-second delay because one second is too small an amount of time in human terms. Being able to trade a fraction of a second earlier is of no benefit. Being able to trade quickly on human timescales is the great benefit of algorithmic trading.

Informed traders, such as market makers, even market-making algorithms, also shouldn't care about a one-second delay. Their contribution to the market is the price. If you are a very good market maker, and you can quote prices better than anyone else in the market, your prices will be the better ones anyway, regardless of how the market is structured. And the profits are the market's rewards for quoting good prices, not trading quickly.

I would say the general idea is that liquidity providers and liquidity consumers compete, above all, on the price, and a one-second delay does nothing to hurt their incentives.

2) Again, I think it's quite obvious that on human timescales, on which actual economic decisions are made, there is no benefit to information arriving just a fraction earlier than it would otherwise. If you think of a market's purpose as revealing the price of an asset, it really shouldn't matter if the price is revealed 1ms earlier. What matters is that it is revealed at all, and revealed correctly with as many market participants as possible.

3) Because arbitrage becomes better when it contributes information to the market, not when it contributes information to the market 1ms earlier than another arbitrageur.

I think your points are completely valid arguments for algorithmic trading. Algorithms providing market participants with better prices than humans are indeed better for everyone. But for high-frequency trading, with the emphasis on high-frequency, I don't think this is very convincing.

cynicalkane• 12 years ago

HFT gives me a minor amount of hope for our regulatory environment: the forces and popular opinion of Big Finance all oppose it, but regulators continue to allow it.

The short answer is that HFT is like Wal-Mart: it's trendy and cool to complain about it, but in terms of providing better and faster inventory to ordinary customers, it works better than everything that came before.

Except Wal-Mart is hated for being (allegedly) bad for poor workers, electronic trading's victims have primarily been the Wall Street old boys' club. Why the public is convinced this is a bad thing is beyond me.

harryh8• 12 years ago

This is what really sucks about that Michael Lewis book. He could have told a great story to the public about how everything they had been told was wrong and that the HFT nerds (just like the baseball nerds) had used technology to make the world a better place. He has the talent to do this so well.

But he fucked up and bought the story the buy side scammers were selling him hook, line, and sinker.

It's such a bummer.

guiomie3• 12 years ago

Can someone help me understand the following statement (specifically the don't branch part) ?

"Don’t touch the kernel. Don’t touch main memory … Don’t branch.’ (That third commandment means don’t fill your program with ‘if’ statements – ones with the generic form ‘if A then do B else do C’ – because they get in the way of a modern microprocessor’s capacity to do several things in parallel.)"

kasey_junk24• 12 years ago

It is probably referring to the fact that code that branches allot will make the prefetcher/branch predictor fail more frequently which causes slower memory loads and less ops. Not sure why it mentions parallelism though.

bluecalm4• 12 years ago

Obligatory link: http://stackoverflow.com/questions/11227809/why-is-processin... see most upvoted answer for short lecture about cost of branching in modern CPU's.

NickStefan• 12 years ago

HFT is a two sided coin. For every person who complains that it's screwed them out, they're forgetting all the times they've saved money on narrower spreads due to extra liquidity. Possible?

Donald MacKenzie on high-frequency trading

Related Stories