Online Algorithms in High-Frequency Trading

http://queue.acm.org/detail.cfm?id=2534976

by stevewepay • 11 years ago

116 90 11 years ago

queue.acm.org

expertonkappa• 11 years ago

Is there a mirror? Seems like too many connections at the moment.

galaktor• 11 years ago

google cache should do the trick if it's down

http://webcache.googleusercontent.com/search?q=cache:WqysSpS...

amelius• 11 years ago

Could somebody explain what the use is of HFT to society? Because I don't get it.

Links are also welcome.

res0nat0r• 11 years ago

Shortening the bid/ask spread and increasing liquidity among other things.

http://en.wikipedia.org/wiki/High-frequency_trading

reverend_gonzo2• 11 years ago

Before HFT, there was only a few market makers, which meant the bid-ask was high (if you wanted to buy or sell right now, you'd pay more), liquidity was low (it'd be harder for you to get in or out of a large position and in hard times it would take longer for the market to recover - See 80s crash vs flash crash).

With HFT, there's a lot more competition over the bid ask spread. Trading firms are essentially in an increasingly expensive fight over pennies, which is good for everybody else, since it's cheaper prices, more liquidity, and a more efficient market.

The people it's bad for is the old-school style brokers (which are effectively gone now) and large investmant banks who have a fundamental view on the market (they need to be smarter about executing since the market is more efficient and reacts much more quickly to their buying and selling).

djrobstep• 11 years ago

Which brings us to the next question: What is the social benefit of a slightly smaller bid-ask spread and a few milliseconds of faster liquidity? And are these benefits worth the massive amount of wealth extraction from the rest of us that HFTers are engaged in?

bradleyjg• 11 years ago

The people it's bad for is the old-school style brokers (which are effectively gone now) and large investmant banks who have a fundamental view on the market (they need to be smarter about executing since the market is more efficient and reacts much more quickly to their buying and selling).

What about index funds rebalancing their portfolios? (Honest question)

cheepin• 11 years ago

Unfortunately, there are a couple types of HFT. At it's core, it just means trading using a computer to execute more/faster trades. That seems justifiable in making the markets more efficient. Where most people including myself have issues is when they get a pass in certain markets to frontrun normal sellers and eat some of their margins simply because their latency was low enough. An earlier HN post that I can't find had a discussion on how this type of behavior can be countered by sending fake buy offers without intent to actually buy, but for whatever reason, that is illegal.

kasey_junk6• 11 years ago

Where most people including myself have issues is when they get a pass in certain markets to frontrun normal sellers and eat some of their margins

Luckily that is not allowed (nor technically possible) on any credible exchange.

An earlier HN post that I can't find had a discussion on how this type of behavior can be countered by sending fake buy offers without intent to actually buy, but for whatever reason, that is illegal.

You probably meant: https://news.ycombinator.com/item?id=8977198

Spoofing, which is what you are describing, is not useful for countering HFT, it is used to change the market price of an instrument by exaggerating the demand for it. This has obvious detrimental impact on any one, HFT or otherwise, who is trying to price the instrument.

Paradigma112• 11 years ago

Is it necessary for things to "have a use to society" to be allowed?

Where is the commitee that decides on the allowed actions?

calibraxis• 11 years ago

The committees are called "governments" and rule life in nation-states. The dominant ones have a capitalist morality and therefore make laws to create and enforce it.

gadders• 11 years ago

This guy's blog is pretty good:

https://scottlocklin.wordpress.com/2011/03/01/the-arguments-...

https://scottlocklin.wordpress.com/2014/04/04/michael-lewis-...

lmm2• 11 years ago

We see crazy competition on latency because these firms can no longer compete on price. HFT has taken the spreads down from $.25 to $.01 (putting a lot of market-makers out of business, saving a lot of money for anyone who is actually trying to buy or sell shares). But since the price isn't allowed to go below $.01 (sub-penny rule) the firms are now all in this wasteful zero-sum race to get that $.01.

wodzu• 11 years ago

While you will hear many negatives here I believe there are few positive outcomes.

The needs of HFT are such that technology barriers are pushed to the limits. Some of the technologies developed in the process make it back to the general public.

Adlai4• 11 years ago

https://www.chrisstucchio.com/blog/2012/hft_apology.html https://www.chrisstucchio.com/blog/2012/hft_apology2.html https://www.chrisstucchio.com/blog/2012/hft_whats_broken.htm...

The second link specifically talks about social utility.

innguest• 11 years ago

There is no benefit. HFT does not lower spreads in reality, only nominally. They put in orders but back away when a matching order comes in. The myth is that HFT improves liquidity. It may look like that. But those orders aren't real and can't be fulfilled.

This is again how scientificism goes. We think machines improve everything. "If only we trade faster, that will help the markets!" It's not hard to see this is BS.

throwaway2837192• 11 years ago

The trouble with this argument is that, on basically every exchange in the world, you do not get to find out if someone has matched with your passive order until after the trade has happened.

There is no possibility to "back away when a matching order comes in" because the first time you find out about the matching order is when you are informed about the trade that you just did!

What can happen is that a market maker will post liquidity on multiple exchanges A, B and C, and will remove their orders from exchanges B and C if they are filled on exchange A (assuming they are fast enough to do it!)

This is not shady or nefarious. In fact, it is the only way that a distributed market could possibly work.

The market maker has some finite capacity to provide liquidity (e.g. they have position limits, they need to post margin, they have certain risk tolerances). They want to maximise the chances of their orders being traded on (i.e. providing liquidity), so they post liquidity on all available markets.

When they match on one of those markets, their capacity to provide liquidity has diminished, until they manage to offset their position somewhere (either on the same market, or on a different one). Moreover, they have just received a new piece of information about the supply/demand imbalance for this particular stock. For both of these reasons, it is natural to assume that they would want to adjust the prices they are offering on other exchanges. The market cannot function any other way.

aet2• 11 years ago

You have the wrong question. The question is: What is the optimal market structure? Trying to strike a balance between the benefits of competition and the troubles of market fragmentation. Fragmented markets = HFT + competition. Unified markets = monopoly power + liquidity. You choose...

tptacek17• 11 years ago

Wait, why would monopolies on trading venues and market-making provide more liquidity? Shouldn't liquidity be on the other side of the comparison?

vasilipupkin5• 11 years ago

I find this question soooo frustrating. I mean, we live in a free ( relatively speaking ) capitalist society. Why does there have to be use to society for anything, unless the issue in question has negative externalities on others ?

pgeorgi2• 11 years ago

HFT may have negative externalities on others. The money they make has to come from somewhere. For example: Buyers pay more or seller get less than they'd get without HFTs acting on the market.

The (quite common) liquidity argument isn't a very good one IMHO, given that HFT have no interest in holding positions for any long time, so they won't buy what they're not reasonably sure they can sell quickly (ie. where there already is liquidity).

The other argument here is that before HFT, traders wanted even larger fees. HFT can't be the cheapest possible option out there, if they can afford to lay their own fiber between continents to operate.

So what is it that makes HFT desirable enough that we (as a society) shouldn't find means to make trading happen without them?

abeppu• 11 years ago

The first section of this which relates HFT to one-pass algorithms is kind of useless. Online estimation algorithms are much more broadly applicable than that, and I'd expect that their relevance to HFT would be immediately obvious. I've never worked in HFT, but I've probably written the online variance algorithm at 3 of 4 companies I've worked. If you care about metrics such as "how quickly are each of my customers sending me requests?", then this stuff is relevant to you.

But more to the point, is there anyone in these fields who doesn't know this stuff? At a basic level, the things that you can do a really good job estimating in an online way are closely related to sufficient statistics, which every undergrad stats course should cover. And hopefully anyone actually involved in writing HFT systems has way more stats background than that -- right?

kasey_junk6• 11 years ago

a) there is a pernicious and wide spread myth about the elite status of people working in HFT. My experience is that it is largely the same as any other technical profession. That is to say, it is no more or less likely for people in HFT to know about online estimation than any where else.

b) Given the reactionary view that many HFT participants have to information sharing, widely known/documented algorithms are often thought of as secret sauce and are frequently not talked about.

c) I agree with your sentiment specifically to one-pass algorithms being not terribly novel but am very happy that people with HFT backgrounds are publishing anything at all as it will hopefully help to remove the backwards thinking around secrecy so prevalent in the industry.

vasilipupkin5• 11 years ago

regarding a), not all technical professions require deep knowledge of massive data processing and time series modeling, so this statement is misleading

regarding b), the details of how exactly to use the algorithms are the secret sauce and those details matter.

tptacek17• 11 years ago

I'm a systems programmer with a pretty primitive intuitive sense for maths. I "agree" with you, in the sense that the algorithms this article mentioned were intuitively obvious to me and thus seemed like they couldn't be sophisticated. What's the state of the art for online estimators?

I've been playing with simulations of market maker algorithms from the literature, and simple Bayesian stuff and MLEs are all I've really seen, which makes me think that the interesting stuff isn't in the literature.

cdavid• 11 years ago

There is a lot of literature, but it may not be easy to find out, and different domains use different names for it.

One keyword to look for is 'stochastic approximation', with a long (> 50 years) history in engineering and stats. Another one is 'stochastic gradient', which got a resurgence following the whole deep learning craze. There is also 'particle filtering' in signal processing.

One generic article I may recommend is http://leon.bottou.org/publications/pdf/online-1998.pdf, it does not require much math knowledge, but that may be less practical than what you want.

The 'onepass' keyword mentioned in the original article is another keyword to look for (see e.g. interesting work to do Monte-Carlo based linear algebra operations for very large matrices, this used to be a reference, not sure what's the state of the art is: http://cs-www.cs.yale.edu/homes/mmahoney/pubs/matrix1_SICOMP...)

minimax• 11 years ago

There is another article in this same edition of the ACM Queue.

http://queue.acm.org/detail.cfm?id=2536492

It goes a bit further into the mechanics of market maker profitability. Does your simulator reflect what your queue position would be in a real electronic market? Does your market making algorithm have logic try to scratch out when it looks like the market is going to move against its position?

wglb• 11 years ago

The particular emphasis on one-pass algorithms might seem self-evident, but the thread running through all of this is an engineering challenge that is very relevant to HFT. If your algorithm takes 60ms to evaluate your current position on each tick, by the time you are finished, the market has moved to the next zip code. Thus, to keep up with the market, you might find yourself rewriting the network stack for better response time.

CyberDildonics7• 11 years ago

Anyone who thinks HFT is something needed should go read Flash Boys. HFT is basically front running with some variations. There is a small amount that is necessary arbitration that will have to be done at some speed, but by and large it is not performing any sort of a service, it is taking advantage of the people trying to buy and trade stock by being a manipulative middle man.

tptacek17• 11 years ago

There are three sentences in this comment. The first two are wrong, and the third is hard to understand, but if I understand it correctly, it too is false.

1. _Flash Boys_ has been panned by technical experts and makes very little sense to people with a basic understanding of how the markets work. I can link to technical reviews if you'd like to see them. I'd be interested in any technical reviews of the book that defend it; I haven't seen any. As a fan of Michael Lewis: this book was surprisingly bad.

2. Front-running is an agency problem. It's what happens when your broker knows you're about to move a lot of stock and, acting on that insider knowledge, jumps in front of your trade. Market-making isn't front running. To make a colorable argument that HFTs are frunt-runners, describe a hypothetical sequence of trades in which that happens ("BOB BUY 10 @ $10, CAROL SELL 10 @ $10.01, &c"). If HN history is any guide, that exercise will quickly deflate the notion that there's front-running (or even any unfairness) happening.

3. HFTs were preceded in the markets by the human specialist system, in which people deliberately increased the spreads to scalp money from trades. HFTs fight over pennies. Human specialists rigged the stock markets so that every share included a premium denominated in dimes and quarters paid directly to the specialists.

jaybaxter• 11 years ago

Could you please link to the technical reviews you mentioned? I would be very interested to read them because as an outsider, Flash Boys seemed overly simplistic, but I haven't been able to figure out what the real story is yet.

kylebrown3• 11 years ago

The 2nd-to-last slide[1] compares the IEX "speed bump" to other anti-HFT measures: "Ingenious, eliminates sniping. But, only works while IEX is small relative to the rest of the continuous market (free-rides off price discovery elsewhere)."

1. http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre...

CyberDildonics7• 11 years ago

So if all this is so great, how come so many people are going to great lengths to eliminate HFT with the exception of the people that make money from it? If it was offering such a valuable service, wouldn't people want it to be there even if they weren't the ones profiting from it? Flash Boys has been panned by HFT people, not people trying to trade stock fairly.

CyberDildonics7• 11 years ago

Nothing in this comment contradicts or disproves that front running can and is happening. Flash Boys described a large order going to an exchange and HFT firms seeing that, buying stocks on other exchanges before the real purchaser, and selling them back to the real purchaser after the first exchange they get to can't fill their large order. HFT firms know there will be buy orders at the other exchanges and they run out in front of those buy orders to become a middle man. That is the very definition of front running and it was obviously happening. Do you have information that directly contradicts this?

dllthomas5• 11 years ago

Mistakes aside, the weirdest thing about Flash Boys, to my mind, is that it's cast as a David vs. Goliath narrative - with the role of "Goliath" filled by the awful HFT shops and "David" by Goldman Sachs and J.P. Morgan. When "David" is orders of magnitude larger than "Goliath" and tends to swing the industry by the tail, it's an odd framing.

tptacek17• 11 years ago

The sad thing about any thread including the term "High-Frequency Trading" is that we won't be able to talk about the technology, and instead will need to relitigate automated trading.

I think we'd have been better off if the title of this thread had been "Online Algorithms in Automated Trading Systems". As far as I understand, that title has exactly the same meaning, but doesn't set off a crowd of HN commenters who have never placed a limit order jumping in to crow about _Flash Boys_.

I'm a little frustrated, since this is a technical topic that happens to be very relevant to me right now (I'm not a trader, FWIW).

CyberDildonics7• 11 years ago

So what is your opinion on the book Flash Boys?

throwaway2837192• 11 years ago

tptacek already responded to your previous comment about Flash Boys below [0]

[0] https://news.ycombinator.com/item?id=9045150

kylebrown3• 11 years ago

The High-Frequency Trading Arms Race: Frequent Batch Auctions as a Market Design Response[1][2] is a must-read. The tldr; is that a continuous mechanism (sequential trade processing) only looks efficient in time-space, but is actually inefficient in volume-space (most of the volume trades at stale prices). The solution is discrete batch auctions (batch trade processing).

1. slides: http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre... 2. paper: http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre...

tptacek17• 11 years ago

This is really interesting, but doesn't it presume that there's only one trading venue for each instrument, and that instruments aren't correlated?

You can make time discrete for a single auction, but the time between auctions still has to be continuous unless everything's traded in a single exchange, which can't in reality happen even if we want it to (and we probably don't).

As I understand it, most of the most dramatic HFT stuff is already targeting multi-venue trades or correlations.

NhanH• 11 years ago

I'd share the same sentiment. I was really disappointed yesterday when the only comments (after 5 hours of nothing) was a seemingly snark question on the evil of HFT.

And to everyone who feel the need to voice their disapproval of HFT: if this was an article about improving the penetration of M1 Abrams, are you gonna be voicing your disapproval about ... something? Why/ why not?

Adlai4• 11 years ago

Should we really be putting engineers to work designing more efficient techniques for punching plasma holes through sheet metal? They should instead work on improving sheet metal technology, as it is useful both for protecting both against natural disasters, and against those pesky hole-punchers.

bladecatcher• 11 years ago

It's interesting how you consider placing a limit order as a criterion for being eligible to comment on this topic.

crazypyro• 11 years ago

Needs a date stamp (2013).

Online Algorithms in High-Frequency Trading

Related Stories