A High Frequency Trader's Apologia, Part 1

http://www.chrisstucchio.com/blog/2012/hft_apology.html

by yummyfajitas • 14 years ago

375 315 14 years ago

sid6376• 14 years ago

What I would love to hear in the follow up to this post is what are the typical technical challenges which a typical HFT encounters? If someone wants to be an HFT what are the core skills one must have? (Strong in algorithms, good c++ skills?). How much math should one know and what kind of math?

netvarun• 14 years ago

There have been a couple of HN discussion threads regarding your last two questions.

1. http://news.ycombinator.com/item?id=3177815

2. http://news.ycombinator.com/item?id=966716

radikalus2• 14 years ago

I think to be successful your team needs to be strong in four key areas:

- Trading (Strategy) - Software - IT - Research

There isn't necessarily one of the four that trumps the rest, and different players have different relative strengths across those four disciplines. (Which are as you would imagine, not terribly specific to HFT)

I imagine the question is developer-focused, so on the software side, I think I value most highly:

- Don't fuck up. This is obvious, and the trading adage of "Don't lose money" is overdone...but essential. Assume you will fuck up and have nets in place to catch you. Doing safeties well isn't sexy and doesn't add any edge, but I've witnessed "unlikely" safeties save trading systems.

- Understand the hardware. Undestand the strategy. You cannot write fast "generic" code. Have a sense for how fast you could optimally be on the hardware and network stack you're using. This requires interfacing with IT and, sometimes, trading, as some "strategies" traders come up with aren't easy to do FAST; you might be able to suggest something similar that is potentially much quicker on the critical path.

- If you really want to compete in low-latency, C++. If you just want to be pretty fast and your strategies aren't competing for specific orders with the other market makers, I don't think it matters. You can probably do a lot of strategies in 100us in any language. GUI can be done in anything, but it should be able to scale. Most of the GUI pieces are pre-existing. Sortable tables for fills. Ladders for displaying markets, etc. I've used Java/C#/C++ GUIs -- C++ has always worked best for me, but I wouldn't be opposed to doing the front-end in JS/HTML honestly.

- Understand the critical path. This and, to some degree, only this, has to be FAST. Obsessing about how to optimally build a theoretical value cache more quickly etc etc is often not that big of a deal. (It's not on the path)

- Linux kernel and TCP/UDP stack DETAILED understanding. (Or verilog I guess...)

- Locks. When does a mutex really need to be locked? Obviously depends on system architecture. And can often dictate architecture.

I think almost anything else I'd say would be either vague or generic. I think there's a lot of niches one can carve out in HFT and, while I personally value generalists who can holistically understand the goal (make $), there's a ton of people doing really well by just being the most badass at X/Y/Z.

jrockway2• 14 years ago

C++? Everyone is now building their own hardware instead.

anonDataUser8• 14 years ago

This offers no insight into HFT and is not an apology. Why is it here?

For people interested in HFT, start your exploring with a google search for "latency arbitrage".

joshu2• 14 years ago

"an intro to limit order books"

Without looking, as it has been years since I did this professionally: is latency arbitrage looking at price moves out of sync with different ECNs and liquidity providers?

anonDataUser8• 14 years ago

Sorry for the lack of guidance in my original response.

Latency arb was born out of the fragmentation that was introduced when the SEC passed Reg ATS.

For those unfamiliar with trading, there are many exchanges, not just one. All of those exchanges are obligated to trade at the same prices. To ensure this, Reg ATS establishes the NBBO: National Best Bid/Offer. If you're buying, and one exchange has an offer at $10.00 and another has an offer at $9.00, $9.00 is the best offer. If you send an order to the exchange that has a $10.00 offer, they are legally obligated to send that order to the exchange with the best price.

Latency arb is the process of keeping quotes/prices in sync by trading when a specific exchange's prices don't match NBBO. Many of the largest HFT players work on a very simple principle: they subscribe to all of the exchange feeds and construct their own NBBO that is faster than the best commercially available NBBO feed. With this, you know what direction stocks are moving as it happens.

Keep in mind that a "price movement" rarely occurs at a specific instance in time. It occurs over the horizon that it takes for the market to synchronize -- which happens as fast as possible and is based on the infrastructure investments of high frequency traders. If you can do it faster and you build the infrastructure the world is yours. Like Vanderbilt with railroads.

firebones• 14 years ago

It's an apology in the second sense of the word apology:

"2. a defense, excuse, or justification in speech or writing, as for a cause or doctrine."

Basically, apology as "explanation and defense of an idea".

rauljara• 14 years ago

It isn't an apology in that sense of the word either. There is no defense here. Just a not so technical explanation.

Though, the article promises a defense in a later post, so perhaps it will turn into an apology then.

cynicalkane• 14 years ago

The word "apology" almost always means saying sorry, even to people who know the other meaning.

The author might consider the word "apologia", which does not obviously mean saying sorry and has the additional benefit of sounding Greek, stirring up memories of Socrates' famous defense at his trial. But again, there isn't even a defense in the author's post, just the promise of one forthcoming.

RobertKohr• 14 years ago

So if you remove the restriction on subpenny increments, you remove the need for HFT?

RickHull• 14 years ago

HFT would still serve a role with subpenny increments. Nonetheless, the subpenny rule is a very important mechanic in the construction of HFT.

AndrewHampton• 14 years ago

I agree. I think there is currently a balance to be struck between accuracy of the price determined by the algorithm and the speed at which it obtains that price. I think removing the subpenny rule would just reduce the importance of speed and increase the importance of accuracy. I'd imagine trades would still happen extremely fast though.

deathflute5• 14 years ago

Thanks, this is a very clear article on the most important ingredients of HFT. I am curious to hear the apology but I am not sure there is anything to apologize for. The market marker is taking on substantial risk of adverse selection and is being compensated for it. As long as legal means are used to achieve that end, this is not really different from any other business strategy. Causal implied benefits to society from any business are subjective and the most important consequences are usually latent.

Anyways, I am curious to know if elaborate prediction models are used here or it is mostly a game of adjusting prices based on correlated instruments? Also, how does one build a simulator for such strategies?

wglb5• 14 years ago

Apology in the sense explained in comment in another thread: http://news.ycombinator.com/item?id=3852595

ChristianMarks4• 14 years ago

I'll believe that HFT adds liquidity to the market when the typical start of employment to retirement period is 15 milliseconds.

yummyfajitas22• 14 years ago

I'll believe that Amazon's network engineers (and whoever else is involved in making their page fast) add value when it takes 200ms to ship products to my home.

lrm2429• 14 years ago

An expected and uneducated statement. It's simple really: any participant who rests an order on the book is adding liquidity. Period. Many HFT strategies are largely passive (market making, for example) and therefore entirely rely on resting orders (as Chris' article points out). Therefore, they add liquidity.

Now, if you want to get into a discussion as to the quality of that liquidity then you have to go educate yourself a bit more. Once you do you'll find that the liquidity being added is not toxic nor fleeting. However, what you and most others don't understand is that it is also significantly more informed passive liquidity than it used to be.

This may lead folks to believe that the liquidity is fleeting because they can cancel away before you can execute. However, that simply means they are better and faster than you, not that they are doing anything wrong.

Why, then, don't we simply require them to rest their orders for a longer period of time? Simple: you want tight spreads. As spreads shrink market making becomes less profitable. When market making is less profitable market makers have to be more risk averse to avoid getting adversely selected and losing money. More intelligent models (they become more informed) and less latency (they can leverage that informedness to avoid being selected) is how market makers stay profitable in a market where the average spread is 1 penny.

_delirium3• 14 years ago

How about rather than adding regulations, just discretize the market's clock? Would a market that performed a trade-resolution tick once every 100ms lose anything important? I believe arbitrage improves efficiency, but I'm less clear on the benefits of extremely high temporal resolution. Currently that's what the incentives encourage optimizing for, among other things. If you predict a market movement 10ms ahead of time, you can profit from that brief 10ms temporal-arbitrage window. And like any arbitrage, that does indeed improve the pricing signal in an absolute sense, in this case by taking a price change that was 10ms later than it "should've been", and moving it up in time via your trading. But it's not clear that 10ms-level pricing inefficiencies are actually something particularly important to smooth out via arbitrage, when you can just define them out of existence by going to a discrete-time market (which are fairly well-studied in the mathematical literature).

mattmanser2• 14 years ago

There's no counter argument here nor does this come across as informed or educated. All you've done is talk about the process with as many technical words as you could without actually saying why 15ms liquidity gives us anything.

Apart from mini-crashes. And massive profits for companies that create nothing and actually were the cause of the recent financial meltdown.

Which is why many of us sit here scratching our heads or make sarcastic comments like the op.

davidw4• 14 years ago

> However, that simply means they are better and faster than you, not that they are doing anything wrong.

So it sounds like "the only winning move is not to play" if you are a small fish, because you're going to get screwed by people who invest a lot in this stuff?

At least that's what I'm reading in that sort of message. I'm quite willing to be proven wrong, given I don't know much about the subject.

paulsutter8• 14 years ago

I'd like to get a better understanding of this liquidity.

Assume the subset of market participants whose holding period is always > 1 day ("slow"). Assume another subset of market participants whose holding period is always < 1 minute ("fast"). Ignore everyone else in the market.

Do these "slow" participants see any practical liquidity benefit from the "fast" participants? I don't see how.

Financial guys envision these "slow" traders as grandmothers, but remember that most entrepreneurs have holding periods of > 5 years.

dpark12• 14 years ago

> It's simple really: any participant who rests an order on the book is adding liquidity. Period.

If a stock is trading at $100.00, and I put a buy order on the book at $0.01, how have I added liquidity?

> This may lead folks to believe that the liquidity is fleeting because they can cancel away before you can execute. However, that simply means they are better and faster than you, not that they are doing anything wrong.

The question is not whether it's right or wrong, but whether it is beneficial. I don't buy that HFT is intrinsically evil, but there seems to be little convincing evidence that it's actually beneficial, either. The system as you've described it basically involves scraping pennies off transactions that would have occurred anyway. That's not intrinsically evil, but it doesn't seem to actually benefit anyone except the HF trader.

> Why, then, don't we simply require them to rest their orders for a longer period of time? Simple: you want tight spreads. As spreads shrink market making becomes less profitable. When market making is less profitable market makers have to be more risk averse to avoid getting adversely selected and losing money. More intelligent models (they become more informed) and less latency (they can leverage that informedness to avoid being selected) is how market makers stay profitable in a market where the average spread is 1 penny.

What's missing here is an explanation of why we should build a system that enables HFT to be profitable. That's not something I care about. If lower profits for HFT means a better market overall, that's fine with me.

Frankly, in markets where the spread is 1 penny, it seems like there's enough liquidity already.

chernevik2• 14 years ago

If you cancel faster than the market, you can stand first in line for everything and leave the line for those occasions where the trade isn't profitable. I don't see the difference between jumping to the head of the line for Good and standing in every line but jumping out for Bad. And I don't see how line-jumping adds liquidity. A bid I can never hit because it will be gone any time I'd actually want to hit it isn't much of a bid, at least as far as I'm concerned. If it's gone before anyone can hit it, then I think there's a definitional question of whether that's really a 'bid' for the intents and purposes of liquidity provision.

Now I'm not saying such bids are Bad. They could be helpful in all sorts of ways. But I'm not sure that provision of liquidity is one of those.

lrm2429• 14 years ago

A few notes:

(a) Orders for stocks priced below $1.00 are permitted to display in sub-penny increments. Not that this is really relevant to HFT, but it is often a point of confusion.

(b) Orders may rest on the book and display at a price but execute at a separate price. For example, if a market is 20.01 / 20.02 and I believe true value is 20.017 I can bid 20.01 but tell the matching engine that I'm willing to accept 20.015. That you are willing to trade at the midpoint also places you at a higher priority in the queue, above those only willing to buy at 20.01.

yummyfajitas22• 14 years ago

Regarding b), I believe it's the MPL order you are referring to.

http://usequities.nyx.com/markets/nyse-arca-equities/order-t...

Some dark pools also allow trades at subpenny increments, although of course (being a darkpool) the orders are not displayed.

lrm2429• 14 years ago

Midpoint orders exist on all the venues. All but EDGA/EDGX provide some rebate for resting a midpoint order. DirectEdge charges both sides for what they refer to as "midpoint match".

georgek• 14 years ago

Transactions that occur in dark pools (over half the liquidity on NYSE listed stocks these days) typically get posted to the exchange after the fact. This used to cause anonymous hedge fund I used to work for all kinds of problems in the leas liquid markets as half the daily volume would suddenly post dive minutes

deathflute5• 14 years ago

To add to (b) Midpoint orders have to be hidden though because lit markets cannot allow displayable sub-penny orders.

lrm2429• 14 years ago

That's correct. They display at a whole penny price but execute at the midpoint. They may also execute at an offset for the market or primary.

jmilloy2• 14 years ago

Note (b) is surprising and I'm not sure I understand. Can you add more details to your example?

If someone sells at 20.01, and I'm higher in the queue because I told the matching engine that I'm willing to accept 20.015, why would I ever not bid 20.0199...? Can someone sell at 20.02 but tell the matching engine they're willing to accept 20.015? Do we match? What does it execute at?

lrm2429• 14 years ago

They are only accepted at the midpoint.

EDIT:

It's just a feature of the matching engine. It's not really that complicated. The venue, let's say BZX, accepts an order from you to display a BID for 100 at 20.01 and you flag the order and tell the matching engine you'll accept 20.015.

When someone comes along and wants to sell, let's say Grandma Sue, she might say: Sell 100 at MARKET. The displayed market is 20.01 / .02. With your bid in the market Grandma will get matched at 20.015, a .005 price improvement over the displayed bid of 20.01. This is a good thing.

Strilanc• 14 years ago

It sounds like they display at a lower accuracy than what you may actually specify, and there's a bit of jiggering to allow the displayed request and true request to both be 'accurate'.

javert16• 14 years ago

So basically you don't know the actual midpoint where the market is trading.

Doesn't that mean people can get an advantage by guessing where that point is and bidding just below it? (Not too far below, because then you're wasting money.) And if so, do they use strategies to try to determine where the midpoint is? (e.g. offer a very small number of shares at many different points, see which ones execute)

cheddarmint2• 14 years ago

What the linked article doesn't discuss is that it is possible to get a real-time notification when a trade occurs. The price of the last trade is the mid-point that you are referring to. Algo trading/HFT shops are listening for trades as well as quotes - this is how they know the price that trades are actually happening at.

lrm2429• 14 years ago

There is no need to guess the midpoint. The midpoint is simply: (best_bid + best_ask) / 2.

grandalf5• 14 years ago

the midpoint doesn't matter at all. There is no "price" only a price available for a certain volume.

jboggan• 14 years ago

Will the following installations address the role of high frequency trading in flash crashes? Although high frequency trading seems like a very good strategy when arrayed against human opponents I have to wonder what happens when all the inputs and outputs are controlled by HFT algorithms trading against each other. How do you debug a HFT algorithm?

vasco• 14 years ago

I am also interested in the debugging part.

wglb5• 14 years ago

The rapidity of recovery from the flash crash can be credited to HFT.

toomuchtodo• 14 years ago

Lets all rejoice in how fast those who created the disaster fixed it. rolls eyes

digler99• 14 years ago

as I understand it, the flash crash of may 2010 was by a bug in the algo that said "in order to allow the market to absorb a large order, simply divide it into X chunks and put them on the market one at a time". The bug was that this algorithm worked if your total shares T were small enough that T/X didn't affect the market. But on that day, T was so big that T/X still moved the market, and the algo didn't take that into consideration. You could protect against this by having sanity post-checks, but once something hits the market there's basically no CTRL-Z button.

One interesting thing about the flash crash was that the original "stock" wasn't a stock at all, but S&P e-minis, which are tied to a whole cross-section of stocks but are on an exchange that is mostly traded by pros. We have circuit-breakers now on individual stocks that halt trading if a single stock moves too much too fast, but that won't be effective if the whole index gets pushed down like it did that day.

veyron2• 14 years ago

What caused the flash crash is a poor implementation of stop and market orders.

Essentially, the exchanges do not natively have a concept of "market orders" -- filling now. Instead, some brokers implement them by selling at a penny or buying at some enormous price, essentially hitting the bid or offer. Under normal circumstances this is not a problem, as the prices are reasonable. However, during periods of extreme dislocation, when traders pull out their limit orders, all that are left are stub quotes at a penny. And because those brokers used algos that would trade at a penny, those trades happened.

You debug an algorithm like you debug any other piece of software :) I "debug" by using a regression against a pure python implementation, which was implemented in the most obvious and correct way. I discussed it here: http://veyronb.wordpress.com/2011/11/02/prototyping-with-pyt...

yeureka• 14 years ago

Testing/Debugging can be achieved by running your own mock exchange with historical or generated data. Also, most exchanges provide a simulated environment where you can do fake trades and make/loose fake money.

dbecker• 14 years ago

I've seen so many news articles full of hyperbole about HFT, and this is the first legitimate explanation I've seen. Thank you.

joejohnson3• 14 years ago

Last week, Bloomberg published this op-ed defending HFT: http://www.bloomberg.com/news/2012-04-10/high-speed-trading-...

The piece is largely propaganda. A nice rebuttal is found here: http://www.ritholtz.com/blog/2012/04/hft-pirates-and-their-a...

Can anyone explain why multiple exchanges exist (besides for historical or political reasons)? It seems anti-competitive that exchanges can be private at all, let alone that exchanges can be allowed to operate as monopolies in a market or country.

HeXetic• 14 years ago

That's not a "nice rebuttal", that's some bullet points spat back. Whereas the original linked article in this discussion provides some context and explanation, the bullet points in your second link are meaningless to the layperson (e.g, me).

joejohnson3• 14 years ago

...the original linked article in this discussion...

Do you mean the Chris Stucchio piece? Because, despite it being an apology, it doesn't justify how HFT is socially beneficial. This piece makes hand-wavy claims about liquidity, and says that it will better justify HFT in a coming blog post.

As for the Bloomberg piece defending HFT, it fails to address a few issues that the SEC and critics have with HFT. These issues are enumerated in the Ritholtz blog post. Pointing out certain facts that an argument ignores is a valid rebuttal.

yummyfajitas22• 14 years ago

Can anyone explain why multiple exchanges exist...allowed to operate as monopolies in a market or country.

Which are you complaining about? The fact that we have multiple privately operated exchanges, or the fact that we don't have a single government-run monopoly on exchanges?

paulsutter8• 14 years ago

I don't think he was complaining. I think he asked a really interesting question. What's the benefit of multiple exchanges?

It isn't obvious to me that multiple exchanges are either good or bad. If it is obvious to you, I'd love to hear a technical explanation.

cdk• 14 years ago

I believe multiple exchanges do foster competition and innovation. I believe NASDAQ was one of the first computerized exchanges in an era when human traded in the pits.

Multiple exchanges allow customers to choose from better trading fees, execution times, colocation offering, etc.

anigbrowl• 14 years ago

Multiple exchanges allow customers to choose from better trading fees, execution times, colocation offering, etc.

I'm not sure how big of a plus this is. Most people wan to expend effort on picking the right stocks to buy or sell, and while they want low transaction costs the price of too many alternatives is increased complexity.

javert16• 14 years ago

It seems anti-competitive that exchanges can be private at all

"Anti-competitive" is a false (i.e. nonsensical) concept. I mean, if I am a really good golf player, and I just keep getting more and more competitive, do I suddenly cross a bright red line that makes me "anti-competitive" because I'm just too competitive?

It's just a made-up term used to attack business that lots of people have unknowingly bought into.

If it were to actually mean something, it would mean using the government to grant a monopoly. That's the opposite of any kind of private business where two or more parties are voluntarily agreeing to trade. Like, for example, a private exchange.

Cushman3• 14 years ago

Anti-competitive behavior would be literally cheating at golf.

In both cases, it has been observed over time that allowing individuals to do whatever they want has a negative effect on the game. You are not required to believe that the rules are necessary, in either business or golf, for them to exist.

rayiner2• 14 years ago

"Anti-competitive" is a well-defined term in the law that reflects behaviors that undermine the "perfect competition" assumption of certain economic theories regarding efficient markets.

It's definitely possible for private agreements to be anti-competitive in the sense of undermining efficient markets. Collusion to fix prices is the pedagogical one.

sailfrog3• 14 years ago

The rebuttal at ritholtz.com is largely propaganda as well. It was penned by the brokerage firm Themis Trading who has been on a crusade against private exchanges, HFT, reg NMS, and pretty much any change in market structure that could negatively impact their business model. Not only that but the post simply lists properties of the equities markets and implies they are bad. For example:

"-the IOI’s that dark pools send to each other"

If IOI's are bad, why exactly are they bad? The post fails to explain the "why" for any counter argument it presents.

One additional rational not previously mentioned for multiple exchanges is redundancy. Reg NMS allows trading venues to match orders of listed securities from other markets. Because these markets compete as liquidity centers it behooves large market participants to have direct connections to all of them. If the NYSE has technical problems (this happens more often than you may realize, to all the exchanges) trading in NYSE listed securities continues unabated on the other platforms.

javert16• 14 years ago

I would be really interested to know the author's motivation for leaving HFT. For example, did you feel that you could make more money elsewhere?

Eventually I’ll even put forward a policy prescription which I believe could cause HFT to focus primarily on socially valuable activities

I'm really against the government making rules ot try to force people to serve society in some fashion. That's totally contrary to the principles of individual rights and pursuit of happiness upon which the country was founded.

yummyfajitas22• 14 years ago

I would be really interested to know the author's motivation for leaving HFT.

I left to do a startup. I am hoping to make considerably more money than I would have in HFT.

I'm really against the government making rules ot try to force people to serve society in some fashion.

I'd suggest you should stay tuned to the next post, since you will likely not object to my proposal. It's not a punitive measure designed to express disapproval, it's a minor tweak to market mechanics.

javert16• 14 years ago

I'd suggest you should stay tuned to the next post

Definitely looking forward to it.

deathflute5• 14 years ago

It is interesting that you believe that E(compensation in a start up) > E(compensation running a HFT strategy). The popular media is awash with reports of billions of dollars being made by HFT traders. Are they completely off?

What about HFTs that are running market taking strategies?

jrockway2• 14 years ago

It's a myth that the financial industry pays amazingly high salaries. The maximum programmer salary at a bank is probably bigger than the maximum programmer salary anywhere else, but the average salary is about the same. I worked on electronic trading systems at BofA, but Amazon and Google both offered me more money to work for them. (Some people get good bonuses, but the best way to guarantee that is to threaten to leave for somewhere that will give you XX bonus on paper. I did that but realized I actually wanted to leave :)

someben• 14 years ago

Kinda an obvious introduction to order books. Later I hope he gets into the rebate mechanism:

http://www.interactivebrokers.com/en/accounts/fees/NYSEstkfe...

Note that you get PAID for placing certain types of orders.

caladri2• 14 years ago

Men buy, women sell, and also if women are sad you can feed them strawberries and get off on it. Got it. Now I understand HFT perfectly.

francoisdevlin• 14 years ago

It's a firefly reference.

caladri2• 14 years ago

That much was obvious. When you build your examples around a sexist experience of sexist media, your examples will themselves end up sexist. I mean, why isn't Zoe buying, too? What is it that Inara's selling? I think we know, and from the heternormative misogyny rises the situation in which a cheeky description of market making has exclusively men buying and exclusively women selling.

johnohara2• 14 years ago

The astute reader will probably ask this straightforward question: -- "Jubal, does that seem right to you?"

tptacek29• 14 years ago

The concept of a market maker? Market makers are not an HFT "innovation"; they're kind of central to the whole concept of a trading exchange.

He hasn't yet explained anything more than what a market maker is or where HFTs fit into them. This post is the first in a series. It's background material. It would be a little weird if anything in this particular post got your hackles up.

johnohara2• 14 years ago

Mal, Inara, Jayne, Kaylee, River, Simon and Wash are all characters from the television series "FireFly." Jubal Early (Richard Brooks) is the bounty hunter in the last episode who ultimately ends up as an object in space.

Throughout the episode, his hypothetical situations always end with the rhetorical question, "Does that seem right to you?"

My comment was a tip of the hat back to Chris Stucchio. I enjoyed the article and also look forward to more.

lifeformed2• 14 years ago

Not making any value judgements here, I just find it fascinating how much we have abstracted away the original concepts of the stock market.

grandalf5• 14 years ago

Why do you say that? How would a basic marketplace work without the same rules?

lifeformed2• 14 years ago

I'm not saying different rules are necessary, just that it's interesting how abstract the whole concept is. I think about stock like how it was explained to me a long time ago, in very simple terms: Bob runs a restaurant worth $9000, and he needs $1000 more to buy a new oven. 10 of his friends lend him $100 each, in exchange for 1% ownership of the restaurant. Now with the new oven, Bob can bring in more money and he shares 10% of that with his ten friends.

Now we have this gigantic hurricane of integers that we call the stock market, and people make millions by running complex programs that find numbers that are slightly bigger than other numbers, and then run some code. An electrical signal is sent across the country, and in a database somewhere, transistors that represent somebody's bank account are changed into a different configuration. Everyone is sitting behind a desk, or standing at a terminal, pushing buttons. Based on what pattern of colors the screen flashes, people's lives are ruined or made.

xaa• 14 years ago

Do HFTers pay any sort of commission or transaction fee (to the exchange)?

zrail• 14 years ago

(not a trader but I've researched it a bit) Yes, often per share traded. A lot of traders use Interactive Brokers which has a few different pricing structures[1].

[1] http://www.interactivebrokers.com/en/p.php?f=commission

anonDataUser8• 14 years ago

Sub-pennies, negotiated directly with exchanges. They do the same negotiation that Interactive brokers does, except unlike interactive brokers, HFT players can trade 10% of market volume which give them significant leverage.

sailfrog3• 14 years ago

The common way transaction fees are handled in US equity markets is called the "maker/taker" model. When a participant adds liquidity to the exchange order book, AND that order is executed, they are given a very small rebate per share (they are the "maker"). The other side of the order, the firm who "removed" liquidity from the order book, is charged a very small fee. The difference between the rebate and the fee is the spread the exchange earns for executing the trade.

Note that not all orders that add liquidity qualify for rebates, the rebate may depend on particular order properties as well as the way in which the trade executes.

radarsat12• 14 years ago

What I don't understand about HFT is why its efficiency does not seem to be limited by transaction fees. I've tried a bit in the past to play with some algorithms that buy and sell (on paper) and it seems like the faster you go, the more you need the market to move in order to scrape a few pennies of profit, because the movement has to be sufficient to cover the spread, the transaction fees, _and_ provide some profit. Of course with a simple example like "buy at $20, sell at $20.10", it seems simple enough to extract some profit, but in fact it would need to be enough profit that it covers the transaction fees, which seem to be a kind "friction" that limits the smallest change in price that is useful. From what I can tell this kind of movement just doesn't generally occur in the sub-millisecond time frame, so how does it work? Or is it just that they are trading so much $$ in one shot that a very tiny change in price covers the transaction fee without issue? (Meaning they are taking very large risks..)

yummyfajitas22• 14 years ago

The sub-millisecond timeframe is part of a race to be at the top of the book or a race to pull orders. Very few HFT's literally trade a single security repeatedly on a sub-millisecond timeframe.

The general strategy is "our signal changed - move fast, get to the top of the book in milliseconds". Then you wait, and seconds to minutes later someone will fill your order sitting at the top of the book. Then you play the same game on the opposite side to unload your position.

The game of shaving milliseconds is solely about beating other HFT's to the punch, it's not about actual market movements.

minus1• 14 years ago

Great article. I'm curious what your thoughts are on the Sub-Penny Rule.

gbhn• 14 years ago

Thanks for the article. It seems to me like a missing piece is a discussion of volatility. That is, I get it that HFT drives spreads lower, but spreads are the wrong direction to make money, even in volume. That is, if pre-HFT spread is that the best buy price is $10.00 and best sell price is $10.05, and post-HFT best buy = $10.03 and best sell = $10.04, you still lose a penny on every trade instead of a nickel -- you can't make that up on volume! :-)

It seems to me that volatility is required for any market maker to make money, and that's a fundamental conflict, isn't it? That is, "outside" traders would prefer the market to be smooth, whereas "inside" traders want it to fluctuate.

I can see the argument that, in actuality, HFT on the whole needs less fluctuation to extract enough profit to provide liquidity, so in theory, it would be expected to be a more stable market maker than human operators. Is that basically equivalent to what you're saying in the article? (And is there any data on that hypothesis?)

deathflute5• 14 years ago

"Then you play the same game on the opposite side to unload your position."

Don't you have to play the game at both sides of the book simultaneously? Otherwise, you would risk adverse selection.

Tuna-Fish• 14 years ago

The transaction fees large players pay are negotiated to nothing. The exchanges want liquidity, especially in the derivatives markets. HFT does provide a ton of that, so they can negotiate away the transaction fees (in exchange to agreeing to maintain some minimum volume.

nvarsj2• 14 years ago

Yep. Most large market makers enter into obligations with exchanges to make markets in a wide spectrum of funds. Even those which are not that profitable. In exchange, the market maker gets much lower fees.

wglb5• 14 years ago

In fact, some traders have a negative fee--huge providers of liquidity are paid to always be there in the market.

DomBlack• 14 years ago

I've also looked at this before, and with transaction fees of 3% of the total traded, I can't see how they make money. Unless they have a deal with brokers for lower transaction fees or something.

drubio• 14 years ago

The OP provides a good technical explanation. Unfortunately he fails to mention another closely related term to high frequency which is what makes this type of trading highly lucrative: 'Front running'. See http://en.wikipedia.org/wiki/Front_running .

Front running is illegal, but if you look for successful cases of high frequency trading they are generally tied-to/accused-of front running. And as you might imagine, in order to do front running you need to be high-up on the food chain (i.e. be a market maker)

The missing key about front running in the article is the 'anonymous' bid-ask: "The matching engine takes his order and displays it (anonymized) to all other traders with a data feed." and "She places her orders, and it is again displayed to the world (anonymously) and stored.".

If you have forehand knowledge of the bid-ask (i.e. non-anonymous) the market maker can front-run and with high-frequency make a considerable profit.

tptacek29• 14 years ago

Frontrunners have knowledge of specific trades that are about to occur. You frontrun a specific block trade. The definition you imply here suggests that all market makers are "frontrunners".

anonDataUser8• 14 years ago

HFT is not front running. Search comments for my remarks on latency arb - that may seem like front running, but it's entirely different. Latency arb involves information that is publicly available. Front running does not.

scott_s3• 14 years ago

My understanding is that front running has been around as long as there have been market makers - and there have been market makers as long as there have been markets.

ChristianMarks4• 14 years ago

The practice of placing a large order and canceling milliseconds later to induce an exploitable price movement accomplishes this, sanctimonious protest to the contrary notwithstanding.

SomeCallMeTim• 14 years ago

Transaction fees for most brokers are pretty high. Even "discount" brokers like Scottrade charge $7 for every stock trade.

There are other brokers that charge $0.01/share traded or less.[1][2] A $0.10 spread becomes easy profit if the buy/sell only costs $0.02.

The linked broker below requires you to be an "expert" before you can sign up -- you have to have made at least 100 trades -- but otherwise just about anyone can sign up.

I have NOT tried to use them for day trading (or at all, for that matter), so please don't consider this a full recommendation. Just trying to show that there ARE other options that are cheaper.

[1] http://interactivebrokers.com/en/general/education/comparebr... [2] http://interactivebrokers.com/en/accounts/fees/stocksPricing...

namityadav• 14 years ago

Yes, their trades are for much bigger amounts, and they also don't pay the hefty brokerage fees that casual investors like us pay. One example, http://en.wikipedia.org/wiki/Direct_access_brokers which cost around $0.004 per share. And maybe HFTers don't even pay that much.

acslater00• 14 years ago

They don't pay "transaction fees" in the way you think about them. The way it generally works is you pay when you "remove liquidity" from the book and you get paid when you "add liquidity" to the book.

What does that mean? Suppose the bid/ask is 20.00/20.10 as in the example given. The order book will show a bunch of people willing to pay 20.00, 19.99, 19.98, etc, and a bunch of people willing to sell for 20.10, 20.11, 20.12, and so on. All of those people have added "liquidity" to the book by making orders which have not yet been placed. The idea is that a huge buyer could come in and try to buy 1 million shares at 20.25, and that order could get immediately executed because of the backlog of non-executed orders.

If you bid 20.10 (i.e. offer to buy the stock at 20.10) the seller at the top of the book will have his order executed (with you) and you will have removed a tiny bit of liquidity from the book. On the other hand, if you bid 19.95, your order just gets added to the system, and you have added a bit of liquidity.

The exchanges will charge you slightly more for removing liquidity (say .05 / share) than they pay you for adding liquidity (say .04 /share) and so make a spread. However, high frequency strategies can be clever about the way they add and remove orders to the book in order to minimize their net transaction costs, and so they end up being rather low.

On the other hand, when a retail investor purchases a stock through Schwab or something, Schwab (or in many cases some other larger bank) is dealing with exchange fees directly, and simply charging their customer a flat commission on top of your trade.

anonDataUser8• 14 years ago

Your numbers are too high. Actual rates are sub-penny or free. Some markets give a rebate for taking, others for posting.

radarsat12• 14 years ago

Thank you, I haven't heard that explanation before, it makes a lot of sense. So the normal transaction fee that you or I would pay is an additional fee by the broker, rather than a direct market fee?

achy2• 14 years ago

From my understanding of this, HFT basically makes it impossible for a non-HFT trader to buy at anything but his maximum buying price and to sell at anything but his lowest selling price. This seems like it would significantly undermine the profitability of the market for non-HFT traders.

tptacek29• 14 years ago

HFT makes things very bad for day traders. It makes things marginally better for people who want to take or sell a long-term position. It doesn't change limit orders.

achy2• 14 years ago

Can you please explain what you mean by 'It doesn't change limit orders'?

UK-AL• 14 years ago

I guess a lot of anti hft literature comes from traders? Yes?

toddh• 14 years ago

Accepting the earliest bid seems quite arbitrary. How would the system change if a random participant was selected? Or perhaps you could buy preferential treatment or use an algorithm weighted by some good actor metric.

3pt14159• 14 years ago

It is in the market's best interest to reward liquidity, so by giving it to the earliest bidder they reward those that have maximized the market's liquidity over time.

mayoff• 14 years ago

You can buy preferential treatment: If you set a higher price for your buy order, then you are preferred over every lower-priced buy order.

If you select a participant at random, you are encouraging each trader to place many orders (at the same price level), to increase the probability that one of his orders is chosen randomly.

If you use a pro-rata algorithm based on order size, you are encouraging each trader to inflate his order quantity, and cancel very rapidly after all (or nearly all) of his desired quantity is filled.

Fando• 14 years ago

very interesting

losvedir3• 14 years ago

Huh, I'm glad this was posted!

1) Didn't realize when I read this it was by yummyfajitas, whose comments I always respect.

2) Do I understand correctly that the order book is public (albeit anonymized)? That surprised me -- it seems like this would lead to meta-games and jockeying for position and such, as opposed to different parties just submitting into a black box what some security is worth to them.

3) Are there any good papers/articles modeling different market set-ups? For instance my black box one above. Or one where orders are matched randomly rather than chronologically. Or one where trades happen in one batch once per day. I can't say I'm opposed to HFT, it just seems to sap a very large amount of engineering brainpower for not that much societal marginal benefit anymore. If I could snap my fingers and give up a bit of the liquidity and get all those engineers back I probably would.

4) I once worked in a sell-side equity research shop which traded stocks the old fashioned way (based on fundamentals) and had a non high-frequency trading desk, etc. That type of company is getting hurt by the lower spreads offered by HFT. But I never did quite comprehend why it made sense to pay for our product (research) with trading commissions. Seemed to cross two unrelated services, although that kind of business model seemed deeply entrenched in the market.

Chirono• 14 years ago

Regarding 2: You're correct, the main order book is public and usually anonymous. There are also non-public books called dark pools where you cannot see what other people are bidding. You are also correct in supposing that a lot of meta-games go on, but dark pools don't remove this entirely; they just change the rules. For instance you could, if it is allowed, submit a tiny orders in increasing price until it gets filled; then you know something about the price in the pool.

losvedir3• 14 years ago

> For instance you could, if it is allowed, submit a tiny orders in increasing price until it gets filled.

Ha, that's clever. I suppose any system that people come up with will always have ways to game it. Reminds me of Arrow's Impossibility Theorem[1] which basically proved that there's no possible perfect voting system. Ah well.

[1] https://en.wikipedia.org/wiki/Arrow%27s_impossibility_theore...

yummyfajitas22• 14 years ago

Do I understand correctly that the order book is public (albeit anonymized)?

Yes.

...submitting into a black box what some security is worth to them.

These are Dark Pools. https://en.wikipedia.org/wiki/Dark_liquidity

Are there any good papers/articles modeling different market set-ups?

These are often called Crossing Networks. https://en.wikipedia.org/wiki/Crossing_network

losvedir3• 14 years ago

Wow, that's exactly what I was asking about. Thanks for the interesting links.

tolitius2• 14 years ago

there are several books including a "public book" that all exchanges publish and an "internal book" that each exchange has.

in case of the "internal book", it is always ahead of the "public book", as changes to it "published" after events occur internally and this "internal book" is updated.

this is important bit, as an individual exchange "knows better/earlier" than the "public".

MikeCapone• 14 years ago

Question: Are all HFTs only market makers like this, or do some of them use other strategies (front-running?) that actually make transactions more expensive for others, or move markets a lot (especially in small caps, I would guess)?

Nrsolis• 14 years ago

HFT is a badly abused term that gets applied to any firm that uses computers to trade.

In reality, there are many other strategies that use automated techniques to model the values of securities using internal metrics and then trade when the price is above/below the internally calculated value. Such conditions usually indicate that the market has "mispriced" a security and will quickly realize it's mistake and return to fair-value pricing.

Many "algorithmic traders" use statistical techniques to determine optimum times to trade so they capture some profit margin with a high degree of certainty. Those statistical techniques can involve anything from automated balance sheet analysis and natural language processing of news items, to pattern-matching techniques designed to notice when a large buyer/seller has entered the market and needs to move a large block of shares.

photon1372• 14 years ago

Can you say how feasible it is to build an HFT startup - the operating costs of execution/connectivity/colocation(!) aside - do you think regulatory authorities make the barrier-to-entry too high?

SEC/FINRA have Series 7/56 requirements for employees, UK has FSA-related principal investment regulations - both require a company to sponsor a person to trade. Also, the MiFiD regulations in EU require an "adequate capital" of 700k euros - is it really possible to get venture/seed capital covering these bare costs so that you just remain "legal", let alone operate?

veyron2• 14 years ago

Thanks to sponsored access and the large trader rule, you really just need the blessing of a clearing firm and pay some colocation fees. As a one time cost anyone can get in with 200K deposit (on a 6:1 levered setup) and about 50K in startup expenses, with a 25k/mo recurring charge.

P.S.: that is what I am doing now :)

photon1372• 14 years ago

That's useful to know - thanks!

djhworld• 14 years ago

Not sure where the apology is...or if there is even one needed in the first place.

I really enjoyed your article, it explained (in laymans terms) something that's often been bewildering to me as an outsider to the financial sector.

Maybe this is why people outside of the sector are hostile to the idea of HFT - because they haven't got a clue how it works?

adamsmith• 14 years ago

The way the system is set up makes it a tax on global efficiency. All of these resources (crazy smart people, money, equipment) are needlessly wasted on getting lower latency when really it adds no systematic value.

Instead, just as the system disallows fractional penny bids, it could discretize order times to the nearest e.g. 5 seconds. Boom. Suddenly you have more MIT/etc grads solving better problems for the world.

market_hacker• 14 years ago

no.

that would only work if every traded product on the planet was in a single matching engine.

noname1232• 14 years ago

Some exchanges already do that. But the issue is also liquidity not latency. When you disallow fractional penny bids and match trades at set time-interval, this causes market-makers to widen their bid/ask spread because of the inventory risk. Ideally, as a dealer you'd like to buy low and sell high and move the merchandise on your lot before the prevailing market condition moves against you.

The spread even at highly liquid stocks used to be 0.10+/cents before the introduction of penny pricing and HFT; this saves money for retail investors.

Also, introduction of set-time matching also means that algo's will just be tweaked to new market conditions. New arbitrage algo's will be introduced to exploit how some exchanges will be slower to react to price-actions. Predatory algo's will exploit how some institutional VWAP orders cannot be canceled quickly against volatile conditions. It's anti-virus vs. the virus writers. Oh not to mention the exchanges are in on this game, more trades mean more revenue for them. Exchanges make money by volume of trades, not by P&L of their customers; and like affiliate marketers, they actually offer liquidity rebates to any traders that offer trades that get taken at market price to boost their volume.

bradleyjg2• 14 years ago

The spread even at highly liquid stocks used to be 0.10+/cents before the introduction of penny pricing and HFT; this saves money for retail investors.

----

It saves money for retail traders (i.e. fools). It has negligible effects on investors in well diversified passively managed funds.

ricardobeat• 14 years ago

If Adam Smith says so. Totally agreed.

traversal4• 14 years ago

Whenever HFT comes up on HN, someone asks why we don't just remove the incentive for low latencies by matching orders less frequently. It's a fair question, but I think there are good reasons not to do this.

Let's say we discretize the exchange to, say, 1 Hz, so all the orders are queued up and then executed simultaneously at the next clock tick. Now, on a given tick, there will almost always be a mismatch between the number of buy orders and the number of sell orders, so some orders will go unfilled. How do we choose which ones? (For this discussion I am ignoring price. Obviously we will fill more aggressively-priced orders first; the question is how to prioritize orders of equal price.)

We could give priority to the ones that arrived first, but of course then we're back to traders racing each other. The only other scheme I can think of [1] is to fill every order in proportion to its size. So if Alice wants to sell 200 shares, Bob wants to buy 100 shares, and Charlie wants to buy 300 shares, we give 50 shares to Bob and 150 shares to Charlie.

Well, this scheme has its own problems -- arguably worse ones than the current system! If Bob really wants to buy 100 shares, and expects to be competing with Charlie, he has an incentive to place a much bigger order: in this case, if he knew Charlie's bid size, he would also ask for 300 shares, expecting to get 100. Of course, Charlie will be playing the same game and inflating his own bid size. The equilibrium is that everyone asks for way more size than they actually want.

There are two things wrong here:

1. Traders are spending mental energy trying second-guess each other's order size; those who don't are crowded out of the most competitive (and, typically, profitable) trades.

2. Since traders can't guess correctly every time, they will sometimes end up buying or selling much more than they want, which means they have to turn around and do the opposite trade, paying the spread in the process. If this happens during during a large price movement, these traders can lose even more money, and in their haste to reverse their trades, they will drive the market even further in the same direction.

And by the way, this is not just speculation. While I'm not familiar with any exchanges that discretize their clocks this way, there is a product with a similar system: the CME Eurodollar contract [2]. In this contract, unlike the other futures products traded on the CME, resting orders do not execute on a first-come-first-served basis; instead they are allocated "pro rata", or in proportion to their size, much as I described above. Based on my conversations with several people who trade them (including HFTers), the results are also as I described: traders routinely over-order and suffer the consequences.

Hopefully this sheds some light on why this seemingly obvious solution is not widely implemented.

[1] Of course I can think of many other schemes, but they either have obvious problems or basically reduce to this one.

[2] There are other products matched this way, but I'm not as familiar with them.

pak• 14 years ago

Why not use a random order for order fulfillment at each tick? No skewed incentives with that rule.

If that creates too many concerns over security of the RNG, then use deterministic rotating ranks. Traders may be able to predict who they will be able to beat out on the next tick, but the high ranks rotate throughout the herd, so everybody gets a fair shot at beating out others at the same price.

btilly• 14 years ago

How about the following variant?

Every market has a book, and a price. If a sell comes in at under the price, and there are unfulfilled buy orders anywhere above the price, the sell executes at that price for the order that has been above that price the longest. Conversely if a buy comes in above the price and there are unfulfilled sell orders below that price, the trade executes at that price for the sell order that has been below that price.

The price itself drifts upwards or downwards at a set rate depending on whether there are currently an excess of buy or sell orders that would execute at the current price.

What downside do you see in such a scheme?

nandemo2• 14 years ago

Tokyo Stock Exchange uses the "itayose" method for its opening auction:

http://www.tse.or.jp/english/faq/list/stockprice/p_b.html

But yeah, it wouldn't be a good method for an every-N-seconds auction.

hristov• 14 years ago

You can do it based on time. Yes it there will still be some incentive to be faster, but the incentive will not be that great. And you can remove all incentive to be faster by simply increasing your buy price by a penny. This means that all the incentive for speed is less than one penny per stock.

This would result in much smaller incentive for speed and less money and energy spent on being faster, and more fair markets, etc.

grandalf5• 14 years ago

A very well-reasoned comment. I wonder if anyone has done simulations of the various scenarios.

InvalidUserName• 14 years ago

Yeah, that and SEC Rule 612 (no subpenny prices, as the article mentions) both make bucketing sensible. Have a (1s,$0.01) granularity and HFT pretty much goes away, the National Best Bid/Offer makes sense again, and flash crashes only happen once a decade. If regulators actually cared about about market stability, they'd be pushing for something like that...

The London Metals Exchange - being an outlier as always - has gone for a different approach: they throttle traffic to 40 actions/client/second and smooth it out over as many seconds as necessary - so a 10,000 quote burst will be smeared over 4 minutes... good luck trying to manipulate the market with that, HFT boys.

deathflute5• 14 years ago

I am also confused about how latency arbs work. If you are co-located at one exchange, wouldn't you be further away from other exchanges?

wglb5• 14 years ago

Big players are co-located at every exchange.

tolitius2• 14 years ago

market makers work (have deals) with exchanges. e.g. a market maker M guarantees to provide liquidity (e.g. orders) for a certain set of securities to an exchange E, and gets certain benefits in return.

This market maker M makes sure that he is as close as possible to the exchange metal, so he has an upper hand.

The importance of colocation is directly proportional to the liquidity (amount of orders) you provide. e.g. if you send orders manually every minute or so, it does not really matter where you are in comparison to a large bank that sends orders every several microseconds..

polynomial• 14 years ago

> The rise of algorithmic trading is merely a special case of machines replacing humans.

One of the article's more salient points.

ptolosav• 14 years ago

Very interesting. As an undergraduate looking to learn more and hopefully get some knowledge so i can get that "edge" in the field when i graduate, what books do you guys recomend on trading? I know HFT is very specific and probably there isn't lots of literature on it, but something more general will do.

Thanks a lot.

surement3• 14 years ago

As I understand it, it's better to learn assembly, algorithms and parallel computing. Some firms even look for PhDs in machine learning. The finance side you can learn when you find a job.

redwood• 14 years ago

The author describes changing bid/sell prices based on new information (like a new press release putting pressure on value, say). He doesn't mention whether people are feeding the algos up/down pressure based on these press releases or if they're automatic.

I assume they're at least partially automated. In other words for every article that comes out about a given company, some algo (or human) gives it a plus/minus ranking and magnitude based on what it means for the company.

Assuming this is partially done by algos that crawl press releases looking for signals, or similar (in order to be first to market with latest news) one can imagine a new type of hacking.

Hack the press releases and watch the algo trades go haywire....

LinXitoW• 14 years ago

While the article itself is very interesting to someone like me that knows squat about diddledy, i must commend the author for including FireFly references.

tjmania2003• 14 years ago

Really enjoyed reading this, great stuff. Even better Chris spent some time at Snevets :)

A High Frequency Trader's Apologia, Part 1

Related Stories