HFT Quote Stuffing is a software bug

http://www.chrisstucchio.com/blog/2014/quote_stuffing_is_a_software_bug.html

by jpdus • 12 years ago

62 63 12 years ago

mikeash4• 12 years ago

That sure is a lot of hand-wringing for what is a fairly simple and obvious problem. You're reacting to the extremely short-term derivative of a noisy signal. Of course things are going to go crazy from time to time.

It's odd that he touches on the obvious solution of rate limiting, but doesn't actually discuss it as a way to stop this problem. If quote stuffing is a bug that harms the people carrying it out, then why don't HFT programs have a rate limiter on them that make them stop? He even goes so far as to say that you can get shut down for doing this, so why isn't there a tripwire to make sure you can never, ever do it?

lucisferre• 12 years ago

The low level of opacity when it comes to electronic trading makes it difficult to make any sense out of exactly what is (or perhaps isn't) happening, which I think is fundamentally the problem.

sambe• 12 years ago

Do you mean the high level of opacity? There is considerably more opacity than in non-electronic trading, surely (everything on record, all data public, no effect from personal relationships etc)?

dllthomas4• 12 years ago

Though it's worth noting that transparency wasn't the hallmark of earlier systems.

yummyfajitas7• 12 years ago

If quote stuffing is a bug that harms the people carrying it out, then why don't HFT programs have a rate limiter on them that make them stop?

They do. But a rate limiter is just another signal - you still need to set the rate limit high enough that you can correctly respond to market conditions, but low enough that you don't go wild.

"Never, ever do it" is like "zero tolerance for bugs". It's a goal to strive for, but it's not achievable.

mikeash4• 12 years ago

It's completely achievable. Rate limit to one trade per second. Done.

You don't want to do that, because it doesn't let you make as much money as you otherwise could? I totally understand. But in that case, quote stuffing is not a bug, it's a deliberate tradeoff.

But I suppose that "we let this happen on purpose because it allows higher profits, even though this particular behavior doesn't help us" doesn't sound as nice.

gd14• 12 years ago

Obviously, well written HFTs avoid this by introducing some hysteresis to the signal. i.e. Insert an order when the signal crosses 0.5, but then only delete it if the signal falls below 0.4.

Chris: You've also missed the most common cause of these issues in my experience - feedback from identifying your own orders in the public stream. HFTs need to filter their own orders out of the orderbook they are trading on, or a feedback loop can begin. This can be difficult, since some exchanges don't provide a clear way to identify your own order, or it may be the case that you receive the public UDP packet before the private TCP packet that contains the ID required to match the two.

For example (a basic one), say I have an algo that calculates the midpoint based on the best bid and ask volumes. And can only insert within x distance of the midpoint. I add a bid to the top of the book for a 1 lot, get the UDP public depth update, don't know whether this is my order or someone elses. So I re-calculate the midpoint, which pushes the midpoint upwards (greater bid volume), which means I then pull my bid, which means I recalculate the mid again, then re-insert, etc. etc....

yummyfajitas7• 12 years ago

I actually mentioned this briefly - see the discussion of the "reconciliation step" in the "Quote Stuffing is not a DOS attack" section.

I guess I assumed everyone's reconciliation system was as good as the one I worked on. But you make a very good point, as well as a meta point - this stuff isn't easy.

gd14• 12 years ago

Sorry, missed that bit

hft_throwaway11• 12 years ago

Sure but what if someone else is doing something in the market that perfectly lines up with the edge conditions for your place and cancel thresholds? Nothing is foolproof. If you are operating at a large enough scale this will end up happening a few times a day, especially when you have multiple agents interacting in ways that can't be predicted. A feedback loop can still happen between two unrelated traders running similar rules like "if I'm not the best offer, improve it by a tick".

hft_throwaway11• 12 years ago

http://zacharydavid.com/2014/04/on-hft-part-ii-bugs-features...

This blog post details similar issues that can lead to rapidly canceled orders. Generally, firms that are smart avoid moving their prices around "too much" since:

a.) You lose your spot in the queue. If I am near the front of the price/time queue buying at 100.01 and move to 100.00 and back up again, now I'm behind everyone else and will either lose opportunity or face more negative selection (I'll only trade when someone wants to sell all the contracts bid at 100.01, whereas before I would get some trades where someone only wants to sell a few).

b.) Sending orders is more computationally expensive than doing nothing. If you send enough orders you will slow down your own system.

c.) Depending on market rules and behavior the messages you send may cause your gateway to queue up or block you from sending more messages which hurts you.

I don't think quote flicker can provide any "evil" benefit to the person doing it. It doesn't make sense to do it to slow down competitors, since most feeds are multicasted to everyone and you still have to process all the orders you send, both from the gateway and over the data feed. If you are faster at handling this situation than your competition then you are going to beat them anyway so what's the benefit in slowing them down?

I do think it can be disruptive in the sense that it places extra load on the exchange and can annoy participants who have trouble taking prices on their screen. Some markets have quote/trade rules to encourage more efficient messaging which is their right and not a bad idea if the limits are set reasonably. Rules like this make more sense than a minimum resting time, which just penalizes makers at the expense of fast takers. Imagine I am putting quotes out in SPY, an S&P 500 ETF, but they need to rest for half a second. Someone sees the S&P 500 futures or the individual stocks move and then my quote becomes a free option to trade at an unfair price. In a market like this, market makers would be forced to put up wider prices to compensate for these losses.

A high quote/trade ratio in itself is not disruptive. A lot of the Nanex charts use thinly traded derivative products like ETFs as their whipping boy. The underlying value of these products will change whenever another stock or commodity changes, so market makers need to update their quotes quite rapidly to avoid trading at unfair prices, even if no trades have occurred in the product itself.

lmm• 12 years ago

If you are faster at handling this situation than your competition then you are going to beat them anyway so what's the benefit in slowing them down?

Suppose your fund has invested a lot in some hefty compute hardware but your trading strategies are on the weak side, whereas you know your competitors have better strategies but their computers are slower. Isn't it then in your interest to stuff the order feed and make everyone more dependent on computer power?

determinant• 12 years ago

Whatever action you take will generally come back down the multicast feeds back to your own systems to process.

hft_throwaway11• 12 years ago

Maybe? Most guys who are really fast are also really smart. It's hard to make money purely being fast these days.

I can't speak for everyone but most "HFT" models are built around providing some kind of service to the market, generally either making markets to try to make the bid-offer spread or arbitraging away inefficiencies between related products.

Even if you are evil, building a model that solely seeks to take advantage of another trader doesn't make sense in the long run. Eventually they will alter their behavior or exit the market. It's not sustainable.

lmg643• 12 years ago

this is an interesting way to look at the problem but in my reasonably informed opinion, flawed.

it seems like this is working backwards from the assumption that quote stuffing must be benign activity, and then trying to rationalize it with signal/noise theories about finding the fair price level.

it is totally inaccurate to say that there is more load on the sender than the general public, and also misses the mark on what the DOS theory alleges.

the exchange bears considerably more load than the sender of orders. the exchange must (1) ack the order, or reject if any flags are populated out-of-spec, (2) add the order to the order book, (3) check for potential matches based on price of order, if so, cross and update feeds, (4) if non-marketable, publish out to (a) depth-of-book feed, (b) SIP feed, etc.

the DOS theory is that excessive activity in a single partition can slow down the publishing of data in that symbol range, which would mess up the quoted price for the exchange in a symbol, which would affect orders priced on the exchange, or anyone using the price in a derived manner (at a dark pool etc).

yummyfajitas7• 12 years ago

I'm not making the assumption that it's benign. It's based on my own experience - I've quote stuffed hundreds of times in simulation, and I'm sure it happened in production as well. It's a hard bug to completely avoid.

Interesting - the DOS attack is on the exchange rather than other market participants. But could you be more specific on how this helps you make money?

kasey_junk3• 12 years ago

There used to be games where you would attempt to get one order connection on one exchange gateway and then another connection on a different exchange gateway. You'd spam with one connection to slow down all the other participants on that gateway, and use your other connection for your real trades.

Other games involved spamming open auctions, settlement trades etc.

This gaming was a very brief phenomenon (because it is stupid, doesn't make as much money as traditional trading, and trivial to foil). The exchanges got rid of it almost immediately but it still gets passed around as folklore for one of the "predatory" activities that HFT does.

hft_throwaway11• 12 years ago

Most exchanges are structured something like:

a.) Each trader has one or more gateway processes that do the order checking/rejecting.

b.) Each gateway maintains a queue of messages for the matching engine.

c.) The matching engine/order book for a product pulls one message at a time from each gateway that has a message waiting on its queue and processes it before spinning back to the first gateway again.

So a trader blasting the exchange with orders isn't going to impact other traders who also place orders around the same time. She'll only be slowing herself down.

mmaldacker• 12 years ago

DOSing the exchange will get you kicked out in no time.

personZ• 12 years ago

The 2. in the series, if I understand it right, is that they overwhelming the derelict SIP feed between market locations, increasing the benefits that their own high speed connections brings. If true, that does make money because they can act "in advance" at other markets before the artificially slowed SIP data gets there.

rsync3• 12 years ago

I always assumed that step 2 was "DoS your competitors".

This is crudely done with volume of traffic, but more elegantly done with frequency and amplitude (and so on) given some set of (possibly wrong) assumptions about how competing algorithms will respond to this rapid fire order placement.

I know almost nothing (beyond ZH and chapters 1-5, so far, of _Dark Pools_) about HFT, but I can trivially think of ways you could be evil with quote stuffing as it is described in the OP.

wglb3• 12 years ago

So since all orders have to go to the exchange first, how does DoS your competitors work?

rsync3• 12 years ago

You're thinking of a crude, direct DoS attack wherein the "traffic" generated by one HFT directly overwhelms another HFT.

But there are other ways to DoS - what if I observe your trading activity and I discern something about your underlying algorithms and how expensive those algorithms are in terms of memory or CPU or latency, and I make a bet that a certain trade pattern will dramatically overwhelm your resources via these second and third order effects...

It's just speculation, but the point of the OP was that there was no way this could be useful to an HFT algo and I am pointing out that in a few minutes we can come up with all kinds of ways that it could be.

hft_throwaway11• 12 years ago

Manipulation definitely happens where traders will show non-bona fide interest on one side of the market to provoke other traders or algos to react. It's known as layering or spoofing and like any behavior where you're placing orders to influence others rather than out of an interest to trade, it's illegal.

Most cases of this or similar behavior like stop running have involved human traders at loosely regulated click trading shops or smaller electronic firms that don't play by the rules:

https://www.finra.org/Newsroom/NewsReleases/2012/P178687

http://www.cftc.gov/PressRoom/PressReleases/pr6649-13

Aside from the regulatory risk, it doesn't fit the business model for most "HFT" firms, who generally are making many low-risk, low-reward trades with a small statistical edge based on proprietary models. Spoofing usually requires showing large false interest which entails considerable risk if you actually get elected. If you are fast enough to make markets or arb crude oil, you can make way more doing that than manipulating the DOM hoping to push algos into doing something stupid until they figure your tricks out.

Predicting markets is hard. Predicting how thousands of heterogenous agents will react to your order is even harder, if not impossible.

rsync3• 12 years ago

"Predicting markets is hard. Predicting how thousands of heterogenous agents will react to your order is even harder, if not impossible."

My very small amount of knowledge on this topic leads me to believe that the idea that one algo would specifically target another algo is not at all farfetched. Who cares how the market responds to your weird trading if you're trying to create distortions for one, or a few, actors ?

kasey_junk3• 12 years ago

So again with all HFT conversations we have to be clear on what we mean with the language we are using.

I've seen quote stuffing used to mean what Chris mentions here. The implication being that HFTs were using large volumes of low quantity orders for nefarious purposes such as DoS the exchange & SIP or confusing your competitors by making the order flow to heavy to process. These kinds of games did occur, but it was very briefly a long time ago because the exchanges themselves had a vested interest in stopping it and an easy mechanism which to do so (fill ratios). I prefer the term quote spamming for this, and I agree with Chris that most of the time when this happens it's just a bad order management bug.

Quote stuffing can also refer to the practice of putting a few large orders into the order book with no intention of trading them in order to make the volume on the book look larger than it is, and then cancelling them quickly to take advantage of the "real" volume. This is a predatory activity designed explicitly to take advantage of market making strategies and is already illegal. Your mileage may vary when it comes to enforcement...

throwaway13qf85• 12 years ago

The second kind of quote-stuffing you mention, where you place larger orders outside the best bid/offer with the intention of distorting the order book (aka spoofing, see comment below) is certainly nefarious. But it's a special kind of nefarious behaviour that really only affects other market participants who can react quickly enough to the information - i.e. it's mostly HFTs taking money from other HFTs.

lrm2422• 12 years ago

Using large non-bonafide orders to lean the book in the direction you want to trade is called 'Spoofing', not quote stuffing. It's been illegal for a long time.

hft_throwaway11• 12 years ago

I think it is more short-term day traders taking money from naive HFTs. I'm sure some people automate this type of manipulation, but I seriously doubt that any of the "big name" market making firms do it. They are under a lot of regulatory scrutiny, make more than enough through legitimate trading, and have the technology & data to enable their internal compliance auditors to detect it.

I also don't think it can make that much. It may work for a little while, but no algo is going to sit there bleeding money to you forever without hitting a risk limit or being reviewed/adjusted. If your business model depends on somebody else doing something dumb, it's not going to work for the long-haul.

001sky• 12 years ago

But it's a special kind of nefarious behaviour that really only affects other market participants who can react quickly enough to the information - i.e. it's mostly HFTs taking money from other HFTs.

This kind of behaviour that undermines the integrity of markets should not get a pass based on who it does or does not hurt, tho. If nothing else its bad PR that keeps people out of the markets or creates other "absences" or dark patterns from people not showing up to trade, fro whatever reason.

jpdus• 12 years ago

Nanex linked an interesting CS research piece[1] which also discusses other possible reasons for (malicious) quote stuffing including:

- walking others into the book,

- creating false midpoints and

- trying to cause stale pricing and slowing others down.

[1]http://de.scribd.com/doc/116761218/CS-HFT-DETECTION

hft_throwaway11• 12 years ago

But even they admit that a market flashing between 100.01 offer and 100.00 offer is sometimes beneficial for them. It says their execution algo will attempt to "pick off" the 100.00 offer with IOCs if they are representing a buyer.

Flashing orders with the intent to manipulate is wrong and is illegal already. It doesn't matter if you do this by hand or with a computer. Putting up a tighter best price, even if it doesn't stay put very long, can only be beneficial. At least some of the time, other traders will get filled at a more advantageous price than they would otherwise.

anigbrowl2• 12 years ago

Except, once in a blue moon this happens when the mean signal approaches the threshold and stays there for a while:

I'm having some trouble with the contention that this is just such a rare thing. I do a lot of audio signal processing so I'm used to dealing with noisy signals at Khz rates. Threshold fluctiations are a fact of life if you're trying to do things like pitch detection, envelope following, or sidechain compression (all of which have obvious analogs in the trading context).

Obviously you don't want to impose too brutal of a low-pass filter on your input because you would miss important signals, but what prevents you from running the input signal to an LPF in parallel and keying a noise gate when the amplitude of the low-frequency (trend) signal drops? This is such a common problem in dealing with audio that we use specific devices called de-essers for recording vocals, because sibilant sounds otherwise tend to throw off other components of the signal chain: http://en.wikipedia.org/wiki/De-esser

Ironically, you mention using XKCD-type graphs because 'nobody reads' the explanations and caveats, but just looks at pictures. I wonder if you aren't missing a trick here by over-engineering your HFT systems for speedy response at the inevitable price of positional confusion. Is it possible that you're using Infinite Impulse Response filtering and running into denormalization problems as a result? If so, can you not rectify the problem by adding a small amount of noise?

I get your point about the negative aspects (for the HFT trader) of quote stuffing, how it can easily lead to being on the wrong side of a trade or punishment by the broker for flooding the output channel with unintended orders. But, you want to make money from automated signal analysis, that's your problem in much the same way that a sound engineer is responsible for preventing transient pops and feedback howls in an audio network. Sure, these problematic signal conditions are statistically rare, but when you're dealing with a high volume of samples you'r going to run into statistically rare conditions on a regular basis - and indeed you mention that this is the sort thing you might expect to happen once a day.

hft_throwaway11• 12 years ago

I think you are right on the money and most traders do use techniques analogous to the ones you describe when designing models:

http://www.decal.org/file/2945

"Alpha is often nothing more than taking commonly available data and mathematically encoding it in a signal correctly. Correct often means something as simple as using an rate of change instead of a difference, normalizing a value, smoothing a chaotic signal with an EMA, using a heteroscedastic weighted linear regression instead of a simple regression, or handling all numerical errors or edge cases."

It's not always easy to get this completely correct in every situation and traders face a reward function that doesn't necessarily reward correctness so much as avoiding Type II errors.

anigbrowl2• 12 years ago

I feel like I'm in the wrong job - clearly I should be routing ticker prices into my mixing desk and playing the market like a cheap violin instead of being an underpaid sound engineer.

austengary• 12 years ago

This is fun

https://twitter.com/nanexllc/status/458663948800520192

HFT Quote Stuffing is a software bug

Related Stories