Barbarians at the Gateways: High-frequency Trading and Exchange Technology

http://queue.acm.org/detail.cfm?ref=rss&id=2536492

by rupelloy • 13 years ago

117 141 13 years ago

queue.acm.org

33273• 13 years ago

This is a well informed and put together article. Up to date technically and the Author knows what he is talking about. As a former HF trader I endorse it, great piece.

rurounijones2• 13 years ago

Out of curiosity. Why "Former"?

33273• 13 years ago

I walked away and moved onto more exciting things like entrepreneurship. As the author states, HFT is like rebuilding gravity models every week, so your product is not sustainable and doesn't grow, cash generation from HFT also is usually not scalable on a per strategy basis.

bjpcjp• 13 years ago

My company builds memory ICs that sometimes sit beside the ASIC or FPGA on an HFT card. The bandwidth & latency numbers didn't surprise me, but the cost of power certainly did. No wonder customers are always crawling up our butt on power consumption.

elecengin• 13 years ago

A great overview of the space. The numbers and techniques are realistic and the article refrains from the moral debate (which you can find elsewhere...)

chollida15• 13 years ago

Some facilities such as the Mahwah, New Jersey, NYSE (New York Stock Exchange) data center have rolls of fiber so that every cage has exactly the same length of fiber running to the exchange cages.3

I remember the first time someone told me that. I thought they were kidding. Then I actually got to see the data center.

The exchanges, and HFT firms, take this very seriously. The speed at which they execute is just unbelievable.

The amount of technology and networking stack knowledge is just mind blowing.

I remember tuning my linux distribution's networking stack to gain an advantage only to find out that everyone had moved onto using asic's and infiniband hardware.

I continue to be amazed at the sheer talent and brain power that goes into this field:)

dclusin2• 13 years ago

Kernel bypass is also common. Having the OS manage your networking stack adds too much latency.

rdtsc• 13 years ago

That is the most common one I've heard of.

Obviously if they are building fpgas into the switch itself then well of course they'd already tried running customized ethernet driver firmware and all.

blumkvist2• 13 years ago

I hope "amazed" is just poor wording.

chollida15• 13 years ago

I think I'm being trolled but I'll bite.

What's wrong with the word amazed?

If I said I worked at Google and I continue to be amazed at the tech behind how they served up ads would I still get the same remark from you?

Maybe I could be doing more with my skill? I don't know but I really love the learning curve I'm on working with cutting edge technology and pushing the performance envelope.

xradionut2• 13 years ago

I think much of the effort is wasted. When you hire embedded designers to program FPGAs and ASICs just to be a few nanoseconds faster in your rent-seeking, it's a bloody shame.

mcphilip• 13 years ago

Spread Networks invested an estimated $300 million in building a new fiber line from Chicago to New York in order to shave 3 off ms. However, they found out after completing the line that faster speeds were already being achieved by microwave connections built in secret. It's an interesting arms race:

http://www.cnbc.com/id/100695563

rfnslyr4• 13 years ago

How do I get into this field? I'm a 20 year old making a shit ton of money by doing mobile work but I find it so boring. Nothing about it is innovative. I feel like a high level computer construction worker. It's brainless for the most part, it just takes time.

I want to get into some really cool stuff. I've been wanting to get into investment banking from a technical standpoint, HFT, algorithmic trading etc, but honestly don't know where to start.

What are the de facto books/websites/reads for this field?

sseveran3• 13 years ago

The worlds of HFT and the rest of algorithmic trading are very far apart. For HFT pickup the exchange documentation for each exchange. Memorize how order types. Get a copy of RegNMS. Memorize that as well. To be effective an HFT needs to exploit the market structure. Learn how processors work. How do you avoid a pipeline stall or a cache miss? Become an expert in c++. Maybe pickup some verilog for those pesky FPGAs.

The rest of algorithmic trading offers a lot more flexibility. In either case read 'Trading and Exchanges'. It will give you a good overview of the mechanics of all different venue types. Then its thinking about how to build better algorithms then the next guy, or get better data.

shogunmike3• 13 years ago

For HFT the learning requirements are extensive.

It is /almost/ essential to have a PhD in CompSci/EE from a top school to do HFT. Alternatively one should demonstrate extensive hardware/networking and optimisation skills obtained from other low-latency industries.

All of the top work is being done on FPGAs and latency is now on the order of microseconds (probably lower).

As for lower frequency algorithmic trading, that is a game that one can play at the 'retail' level if you're willing to spend (quite a lot of) time learning.

I run a website about algo trading. If you want to get a taste for what is involved have a look at some of the articles here: http://quantstart.com/articles/#algorithmic-trading

chollida15• 13 years ago

How do I get into this field?

Well turn the question on its head and ask what can you bring to a hedge fund to allow it to be more successful. That's the test we use.

Hedge funds are similar to start ups in that they are small businesses where everyone needs to pull their own weight and be able to handle their own area without supervision as everyone is busy enough handling their own area of concern.

Or put another way, say the firm has 10 people and each share equally in the year end bonus pool. If the pool is $100 then each person makes $10 in bonus.

Therefor you must be able to increase the bonus pool by atleast $10, to $110 in the next year or two, in order for the firm to break even by hiring you.

So what do you bring to the table that allows the firm to increase its profits? how good are your networking, math, C or C++, optimization skills, what market structure insights do you have?

dopamean2• 13 years ago

I'm wondering if someone here can explain to me the value in HFT for anyone other than the people doing the trading. I went to Wharton and some old friends are in HFT and the one thing they all say, and I hear repeated everywhere, is that they provide liquidity to the market. Something about it just doesn't ring true to me. If it is in fact true I'd love to hear an explanation.

kasey_junk9• 13 years ago

Liquidity has certainly gone up.

Also, it is hard to prove a causal relationship, but with the rise of electronic trading and HFT, price spreads and fees have come down.

I can't prove it, but I believe that insider trading is also much less rampant in modern electronic systems than it was in the older pit based markets.

selmnoo• 13 years ago

The liquidity it provides is often very low quality. When liquidity in the old days was added, the players involved had to operate under stricter regulations -- they had to post a quote that was part of the National Best Bid and Offer a minimum percentage of time. With HFT it's a wild west scenario. There's a lot of uncertainty and risk of failure in many of its steps.

beagle38• 13 years ago

Re: spreads & fees: Price spreads and fees have come down as a result of electronic trading being open to everyone, and is probably independent of HFT (at least, modern HFT): E.g., The decreased spreads and fees were happening in europe in the early 2000s, when fastest updates were at 1/4 second, and slower updates were at 2 seconds. The nanosecond scale was not the reason for this.

Re liquidity: it depends how you define liquidity. If you define it by "the average size at the bid or offer and next few price levels" or "volume traded", then yes - HFT has helped liquidity tremendously. If you define it as "the probability that a large order can complete", then liquidity has NOT gone up. With HFT, it's the same 100 shares/futures changing hands thousands of times per day, and they disappear often in times of uncertainty. If you need to execute a large order, you observe that you do not have the amazing liquidity everyone in HFT is talking about.

Re insider trading: I wouldn't bet on it. With electronic trading, it's now easier to do insider trading slowly without attracting much attention. Furthermore, HFT has created new forms of fraud, so far unregulated: see e.g. http://www.nanex.net/aqck2/4329.html

pedrocr• 13 years ago

I seem to recall that the research on this says that HFT is actually a net taker of liquidity and that their profits are fully funded (and then some) by other investors not really by making the market more efficient (they actually reduce efficiency). I've also heard stated many times that an exchange that only matches trades every X seconds would retain efficiency while eliminating HFT.

This isn't the article I remember but goes into detail on most of these issues:

http://www.demos.org/publication/cracks-pipeline-part-two-hi...

Mikeb852• 13 years ago

They do provide liquidity. I can place a market order through my online broker, and have it execute in the time it takes my browser to refresh.

This wasn't possible before HFT, and it allows retail investors to get much better trade execution.

beagle38• 13 years ago

Just to be clear, HFT these days mean microsecond round trip times between trading algorithms and exchanges.

The situation you describe is thanks to electronic trading, and is also available in financial markets that are not dominated (not to say "infested") by HFT, such as currencies and CFDs.

lmm• 13 years ago

What HFT provides is what other market users ask for; a limit order is essentially "I will pay 0.5 cents/share commission (thanks to the sub-penny rule) to the first person who can sell me shares in x at or below $y)". Since you asked for the first one, you must want it fast, right?

Perhaps other order types could allow market participants to ask for something closer to what they actually want - but new order types are often criticized as giving HFT players an advantage, since they can understand and exploit them faster than other market participants.

Alternately, eliminating the sub-penny rule would allow HFTs to compete on price rather than just latency, which would mean better prices for long-term investors and remove a lot of the profit from the HFT industry.

chollida15• 13 years ago

I'm wondering if someone here can explain to me the value in HFT for anyone other than the people doing the trading.

Well I guess I'm very biased but here's my stab at it.

High Frequency trading is at the for front of alot of technology such as ASIC's, Infiniband networking gear,and low latency OS and networking stacks.

You could argue that they help push these technologies forward by providing the first customer for these areas.

Liquidity has gone up with the advent of High Frequency trading but like someone else has said its hard to disentangle all market factors to say this is due to HF Trading.

Something about it just doesn't ring true to me. If it is in fact true I'd love to hear an explanation.

To be fair, this statement is the equivalent of "I've heard evolution is a well followed theory but something about it doesn't ring true to me."

What sort of explanation would you like that hasn't already been said 100's of times by people more qualified than me?

You haven't said what you don't like or disagree with about the many existing explanations:)

lolcraft• 13 years ago

High Frequency trading is at the for front of alot of technology such as ASIC's, Infiniband networking gear,and low latency OS and networking stacks.

That's not an argument for HFT. If HFT is useless, then it has divested a great amount of research into technologies that no one really wanted otherwise, i. e., that the social cost of HFT is greater. Standard microeconomics.

The question is, what's HFT good for? Do its benefits outweigh its costs? I believe they don't. Low-quality liquidity under the 1s frame is not only useless for traders, it has a great social cost, in the form of research diverted to this inane game. The only plausible arguments I've found for HFT claim that HF traders wouldn't profit if it wasn't useful -- which is really just begging the question; a circular argument.

mahyarm• 13 years ago

It's the whole war funds large amounts of science R&D argument and thus is a great benefit for humanity.

jrehor• 13 years ago

You are assuming that HFT profits come at the expense of investors. In fact, they were taken from insiders.

I'm old enough to remember trading on US stock exchanges in the mid 1990s when prices where quoted in 1/8s and 1/16s and NYSE specialists were the only ones with visibility into order book. Think about it: you as an investor had no idea of the depth of the order book, but the specialist who took the other side of the trade from you had it in front of him. The specialists were minting money. They would lose money maybe one day per quarter, and their ROEs were ludicrous.

HFT and ECN trading killed them. Labranche, Van der Moolen, Susquehanna, Spear Leeds, all gone. Goldman Sachs bought Spear Leeds for $6 billion in 2000 (it is now no more); Labranche specialist business fetched only $25m when it was sold to Barclays in 2010. Van der Moolen went bankrupt in 2010.

Investment bank trading desks, true champions of customer front-running, are shrinking fast. Cash equity trading has become so tough for the banks that they are starting to think of it as a cost center, a loss leader to promote their equity underwriting business.

These were multi-billion dollar businesses, with tens of thousands of middlemen living high on the hog from the spreads and front-running. They're all (mostly) gone. Good riddance.

If you want to relive the old days of trading before HFT, go execute a large trade on Karachi Stock Exchange. Put the order in and watch in amazement.

I would like to see the data showing total revenues of specialists, market makers, bank trading desks, brokerages, and HFT traders, over time. I would bet they have been going down for two decades. This is undeniably a good thing for investors.

tanzam753• 13 years ago

Tighter spreads were made possible by electronic exchanges, penny pricing, and direct access brokerages.

You didn't need HFT for that. You could've gotten the same thing with LFT. If all you want is a penny spread, then nanoseconds don't matter.

RickHull2• 13 years ago

some old friends are in HFT and the one thing they all say, and I hear repeated everywhere, is that they provide liquidity to the market.

More specifically, HFT acts like a market maker in that it will take the other side of trades that other players (HFT or not) want to make. Let's say I own 1000 shares of MSFT, but I need to liquidate them for some reason. A margin call, a new car, or some immediate, unexpected cost has arisen. The more potential counterparties I have -- the more players that will potentially buy my shares -- the more liquidity there is.

With more liquidity, I can hold out for a higher price (with a cost in time), or I can sell more quickly (at a slightly lower price -- with a cost in money) than I could otherwise.

Whether or not my HFT counterparty is executing profitable trades (which it is, presumably), the very fact of its existence as such is a net benefit to me.

tanzam753• 13 years ago

There is so much liquidity in MSFT that it'd trade at a penny spread with or without HFTs. Thus, the only advantage is speed.

So you got your order filled in 5 milliseconds instead of 6 milliseconds. So what? You're a person.

And stocks without enough liquidity to get to a penny spread? The HFTs don't like those stocks, because the "F" is not "H" enough for them to participate. The very nature of HFTs is that they prefer to participate when there is already plenty of liquidity.

ForHackernews• 13 years ago

God, what a waste of talent and technology.

thejosh• 13 years ago

Not really, what comes out of it is nice.

001sky7• 13 years ago

By 2005 we were approaching the one-millisecond barrier for latency arbitrage

Everything you need to know, in one soundbite.

gd18• 13 years ago

Why? What's the response time on an airbag release in an accident? On the ABS in your car? Your graphics card manages to rotate, scale, texture and shade a hundred million polygons per frame in around 1000/60 = 16ms. We're talking here about processing one UDP frame, usually under 256 bytes.

Why is this the only topic discussed by HN where technology suddenly becomes this frightening bogeyman?

001sky7• 13 years ago

Its not the engineering, its the objective. Latency arbitrage is code-word for front running. Woops.

tucson• 13 years ago

""It's legalized front-running. I think it is basically evil and I don't think it should have ever been allowed to reach the size that it did," he said. "Why should all of us pay a little group of people to engage in legalized front-running of our orders?""

Charlie Munger, http://www.cnbc.com/id/100705820

smoorman1024• 13 years ago

What's the difference between HFT and traditional Human Market Makers?

May I suggest that Electronic trading and HFT have made it possible for a few people to do the work of thousands. This has brought down the cost of trading.

anonymoushn2• 13 years ago

Which HFT firms have access to the orders of other participants before the exchange does?

AndrewBissell3• 13 years ago

Charlie's just upset that some of his bridge partners aren't scalping what they used to with their floor-based market making businesses.

manishsharan• 13 years ago

Can someone please tell us what is the return or for an hft fund vs. the market in the long run say 5 years. Also , are there hft funds available to general public for investing.

sseveran3• 13 years ago

There are no funds (that I know of) that except outside investments. Most are proprietary trading groups meaning they don't solicit investment. As far as returns they can be big. RenTech Medallion has thrown up impressive numbers for years, many 50% or more. Most of the firms that I know of are north of 50% in the small to medium sized space. If you want to peek inside a larger one look at GETCOs S1.

HockeyPlayer2• 13 years ago

Are there hft funds available to general public for investing?

No, one of the characteristics of HFT is that the returns are high and the capacity is low.

chollida15• 13 years ago

one of the characteristics of HFT is that the returns are high and the capacity is low.

The second part is correct, the first, not so much anymore.

http://www.businessweek.com/articles/2013-06-06/how-the-robo...

cs702• 13 years ago

So much talent... focused on the buying and selling of securities, instead of creating new things that will make the world better in a directly measurable manner.

Virtually all trading volume today consists of buying and selling old securities -- essentially, legal claims on existing assets. The sale of new securities issued to finance the creation of new products and services -- for example, a company selling new shares via an IPO or issuing new bonds for investment in physical infrastructure -- represents only a minuscule portion of total trading volume.

I'm not sure having so many of our best and brightest minds going to Wall Street (and into high-frequency trading in particular) is a good thing, from a societal perspective.

PS. Whenever I read anything about high-frequency trading, I'm reminded of the following passage, written by John Maynard Keynes in 1936 -- 77 years ago (!): "Of the maxims of orthodox finance none, surely, is more anti-social than the fetish of liquidity, the doctrine that it is a positive virtue on the part of investment institutions to concentrate their resources upon the holding of 'liquid' securities. It forgets that there is no such thing as liquidity of investment for the community as a whole. The social object of skilled investment should be to defeat the dark forces of time and ignorance which envelop our future. The actual, private object of the most skilled investment to-day is 'to beat the gun,' as the Americans so well express it, to outwit the crowd, and to pass the bad, or depreciating, half-crown to the other fellow."

Source: http://ebooks.adelaide.edu.au/k/keynes/john_maynard/k44g/cha...

The more things change, the more they stay the same!

foobarian• 13 years ago

I always wondered whether it would make sense to legislate a random added latency in exchanges in order to create a latency noise floor to remove the need for HFT. Without such a cap it's an arms race that adds no value.

socillion• 13 years ago

Without such a cap it's an arms race that adds no value.

I have seen nobody ever challenge the value of liquidity in these discussions of HFT. Assuming this, the argument is that at some amount of latency the additional liquidity turns into a negative.

It seems more like the additional value is simply extremely small, but meaningful over an entire market.

What's the exact cutoff point for the value of additional liquidity turning negative, and will anyone agree on that arbitrary number?

Assuming HFT does turn into a negative at some point, wouldn't the same apply to optimizations in other markets like online advertising?

If it does apply elsewhere, we should also legislate away the practice of making online advertising more efficient to prevent tech giants like Google and Facebook from wasting billions of dollars and untold man-hours on these activities that are clearly worthless to society.

npongratz• 13 years ago

Why resort to legislation? Create such a market yourself (or with a team) and see if your tweaks naturally drive demand from market participants. No force necessary.

dopamean2• 13 years ago

Someone said something similar about a lot of people in the startup community. That we were sending our best and brightest out to create products that ultimately only served the purpose of pushing ads (Instagram, Facebook, Twitter, etc). I don't know how true this is. There are a lot of very smart people in the world. The luxury of having that many smart people is that we can have a lot dedicated to designing amazing infrastructure that sends satellites into orbit and we can also have Twitter and HFT.

31reasons• 13 years ago

Then why we haven't cured cancer yet or malaria or global warming or psychological disorders ? There are a lot of smart people in the world but we really can't afford to work on silly things.

SkyMarshal2• 13 years ago

Jeff Hammerbacher, CTO of Facebook, now CEO of Cloudera.

That said, actual product companies are making a comeback in Silicon Valley, and are using innovative new tech like 3D Printing, and there's some amazing work going on in Virtual/Augmented Reality as well (Occulus Rift, CastAR, Google Glass), - all physical products.

And lets not forget Tesla upending the world's perception of the economics of producing and owning electric vehicles.

While I agree with the sentiment about smart people wasting their potential chasing cash, I think it's only an intermediary phase before the next wave of "insanely great", life-changing products hits critical mass.

kazagistar• 13 years ago

Theoretically, people would pay for the things they value and problems they want solved, and the market would do the rest.

Practically, most people don't actually have that many major problems that need solving, and it is pretty simple to invent something that isn't what someone needs, and convince them that they need it, then to come up with novel ideas.

Industry and free enterprise does not solve problems much anymore. It sells the process of convincing people they have problems.

analyst74• 13 years ago

Similar things can be said about military research, law firms, medical clinics (not medical research), advertisement firms and most other types of work.

0xdeadbeefbabe• 13 years ago

Especially law firms.

sumodds3• 13 years ago

I disagree with this assumption. The problems with military research, law firms, advertisements are of degree not of its existence. Sure, they have their own problems. But we need military research: which is arisen out of insecurities of nation states, to given protection to their states. Law firms, too have a place so too medical clinics and advertisement firms.

In case of HFT, the argument is not of degree but existence. Military solves security problems of nation states, law firms communal issues of individual citizens, advertisements to help the product reach the intended audience. But what problem does HFT solve other than making a few people very rich.

icebraining• 13 years ago

http://www.marxists.org/archive/marx/works/1863/theories-sur...

fmstephe• 13 years ago

While I wouldn't want to defend HFT on any moral grounds we are at least seeing some of the technology leaking out into open source.

https://github.com/OpenHFT

While I don't like the role of HFT in modern markets I find the technology that it drives very exciting.

fijal• 13 years ago

Most of their technology is incredibly secret (for obvious reasons). They won't even tell you they're using your open source project (they can give you money though).

They're (on average) worse than your average tech shop. But then again, people like google or facebook will never buy software from anyone, so if we're left with just them, you can never create a business writing software that caters for other software companies, not sure what is worse.

pdovy2• 13 years ago

Virtually all trading volume today consists of buying and selling old securities -- essentially, legal claims on existing assets. The sale of new securities issued to finance the creation of new products and services -- for example, a company selling new shares via an IPO -- represents only a minuscule portion of total trading volume.

Leaving aside for a moment the question of the value of HFT, this is a skewed view of the world of finance. The equities markets may be the most visible to the public (quoted on the news, associated most directly with companies and products we all know, etc), but it's just one market.

In fact, the size of global bond markets is almost double that of equities [1] and is a huge source of funding for companies (and governments), and has more frequent new issues for the same entity than in equities. Similarly futures markets are hugely active and important as both a source of price discovery for commodities and as a way for businesses small and large to hedge risk.

I mention all this not to pick on you in particular, but because it seems in the wake of the financial crises of the past few years, this "finance is bad" mentality seems to be pervasive. There are absolutely problems, but I think it's important not to lose sight of the fact that much of what financial markets do is hugely important to modern businesses and governments.

[1] http://finance.zacks.com/bond-market-size-vs-stock-market-si...

dasil003• 13 years ago

Markets are hugely important. The problem is the size of the investment banks who neither supply the capital nor the equities, but due to proximity of the transaction are able to skim more and more cream out of the transactions and devote more and more brainpower to increasing that arbitrage. They have become so instrumental to the lives of the rich and powerful that they can completely destroy economies and be bailed out by tax payers the majority of whom have absolutely no interest in the solvency of said banks other than some extremely urgent, hand-wavy and fundamentally interest-conflicted warning of economic cataclysm.

Obviously regulation doesn't seem to do much good because it just changes the rules which sends the investments games scrambling for new arbitrage opportunities which no regulatory agency has the resources to keep up with until its too late. Given the lack of political fortitude to combat moral hazard by forcing financial institutions to take their own losses on the chin, I think perhaps widespread and rabid investment banker hate could be the best societal medicine we have against the brain-drain to finance.

pdonis3• 13 years ago

Wouldn't a more relevant statistic be trading volume? Stats for that (at least for the US market) are here:

http://www.sifma.org/research/statistics.aspx

It looks like the US bond market daily trading volume is about 25 times higher than the US stock market daily trading volume. However, that statistic is highly skewed by trading in US treasury bonds, which arguably should be a separate category. The same would presumably apply to the stats on market cap.

Also, it's not the relative size of the equities markets that's the problem; it's the amount of effort and talent that goes into what is essentially a zero-sum activity. Bonds and futures (and, to be fair, new stock issues) are positive sum. HFT is not.

(More precisely, HFT is only positive sum to the extent that it improves price discovery and therefore market efficiency; but I have a really hard time seeing how shaving a few more microseconds off trade times changes that significantly. It does, however, significantly change who is able to benefit from asymmetric information by inducing others to take the wrong end of zero-sum trades, which is what HFT is designed to do.)

fin101• 13 years ago

Company MakeWorldABetterPlace.com creates new thing and makes us all feel better in a directly measurable way.

They decide to create an even better new thing that has heavy capital requirements so they issue stock.

Even better new thing is not as great as we all thought, so stock goes out of favor as market participants channel capital to new and more promising tech companies.

Years down the road, MakeWorldABetterPlace.com, trading on old securities decides to enter the mobile market and creates revolutionary device. Market participants catch on and start channeling capital out from less promising securities and into the old and existing claims on assets of MakeWorldABetterPlace.com. Thousands of businesses and jobs are created by new privately held companies that create new things using MakeWorldABetterPlace.com's mobile operating platform.

So much talent focuses on the buying and selling of securities because it is the most social way to vote directly - by allocating capital. And the product of that capital flow is what is observed as the creation of new products that make the world better in a directly measurable way.

If you don't think liquidity is important, then with it goes the ability for countless tech startups to exit (through their acquisition by large corporations in exchange for claims on existing assets) and use the resulting capital to create subsequent startups, that perpetuate the cycle of creativity.

a_c_s• 13 years ago

There more liquidity there is, the less marginal value additional liquidity has.

33273• 13 years ago

This is why I left. It is actually the ultimate brain drain for the United States.

phyalow2• 13 years ago

Virtually all trading volume today consists of buying and selling old securities. I'm not sure having so many of our best and brightest minds going to Wall Street (and into high-frequency trading in particular) is a good thing, from a societal perspective.

Your so wrong. Cash Equities account for notionally c. 50% volume at best the rest being made up from exchange traded derivatives. HFT, MM and Prop firms by speculating and posting contra liquidity lower transaction costs for other market participants. Society benefits from having a fluid and well greased market which reflects all known information.

Also get your Keynesian BS out of here.

kylebrown2• 13 years ago

The sub-penny rule (decimalization) was also mentioned by Chris Stucchio (aka yummyfajitas).[1] And it is extensively discussed in the book "Dark Pools".[2] Because order queues only exist at whole-penny prices, the bots are forced to compete on speed. If it weren't for that rule, there would also be competition on price because you could place offers and get price-priority in the sub-penny interval.

The playing field is basically a DDoS arms race, where the main technique is full-throttle quote-stuffing to push anyone without equivalent hardware and connection away from the front of the queue, and out of the game. (Dark Pools also mentions a conspiracy of special undocumented order types on certain exchanges). The ACM article even admits that the algos aren't even particularly intelligent, just very speedy: "In my experience, most high-frequency algorithms are fairly straightforward in concept—but their success is based largely on how quickly they can interact."

A distributed order book like the one in the Ripple protocol[3] is a much fairer system. The network charges a transaction fee in order to prevent quote spam/DoS attacks, price fractions can be as small as 6 decimal places, and offers are processed in an effectively random order (within each discrete ledger).[4]

1. http://www.chrisstucchio.com/blog/2012/hft_whats_broken.html

2. Dark Pools: The Rise of the Machine Traders and the Rigging of the U.S. Stock Market by Scott Patterson

3. https://ripple.com/distributed-fx/

4. https://ripple.com/wiki/Arbitrage#Rapidly_changing_offers

gd18• 13 years ago

Mind you, if they lower tick sizes, that means a lot more order cancellations. A market-making algo is simply a control system, taking in a set of inputs and spitting out a fair price that it is willing buy/sell something for. The number spat out is a rational one, that is then rounded to the nearest discrete tick. Smaller ticks = more jumps in the discrete output price = more cancellations/modifications. Which is fine, except the anti-HFT brigade seem to think a trading system that cancels 10000 orders and trades 100 times is the devil. It's not, it's about as remarkable as any other control system. You may as well express outrage that the cruise control system in your car adjusts the throttle ten thousand times a second. The only downside is that the exchange needs to cope with that level of traffic.

pasbesoin2• 13 years ago

I'm going to say this again: Exchanges should be put on a stepped clock. I'd suggest a one minute interval. Orders pile up during that interval, and then they're processed in random order. If things start to go really, State-concerning hinky, then authorities "pull the plug" before the next interval is up.

Anything less, to my mind, serves point-shaving and profiteers, rather than real investment.

Then, all the "gearheads" can go back to doing useful science. And maybe getting paid decent wages and benefits for doing so.

/grump

theorique2• 13 years ago

Because of course "real investment" is saintly like the Dalai Lama, and is certainly not looking for a profit.

pasbesoin2• 13 years ago

I take your point, and I didn't downvote you. From my perspective, there is an important difference between taking a position in / with respect to a product (or its organization) and taking a position for the position's sake.

Perhaps it is something of a continuum, not a strict either/or. If so, I find that currently we've gone too far towards the latter.

I'm also concerned that current constructs exhibit chaotic tendencies that put real people and essential systems at risk. I view stepped trading intervals as a potential damper to some of these negative and out-of-control tendencies.

Ultimately, I view investment as backing and furthering production. And I don't see how micro-second based investment appreciatively, substantially furthers this.

P.S. Admittedly, it has been an important factor in pushing computer and network systems development, which may well have had knock-on effects. E.g. is one substantially aiding the other, or the other way around: Trading versus CERN data processing and distribution?

xixi772• 13 years ago

"real investors" do have an option of going to dark pools :)

gd18• 13 years ago

Dark pools exist because these "real investors" you speak of are muppets. If you're a market maker out in the big bad world you are exposed to adverse selection risk - i.e. you're up against smarter, better informed or faster traders who hurt your profits. If you set up a dark pool, you can pick and choose your customers, and keep these people out. You can quote tighter in the dark pool than the wider market, so your customers think they're getting a great deal, but in reality they're only getting this deal because they are (collectively) great big juicy marks who you can make a fortune from.

This is why you see the big investment banks setting up dark pools, they can't compete in the technology race or with smaller more agile firms, so they retreat to these pools.

gd18• 13 years ago

By the way, this is an awful, awful idea that you hear quite a lot. Think about what would happen. What incentive would a participant have for placing an order early in the one minute interval? If you can't cancel it, you're a sitting duck. And if you can't be executed now, why insert the order now? By doing so, you show your hand. You'd end up with a rolling one minute e-bay auction where everyone tries to insert at the last possible microsecond.

MereInterest• 13 years ago

Why would it matter when people place the orders if there is no extra information to be gained during that time? If the orders placed are only revealed after they have taken effect, why bother waiting for the last moment?

Nimi• 13 years ago

Hoped to find such a comment here, this is basically about mechanism design - how can we design an exchange that removes the additional profits such microscopically low latency generates, when the low latency itself isn't providing value, i.e. the low latency => profit phenomenon is an artifact of the rules of the exchange.

I appreciate your thoughts, but regarding your suggestion: random isn't well-defined. It can be random as in "randomly pick a participant, let it buy/sell as much of the stock as it asked, then pick another participant etc.". This encourages Sybil attacks - participants have an incentive to create other identities for themselves, and hope one of the identities will be picked "randomly". Another definition of "random" may be "randomly pick an order to buy/sell a single stock, execute it, repeat-until". This, I was told on an earlier HN thread, strongly encourages placing more orders than one wants, again in the hope of fooling the "random" part of the exchange. I was told this is actually what's happening currently in exchanges built this way.

I had a suggestion I tried to analyze (even as a possible idea for a mechanism design paper): What happens if "random" is defined as "your chance of getting picked (to be able to sell the stock) is proportional to the amount of this stock you own"? At least, we got rid of the incentive for Sybil attacks (and didn't create the opposite incentive, for conglomerates - separate entities don't have an incentive to present themselves as a single entity, then do the bookkeeping internally). But it's unclear what to do when there are more willing buyers than sellers. Any thoughts, anyone?

I have also wondered whether this whole thing is an artifact of quantization - what would happen if the price granularity was allowed to be as small as one wants?

(In case anyone's wondering, a friend I consulted told me this probably wouldn't fly as a paper - useful economic policy if I can solve that "more buyers" problem, but nothing scientifically novel).

p.s. Sorry for the long comment.

unabridged• 13 years ago

Would there be any loss of liquidity if trades were matched in 1 or 2 second intervals? This would eliminate all front running and colocation bias.

kasey_junk9• 13 years ago

This doesn't solve the problem in anyway. The issue is what if you have a different amount of buy to sell qty. Who gets their order filled? This is called matching and each market matches in different ways.

The typical way (and one of the leading drivers of latency races) is first in/first match. If you continue to FIFO match on your intervals there is still a tremendous advantage to being first into the interval.

If you aren't going to FIFO match that is fine, you just need to determine (and publish) your matching routine.

icedchai• 13 years ago

Good to see an article about real engineering, not the usual web app / JS bull...

Barbarians at the Gateways: High-frequency Trading and Exchange Technology

Related Stories