Watch High-Speed Trading Bots Go Berserk

http://www.technologyreview.com/view/428756/watch-high-speed-trading-bots-go-berserk/?ref=rss

by cpeterso • 14 years ago

71 86 14 years ago

technologyreview.com

rgarcia2• 14 years ago

Anyone know what the y-axis measures? The original Nanex source doesn't label it either[1].

[1] http://www.nanex.net/aqck/2804.HTML

lt• 14 years ago

Looks to me like the time of the day.

gahahaha• 14 years ago

You are correct. Felix Salmon comments: http://blogs.reuters.com/felix-salmon/2012/08/06/chart-of-th...

> in 2008, a pattern starts to emerge: a big spike right at the close, at 4pm, which is soon mirrored by another spike at the open. This is the era of traders going off to play golf in the middle of the day, because nothing interesting happens except at the beginning and the end of the trading day. But it doesn’t last long.

rgarcia2• 14 years ago

That's the x-axis.

devingoldfish• 14 years ago

Presumably share volume, though I'm not sure what the multiplier is (my guess would be 1000x).

curiousDog• 14 years ago

Any idea how much Knight ended up losing? I remember interviewing with this company, Allston Trading in college. They were the only ones who showed up at the Career fair with the salary printed out on brochures. $100k for a fresh java developer. During the interview in their plush offices in the Chicago stock exchange, they were casually discussing how they ended up losing $5million in forex last night. All of them were young, brash and looked like they had little regard for investors money. One trading guy was telling us how he flies to vegas just for the weekend to get with hookers (during a lunch interview!).

The managers seemed like your typical alpha-male..bullish, dominating and little regard for morals. Although some computer scientists that they had seemed brilliant and were genuinely interested in building safer, better algos. So i wouldn't be surprised if it's these wall st. types that are pushing the limits further even if deemed unsafe.

mhurron• 14 years ago

>i wouldn't be surprised if it's these wall st. types that are pushing the limits further even if deemed unsafe.

It's human greed. Those 'wall st. types' and those brilliant computer scientists are all in it to make as much money as possible. Those you looked up to as hero's were not corrupted by those you wish to look down on, they all went into it wanting the same thing. The environment changed allowing them to try and get it faster.

theorique6• 14 years ago

Consensus number seems to be $440M in losses on the bulk liquidation of $4.5B in unwanted positions to Goldman Sachs. This represents about a 10% loss.

GS does bulk liquidations like this and typically takes a haircut of 2-5%, presumably based on a risk and liquidity assessment.

damoncali4• 14 years ago

They lost $40 million and 73% of the company plus 3 board seats. Ouch.

tectonic• 14 years ago

Does HFT create value in any way? It strikes me as purely driven by greed and so utterly irresponsible for society as a whole.

damoncali4• 14 years ago

It helps liquidity. How much that matters is up for debate.

rdtsc• 14 years ago

I think the implication was, is the liquidity level of 2007 inadequate and we need the liquidity level of end of 2011 (which is about an order of magnitude higher).

In the end maybe I don't care if these firms just transferring money between themselves but when people's retirement accounts and the fate of whole companies in their hands it seems there should be something preventing this (another meta-algo that punishes aggressive and pushy algorithm...?)

slapshot• 14 years ago

As has been pointed out a few times in connection with this incident, Knight is largely a market-maker (not an HFT firm): Knight keeps open quotes on both buy and sell sides of transactions at all times and provides liquidity. Their strategy to make money is the "spread" -- they price the buy a couple of cents lower than the sell and on average they end up buying a couple cents lower than selling.

They are part of the infrastructure of the major markets; a company listing on NASDAQ or NYSE is required by the market to choose at least one market-maker who will be designated to always keep open buy and sell orders on the books.

As it happens, Knight does its market-making through computers; this replaces sweaty men who would stand in the "pit" of the trading floor and trade by waving fingers at each other (including, occasionally, the middle one). It looks like HFT in that computers trade quickly, but it is intended to be a liquidity-providing service to the market: there is always somebody willing to buy or sell anytime the market is open.

(In contrast, HFT does not require always-open buy and sell quotes.)

white_devil4• 14 years ago

Yeah, "liquidity" is the standard bullshit distraction trotted out every time HFT is discussed here. Strangely enough, we've got "yummyfajitas" defending it again, which maybe, just maybe, might have something to do with the fact that he works in HFT.

As for liquidity, since the bots only make trades with the goal of shaving off tiny little slivers of profit on millions of trades, why and how would one of those sweaty guys ever buy anything from them? Even if one wanted to, the opportunity to buy might pass in a couple of microseconds.

All in all, working in HFT or even the financial "industry" at large, is complete bullshit. No self-respecting, decent human being should do it. No matter how you look at it, your mission there is to help world-raping scumbag bankers and the like make more money. They certainly don't need it.

sp332• 14 years ago

https://news.ycombinator.com/item?id=3855610 And yes, ~100% of the trades at an exchange are driven by greed, with or without HFT.

hexagonal2• 14 years ago

It creates liquidity, which makes it easier to buy and sell shares when you want to buy and sell them.

Unfortunately, the only time you really want liquidity is during a crisis, which is when all the HFT firms exit the market, since they're in it to make money, of course, not to act as a regulated and guaranteed market maker.

theorique6• 14 years ago

Regulated and guaranteed market makers are both (1) required to post a bid and ask even in crisis situations (2) high-frequency and high volume traders because they need to deal with a great amount of market order flow. These things are not necessarily exclusive.

samstave4• 14 years ago

HFT should b illegal. Period.

What value is there in any system where trades over "pennies millions of times per second" is provided at all to real world conditions/humans.

This is simply a tool by a few to make money in an invisible, unregulated and uncontrollable space.

This is simply not sane.

cpeterso• 14 years ago

If we agree (and I do) that trading should be throttled, what is a reasonable speed limit? Once per second? Per day? Is there a precedent?

samstave4• 14 years ago

Maybe we need much tighter rules around how the trades can be made.

I have no clue at this point what the model should be - but HFT is something that is completely undoable for a human, though it can have impact on humans at large scale and should be thought of more critically before they can be deployed into production.

I say impose a moratorium on all HFT until a determination of their value, impact, risk etc can be done and published and debated widely.

koglerjs• 14 years ago

What I've heard, and this is almost worse than anecdotal evidence, is that HFT helps stocks converge on their true 'value' or 'equilibrium' much faster.

This could, of course, be utter bullocks, and I have no way of tracking down the source, but it's at least an interesting idea.

MarkPNeyer8• 14 years ago

high frequency trading is very important!

without it, investors would not be easily able to change their minds about how much other investors think wildly complex financial instruments are worth!

think of the children!

theorique6• 14 years ago

Value: accurate pricing, reduction of bid-ask spreads, transaction automation causing reduction of commissions.

MarkPNeyer8• 14 years ago

a bid-ask spread is a measure of uncertainty.

by reducing the bid-ask spread, hft shops are adding more "certainty" to the market - but that certainty could be (and is) sometimes wrong.

in other words, we're making financial decisions faster, at a cost of adding more more false certainty.

if the economy is a computer, HFT is overclocking the shit out of the processor.

zacharyvoase3• 14 years ago

This is from the department of “I don’t understand/like this, so I’m going to use the coercive force of the State to stop it from happening”.

People like you are the reason cryptographic algorithms still get banned in countries around the world.

samstave4• 14 years ago

Thats bullshit.

Hav you not noticed the banking scandals in the past few years? Have you not noticed that NOTHING has been done about them?

Also, if you look lower down the thread - I stated that a moratorium on bot based HFT should be put in place until we can fully understand how to regulate these properly.

Equating me with dictatorial banning of cyptographics is ridiculous.

scrumper• 14 years ago

That's rather simplistic, not to mention completely wrong. US markets are highly visible (you just saw a free visualization of HFT activity since 2007), very tightly regulated and well controlled. Things go wrong, as they do in all complex systems, but rarely do they go wrong to the extent that ordinary investors are harmed.

Read this, before jerking your knee any further. Think of it as turning on the bedroom light, showing the monster in the corner to be nothing more than a pile of clothes:

http://www.chrisstucchio.com/blog/2012/hft_apology.html http://www.chrisstucchio.com/blog/2012/hft_apology2.html

Financial markets have been moving towards zero latency trading since their inception. First, men gathered under trees or in coffee houses, within shouting distance of each other. They fought over the best spots. Then, traders became some of the earliest adopters of telegraph technology, running private lines from remote cities to the exchanges. Ticker tapes appeared in distant offices. Computerized price dissemination followed, together with electronic trading connections to the exchanges themselves. All in the name of getting information and acting on it before the next guy. It's the reason you, as an individual, can trade global markets cheaply and instantly. It helps you take control over your retirement and savings, rather than paying through the nose for some managed scheme where only a privileged few can access the markets.

Where is your arbitrary line where progress stops?

samstave4• 14 years ago

I think you are arguing something different than I. I am open to being wrong, but I think I am making a different point.

You are saying that by having a zero latency trading capability, financial markets are the sign of progress and it is an inevitability that, not only is desirable, but preferred.

That's fine, but not what I am am taking issue with is fully automated system of extremely high volume, low margin, trades that are conducted by bots and have no human interaction other than those that are profiting from them.

I admit I am not savvy enough in this area to argue about whether or not HFT provides more equilibrium to the price of a stock, but I don't see that as a wholly convincing argument as to why HFT is important and adds value.

I am skeptical of that claim, mostly due to my ignorance, so please educate me on the following: The claim that HFT rarely goes wrong such that "ordinary investors" are harmed; who then is the non-ordinary investor who is benefiting from the HFT, and what value are they providing in the markets, other than profiting from HFT.

It appears to be making the claim that HFT is a meta market that the ordinary investor shouldn't even be concerned with except in cases of extreme rarity where a flaw causes them some financial harm/risk.

Again, if this is true - what true concrete value to the world does this meta system provide? I simply cannot see it, and again, I claim awareness of my ignorance, so please explain like I am five.

Thanks!

joering2• 14 years ago

This should be illegal long time ago -- think Captcha for buying/selling stocks.

The real issue here is that -- whoever was in charge of those stock exchanges hasn't properly addressed this issue as we all can see now its been growing slowly but constantly since 2007.

pavel_lishin• 14 years ago

> think Captcha for buying/selling stocks.

I like that the side-effect would be a drastic improvement in OCR technology.

radikalus2• 14 years ago

This would be awesome. (Time to bust out your DBNs)

Symmetry• 14 years ago

Or just only permit trades at 100ms intervals or something. There are simpler solutions than Captchas.

waqf• 14 years ago

I'm imagining this would mean an auction (something like the current opening/closing auctions) every 100ms ... I like it. It seems more scalable than the current system.

hexagonal2• 14 years ago

Blogspam. OP is http://blogs.reuters.com/felix-salmon/2012/08/06/chart-of-th...

powertower• 14 years ago

Do HFT firms have contracts with larger banks to take their positions with a discount in situations like this, so the stocks are not imidiatly dumped on the market causing havok? Or to insure them against loss?

Sounds like we could segment, rate, and pacakge these contracts up into insured "credit-default-swap" type-of vehicles and make money making/selling/buying/trading them.

danso• 14 years ago

I know HN's frontpage has had a few posts dealing with what financial industry coders do and the validity of it. The ones who claim to be HFT-bot coders argue that they're highly-skilled coders...but it was not always clear if their skill is in algorithm-implementation, or in engineering. That is, they may be great at working with the traders in coming up with a highly performant bot, but they have less ability in exception-handling and design, which may be things that their bosses don't really think about. (I honestly don't know, was just wondering if someone could competently recap the relevant discussions.)

radikalus2• 14 years ago

Most everybody I know (in the field) is in disbelief concerning the length of time the Knight disaster unfolded over. I've had systems "loop" in BAD ways, but never for more than 1-3 seconds.

Safeties are paramount in the field. There's many layers of protections to avoid *ing yourself. Most of these are deployed in ways that are not terribly expensive on the 'critical path' -- that said, there are a LOT of people doing REALLY stupid things in the field. (As there are in any field, it's just more 'scandalous' when an error in a financial application occurs)

Algorithm implementation is largely trivial, and the high pricetags paid developers in the industry are usually for specific engineering skills along with the semi-requirement that developers be 'all-hands-on-deck' when the situation calls for it. (There's a bug in the new strategy...well you need to fix it before tomorrow's open etc) At least, that's how I see it.

MarkPNeyer8• 14 years ago

i worked in electronic trading for a few years. we weren't high frequency, though - it took us a few milliseconds to respond to price movements, which is why i left for the startup scene in 2010.

exception-handling is something we took very seriously, because the consequeneces of a fuckup are so bad. in late 2008, our firm lost around 300k in ~2 seconds. one of the exchanges sent a signal that the interest rate was now 0, which, according to their spec, meant that something wasn't right. our system didnt' handle that properly, so the algos went nuts because hey money is free! the fuse system caught it a few seconds later, but not before we shot orders at everything in sight.

i wrote a system that handled federation and service location, and as part of the design, i constructed four state machines which were used to handle the four possible cases in the formal model we developed. i went over the state machines many times by hand, and coded unit tests for each possible (State, Input) transition, followed by a battery of stress tests.

these guys are good - but nobody's good enough to never make mistakes.

yummyfajitas5• 14 years ago

Lets stop and think for a moment about what we are discussing. Knight gave other people about $400M by making some bad trades. Also, some graphs wiggled.

The net impact on the market was negligible. If you buy on the timescale of days to weeks, you can completely ignore HFT. Here is a blog post with graphs demonstrating the market impact of various HFT events:

http://www.chrisstucchio.com/blog/2012/flash_crash_flash_in_...

photon137• 14 years ago

"If you buy on the timescale of days to weeks, you can completely ignore HFT"

Not if you have to deal with intra-day margin calls or stop losses.

The trouble with most HFT-based algos is that most of them rely on a set of common signals (regardless of how "diverse" their creators may claim them to be) and end up making the various markets and assets highly correlated.

So when stops trigger, they cause a massive cascade. This needs to be protected against at the exchange-level (edit: they do so via various mechanisms). Most bots usually don't have the sophistication to protect against tail-events.

MarkPNeyer8• 14 years ago

the technology used by HFT firms to make markets in microseconds makes the web development world look like children.

pubsub is the hot new thing here in the valley - welcome to wall street, circa ten years ago.

asynchronous is the new synchronous? what is that - a new album by smash mouth?

if you completely ignore HFT, you'd think there would be way more programmers and mathematicians available for hire.

zanny3• 14 years ago

> if you completely ignore HFT, you'd think there would be way more programmers and mathematicians available for hire.

Can you clarify this? My interpretation is that if you discount mathematicians / programmers working in HFT, you would expect more of them to be in the market for jobs?

dasil003• 14 years ago

On the other hand, web development is solving many many more problems for many many more people.

HFT firms are solving the single problem of making their owners rich with the authoritatively, hand-wringingly repeated justification about how critically important that liquidity is to everyone and their grandmother.

Aloisius• 14 years ago

As someone who has designed, built and ran systems that operated at a scale that made the software that HFT firms look like toys and built several web companies, I can honestly say that you have absolutely no idea what you are talking about.

Web development and low-level systems engineering are entirely different and neither one is particularly harder than the other. If anything, both are simply tedious.

And pubsub was old by the time wall street got a hold of it.

thetabyte• 14 years ago

Yes, but does the fact that "nothing bad has ever happened" mean that "nothing bad will ever happen"?

As a developer, I'm familiar with the fickleness of software. What happens when we see HFT algorithms error in unison? What have we not thought about?

SoftwareMaven• 14 years ago

Things like the Flash Crash[1] are what occur. Te scary thing in my mind is that error states are the hardest to test for and tend to be multiplicative in cascading severity. The HFT world is hiring a lot of really smart people. We have no choice but to trust that enough of that brain power is going towards addressing those error states.

1. https://en.wikipedia.org/wiki/2010_Flash_Crash

jobu2• 14 years ago

They're sucking a few pennies out of every transaction on these exchanges, the long-term effect is that of a parasite in a larger host. Sure they may have some small benefit of improving liquidity to the market, but overall they're causing harm by reducing the returns of legitimate traders.

zacharyvoase3• 14 years ago

Define ‘legitimate’.

dllthomas• 14 years ago

If they're adding a lot of liquidity, I'm happy with them reducing my returns a little. If they add a little liquidity, I'm not happy with them reducing my returns a lot. There are questions to be answered that we can't get at from an armchair (at least, the metaphorical armchair).

mark-r• 14 years ago

If there are people making money off those HFT trades, then somebody else is losing money. It's a persistent tax on the entire market. Just because it's in the background where you can't see it doesn't mean it isn't affecting you.

yummyfajitas5• 14 years ago

If there are people making money off those barbershops, webapps and grocery stores, then somebody else is losing money.

damoncali4• 14 years ago

They shut down the markets. That's bad.

sprobertson• 14 years ago

If computers are running the markets, and the markets have such a large influence over society, does that mean the computers have already quietly started their takeover?

angersock• 14 years ago

what does that even mean go away

zacharyvoase3• 14 years ago

When I was in high school, my math teacher instilled me with the value of always labeling axes when plotting graphs. This GIF provides us with exactly no information on anything, but it sure seems scary enough. The line is getting higher and more erratic! This can’t be good!

Let’s piece apart some of the language here:

* “High-Speed Trading Bots Go Berserk” — doesn’t mean anything, they’re processes, they’re just following their algorithm.

* “…unnerving the financial markets.” — doesn’t mean anything, ‘markets’ don’t have nerves. In fact, I once wrote about the dangers of reification: http://blog.zacharyvoase.com/2012/03/31/reification/

* “activity becomes much, much more frenetic and erratic” — to a human observer. These algorithms don’t have concepts of ‘frenetic’ or ‘erratic’; neither does the universe.

* “…so complex that its potential systemic repercussions are literally unknowable” — suggests that systemic repercussions of anything were ever ‘knowable’. If they were, we wouldn’t have markets. c.f.: http://mises.org/daily/2608/

Don’t just read. Think. Thanks.

jobu2• 14 years ago

Based on the Nanex article (http://www.nanex.net/aqck/2804.HTML), it appears the chart is showing thousands of quotes per second on the left, vs time of day on the bottom.

mvzink• 14 years ago

Yeah, but you have to admit that this is pretty skilled sensationalist journalism :P

How convenient Nanex didn't include labels in the graphics they provided! [1] In fact, they specifically say that high frequency trading doesn't concern them, and are talking about quote spamming. Not. The. Same. The animated graph used in the article combines metrics of both, as far as I can tell, but Nanex makes it clear that High Frequency Trading is stalled or declining while High Frequency Quoting is on the rise.

Furthermore, the bit saying "activity becomes much, much more frenetic and erratic" is just... totally silly. Partly because of what you mentioned, but also because... well I don't know what else to say, it's just so utterly, embarrassingly silly. I'd wager the graph's movement isn't even nearly as 'frenetic' as the stock market itself.

In any case, it's quite clear that the author didn't even peek at the page he got the graph from.

Ugh. This kind of 'journalism' makes me sick.

[1] http://www.nanex.net/aqck/2804.HTML linked all-too-subtly in the article

coolgeek• 14 years ago

> ‘markets’ don’t have nerves

Seriously? Market participants certainly have nerves, and it is perfectly clear to anyone without an ideological axe to grind that the term "market" is used here as shorthand for "market participants".

Don't just write. Think. Thanks.

TomGullen• 14 years ago

If the chart is showing frequency of trades it's just showing that there are more trades right?

I find it a little misleading that the graph is presented as somehow being connected to market instability. If they are correlated, they haven't demonstrated the relationship clearly to me. Am I misreading this?

Also I fail to see how someone else's algorithm going nuts could impact regular Joe traders as the article also suggests. If a stock price starts plummeting because of a bug in a trading algo goes haywire, a savvy trader would look to their news sources and realise nothing has happened. Or if they are passive investors probably by the time they mobilise to panic sell it's already corrected itself.

ShabbyDoo• 14 years ago

I'd like to see a graph of average spreads between 2007 and now.

ck2• 14 years ago

I don't know stocks but this strikes me to be the same mindset as domain tasting and should be ended entirely, not regulated, just killed, regardless of any claims of industries being built around it - it's just wrong.

kerryiob• 14 years ago

Most of comments here are missing the big picture.

Most of the general public and individual investors do not have clear insight into how HFTs work, or if they are beneficial or not. Most are inclined to be scared of crashes and erratic behavior traced back to HFTs. Trading off liquidity for a loss of trust in capital markets is a bad deal.

When investors lose trust in the system, it all falls apart.

retrogradeorbit• 14 years ago

Tech journalist discovers zerohedge. They ran this story back in February.

http://www.zerohedge.com/news/presenting-rise-hft-machine-vi...

Watch High-Speed Trading Bots Go Berserk

Related Stories