hft_throwaway

https://gigaom.com/2012/01/16/sarah-lacys-pandodaily-launche...

http://techcrunch.com/2013/05/23/lyft-a16z/

http://www.cnbc.com/id/102008782

That assumes there is no bias to their signal returns over time, which is extremely unlikely. 90% of the moves that go their way could happen in the first second, for example.

Many HFTs hold positions for seconds or minutes, and predict prices out over similar time horizons, but they'll lose their best winners to competitors if they aren't fast and adept at executing.

More than when I last looked, but to put that in perspective, you could trade $40mm in a few price ticks in major currency pairs and CME avg daily turnover alone is > $100bn: http://cmegroup.mediaroom.com/2013-06-07-CME-Group-Sees-Reco...

Public spot FX markets and OTC trading do huge volumes too.

Some BTC exchanges charge like 60 bps per trade. Finding a signal to overcome that cost and do enough trading to make it worthwhile would be quite difficult.

A simple signal would be more shares bid than offered at the inside market. Imagine a market with 1000000 shares bid at $4 and 100 shares offered at $4.01, it's more likely to tick up than down in the very near-term. This isn't tradable though since the predictions aren't large enough to overcome the spread and you can't expect to join the 1000000 share bid and trade when it's good (the 1000000 shares that made you think it was going to move up all have to trade or cancel for you to trade, not looking so great anymore).

Real HFTs use other features of the order book, movements in related products (maybe a move in oil futures in the last minute impacts the price of an airline stock in the next 10 seconds, etc.), and so on. The signals are not really that complicated, but competitors eventually converge on knowing the same signals and compete them out of existence/profitability. Also, how you execute around your prediction matters just as much if not more.

http://queue.acm.org/detail.cfm?id=2534976 and http://queue.acm.org/detail.cfm?id=2536492 have some more examples. http://www.decal.org/file/2945 also has some ideas for old signals that probably won't work anymore.

I see no mention of the spread, costs or even ability to put on a short position, exchange lag variability (huge issue when simulating even on modern exchanges, let alone fly-by-night bitcoin markets). Additionally, it is very easy to find trend-following signals that "work" but break down when conditions change very quickly. That seems to make up most of the prediction, along with an order book imbalance signal that might be more stable.

I think a better approach would be looking at order book features and lags vs. other markets. The costs are high on BTC markets so it would be pretty tough to overcome those though. Anyone with experience to do this is probably doing it somewhere more lucrative. I think BTC markets only trade a few million USD a day.

I don't believe they were trading with themselves. They were accumulating stock in the opposite direction to hedge their imbalance-only order. However, the way they accumulated that stock was done in a way that would impact the price received on the closing order.

So imagine they sold 10000 shares imbalance-only in the auction at 3:50. They would then buy 2500 immediately, slowly pick up another 2500 in the next 9 minutes, then do the last 2500 in the last second. The initial trades would get them into their hedge at a low average price, and the last-second ones would push the price of the closing auction up, maximizing their return on their imbalance-only order.

If you read the case, it seems like this didn't always work if they had slippage on the close or they got so aggressive that they would "flip" the imbalance and not get filled on their closing order at all, instead being stuck with a bunch of shares from crappy prices. The people trading with them in the last two seconds could also keep replenishing their orders and prevent them from spiking the price. Sounds like a dangerous game of high-stakes chicken.

Matt Levine's Bloomberg article is a good layman's explanation of the issues: http://www.bloombergview.com/articles/2014-10-16/high-speed-...

They were wrong and got punished, but the market would have punished them eventually anyway, and it sounds like it was already doing so. The SEC filing states that this scheme lost them $3mm on an index rebalance day when activity in the closing auction peaks. That should be a great opportunity for anyone intermediating between the continuous and auction books, so it is very surprising that they'd lose. Someone probably caught on that they were manipulating the close and waited for a big day to take advantage of them, or there was so much activity that they didn't have enough ammo to goose the price like on slower days.

Also, to be clear, there is nothing illegal about HFTs intermediating between the continuous and auction books or the majority of HFT activity. These guys were charged because they committed fraud, the use of a computer was ancillary.

It is stuffy, but I wouldn't call it backward-thinking, unless mindless hedonism is what passes for forward-thinking these days. Raising children without marriage would be akin to starting a business without incorporation or contracts. Strong bonds, group loyalty, and long-term planning/character are what let man build something larger than himself.

The claim that market makers pass costs on to end users is only true if they have pricing power. In reality, on-exchange liquidity provision is basically the kind of perfect competition that only exists in economics textbooks. Market makers are selling a commodity product (you don't care or control who you trade stocks with) in a market where buyers are purely sensitive to price (tightest market always wins and is enforced by exchange matching rules).

So what actually ends up happening in a market with multiple competitive market makers? To make money, a market maker needs to trade a lot of volume. The only way to trade a lot of volume is to put up the most aggressive (worse for the market maker, better for end users) prices at any time. Market makers can only do this by charging a smaller spread than their competitors. They can only charge a smaller spread by either reducing their margins or getting smarter at deciding when to be in or out of the market, usually a combination of both. The end result is extremely tight markets that react to information very quickly (i.e. cheap to trade and very efficient).

Competition keeps markets honest. If you had one very fast guy, he would clean up, but when you have a dozen guys who are roughly equally fast, they all compete one another down to barely making profit above their cost of doing business. Only the most efficient can survive. If anything, we want more HFT by removing barriers to entry rather than creating a lot of regulations that would ironically help incumbents by killing off weaker competitors.

That's not what's happening here. Traders are arbitraging and reacting to public trades and orders on multiple markets.

If you walk through a physical market where 8 apple carts are lined up, all selling apples for $1, buy every apple at cart #1, then buy every apple at cart #2, and so on, would you be surprised to find the price moving up or sellers stepping away as you approached carts #7 and #8?

The same thing happens when trading. Securities trade on multiple markets and multiple exchanges cannot match cross-market trades atomically. It's absurd to suggest that one side of the trade should be expected to close his eyes to what's happening in the world around him and sit tight while a huge trader runs his quote over. Why is one party more deserving of a good price than the other?

If you route to one exchange only there is no way for anyone to see or react to your marketable order before it executes, ever. If you route your orders intelligently, it can be very difficult or impossible for anyone to pull away before you get your fills. That's the executing broker's job. Instead of getting better at his job, this broker would rather complain to a very vocal conspiracy theorist who has been proven wrong many times in the past by people with actual experience and data: http://zacharydavid.com/bad-research/the-hunsader-follies/

Cool idea!

Some questions/thoughts:

Do you provide any access to raw tick-by-tick event data or just bars?

How do you secure your users' scripts so they don't worry about you stealing their ideas or front-running them?

It might be useful to provide FX rates as a feed so users could come up with a synthetic price for non-USD pairs.

For guys who want to work passive orders, an API where they can pass a dictionary of prices/sizes they're willing to buy/sell at into a goal seeking algorithm that handles canceling/placing individual orders might be a better match and more convenient.

That should create opportunity for automated trading. Identify large orders and place small orders a de minimis amount ahead of them when the bid-offer spread is sufficiently large. If someone trades with your bid (offer), try to be the best offer (bid) to exit and make the spread. If the large order behind you starts trading, trade with it to lose 0.01.

I've traded these markets by hand before using relative value signals from other exchanges (e.g. buy on bitstamp if btc-e moves up and bitstamp hasn't yet). You can't really do pure arbitrages since it takes so long to move money around but if you can identify which market leads you can use that as a starting point for coming up with a fair price to trade around. I made money easily but gave up on trying to automate it since the volume is so low, bid-offer spreads are often tighter than 2*trading fees, you can't short-sell easily/cheaply, exchanges seem incompetent or downright crooked, laughably bad APIs, etc.

These markets are wildly inefficient though. For a home day-trader or hobbyist there's easy money to be made. It's absurd how slowly price discovery trickles from one market to another. Most modern markets don't even get mispriced by a penny for more than a few microseconds, but there are nickels, dimes, quarters and even dollars of mispricing to capture in BTC/USD markets clicking on the screen.

You could probably dominate with an algo, but how much can you really make in a market that trades 10s of millions in notional value a day? There are also way too many exceptional conditions to let it run even semi-unattended since the exchanges are too immature. Anyone who's capable of doing it can make more doing the same thing on a real market.

ETA: Exchanges in this space are really, really bad. Some don't even know about rounding errors when using floating point for prices. Or would you trust trading on a market that can't even match trades as an atomic event? http://www.reddit.com/r/Bitcoin/comments/1r4d6t/bitstamps_st...

It likely was happening.

If I'm a market-maker, I want to trade with retail orders or slowly sliced institutional orders that are trying to rebalance a portfolio throughout the day. When I sell stock to them, the price doesn't move very much in the short-term and I have a chance to buy back on the bid to make a spread. I don't want to trade with aggressive arbitrage or stat-arb traders capturing short-term price inefficiencies, hedge funds that want to buy huge blocks of stock all at once, etc. If I sell stock to them, the price ends up moving up more than the spread very rapidly and I lose money.

When trading on a public market, I have no way to control which of these groups I trade with. I have to employ countermeasures to predict whether it's likely that I'll trade with an informed or uninformed trader and adjust my market accordingly. This is why you see market-makers rapidly flash their quotes for seemingly pointless reasons (e.g. I may adjust my prices in an airline when oil futures move) or flip out of positions aggressively when their model believes they were hit by an informed trader.

Now if I'm able to trade on a market where I know I'll only trade with long-term fundamental traders, I can quote much larger, tighter, and more stable markets. This is good for the institutions since their execution costs are lower than going to the exchanges. It's bad for arbitrage traders and hedge funds with short-term information since they effectively pay higher execution costs in a tiered market (if more "good" order flow goes to dark pools, less reaches displayed markets, and spreads on displayed markets will increase to compensate). When people want to end dark trading or internalization, they really want to have retail and institutional flow cross-subsidize fast speculators.

Ironically, it's probably HFT market-making firms who were the most impacted by Barclays giving other high-alpha HFT firms manual overrides to put them in a lower tier. If anyone would be excited at the promise of removing high-alpha traders from the pool, it would be market-makers who want to avoid being "picked off", not institutions who have a much longer time horizon.

But isn't that because C++ itself evolved from C? The original "C With Classes" wasn't much more than a convenient syntactic sugar for using structs and function pointers as a poor man's object system, something that was and is common in large, abstract C programs (e.g. Linux's VFS layer, Gtk+).

I think you may be confusing is with ought here. Everybody sane knows C++ has flaws; I think Meyers even feels this way in his presentation. C++ is popular because it evolved and thus contains a lot of compromises. To draw an analogy, the Mormon Church is immensely popular in the US for similar reason--they basically said, "Hey the New and Old Testaments you spent your formative years learning are all good, but we've got some swell new stuff here too!" I'm sure Zen Buddhism is more theologically pure, but people like what's familiar to them.

Programming languages are a network effect problem first and foremost. Just like it's hard to unseat Craigslist despite its crappy UI, it's difficult to get users for a new language that may be only marginally better in terms of features, productivity, safety or convenience. Even if it's a significant improvement existing code bases, library availability, programmer availability with domain knowledge and so on matter much more for serious projects. There's a reason why Facebook took the time to write a PHP VM instead of rewriting their code.

Guys who write HFT systems in Java are basically programming them like one would in C or C++. They pre-allocate byte buffers for everything and never run the GC. If you write C-style code in Java, it's going to run pretty quickly. They aren't doing architecture astronaut AbstractMetaClassFactory stuff or using much of the provided libraries for latency-sensitive code.

FWIW, I don't think Jane Street is a competitor in the "ultra HFT" space where every nanosecond counts. AFAIK they are more of a statistical and quantitative trading group, so they may have less need for things like talking directly to hardware, keeping tight control over memory layout, deterministic latency w/o GC pauses, etc.

Almost everything that is popular has evolved over time, and making these changes once you have established users results in compromises that can be sort of ugly. C++ has patina.

Yes. Alinea was a touch more subdued. ;)

I think it's a bad idea for a restaurant. Auctions generally involve "winner's curse" where the top bidder almost always overpays vs. fair value. Restaurants want to give customers a good experience to keep them coming back. Feeling like a sucker leads to buyer's remorse. Getting a "difficult" reservation feels like a prize.

Sure Next tickets may be underpriced at times. I imagine the Trio menu will be a very hot ticket when it first comes out, but pricing so customers get a good value for the duration of the menu is a better long-term play. The guy who scores "underpriced" tickets will feel great about the experience, brag to his friends, and likely return.

Are people who paid $800 for a PS4 happy about their decision today?

Hear, hear.

I hacked up something with Asterisk to auto-dial Schwa until I got something other than the dreaded machine. Alinea is like walking into a fantasy world on acid but eating incredible food with Aesop Rock blasting while Chef swigs Jameson 10 feet away from you was as if some crazy person got inside my head and knew exactly what I wanted.

How many do you have? What is "sufficient" collateral?

If someone has the cash, is able to withstand volatility, and the rates are low enough this trade is a no-brainer. BTCs have no intrinsic value and are propped up by speculators. Aside from speculative interest, they only have value in so much as they can be exchanged uniquely online, but they are just as useful for doing that at $0.0001/BTC as they are at $1000/BTC. Long-term I would not be surprised to see them go to 0 or some nominal amount.

ETA: Odds are nobody will loan you enough at a low enough rate to make this trade worth putting on. It's like how you can short leveraged long and short ETFs and make money due to volatility decay, but nobody will lend them cheap enough for it to be worthwhile.

I see a lot of people make claims like this, or even stronger ones such as claiming it has little/nothing to do with environment. Charts like this bear that out somewhat with the poorest Asian and white students scoring higher on the SATs than well to do blacks: http://1.bp.blogspot.com/-1a52vkpjans/UmjAc5fGxtI/AAAAAAAAA6...

Generally, people I see who bring this up are usually using it as a thin veneer over racist beliefs. When asked what end discussing minority IQ serves, they usually suggest defunding inner-city (read: black) schools or similar measures.

I can't get on board with that. I think the US certainly fails at providing equality of opportunity to many groups, and there is still widespread discrimination against women and minorities. We can certainly do better and we should view people individually rather than treating them in a prejudiced way based on group membership.

I also have a hard time getting on board with a "blank slate" view of humanity. I'd love for someone to prove me wrong so I could fire back at "race realist" reddit commenters, but it seems plausible to me that different groups, especially men and women, are biologically predisposed to certain traits, on average.

I know in my career that requires both a certain amount of aggressiveness and quantitative aptitude that my colleagues have overwhelmingly been Asian or white men. I've worked with women and "under-represented minorities" who've made me feel like a chump trading, and I certainly don't harbor prejudices about them, but they're difficult to find.

It's a Britishism that's a more polite way of saying that: http://en.wikipedia.org/wiki/Tired_and_emotional

FWIW it seems unlikely that this type of behavior would be brought about by drinking alone, especially once the media took notice. I known people who have made idiotic career-limiting moves while drunk (like brazenly hitting on the boss' daughter at a party) and they usually get deer-in-headlights embarrassed once someone wiser advises them to knock it off. Maybe drugs, stress, or some type of medical condition?

In any case whether caused by substance abuse or some other problem, it seems irresponsible to comment on publicly like this. It doesn't even have to be caused by mental illness in a traditional sense. There have been cases of people with brain tumors acting out in wacky ways. It's not really fair to disparage someone who's acting unusually and may not even be aware or in control of their behavior.

Or maybe it's just some bizarre attempt at "going viral", who knows these days?

That's very true in this case. The issue here was that Knight isn't just trading for its own account. They're a broker where they likely have some SLA-ish agreement with clients, or face repetitional risk at the very least. As a registered market-maker they're obligated to quote two-way prices. Shutting down costs them money and exposes them to regulatory risk.

The code was QAed, but they didn't test old and new versions against each other. Version A could accept a flag and run obsolete logic that would lose control of its orders but never sent it, so this problem never happened. Version B sent this flag and the receiver would send RPI orders with it. Put a Version B sender and a Version A receiver together and you end up with a disaster.

From a systems perspective, my takeaways on this are:

-Don't re-use a message for a semantically different purpose in a distributed system where you're running different software versions (even in cases where you don't plan to, really, since you may roll back or end up running the wrong code by mistake)

-Version your messages so anything that changes their meaning can only be accepted by a receiver that follows that protocol

-QA old and new builds against one another

If you really want to look at the root cause of this, it's cultural. Trading desks don't want to spend development time on things that don't generate PnL. Traders want to try lots of ideas so many features are built that don't get used. Code cleanup gets put on the back burner. Developers do sketchy stuff like re-purposing a message field because it's annoying or time-consuming to deploy a new format. If traders aren't developers themselves, they may underestimate the risk of pressuring operations & devs to work more quickly.

Things like this are probably the biggest risk faced by automated traders, and the good shops take it very seriously. I've never been scared of any loss due to poor trading, but losses due to software errors can be astonishing and happen faster than you can stop them.

You know that saying about finding yourself in a hole? A "close out my risk" button is fine for situations like losing money or whatever, but if you have no clue what your orders, trades and risk even are, the only sensible thing to do is stop making it worse.

I think you are right on the money and most traders do use techniques analogous to the ones you describe when designing models:

http://www.decal.org/file/2945

"Alpha is often nothing more than taking commonly available data and mathematically encoding it in a signal correctly. Correct often means something as simple as using an rate of change instead of a difference, normalizing a value, smoothing a chaotic signal with an EMA, using a heteroscedastic weighted linear regression instead of a simple regression, or handling all numerical errors or edge cases."

It's not always easy to get this completely correct in every situation and traders face a reward function that doesn't necessarily reward correctness so much as avoiding Type II errors.

Because the market has many actors. If I try to push the market down to shake out stop losses, another trader may say, "Hey, this price move doesn't make sense considering recent activity, how the broad market is moving, etc., I'm going to buy more" and prevent me from doing so. Likewise if I'm shifting the depth with a big false order. I might induce some actors to do bad trades, but another actor might see my order as a great opportunity to trade in size and trade against it, so I'll take a loss.

Liquid markets with a diverse set of actors are more resilient to manipulation attempts. It's hard to do unless you have more capital or are willing to take risk that other actors in aggregate are not.

Misuse of language by opponents of automated trading is intentional. When presented with examples of beneficial computerized trading like automated market making, they throw their hands in the air and say, "oh no, when I say 'HFT' I only mean these bad kinds":

"The practice of spoofing, or sending in fake orders in order to gain information about what other investors, traders and other algorithms are doing, is the corner stone of most High Frequency Trading strategies (HFT)."

http://www.nanex.net/aqck2/4371.html

I think it is more short-term day traders taking money from naive HFTs. I'm sure some people automate this type of manipulation, but I seriously doubt that any of the "big name" market making firms do it. They are under a lot of regulatory scrutiny, make more than enough through legitimate trading, and have the technology & data to enable their internal compliance auditors to detect it.

I also don't think it can make that much. It may work for a little while, but no algo is going to sit there bleeding money to you forever without hitting a risk limit or being reviewed/adjusted. If your business model depends on somebody else doing something dumb, it's not going to work for the long-haul.

But even they admit that a market flashing between 100.01 offer and 100.00 offer is sometimes beneficial for them. It says their execution algo will attempt to "pick off" the 100.00 offer with IOCs if they are representing a buyer.

Flashing orders with the intent to manipulate is wrong and is illegal already. It doesn't matter if you do this by hand or with a computer. Putting up a tighter best price, even if it doesn't stay put very long, can only be beneficial. At least some of the time, other traders will get filled at a more advantageous price than they would otherwise.