How I made $500k with machine learning and high frequency trading

http://jspauld.com/post/35126549635/how-i-made-500k-with-machine-learning-and-hft

by jspaulding • 14 years ago

551 308 14 years ago

jspauld.com

jspaulding38• 14 years ago

If anyone has questions for me happy to answer as best I can.

fr0sty4• 14 years ago

Wouldn't the term "Statistical Arbitrage" be a more apt description of what you were doing?

If you are relying on a broker-supplied pricefeed over the internet you are far outside the real of what is traditionally understood as "High Frequency Trading".

jspaulding38• 14 years ago

It wasn't anything over the internet. I had a server rented at my broker who were situated close to the exchanges in Chicago and had direct lines. For sure I was not the fastest but only behind by a couple milliseconds perhaps.

kamaal2• 14 years ago

Do you know of any resources/books to learn trading/HFT in general?

jcromartie• 14 years ago

I have one question:

Why doesn't every hacker do this to make extra money? Is it within the grasp of anybody who can program to automate trading like this?

EDIT: Sounds like it's not really for everybody. You have to own or rent a server with access to direct lines to the exchanges, or else your lag will be such that profiting from HFT is impossible. How much do these cost?

amalag2• 14 years ago

I tried at one point to do it with Ninjatrader. The programming skills for the trading software is not complicated. What is complicated is tweaking it so it will make money, there are tons of indicators out there and many people have tried this with neural networks and the like. A lot of effort is put into it. I personally thought there were smarter people than me who barely had an edge. I honestly didn't think it was within the grasp of a single programmer nowadays, but this author has proved me wrong. He of course has much more sophisticated algorithms than what I was attempting.

jetti2• 14 years ago

"Why doesn't every hacker do this to make extra money? Is it within the grasp of anybody who can program to automate trading like this?"

I have flirted with HFT in a hobby-like manner and it isn't the programming that will get you the money, it is the domain specific knowledge coupled with the programming. On top of that, there are quite a few risks and potential to lose a lot of money.

beagle318• 14 years ago

> You have to own or rent a server with access to direct lines to the exchanges, or else your lag will be such that profiting from HFT is impossible.

A profitable predictor is a much, much harder problem.

At a place like Goldman Sachs, a quant with a working predictor gets paid 5 times as much as the IT guy who makes that predictor talk to the market quickly enough.

Because, as your question implies, it is (relatively) easy to do the IT work or hire someone to do it. Not so for the predictors.

hooande• 14 years ago

Because it's gambling. Some are better gamblers than others, but no individual can consistently have more ups than downs over a period of years.

"I was making a lot of money but now I've stopped" is the same thing as "I was lucky until I wasn't". Making a living by gambling pretty much sucks, which is why most hackers don't do it.

(You'd think that something as complex as markets would attract hackers trying to "figure it out". The problem is that due to the changing nature of the other participants, all hacks are temporary. Makes for great blog posts, but not a long term strategy.)

pnathan• 14 years ago

You have to have the capital for the server and access to the data feeds, as well as time to burn. Also, you need to find finance interesting enough to spend time with it.

vellum• 14 years ago

1.)Getting a server close to the exchange

2.)Finding a good predictor. Predicting up or down is easy on paper. Finding a predictor that a.)beats the spread, b.)factors in lag time to execution, and c.)factors in commission, is hard.

3.)Market regime changes. What works a few months ago isn't guaranteed to work now.

juiceandjuice• 14 years ago

To really be able to do this with a lot of success, you need to be able to do intraday trades, which would make you a day trader under SEC rules, which also requires $25,000 in your brokerage account.

incision3• 14 years ago

>Why doesn't every hacker do this to make extra money?

I wondered the same thing and tried to answer it for myself not that long ago.

In short, it's hard, time-consuming, stressful and costly.

1) Your code has to work going forward. Coming up with something that works in backtesting is easy, doing it moving forward while staying within your risk envelope and considering all the associated costs is hard.

2) Closely related to #1. Whatever you put together is surely going to need plenty adaptation and oversight. It's a very fluid problem, you're just one player among countless others. How much time do you have in the day? Where is your capital coming from? For most people, the source of capital is a job that they probably can't ignore while fiddling with a trading program that would have to be remarkably successful to replace that income.

3) Everything costs money. You need a certain amount of capital to start with and there are all sorts of running costs. From the basic costs of execution, to market data and eventually co-location if you get that far. The fact that the OP kicked off with an amount that's essentially the barest minimum for any kind active trading is exceptional.

4) Can you maintain discipline and continue executing on all of the above while losing money? How many people struggle to simply get up on time, control their desire to check a particular website or to get over and move on from some frustration? Even if there's automation involved, trading will test your mental fortitude.

dude_abides• 14 years ago

Thanks for the post. Is this the software you used?

http://www.ctsfutures.com/t4.aspx

Would be even more awesome if you could share some code on github.

jspaulding38• 14 years ago

Yes, however, I forgot to mention that pretty early on I converted my program to use an API from https://www.tradingtechnologies.com. It's odd, but I can't remember exactly why. I think it was simply because I found a broker who could offer me a lower commission rate and they only supported TT. With regard to posting code yes I may do that. We'll see.

mej102• 14 years ago

Do you think you got lucky or that your skill as a trader made you this money? Or rather, which of these do you think mattered most?

jspaulding38• 14 years ago

1) Being connected to friends who taught me a scalping style of day trading 2) Being really good at designing algorithms. 3) I only wish I could have started automated trading back in 2001

retube• 14 years ago

why do you think your technique stopped working over time?

psionic_• 14 years ago

I wonder if running your program nowadays can have same successful results as two years ago.. anyone is going to try? :)

mgl3• 14 years ago

How do (did) you cope with increased stress level? Trading futures, especially in a an automated way, can easily drain your margin unless the algorithm is really well tested for edge cases.

jspaulding38• 14 years ago

My automated program was much less stressful than trading manually. The best was going to Hawaii, waking up, and having the entire day done. I was making like 6k every day on that vacation. The best!! :)

stickydink2• 14 years ago

Looking at your first chart there, is there a reason (other than market conditions) you were making significantly more at the end of '09 than mid '10?

jspaulding38• 14 years ago

To be honest I don't know exactly what happened. My theory is that over time more and more market participants started integrating the types of analysis I was doing which rendered my program ineffectual. It's a pretty normal pattern that there is some inefficiency in the market and over time it disappears.

mgl3• 14 years ago

It's possible that your algorithm is sensitive to market volatility.

stickydink2• 14 years ago

Being a machine learning program, how much of it did you tell it to forget?

Were you compounding the data always, or telling it to forget what was going on several months ago? Or somewhere in between?

(I'm pretty unfamiliar with machine learning, apologies if this is obvious or something)

wtvanhest• 14 years ago

You said "Growth stopped" but your P&L shows negative growth approaching zero. Would it be more fair to say that your profitability turned to zero?

jspaulding38• 14 years ago

To be honest I'd given up and moved on to other stuff well before I actually shut down the program.

nrmn• 14 years ago

Could you comment on how your "curve fitting" algorithm worked? Did you end up with an equation for each curve? Im working on something that requires curve fitting and any kind of tip would be helpful.

jspaulding38• 14 years ago

I basically just brute forced it. I came up with a cost function which would measure the difference between a possible curve and each data point. I think you're supposed to do the squared difference but I can't remember if I did that. With a cost function in place it's just a matter of zooming in on variables that minimize the cost function.

davidw2• 14 years ago

Looks like it's written in C# - is that correct?

binxbolling• 14 years ago

Doesn't the post explicitly say it was?

jetti2• 14 years ago

It stated in the article it was C#.

"In 2008 I was “manually” day trading futures using software called T4. I’d been wanting some customized order entry hotkeys, so after discovering T4 had an API, I took on the challenge of learning C# (the programming language required to use the API) and went ahead and built myself some hotkeys."

davidw2• 14 years ago

Yes, that's what I was going off of. I was curious about that aspect of it, if it had stayed C# or what.

amalag2• 14 years ago

What contracts did you trade? Looks like you did futures contracts? Did you look into forex at all, or was this strictly equites.

jspaulding38• 14 years ago

I was trading stock market index futures: Russell 2000, DAX, and a bit of Nasdaq. I experimented with other stuff as well.

pdog• 14 years ago

Would you be able to open source any of the code behind your trading system? Maybe not the "secret sauce", but it would be interesting to see how you processed the data feeds, modeled the data, entered orders, etc.

jspaulding38• 14 years ago

I certainly could open source it. I may just wait a bit on the off chance that somebody wants to purchase it.

Juuumanji• 14 years ago

that would require some proof that it works today.

Evbn• 14 years ago

1. Hook

2. Line

3. ...

Profit!

sturadnidge• 14 years ago

Relax, nobody is buying your program.

Other than sheer luck, the most plausible explanation for your diminishing returns is that you found a strategy that worked _at that point in time_, other people copied it (starting with your broking firm), and as that strategy became more common your ability to make money disappeared.

I work in the industry, this happens all the time. Trading strategies have a shorter half life than you may think.

tarr112• 14 years ago

This is just glorified gambling. I am not sure what special insight or advantage he had, other than his own model. Every trader has a model.

It could have easily been called "how i lost 500k with machine learning". Like gambling, it's easy to manipulate statistics to show that you did well in some period of time.

I worked for a large investment bank about 10 years ago, writing trading programs for quant traders who were market makers. The quants called guys like him "retail" investors and they gleefully picked off all those trades. It's how they made all their money.

So, everyone else, beware of making this a case study in how to make lots of money really fast. You are more likely to lose money.

marshallp2• 14 years ago

I don't get that. It would be true if he just made a few trades, but the author claimed to be making 2000 trades a day. Over a period of months winning that wouldn't qualify as blind luck.

DanBC• 14 years ago

>Over a period of months winning that wouldn't qualify as blind luck.

Why not?

tatsuke95• 14 years ago

Definitely not blind luck; but boosted by the fact that SPY was on a massive bull run at that time.

This is really cool, any way you cut it.

mgl3• 14 years ago

It's more a risk management game, not particularly gambling, just look at EUR/AUD: http://stooq.com/q/?s=euraud&c=3y&t=l&a=ln&b...

Would you dare to "predict" the direction of this FX rate movement in the next month? Then it's a matter of calculating potential profit/loss factor and adjusting your trade value. And yes, as with any high risk investment, putting all your eggs in one bucket is not a brilliant idea. Just like taking all your savings to Vegas.

washedup2• 14 years ago

I disagree. The point of the article was the show the steps required to develop a statistical advantage in the market place. If you develop a robust model and are very diligent in how it executes and learns, you can be successful. However, what you say about market structure is true. It goes through periods of stability, followed by abrupt changes. Any model that a trader has developed has been developed on such a short time-scale of market activity, that it can turn out to be a bad sample size. Depends on the scale of time and trades.

jspaulding38• 14 years ago

I tried to address this concern at the start of my post. If you have some idea of how I manipulated the statistics I'd be happy to respond.

Having said that I can agree that my case is pretty unusual and that everyone should beware of attempting to do something like this. Even for myself I couldn't do it now. (There is a reason I turned my program off.)

147• 14 years ago

Why couldn't you do it now?

tarr112• 14 years ago

Ie., if you made 500k in one period, but then lost 50k a year for the next 10 years, you've made a net profit of 0. All I know is that you had one good run, similar to how some mutual funds have a good run for a while.

How much did you spend before you "tuned" it? How much did you spend afterwards? What were the tax consequences of your trades? Did you make exactly 500k? Have you traded at all since then?

You mentioned that you occasionally "sat in" and took some large losing positions. Were these on purpose? Bugs? Was your exposure actually much higher than you thought? Was limiting contract size enough risk management?

yajoe2• 14 years ago

This. I love crazy projects and Show HN's until the cows come home, but this one is dangerous that I must repeat the warning to others.

I cannot emphasize how important it is to understand that people who trade using price action (http://en.wikipedia.org/wiki/Price_action_trading ) are just speculating based on where they expect the price to move. It's no different than people who play Texas Hold'em online and speculate what cards others have based on betting patterns. If you get good at spotting the patterns (like this guy did) you can go on a winning streak, but when the game changes (as it did for this individual after 2009) then you either go home or go broke.

This guy found one edge in 2009. It won him 500k. Fantastic. More than any edge ever won me. But, the market has changed so much since then, with HFT becoming so prevalent (http://www.theverge.com/2012/8/7/3226187/high-frequency-trad... ) that please be careful before you follow this course. His code is unlikely to be worth much today unmodified, and when you modify it you'll realize, as I have, that when the other players have access to the order books and can jump the line you have no chance in the game in 2012.

One last nit: Please, please post recent data when you talk about projects like this. 2009-2010 is 3 years ago. Since then there was significant turmoil in the US, Asia, and the EU. How are these returns relevant for today?

beagle318• 14 years ago

The Instagram guys found an edge. It won them 730m. Fantastic. More than any edge ever won by me. But the market has changed so much since then, please be careful before you follow this course.

You are not wrong, but what you wrote here is applicable to any success story posted on HN.

Caveat lector. Always.

DennisP• 14 years ago

This is a bit off-topic, but it's actually quite feasible to get a real edge in Hold'em, and it's not just about spotting other people's patterns.

To start with, there's simple probability: knowing the odds of making you hand vs. the payoff in the pot, or the chance of winning with various starting hands. This is pretty basic but a lot of low-stakes players screw it up. If you get it right, their mistakes are your gain.

At a more advanced level, game theory comes into play, using bluffs and so on. The game is complex enough that it's not completely solved, and it's an active area of research. The University of Alberta is doing a lot of working developing poker bots using game theory. By playing a good strategy, you can prevent other players from exploiting your patterns.

Only after you've got a good grasp on all that should you really think much about exploiting a particular player's weaknesses. The Alberta guys are doing work on that part too. Exploitative play can improve your profit but also makes you more vulnerable.

For a good overview of this stuff, the book Mathematics of Poker by Ankenman and Chen is a good place to start.

I agree though that HFT is awfully competitive these days. If I had to choose between the two I'd play poker.

jspaulding38• 14 years ago

Hey, I didn't actually intend this to be a course. I do not make any money in the market right now so am certainly not qualified to teach a course on it. And of course, if I was making money in the market I wouldn't have posted this at all. So please everyone remember that. These comments have made me realize it's probably for the best if I do not post the source code. Basically you are competing against armies of PHDs who are buying buildings next to the exchange so they can get their executions slightly faster. It is indeed surprising to me that I was able to make money in the first place. But I do know for a fact that I did make money and I also know that I was not at risk of losing a bunch of money. As mentioned the most I lost in one day was $2000. That's all I was risking.

pja• 14 years ago

If you make that many trades and your total market exposure at any given moment is small yet you consistently make a net profit then you've found an edge. At which point it's not really gambling any more, it's just making money!

If your net exposure is small, but that's only because you're offsetting various positions then you're probably picking up nickels in front of the volatility steamroller & if you stay in the market long enough you'll get squashed at some point.

beagle318• 14 years ago

It is only gambling in the sense that any business is gambling: Your customers might stop coming tomorrow because the fad wore off, or a competitor provides a better/cheaper/hipper alternative.

(And indeed, living is gambling. It's all just a matter of the risk/reward portfolie).

But jspauld has apparently made $2/trade after fees on 250,000 trading, with a very small standard deviation (I would guess less than $2/trade) - which makes it one of the best businesses one could ever have.

You can't live without gambling - by e.g. going to be a salaried employee for Yahoo rather than Google or that weird newfangled "TheFacebook" thingy back in 2004, was a gamble.

jspauld, statistically speaking, has made less of a gamble there than almost anyone else posting on HN.

> So, everyone else, beware of making this a case study in how to make lots of money really fast. You are more likely to lose money.

True. But that's true for every single success story posted on HN, reddit, or USAToday.

martinced2• 14 years ago

Exactly. I've played millions (!) of real-money online poker hands and won some money in the process (not anywhere near what most people who've played that many hands did).

Most people don't understand that, when you're able to recognize patterns, playing millions of hands while never exposing more than 1% of your bankroll on any deal is not "gambling" but "printing money" (a tiny amount of money in my case compared to consulting but that is not the point).

At the same time the very fact that obviously (seen most of the posts here) most people don't understand basic bankroll management, risk management, standard deviation, expected value, variance, etc. means there are probably quite some opportunities out there to make money for those who do understand that ; )

OldSchool3• 14 years ago

Very true, every time you hire someone or spend money on Adwords you're gambling that you'll net more in $$$ or at least lifestyle improvement (in the case of a new employee) than your cost. That is, if you're in business to make a profit, not just spending OPM to build your brand. Even acquiring a customer is a gamble.

gweinberg• 14 years ago

No. Every business has a risk element, but what makes this gambling is that there is no good or service being produced. It's a game of trying to outguess the other players, with one trader's gain being another trader's loss (relative to market returns).

Because there's a commission on trades, and because you pay taxes on net gains but your minimum tax is zero, high frequency trading by its very nature must a loss for most players.

motters• 14 years ago

Correct. The far more common story is "how I lost $Xk on the stock market". Probably most people with programming/AI skills have tried their luck with the stock picking problem at one time or another. I have a few times, but only in simulation. Even using very elaborate machine learning methods and a lot of training data, making money from automated trades is a difficult problem, and my impression is that it's very much like betting on horses or football games.

agilebyte• 14 years ago

How is it like horses and football games? Don't the latter have a lot more people playing with their emotion rather than utilizing an algorithm? Something that you can take advantage of?

martinced2• 14 years ago

It could have easily been called "how i lost 500k with machine learning".

If you've really worked in that field than it's very surprising you've never heard about what professional poker players call bankroll management (and they "stole" the concept from professional traders).

The whole point is that you can --either if you gain an edge or get lucky-- win big. Very big. But you're never exposing a large part of your funds in the process.

Maybe OP had an overall "stop loss" at, say, $10K. Had he had five minus $2K days in a row at the start, he'd be out. He wouldn't be broke. He wouldn't be without a car and without a bank account. He just would have lost $10K.

But there's no upper limit as to how much you can win.

All you need is discipline and sound bankroll management.

And, yes, I've won a five-digits figure (hence not anywhere near what OP did) real $$$ at online poker. Starting from $0.01/$0.02 small blind/big blind tables and then working my way up using bankroll management.

It's assymetric. You're foolish if you think that succesful traders who won $x were as likely to lose $x. This is simply not how it works and it's very well explained in OP's article. He's detailing what his maximal daily exposure was and it was tiny compared to what he made.

jspaulding38• 14 years ago

I have some friends that have made a lot of money playing poker. The analogy is very good. But, in poker you have to be willing to deal with much bigger ups and downs than what I had to deal with. You can hit a bad streak and it can hurt - even if you are in fact a really skilled player and do in fact have an edge.

With my program I didn't really have bad streaks because my P&L was averaged out over thousands of trades per day.

incision3• 14 years ago

Well put.

Risk management is probably the single most important thing to understand in trading. Unfortunately, it's something that lots of people learn late, if ever.

Folks get caught up in the romantic notion of betting it all and winning big, but end up losers. Meanwhile, the consistent winners they aspire to be are exposing perhaps 0.2% of their roll at a time.

halcyondaze• 14 years ago

Yes this is the point I was going to write myself. I played online poker for four years and won over $100,000. I wasn't very good compared to the top 5% of players at my stakes, but I was much better at bankroll management, tilt control, and all of the other soft skills. The way I structured my bankroll made it actually impossible to go broke as well.

smrtinsert• 14 years ago

Stop-losses are not as effective or nearly as simple as they are described in typical financial media.

http://falkenblog.blogspot.com/2011/02/stop-loss-myth.html (with additional citations in the blog)

hogu• 14 years ago

Uh, if you look at his daily pnl charts, it looks like gambling with some extremely great odds, he rarely looses any money. That pattern is typically associated with HFT, If you can do many small trades and your strategy really has positive expected value you'll get great returns. I don't expect it would work now though, the HFT market is much more competitive these days.

downandout• 14 years ago

"Glorified gambling" is a pretty accurate description of this. However, that isn't necessarily a bad thing. If you have a situation where you can gamble with a long-term positive edge, then the proper strategy is to play with as much money and for as long as possible. The determining factor here is whether or not the combination of a particular investor's strategy, algorithm, and ability to execute will give them a long-term edge over others in the market - not whether or not this may be a risky activity in the short-term.

tomjen3• 14 years ago

You know, that is a really good argument.

Except two things:

1) He didn't lose money, he made 500k.

2) If this worked reliably, you would be out of a job.

cheeseprocedure• 14 years ago

Er... it doesn't work reliably, and by definition, cannot work reliably. If market inefficiencies exist to be exploited, then someone is ultimately getting the short end of every stick.

brianjyee• 14 years ago

Gambling can be done intelligently and profitably. Don't be so quick to label gambling as a pitfall to be avoided at all costs. I am not a financial expert, but as someone with a background in math/probability based gambling strategies, I can see similarities in the financial markets.

ww2• 14 years ago

No. This is not. Not ONE gambling. It is thousands of thousands of gamblings with a consistent winning ratio. cannot be oversimplified to a symmetric chance of win or lose.

HockeyPlayer4• 14 years ago

I run an HFT group, and what he describes isn't what we'd call "retail". He was doing a number of things that professional shops do, including making markets to avoid paying the spread and paying attention to queue position to predict execution

With a bit of luck and a good partner, this guy could have built a sustainable business.

marvin2• 14 years ago

This is a very mean and unconstructive comment to someone who made the impressive achievement of building his own automated trading system and actually making money from it. I've started calling out comments like this one, because they cause a bad environment for useful discussion.

The only argument in your comment that isn't your own unfounded opinion is that market makers make money from people who execute trades. But this is true by definition.

The traders who "gleefully picked off all those trades" weren't outsmarting anyone, they were simply profiting from the difference in the asking and offering price in the market. This is the role of a market maker, and actually makes it cheaper for people like OP to execute a large number of trades. So even though this comment sounds like a sensible rebuttal of the linked article, it doesn't really say anything at all.

Again, sorry for creating a negative reply and contributing to a bad tone, but I really the right thing is to call out these kinds of replies. They discourage honest sharing and discussion.

yajoe's comment is an example of how to criticize a post like this in a useful way. http://news.ycombinator.com/item?id=4748989

KingMob• 14 years ago

While writing his own trading system is a decent accomplishment, due to things such as an overall rising market in the time period involved and survivorship bias, the original author is likely to be completely mistaken about the reason for his winnings.

Given that he might convince other people to engage in high tech gambling in a less-favorable market than the one he operated in, strong words are called for in this case.

lurkersmirker• 14 years ago

I once worked for a software shop, and part of my job was writing trading code in a proprietary language for customers, who ranged from low end day traders to 8 figure annual revenue hedge funds. I had access to all kinds of tools, and saw many a varied strategy. There's quite a lot of money to be made selling solutions. I won't day trade.

xanmas• 14 years ago

What kind of solutions?

throwaway1979• 14 years ago

I'm a pretty risk averse guy and my typical reaction is to figure out why something won't work. One of my pet excuses is that assuming markets are efficient, someone must have already figured this out/ come up with a better exploit, etc. Most of the time I'm right. What troubles me is encapsulated in the following parable:

A UChicago economist and graduate student are walking across campus. The student says ... hey ... there is a hundred dollar bill on the ground! The economist scoffs and says no there isn't ... if there was one, someone must have picked it up already.

Sometimes I catch myself thinking this way. I have to remind myself that (a) markets aren't perfect, and (b) the real world has huge asymmetries in information, ideas, and perhaps willpower (by this, I mean while 100 people might think of a great idea, not all will attempt to implement it; even then, people will differ in execution).

That said, you're likely right. This trading strategy will likely lose money today :-p

jspaulding38• 14 years ago

It's easy to fall into that mindset. And in fact, that mindset is right back where I am now. The only reason I had the gall to attempt this in the first place was the the simple fact that I was making money at the time (in 2008) 'manually' day trading the Russell 2000. I thought this 'should not be possible' so I figured there's no reason not to try an automated program.

ScottBurson• 14 years ago

There's a sentence in this article that is critical and yet very easy to overlook: the author had 2 years experience daytrading manually. That already gave him a lot of knowledge of how the markets work and where an edge might be found.

I think that if someone is a good programmer and has some mathematical chops and has that kind of experience daytrading, taking a shot at automated trading is probably a reasonable thing for them to do. Without all of that background, you're right, they're almost certain to lose money.

ishaanc902• 14 years ago

I don't get you haters. sure the title is a bit misleading because he never really discusses what his alphas were. But its a pretty good high level description of the architecture of a hft system. I was a quant at GS and these are not the retail investors you pick off. You have your own set of alphas and most of them are meant to pick on mom and pops clicking away at home. This guy didn't reveal his strategy but nevertheless the graph shows his strategy had a significant edge. The lifetime of a strategy also looks like that. It is another thing that his title for the post is kind of off.

vinayan3• 14 years ago

Have you ever thought of making a trading system that would buy tons of stock when a flash crash happens? It is going to happen again. If your system is ready and you buy before they shut the market down or roll back orders you could make a hefty profit.

crntaylor2• 14 years ago

The difficulty is in identifying what is a 'flash crash' (i.e. a temporary downward blip in prices caused by computer or human error) and what is a genuine downward price movement.

If the market dives and you quickly get into a big long position, and then it dives some more - what do you do? You can either close out your losing trade and take the loss, or hope that the market comes back up, all the while holding on to the risk of further losses.

Also, there's no guarantee that trades in the middle of a flash crash will remain valid after the crash. The exchange could nullify all trades in a certain period of time, which would completely wipe out your upside potential.

beagle318• 14 years ago

> The exchange could nullify all trades in a certain period of time, which would completely wipe out your upside potential.

This is the most important thing: In every single "flash crash", the exchanges have retroactively canceled trades, in a rather arbitrary manner (e.g., "every trade between 16:30 and 16:38 is null and void"). There is some underlying justification, but it is also arbitrary (e.g., "anything below 3% of the price when the flash crash started", with no specific justification for the 3% number, or a well defined methodology for the time of the crash).

That could easily turn a +$100K profit into a -$500K profit, depending on circumstance.

lrm242• 14 years ago

Take a look at the VIX from 2009 until today and you'll understand why you stopped making money.

S_A_P• 14 years ago

http://finance.yahoo.com/echarts?s=%5EVIX+Interactive#symbol...;

very interesting, a spike in sept-dec 2009 that roughly follows his PNL...

jcampbell1• 14 years ago

Without a ton of volatility, any homebrew HFT is going to lose to commissions and spread. Basically, he was trading in one of the few periods where is was possible to make money.

Judson• 14 years ago

This reminds me of an AMA from a few years ago. Really interesting if you are into stuff like this:

http://www.reddit.com/r/IAmA/comments/9s9d7/iama_100_automat...

jspaulding38• 14 years ago

thanks for posting hadn't seen that

tomp• 14 years ago

This article is missing a crucial piece of data: what was the initial investment.. earning 500k with 10k initial investment is genious, with 10M initial investment it's just another year on the stock market.

beagle318• 14 years ago

This is a different style of trading than what investors do. He said that he was never more than a few contracts in. A single contract needs $10K day trading margin usually (depends on time frame and specific contract, but it's a reasonable estimate). Therefore, if he was never more than 10 contracts long or short at the same time, the risk was of $100K.

Making $500K on $100K over 6 months is good business.

jspaulding38• 14 years ago

You are correct. I think I had to have $10-30k in my account. I was extremely low risk so they weren't concerned.

brown9-2• 14 years ago

Especially considering that from it's lowest point in 2009 through the end of 2010, the S&P500 rose by 71%. Need to know what the starting capital was to be able to figure out if his return beat the market.

curiousdannii2• 14 years ago

I'm sceptical that HFT is good for the public. What did you do to ensure your system wouldn't make a flash crash worse?

beagle318• 14 years ago

Why is it his business to defend other market participants?

curiousdannii2• 14 years ago

I didn't say it was. But everyone should be in the business of not being evil and not exasperating the problems of others. In trading that means not contributing to flash crashes.

I am not saying that this guy's trading did contribute to flash crashes! He may have successfully implemented systems to prevent that. I hope he did, and if so I'm interested to hear how.

rhplus• 14 years ago

The charts show he was trading between Jun 2009 and Oct 2010. How much of his gains could be attributed to the market recovery in general? The Dow went from about 7000 to 11000, the Russell from about 600 to 800.

washedup2• 14 years ago

Well, in the article he said tat he did not care about direction, he would simple buy when his expected price was up, and sell when down. However, there could have easily been a bias in his model that "preferred" and performed better during upward movements. If so, he got lucky.

jspaulding38• 14 years ago

Longs and shorts were 50/50 and my program showed no preference for up or down days. High volatility and high volume was what it liked.

mikeash2• 14 years ago

You don't need a bias to accidentally make money when the market is overall moving up, do you? Picking stocks by throwing darts while blindfolded will, on average, make you money in a market that's moving up.

mertd• 14 years ago

That's about 50% gain. Unless he put in $1m that doesn't completely explain his ROI.

rorrr3• 14 years ago

Considering he was making 1000-4000 trades per day, I assume he put more than $1M. Trade fees alone would eat a huge chunk of money. The cheapest trade I can find is $3.95 (optionshouse.com).

So the fees will be $3950 - $15,800 per day.

$15,800 - $79,000 per week.

pbharrin• 14 years ago

Great point, here is a chart of the Russel 2000, incredible growth during that period. http://finance.yahoo.com/q/bc?s=%5ERUT&t=5y&l=on&...

junto• 14 years ago

Out of interest, how much capital did you start off with?

jspaulding38• 14 years ago

$10k loan from dad. Built it up to 30k trading manually before my automated program went live.

intel4004• 14 years ago

What were your average transaction costs per trade? 1000 to 4000 trades per day, lets say 2500 on average, translates to about 625K transactions per year. I assume you did not have to pay something like ETrades 3$ commission per future contract, which would result in almost 2M of fees per year.

danso• 14 years ago

I don't have much experience with finance or working experience with machine learning, but I've always wondered how much room there was for a clever amateur to profit in this space, even as it's crowded with much more sophisticated professionals with much more sophisticated algorithms and machines.

I'm thinking back to Garry Kasparov's piece in the NY Book Review a couple years back: http://www.nybooks.com/articles/archives/2010/feb/11/the-che...

He talks about a chess tournament in which it was "anything goes"...competitors could be human, computers, or humans with computers. The expected outcome was that a grandmaster using a Deep Blue-like computer would win, but the winners ended up being a couple of amateurs with three computers:

> The surprise came at the conclusion of the event. The winner was revealed to be not a grandmaster with a state-of-the-art PC but a pair of amateur American chess players using three computers at the same time. Their skill at manipulating and “coaching” their computers to look very deeply into positions effectively counteracted the superior chess understanding of their grandmaster opponents and the greater computational power of other participants. Weak human + machine + better process was superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process.

So in HFT, how much room is there for an amateur to profit over professionals by having a sophisticated process?

bloaf• 14 years ago

Good point, I also wonder about the potential to exploit the algorithms used by the "professionals." In other words, if you can come up with a reasonable approximation of what the pros will do, can you use that information to beat them?

aggronn• 14 years ago

'Theoretically', no. Its hard to be optimistic about these two ideas because while the chess example is a good story, its not analogous for many reasons, ranging from disparity in available information to players to a difference of several magnitudes in saturation. Not to mention HFT just isn't chess.

HF traders are just as much hackers as anyone on HN (and there are plenty of HF traders on HN). So 'theoretically', they've already done what is being suggested here. If someone comes along and develops a winning strategy, it really shouldn't be considered as having anything to do with 'professional strategy vs novice strategies'. It would just be about one person either getting really lucky or coming up with something that is genius in its own right.

If there are 'professionals' and then other 'professionals' whose strategy depends on information about how other 'professionals' trade (and there is), you end up with strategies at all valid points in the sample space of possible strategies and counter strategies. Theoretically, there should be no other possible strategies. Inevitably someone will come up with one though, and the 'sample space' will grow. But its extremely unlikely that additional unique strategies are successful just because they 'counter' the strategies in the sample space. But then again, this is real life and these things aren't impossible.

Sharma• 14 years ago

Trust me, you earned that much because of your luck. Otherwise Andrew Ng would have partnered with another finance professor and they would have been the richest people on earth!! Imagine trading with their expert systems on global markets.

I traded stocks and Forex for years and my experience says, it is not for everyone. What ever indicators,discipline or model you follow it is going to work only if you have the right intuition or luck!

zdwalter• 14 years ago

I like to trade Forex using mql4, any suggestion? Thanks.

arbuge2• 14 years ago

I am curious as to exactly why the profitability decreased steadily and rather rapidly all the way to ~zero. The article doesn't seem to expound on that unless I missed something.

Is this a result of bots on the other side adapting in some way to what you were doing? I would have thought you would be too small a player for them to notice.

baq• 14 years ago

another comment explained: http://finance.yahoo.com/echarts?s=%5EVIX+Interactive#symbol...;

jspaulding38• 14 years ago

I don't think anyone was adapting to what I was doing in particular but rather simply adapting to the opportunities in the market.

arbuge2• 14 years ago

Thanks. Makes sense... I guess those opportunities tend to get ironed out rather fast then.

dschiptsov5• 14 years ago

Basically this is a story about a guy who was smart enough to script up his trading tool (he discovered that there is an API and wrote some code to use it).

He trade other people's money, using other people's (probably employer's) account and resources, I suppose.

His employer have paid all the fees, and, took all the risks - if there is profit - it is mine, if there is a lose - it is theirs.)

The essence of trading is about having a special (insider) position of even being a market maker, who just collecting fees from every trade other people do.)

But this is just my guess.

mattyppants3• 14 years ago

This is not even close to an accurate summary. He never stated that he had any employer backing, and he wasn't collecting market making fees. In fact he was paying brokerage fees which is the exact opposite.

dschiptsov5• 14 years ago

He used his own money to test his algos? Come on.

Taking a Machine Learning and Statistics course does not make you a trader. You need something else - access to the system.

This is why my guess is that he was an employee.

curationary2• 14 years ago

Basically this is a guy who does what many other have done, look no further than elitetrader.com, but smart enough to add "machine learning" to his story headline, which is the hottest buzz term on the street this year, everyone and his mom talks about machine learning last I checked.

dschiptsov5• 14 years ago

So, it is just plain SEO for his startup - a primitive on-line course aggregator.

iandanforth• 14 years ago

I'm glad to see a healthy respect for investment among the hacker community. It's traders like this who commit to nearly a full 10 seconds of ownership that are the backbone of economic growth for this country.

unreal375• 14 years ago

I think there is a healthy respect for computer code that can let loose upon the world and make money.

mattyppants3• 14 years ago

Your missing the point that owning part of a company for 10 seconds brings zero social value to the economy. Which I also tend to agree with. Slightly off-topic from the point of the article, which was that he achieved it without backing of a fairly large institution.

OldSchool3• 14 years ago

Great work, very interesting to me. Counter to what we're constantly told through the media this stuff can be done. Doing it year after year seems to be the elusive part. Intuitively, once you've proven your technique on 1000+ trades it's not luck.

I developed a fully-automated low-frequency stat arb system that I ran in 2007 based on a perhaps even simpler algorithm. It traded various equities equally to the long and short side regardless of market conditions so widespread rally or collapse was irrelevant. I logged about 20-30 trades/day - much slower.

The results, using no leverage, were +90% in a year with a worst drawdown of 2% and a Sharpe ratio of 2. Total trades were 5000+. Month-to-month the results were very consistent until the uptick rule was nixed in July 2007. August 2007 was a record winner for me, but Sept-Dec 2007 fell flat, not losing, but with greatly diminished profits and the same variation and more frequently getting slammed all-long or all-short instead of a mix that was often near-neutral. Also getting fills better than my orders then completely disappeared, as this was the beginning of the HFT middlemen - including your own brokerage. I shut it down at the start of 2008, keeping the profits intact and moving on to other priorities.

I continued to monitor the theoretical results for a couple of years but the conditions didn't return so I eventually cancelled my data feed.

mempko• 14 years ago

Wow, fuck that guy. no really. Should be title "How I stole 500k with machine learning and high frequency trading".

oijaf8882• 14 years ago

Who did he steal it from?

hafabnew• 14 years ago

I've been considering trying HFT myself for a while. I'm competent with Machine Learning and am a Software Developer by day, so I can program and can sysadmin well enough to get something up and running without any trouble at all.

But, every time I've tried to actually get started, I've always found the amount of research required before being able to begin is just staggering.

It seems like the logical course of single-programmer HFT trading being:

- Find sample data

- Build your trading program using sample data

- When you're happy: connect to live API and set your trading program loose

- Iterate.

However, the first step and the third step seem like the ones which require the most research.

Is there somewhere which has a straightforward dump of timestamped market data available to download (free or not), in order to actually develop a working program?

Likewise, figuring out what to actually trade with, and which service to use is also pretty taxing.

sseveran2• 14 years ago

Nothing in HFT is free. People doing this for a living use precision time protocol in a colocated data center to build their own timestamps. However you can get started by buying ITCH, OpenBook, etc... data which has the full market depth feeds for the various exchanges. There are a couple of brokers out there specializing in the space.

If the easy part was building a working model either you got incredibly lucky or the model is wrong.

aiskunkworks• 14 years ago

Find sample data: https://quantquote.com/ API (Colo): http://www.lightspeedinstitutional.com/automated-trading/

monksy• 14 years ago

Quality sample data and historical data to backtest is hard.

Symbols change, stocks split, dividends are announced etc.

hnruss• 14 years ago

Just because you CAN do it doesn't mean you SHOULD. Even if you don't think of it as "gambling", you're still taking in tons of money without providing any tangible benefit to society.

If you want to make money from investing, why not do so in a socially responsible way? Invest in companies that are changing the world for the better. You might not bring home as much money, but at least you'll be able to sleep well at night.

jspaulding38• 14 years ago

I didn't lose any sleep but I do indeed hope to contribute more to society with future projects.

wyan• 14 years ago

You mean, something like Instagram?

sjm• 14 years ago

Not really defending the parent, but believe it or not Instagram provides something that makes a lot of people happy. Winning a bunch of money on the stock market does nothing for anyone but the winner. Not really a fair comparison.

quant1232• 14 years ago

There is an air of either incredibility or sheer jealousy in these comments. Nevertheless, I just wanted to tell the OP that he did a great job. Thanks for sharing. I work in the finance industry as a quantitative software developer, and it certainly is not an easy job for one person to do. In fact, I tried (independent of my professional work) doing this myself, and I ended up losing a lot of money. If people are trying to do this, please please be careful. Big companies, like ones I have worked at, have technical and human resources that are vastly more powerful.

jspaulding38• 14 years ago

thanks quant123 ;)

loup-vaillant7• 14 years ago

While this was quite fascinating, I couldn't see this form of trading as anything but a zero-sum game. Some players win, the other lose, like in any other game.

Except finance is supposed to be "serious". In most serious, legitimate activities, extracting money means you provided value somehow. So, what value high frequency trading could possibly provide?

loup-vaillant7• 14 years ago

Okay, enough with the downvotes. What's wrong with my question? I didn't mean to bash a field about which I know next to nothing. My question is genuine.

Now I do have an idea where trading could be useful. For instance, a good old merchant doing arbitrage and making a profit is pretty useful: that's how different regions can specialize, do better than they otherwise would, and ultimately lift us from hunter-gatherer tribes where 60% males die a violent death, to our civilization now. And money, as evil as it may be perceived, is to date the best organizing medium humanity ever had.

Investors also have their use: by better allocating money among companies, we could hope to give more money to those who are better at converting it to actual wealth.

But.

The OP didn't make a ton of money because his model of the the companies had an edge, but because his model of the behaviour of other traders had an edge. So I fail to see what useful information his trading put into the system. How his actions resulted in better (or worse) allocation of money between companies. How the (very serious) game he was playing was anything but zero sum.

Maybe there's an error in my reasoning. In this case, I'd happily accept downvotes, but please tell me where I have gone wrong. Not understanding why one's post is being scolded is just frustrating. Like failing an exam and being told to have a hard look at oneself. "Yeah, I get that, but what should I look for?"

gd14• 14 years ago

A strange position to take, I'm guessing you've absorbed it from the media somehow. His machine was in the market enough to trade 4000 times a day (I would suggest passively, or he would have been eaten by the cost of crossing the spread), so he was basically continuously offering a service to the market - an offer to sell and a bid to buy at the price he thought fair. Do you demand to know what value your local 7-11 provides by selling you milk? The value is that you can buy milk 24 hours a day, so you don't begrudge them the 15c a carton they are making. And they probably charge more than your local supermarket for the convenience too. The dirty thieves.

loup-vaillant7• 14 years ago

I refine my position here: http://news.ycombinator.com/item?id=4752742

I do get the value of trading, investing, and some form of arbitrage. Heck, I don't have to search for the farm to get my milk. That's a service, and it does deserve a reward. What the author did is a bit different:

> Most of the market volume was other bots that would only execute a trade with me if they thought they had some statistical edge.

I understood it meant "they would trade with me only if they think I was the sucker". And of course, he would trade with them only if he thinks they were the suckers. It's not providing a service. It's fighting in a zero-sum economy.

Now he did say "most of the market volume". So there's a fraction that's not bots, and probably also a smaller fraction that does not even play the zero sum game, but instead does some positive-sum trading with the mostly-zero-sum players. But this interface boundary seems incredibly thin, compared to the internal zero-sum behemoth. That looks like a highly inefficient use of time, energy, and brains.

And even then, I'm not sure the zero-sum game provides any service to the rest of the world: zero-sum players base their models on the behaviour of each other, not on the actual performance of companies. That would add no new information to the system. At best, that only amplifies the effect of the few that actually predict company performance. And I doubt it does it well.

dwk9080• 14 years ago

If you're interested in working on this kind of stuff in the San Francisco area, send a resume to Headlands Technologies. careers@headlandstech.com

ianstallings• 14 years ago

I could do the same with poker. Or I could lose it all. That variance thing is a bitch.

mrchess• 14 years ago

Cool article but I hope people don't start trying to follow this path. Ask yourself -- why did he stop?

jspaulding38• 14 years ago

yes. and yes.

stcredzero• 14 years ago

Is there a way to do this with Python or Ruby? I could just as well program this in C#, but I have a friend who can code a little, but doesn't really need everything in C# to do what he wants. The value add of offering the simulator, including the taking into account the bid/ask prices and a stochastic model for latency. Combine this with a web based code editor and easy hosting, and I think this would be a viable product.

HockeyPlayer4• 14 years ago

Sure, his program could easily have been done in almost any language. He wasn't competing on speed, which might have excluded languages like Python.

owyn2• 14 years ago

Nope, there is absolutely no way to do this with python or ruby. :)

Seriously though, there are some existing frameworks and products that you could check out. I haven't used this myself, it's just in my bookmarks:

http://www.rapidquant.com/features

A C++ library with python bindings: http://quantlib.org/index.shtml

And a low level library for data analysis: http://pandas.pydata.org

grumps• 14 years ago

If you use their library/SDK/Framework.... how do you actually trade, do you still need a brokerage? Sorry a bit new to this field.

unreal375• 14 years ago

There is a coursera course called "Computational Investing, Part I" that I am taking that aims to build a market trading simulator to test a trading model. It just started so it's not too late to join.

https://www.coursera.org/course/compinvesting1

runjake2• 14 years ago

Relax with the vitriol. The guy is sharing an interesting personal story, not providing a step-by-step HOWTO or recommending people follow his suit. In fact, the article is really an ad for his startup Courseware.

jspaulding38• 14 years ago

CourseTalk. CourseTalk! http://coursetalk.org .. You are right kind of :) But I've made a decision to start reaching out generally so I can attract cool people to work with on whatever projects I may be interested in in the future.

runjake2• 14 years ago

By the way, no offense meant by the advertising thing. I enjoyed the article. Although, as a technical person, would've enjoyed more details on the code and algorithms.

ww520• 14 years ago

Very cool article. Gave a detail explanation of the ENTIRE process. Thanks!

ChuckMcM• 14 years ago

The high point of my trading was October 2009 when I made almost 100k. After this I continued to spend the next four months trying to improve my program despite decreased profit each month. Unfortunately by this point I guess I’d implemented all my best ideas because nothing I tried seemed to help much.

It is of course possible that once you made "real" money with your algorithm it was spotted by the other algorithms which then started working against it. (Aka exploiting it) Having talked with people in that space (hft) I was left with the impression that an insane amount of analysis was done on those trades.

jcfrei• 14 years ago

It is pretty clear from his own graph that this stopped working in october '10. which was an eternity ago in terms of financial markets. algorithmic trading has increased manifold since then, so finding another arbitrage opportunity like he did is only going to be more difficult.

rbc• 14 years ago

I started the infrastructure for this kind of thing a while ago. It's BSD licensed.

It is a software implementation of the Viable System Model (VSM), a model for autonomous systems developed by Stafford Beer. It provides structure, communications, auditing and alerting for autonomous systems.

Part of it is base code for dealing with stocks and options, treating securities positions as autonomous systems that have the scaffolding for running simulations on themselves. It's in Smalltalk and runs under Squeak and Pharo. It can be found at:

http://home.rbcarleton.com/rbc/software/smalltalk/VSA/

I won't try to advertise it as a complete solution, but it might be the start of one.

padobson• 14 years ago

Don't do this with your own money. Found a startup building HFT tools, and then raise money for it, and use other people's money to test your tools.

If the tools work, sell them. If they don't, tweak them, try it again, and sell them until they do.

This is risk management.

nesu• 14 years ago

Wait. This is not HFT. There's a huge difference between automated and high frequency trading. What he does is only automated scalping at best (or at the fastest).

Automated trading is more on strategy, while HFT has more to do with volume and speed. With automated trading, you predict price movements. HFT involves being a liquidity provider. You don't use market technical indicators in HFT, you wait for some really huge orders.

HFT firms won't bother him. Those are dealing with an entirely different set of algorithms. He should have contacted brokers instead.

crucialfelix• 14 years ago

quite true. but HFT has become a buzzword like "the cloud" and even many financial industry specialists claim to do HFT but in fact they are just market making. auto-scalping or even hedge funds that day trade with robots are not HFT.

this: http://books.google.de/books?id=dobO95EBcqsC&redir_esc=y

even though its over 10 years old, is real HFT. moving correlations, windowed FFT (of bid and transaction events), microscopic operators, negative first-order autocorrelation of returns.

it is NOT about supplying liquidity or being a market maker. that's just market makers trying to say they are in on the latest trend.

HockeyPlayer4• 14 years ago

I run a 12 person HFT group in Denver. This is an excellent description of how an individual can bootstrap themselves into success. Great story, and nicely presented.

The one interesting point that he glossed over is what his indicators were. He wrote, "The indicators that were most useful were all relatively simple and were based on recent events in the market I was trading as well as the markets of correlated securities."

Which doesn't really tell you much.

rorrr3• 14 years ago

How well are you guys doing? How much money did you make?

jspaulding38• 14 years ago

Thank you. I'm looking back through my code and there are really a lot of indicators. It would get pretty technical to explain them. They are all explainable it's just that each one corresponds to slightly different market conditions and I just didn't want to get into it.

HockeyPlayer4• 14 years ago

Fair enough. Most people considering trying this probably have a few ideas for indicators.

But for anyone coming to HFT from a coding background instead of a trading background, an explanation of one of your indicators would have been fascinating.

deathflute• 14 years ago

Thanks for an excellent post. I would like to see any one indicator explained in detail as well. I am sure it would be of interest to a lot of people.

mcarvin3• 14 years ago

Very important to understand that making $500k speculatively is not evidence of an 'edge', nor is trading frequency evidence of the absence of luck. From March 2009 through much of 2010, the market was strongly bullish - if his algorithm showed a positive market bias then his returns would primarily be a function of timing (read luck: and there are a million variants on the nature of the bias that could be unwittingly responsible for his returns, despite the frequency of trades).

We cannot even tell if $500k is a good risk adjusted return - we have no information on volatility, nature of the exposure or most importantly how much money he started with?

Not exactly shocked Jim Simons didn't return his email. But completely shocking that he walked away from a successful automated trading strategy... the only thing rarer than a free lunch is a man willing to walk away from one. suspect.

fr0sty4• 14 years ago

1. Author graphed his daily returns which should give you a handle on his volatility. He also stated his max drawdown was $2k.

2. Author has stated elsewhere than he began with $10k in seed capital.

3. The "bullish" market you cite increased only about %70 in that timeframe while the author's returns were multiples of that number.

4. Author walked away from a _previously successful_ strategy that no longer produced profits. His montlhy returns went to nearly zero so he stopped.

Edited to fix numbers.

mcarvin3• 14 years ago

1. I cannot get even a remote sense for the nature of his risk exposure from looking at his daily returns. 2. ok. 3. The point here is that a systematic bias in his algorithm will expose his trading strategy to the good graces of market fortune (luck) regardless of whether he trades a million, billion or once a day. The source of the bias is irrelevant. 4. did not see where he said that but that very much confirms 'timing' / which in this case I interpret as luck as being at least a contributing factor.

namank• 14 years ago

Might've been what it was a couple of years ago but this post, dated today, is the perfect advertisement for the author's current business.

Well done!

PS: no sarcasm intended, it truly is an excellent advert.

hellsten• 14 years ago

A rising tide lifts all boats. "Price move predictions" sounds as effective as tea leaf reading.

doki_pen• 14 years ago

Isn't profit meaningless without knowing initial investment? I read the first few paragraphs and got bored. Why not say upfront what the bankroll was to start?

edit: I found what I was looking for in the comments.

JoblessWonder2• 14 years ago

Just to save someone else from having to look for it like we both did, his initial investment was

  $10k loan from dad. Built it up to 30k trading manually before my automated program went live.

thisismyname• 14 years ago

How much did you start with?

dmmalam2• 14 years ago

Being pedantic, 4000 trades a day isn't HFT. This is stil algo trading, of which HFT is a subset.

I consider HFT to be any strategy where speed itself is the what gives the edge. Colocation is usually a prerequisite, though not sufficient. It's a shame HFT gets all the attention, when it's really a tiny portion of trading activity. Algo-trading in general is 70%+ of market activity in the US.

Also limiting trades isn't really adequate risk management. The tech exists to very accurately model your exposures. This is something I see underdeveloped a lot, and what separates the top trading firms from the rest.

Still I commend you creating a model, working out how to test and execute it automatically and actually trading your own money.

I really think more hackers should be actively managing their money, (in general, not like in the article). We have these amazing liquid markets, all time low spreads/commissions, products like ETFs/derivatives to accurately and cheaply execute a given strategy, and a huge increase in tech to model risk, but personal personal investing is the same as the 60s.

jspaulding38• 14 years ago

I don't know the exact definition of HFT but I did run my algorithm from a server collocated with my broker close to the exchange. I modelled lag time in simulation and not having it collocated certainly would have hurt.

beagle318• 14 years ago

In the US, HFT is mostly synonymous with "all out tech war, flooding the order queue so your less-equipped peers get lags". (Nanex publishes analysis on these events, which are not occuring several times a month and keep accelerating).

In Europe, HFT is mostly what OP describes, because they have reasonable control (e.g., you have to have one execution per 10 orders or pay a fine; in US exchanges, you can sometime finds 10,000 orders submitted in 3 seconds, hundreds of thousands per hour, with 10-20 executions).

> Also limiting trades isn't really adequate risk management. The tech exists to very accurately model your exposures.

That's basically what AIG did with copulas. Unfortunately, the assumptions in these models tend to break during crisis, when correlations go to one. And AIG went bankrupt.

Limiting trades, done correctly (mathematically AND legally) is the ONLY way to do risk management properly. With more assumptions, you can have "more efficient" risk management in terms of leverage (e.g., you can net S&P and RUSSELL exposure by assuming their correlation structure) - but as AIG has shown, that does not mean you are doing a better job of managing your risk.

anonymoushn• 14 years ago

Some US venues impose fees for having low fill rate. As far as I know, most US HFT firms' strategies are not based on creating latency by rapidly submitting and canceling orders, but the bad behavior of some players is much more visible than the orderly behavior of others.

dmmalam2• 14 years ago

When I say limiting trades, I mean naively saying 'I will have at most x positions outstanding'. Each trade has an associated risk (variance), that interacts in complicated ways in a portfolio, which I'm sure you know.

silly example, 100 small positions could be less risky than 1 large position or, 1 long, and 1 short trade will cancel each other out and create a riskless portfolio (with 0 return).

You need to have a risk budget, account for each trade, and work out the risk for the composite portfolio. Obviously this is not fool proof, but it's a way better approximation of the real world.

shizzy0• 14 years ago

And nothing of value was created.

Just kidding. It was a fascinating article. Thanks for sharing.

suyash• 14 years ago

@Author ? Can you create an online course and teach us all? Thanks

photorized• 14 years ago

I am skeptical for two reasons:

1. when you have a good system (even one you cannot "improve" further), you don't talk about it. 2. You don't just stop using it.

There should be more to this story.

frankster• 14 years ago

Can I recommend that you read the article and you will find therein the answers you seek!

gubatron• 14 years ago

of course no firm would respond to his noob low yield model.

The level of the coders doing HFT is beyond the comprehension of this guy, added to the team of Mathematicians, Physicists and computer scientists at your average HFT firm, they probably laugh when they read this.

Good try though, it was awesome that eventually he tuned it to profitability, but there's no way in hell they'd buy that amateur software/algorithm.

Kudos though for taking on the task of learning how to code and making money with ML.

jspaulding38• 14 years ago

This is such an odd comment. What motivates you to write such a comment? And what makes my system amateur and noob in your eyes?

derryl• 14 years ago

This might be a dumb question but... what system does your algorithm interface with?

Last time I checked, NASDAQ and NYSE don't exactly publish API's

nashequilibrium• 14 years ago

Pls do not follow the advice of the OP. I started a hedgefund in 2004 doing HF platform arbitrage and ran it for 5yrs and i can honestly tell you that this is just survivorship bias. This is a very complex field and being off slightly, having a slight bias, a fraction of a point off your execution pricing and a slightly flawed money management system is recipe for disaster.

The biggest issue is the confusion that you can apply machine learning to HF trading. HF trading sub 15min mark is more about playing the deal flow, and only the institutions have an edge on this. This is why goldman had to separate the buy and sell sides in the early 2000's. Above 15mins you are able to find an edge using time series analyses since the market is scaling invariant according to Benoit Mandelbrot and this does not apply to dealflow. Also having access to dealflow allows you to predict volatilty seconds ahead which allows you decrease your risk and increase you reward as well as handle your costs since the volatility will impact your transaction costs even if transaction costs themselves stay the same. There is just so much stuff to cover that a comment will not do justice in explaining what is wrong with this guys logic.

jspaulding38• 14 years ago

There are plenty of shops making tons of money with HFT who do not have deal flow at all - it's got nothing to do with luck.

Survivorship bias would mean I simply got lucky. If you're going to say that you're at least going to need to look at my P&L charts and say how I could possibly achieve that much success with luck alone.

Finally, machine learning has everything to do with my success. There were hundreds of variables in my algorithm that were ALL optimized using ML. If you read the article you would know that I built an accurate model for backtesting that I used to optimize variables as well as confirm that I was going to make money before I even started live trading.

I'm pretty confident that whatever you were doing in 2004 has nothing to do with what I was doing.

quant1232• 14 years ago

agreed. jspaulding got it right although I can see nashequilibrium's skepticism as the US futures market is incredibly crowded (read: competitive, no free cookies).

confluence• 14 years ago

A lot of people are stating that this is like gambling - it is - but not in the sense that you think. Firstly he doesn't use his entire bankroll on each trade, secondly he goes long-short consistently over very short periods of time, thirdly he's too tiny to actually move markets, and fourthly he is in and out within a day - where his max var. is 100% on thousands of tiny trades. Think of it like this - he has $100 - he bets $10 of that each day. He can blow that $10 - no problems.

Worst case he runs out of capital over a period of weeks.

He can't blow up in the way that you think - but he can have large drawdowns over a period of weeks.

Markets are eventually consistent scalable systems - and that is why we prefer them over central planning. In the medium term they prices things correctly, cheaply and efficiently (decade+).

In the short term however (sub-decade) - they can't price jack.

Markets are inefficient period - if they weren't, well then P=NP and you could just put your NP-hard problems into a market and get back cheap, quick, accurate results. Oh - wait - protein folding is actually harder than that.

There are 2 major ways to make money in the markets. Value-Growth and statistical arb (often high frequency). The former (Buffett) is highly concentrated bets on the future of business (I'm value - long TSLA/GOOG/Samsung). The latter is looking for thousands of small diversified statistically significant correlations above 50% (random guesses) and trading costs between securities/price movements over short time intervals (aka statistical ghosts in the data - RenTech/Shaw).

Both work. Both work well. And will continue to do so as long as markets exist.

JuDue• 14 years ago

"Since I'm no longer running my [half million dollar] program I’m happy to tell all"

This does not ring true.

sejje2• 14 years ago

I don't understand trading enough to even understand many of the terms in the article, however I'm curious to one thing: is it possible that a program could be written specifically to exploit yours? And/or is that a potential reason it became unprofitable?

FireBeyond• 14 years ago

Hopefully not buried too deep, but any books recommended for getting into day-trading, either manual, or algorithmic? Kindle preferred, but definitely not the deciding factor.

leminhhai• 14 years ago

Amazing, love your project

dataisfun• 14 years ago

This is an activity that adds no value to our world.

MonteChristo• 14 years ago

The prototypical example of why a tax on financial transactions is urgently needed.

curationary2• 14 years ago

how is it "machine learning"? A few if-else doesn't make it "machine learning", or does it? - @curationary

jspaulding38• 14 years ago

It takes data (last 4 weeks of market update data) and learns how to predict price movements and how to make profitable trades.

vincegata2• 14 years ago

Thanks for the post, it's very inspirational.

Could you explain this part, specifically what do you mean by "bucket".

"To accomplish this I tracked predicted price moves in 50 buckets that depended on the range that the indicator value fell in. This produced unique predictions for each bucket that I was then able to graph in Excel. As you can see the expected price change increases as the indicator value increases."

vincegata2• 14 years ago

Thanks for the post, it's very inspirational.

Could you explain this part, specifically what do you mean by "bucket"?

datashaman• 14 years ago

Excellent article on this posted here: http://www.lrb.co.uk/v33/n10/donald-mackenzie/how-to-make-mo...

How I made $500k with machine learning and high frequency trading

Related Stories