Ask HN: Has anyone used machine learning to trade?

https://news.ycombinator.com/item?id=955702

by haliax • 17 years ago

53 62 17 years ago

news.ycombinator.com

I'm thinking of trading on either the FX or Stock markets using ML algorithms (primarily neural networks) on an intraday timescale. Does anyone have any experience in doing this? Any tips/stories/recommendations on brokers, data, etc? Is it a complete waste of time to try without being a big fish (I've only got about 10K to venture on this)?

cperciva2• 17 years ago

Is it a complete waste of time to try without being a big fish (I've only got about 10K to venture on this)?

Yes. The intraday movement of stocks is small enough that you'd need to be using a large fraction of your capital on every single trade in order to overcome trading costs; and based on the Kelly criterion you'd need to have unrealistically high odds of betting correctly in order to avoid going broke.

haliax4• 17 years ago

Do you think that interday would be more suitable then?

cperciva2• 17 years ago

More suitable, sure. But I still doubt you'd get anywhere -- people who run hedge funds are very much aware of ML techniques, so I'd expect them to run circles around you.

rw2• 17 years ago

A strategy: run a LOT of simulations, keep running them every day. When they are too good to ignore, go raise some capital.

physcab2• 17 years ago

Can't you just download a historical set and use that to test whatever algorithms you think you would use? Then when you are confident enough in their performance, you can take the risk? I wouldn't say it would be a complete waste of time because you would be sure to learn something. If you think you can realistically make money--well then, that's a different question that I wouldn't be fit to answer.

anamax• 17 years ago

> Can't you just download a historical set and use that to test whatever algorithms you think you would use?

Be careful - much of the historical data has been pruned to eliminate some data from companies that didn't survive.

I've heard that one of the major quant houses (Shaw?) can often guess what data set you used to test your strategy by looking at the strategy.

Plus, there's a selection or fitting problem. Your strategy may have worked well on average from 1972-1999, but that doesn't mean that it will work today.

And there's a velocity problem. There are pricing errors in old data to exploit. And, if you had today's computers then, you might have been able to find them fast enough because your systems are faster than what folks had then. However, today you're competing with folks who have faster systems than you do.

physcab2• 17 years ago

You make some good points, but isn't this always going to be a problem when training your algorithms? You can't really train in real time so by the nature of the industry you're always going to be put at a disadvantage by using older data.

seunpy• 17 years ago

Trading with historical data isn't accurate enough. It doesn't account for the way your trades would have influenced the market.

wclax04• 17 years ago

If you're interested in this, why not join a firm that does it? They will have a large capital base to start with and eventually start your own venture. Unfortunately 10k won't get you very far in this game

njl• 17 years ago

Practice some with historical data, get a job with a firm that does this, and eventually strike out on your own. If you prove yourself, people will hand you millions to manage. Starting with your own money seems like a bit of a fools game.

ct2• 17 years ago

Being a "small fish" is actually an advantage since you have less capital you need to put to work and thus can more easily get fills and catch the bigger waves generated by the bigger fish institutions like mutual and hedge funds.

The institutions have to hold on for longer periods and can't really "scalp" trades per se given that the bid/ask volume would collapse if they didn't distribute and "average in" slowly overtime to accumulate a position vs. buying or dumping their entire position all at once.

JimmyL• 17 years ago

Being a small fish means you get killed by brokerage fees, which don't get cheap until the value of your trades is in the millions.

ct2• 17 years ago

True. Each side has their own set of advantages/disadvantages. And so you'd have to adapt and take that into consideration in addition to bid/ask spread. It's not impossible (nothing is impossible), and you'd only get killed by brokerage fees if you're overtrading.

If you can find an algorithm that can trade only a few times a day and ride stocks that has a volatility of 1% or more then even at retail commission levels you can still do pretty well as you slowly build up your account to trade strategies that require more churn.

patio11• 17 years ago

A strange game. The only way to win is not to play.

stavrianos2• 17 years ago

Reality begs to differ.

nir• 17 years ago

If by reality you mean Buffett or Soros, consider the thousands (hundreds of thousands?) of extremely smart and hardworking people who fail to replicate their success.

Create2• 17 years ago

intraday NN is irrelevant because NN are prone to overlearning, so the stuff becomes even more horrible - actually in reality you don't care which way it moves, as long as it moves: this is why the "big fish". It is like owning the place in Vegas: you don't care about the bets (much less about their substance or "direction"), as long as there are bets.

if you really want intraday, think about news.google.com and http://rs2007.limsi.fr/index.php/TLP:Page_12 to automatically tweak the few k base parameters (SVM, associative memory etc).

http://www.bbc.co.uk/science/horizon/1999/midas.shtml

Options, Futures and Other Derivatives, J. C. Hull, Prentice Hall Int’l

Financial Calculus, M. Baxter and A. Rennie, Cambridge University Press

but the ML part is almost irrelevant: it is not the lever in the leverage (think CDO trade).

bravura• 17 years ago

intraday NN is irrelevant because NN are prone to overlearning

I call bullshit. A lot of the most successful applications in machine learning are modeled using neural networks.

To regularize a neural network, you use standard machinery that controls the capacity of the machine. The scale of the weights cannot be too large (this is a famous result from IIRC Haussler in 1992) so penalize the weights using an l2 (Gaussian) prior. This reduces the model complexity and the potential for overfitting.

dpapathanasiou3• 17 years ago

I have friends who do this, and they told me, anecdotally, that for every 10 trades they do, 8 or 9 lose money.

While they do make it up on the rest, you have to ask yourself if you can afford the costs (brokerage commissions and fees) of executing your strategy in that context (they all work for banks and hedge funds).

If you really want to get into this, you might be better off applying to a quant-based hedge fund.

A firm like that will have the capital and brokerage connections to make the game profitable.

Also, take a look at Ed Thorp's web site (http://edwardothorp.com/index.html).

He was one of the original quants and has many interesting articles on the industry.

Tichy• 17 years ago

"I have friends who do this, and they told me, anecdotally, that for every 10 trades they do, 8 or 9 lose money."

At least an AI I would create to trade would probably not be worse than me.

schemer4• 17 years ago

I do this and I make money in 1 out 2 trades. Do you know what the heck you are talking about ?

http://www.brettsteenbarger.com/articles.htm

create_account• 17 years ago

Then you must be fabulously wealthy.

dpapathanasiou3• 17 years ago

Half your momentum trades are profitable? Is that before or after trading costs? How often and at what volumes are you trading?

jakewolf• 17 years ago

You'll need at least 25k in order to be a day trader in stocks. You can get around that by trading futures. http://www.finra.org/Investors/SmartInvesting/AdvancedInvest...

bcl• 17 years ago

You might want to start our by reading "Drunkards Walk", and then walk away and find something a bit less chaotic to throw your money at.

agbell2• 17 years ago

This is not what you are asking but a market with more inefficiencies and less transaction costs is Betfair. There is a simple api and free historical data.

I spent a bunch of time with Neural Nets and SVM and that data. Sadly I didn't find a profitable strategy. I would love to hear about anyone else who is looking in that area.

haliax4• 17 years ago

What type of analysis did you look into? i.e. what were your input vectors / output?

agbell2• 17 years ago

I would generate a back or lay output signal for 'Correct score' bets for soccer matches before the game started. Input would be everything you can pull out of the the data set especially including the odds on other more standard bets like the match odds and the over / under odds.

There are also arb opportunities between the various arcane bets, but that involves speed more than ML.

copernicus• 17 years ago

In general, machine learning would probably not be a great approach to day trading. There are several problems:

1) Limited market data: As an individual, you have limited access to market data. At best, you probably have real time top of book data but without paying for it, you almost certainly aren't getting a real time depth feed. Going off of only last trade data gives you even less to work with. In most cases free quotes are delayed making them worthless to day traders (although not longer term investors).

2) Limitations of market data: Even if you have a lot of historical data with a full order book, that is still of limited utility for simulations. The problem is that everything you do affects the market so any trade you simulate needs to be doing small enough quantity to not make your simulation unreasonable. How to deal with simulating market-affecting trades is a complex problem and the only way to truly know is to run your trades in the market.

3) For day trading, there is a lot of profit to be captured simply from the microstructure of the market (its moves back and forth, etc.). To capture much of this it helps to be fast but you won't be able to colo boxes at an exchange with only 10K.

Machine learning seems more suited to longer term investments and trading on a slower platform than making a quick profit arbing or scalping. You may want to consider this rather than day trading since not only does day trading require much larger amounts of capital, but its also extremely risky.

Also, if you plan to trade equities, enjoy all of the ridiculous SEC regulation (order marking, etc.).

unexpected• 17 years ago

haliax,

If you want more information on this, there was an IAMA on Reddit about 2-3 months ago, on a guy who claimed to do exactly this. It was incredibly informative, and went into a lot of the details that you ask for here.

http://www.reddit.com/r/IAmA/comments/9s9d7/iama_100_automat...

Keyframe• 17 years ago

of course, get MetaTrader4, open up a demo account on FXCM, Alpari, dbFX or whatever and write your own software and backtest it without losing any money. Boston Technologies has a free expert advisor, maybe you can build up on it - http://www.bostontechnologies.com/expertadvisors.html . I am not keen on automated trading though, since I do a bit of TA and FA, no software I know can do this - and where is the fun in it.

randomtask• 17 years ago

I don't know much about stock markets, but I did a consulting gig for a large currency trading company a while back where I was doing a bit of work on their algorithmic trading platform. I learned that unless you are trading at least a million dollars a trade it's very unlikely that you will make money.

The reason for this is that the large companies won't talk to you unless you're trading at those volumes since they charge pretty high transaction fees. The smaller companies that will talk to you all have accounts with the larger companies and make money by increasing the spread between the bid and ask price they receive from the large companies. The expected return on a trade is usually somewhere in the region of 1-10 pips (where a pip is one ten thousandth of a dollar), with 10 pips being a really good return and 3-4 being more realistic. The smaller companies usually increase the spread by up to 3 pips in each direction on the price they get from the larger company which swallows up your return.

While millions of dollars per trade might sound like a lot the leverage in this market is also very high at 50-100 times meaning that with your 10k you could trade 500k-1M though on a sum that low you might get less leverage. However if you lose money on a trade you could eat substantially into your capital. So if your $1M trade goes against you by 10 pips you'd lose 1k.

I'm not ruling out making money on that market, but I'd say it's very unlikely unless you have a lot more money to start with. I am not an expert in this area though. This is just what I learned from the small amount of time I spent working with the company.

ctkrohn• 17 years ago

If you're interested in reading more, take a look at the message board http://www.nuclearphynance.com.

And $10K will almost certainly not be enough. You'll spend all your money paying commissions or bid/ask.

schemer4• 17 years ago

10K is too little. Statistically speaking, a small draw down can wipe out your acount. Most professionals have stop losses to limited their loss to 1-2% of their account. Google "trading money-management". You probably want to paper trade for a while before risking real money. If you are really interested in markets you can follow it as a hobby while working as a programmer. I used Interactive Brokers for trading futures, fx, options and stocks. They are low-cost and provides an API for automation. I write java code for my own personal "proprietary desk trading".

keefe• 17 years ago

My understanding is this is a pretty huge field : http://en.wikipedia.org/wiki/Computational_finance and that if you try to roll your own, you are computing against a variety of math and CS phds that big trading firms hire to do analytics.

caffeine• 17 years ago

The only tip is: do it on paper first, with the exact setup, lag times, etc. that you're going to have when doing it live. Do it for a while.

And realize that this is a fanciful pursuit which is likely to lose you money, because the guys you're going up against have $ millions, ns-scale lag times, free or near-free transaction costs, and PhDs in theoretical physics.

markbao• 17 years ago

One of Ray Kurzweil's companies, FatKat, does quant finance based on pattern recognition. http://www.fatkat.com/overview.html

canoebuilder2• 17 years ago

On a related note, can anyone point to any writings/information on Kurzweil's work in this area?

canoebuilder2• 17 years ago

Noticing comments on the high degree of proficiency of the competition(or so is the perception.)

Anyone know of any particular markets where a small timer might have a better go at it due to insufficient volume to attract the bigger players, or for any other factors?

_cbsz• 17 years ago

If anyone knew, they would be crazy to tell you.

Besides, the bigger players could easily afford to risk a few hundred grand finding out whether they can plug the existing system into the small market profitably.

andr• 17 years ago

Yes, I did work on that. I used a neural network to trade FX. I wrote my own neutral network framework using code generators for x86 and CUDA.

The neural network was a fairly normal feed-forward network with GA-inspired, but heavily modified learning process. It combined layers of neurons with memory and memory-less neurons. It is essential to keep the network size small, because a big network will just memorize the data you use to train it.

Also, I used several other inputs, such as the frequency of trades per minute, overnight rates for each currency, and stock market indexes for the relevant countries.

Finding historic data was the biggest problem. I used Bloomberg data from my university, but that was a pain in so many ways. There is NO free data for most of the inputs I needed. Good data from Thomson Reuters or something like starts at $5k/mo.

Overall, the model worked fairly well, generating about 10% annual returns even with a few handicaps I put to be on the safe side. This was about an year ago and the only reason I stopped working on it was that my partner in the financial startup quit. I'm sorry if some of my terminology is wrong - I haven't been working in that field since then. If you have any questions, I'd be happy to talk about this. My email is in my profile.

Eliezer• 17 years ago

DON'T. All money that can be pumped out of the market by standard ML algorithms has been pumped out by standard ML algorithms.

smanek• 17 years ago

Don't bother.

D.E. Shaw, Renaissance Tech, Goldman Sachs, etc. have already hired hundreds (if not thousands) of people smarter than you to do exactly this. They have more Math, Physics, Econ, CS PhDs than you can count/

steveplace3• 17 years ago

They have more Math, Physics, Econ, CS PhDs than you can count/

So did BSC, AIG, LEH, and MER

crpatino• 17 years ago

More to the point! The big disadvantage ML methods is that they are just fancy statistics. The main faulty assumption here is that they believe the future looks a lot like the past, so they try to find even more complex patterns in historical data to extrapolate future trends slightly faster than everybody else.

In order for this approach to work, you need very precise market conditions. When markets are stable, your edge is zero and you would be better off investing in some sort hedged index (minimum effort, decent result). When markets are not that stable, it is when you can anticipate trends and get a decent profit for your effort. However, when markets suffer big unexpected turns is when fortunes change hands... and it is when your algorithms give you negative advantage! While you are warrantied to keep following the numbers wherever they take you,a savvy finance human agent has a chance to smell the blood and discretely head for the exit before everyone else. So, it is your call guys. In my opinion, this will not be a viable approach until we get some type of semantic-web-ly mine-able news feeder.

steveplace3• 17 years ago

The majority of people who have responded to you with various pieces of opinion/information forgot to answer this question:

Does anyone have any experience in doing this?

So take everything with a grain of salt-- there seems to be a lot of advice going around by people who don't have skin in the game-- especially those who say it can't be done. Unless they've put money down to test this, then their opinion should be discounted.

For those saying commissions are to high for a shorter term timeframe: trading has been commoditized. Lightspeed offers .40 per lot, and IBKR offers around that given a certain amount of trading volume. Forex brokers make money off the spread, and so the commission that you pay is only from the bid ask. Since you are trading small amounts, you will get hit on the comission structure, but if you pay per share rather than per trade it won't hurt as bad. You also might want to look into equity futures as well-- very liquid, good tax advantages, and good commission structures.

Biggest thing about small sizes is that you're subject to the pattern daytrader rule-- you can't make >3 intraday trades if your portfolio is <25k. SEC rules.

I'm a discretionary trader, and not a system trader. I am currently developing some systems to go live but they are not ready as of yet.

I also know of many who are quants, and it's possible and profitable.

johnwatson11218• 17 years ago

I just want to add that I read some of the Turtle Trader books and one more on Trend Following. The last book finally got across to me the point that the trend followers used computers and math not to try and predict where the markets were headed but rather to try and classify markets as rising, falling, or side to side. Their theory is that once a market has a direction it is likely to keep that direction - until it changes.

This is very different from most of the stuff I read about with neural nets. In most of the other books the idea to try and predict when markets are over sold or about to change so that you can be in on the ground floor.

I'm doing a webapp about card counting at black jack and the trend following approach seems a lot more like the reality of card counting. When the deck is hot or in your favor you bet more. Most people think that card counting is a super complicated exercise in combinatorics where the player is recalculating basic strategy in real time and able to prove that certain plays are the best.

Just something to consider.

ad• 17 years ago

How much do you know about trading now? If you know little, I would recommend the textbook Trading & Exchanges by Larry Harris. It's somewhat out of date but still very good. I work at a high-frequency trading shop and that is the first book they shove at you when you walk in the door.

Other than that, I'd like to second what several other people have already said: getting a job at a firm is a good way to learn, you're outgunned in terms of brains and money, and 10k is probably too little. Now, you can do a lot of learning for 0k while you save money, and every startup is outgunned by goog, msft, but that doesn't mean there isn't room in the ecosystem. Like a startup against an established player, you could find a niche that is going unexploited. What makes trading harder is the established players don't tell you much about what they're doing, so you don't know if you are doing something new or walking into a trading strategy perfected 10 years ago by DE Shaw. With a software startup, you have more information about what your competitors are selling, what their products are, how you might differentiate yourself, what's missing from the internet, etc. Software companies promote their products and try to get public attention; trading shops try to keep as many secrets as possible.

gradschool• 17 years ago

You might find it interesting to look at this site, which gives you a lot of the infrastructure for testing your FX strategies with real time data (although not direct access to the data).

http://preview.collective2.com/

Strategies are independently monitored and ranked on the site, and can be switched from simulations to real accounts when you're ready. If your strategy performs well, it will be noticed and you can get paid commissions on it (potentially helpful for raising capital).

For FX and equity data, you might as well use Yahoo finance. I run a Slicehost server that scrapes it once every minute with no problem. It's not ultra high frequency but you can't hope to compete in that space anyway, as others have pointed out. You should start building a historical database now before you do anything else, so that you'll have something ready when you need it. You probably don't need years and years of it.

I worked as a quant for a while and would support the other comments that standard ML techniques are unlikely to outperform, but you might be the exception.

maconic• 17 years ago

UCSC Professor Helmbold used machine learning to discover optimal portfolio allocation: http://users.soe.ucsc.edu/~dph/mypubs/OnLinePortfolio.pdf

known• 17 years ago

Please read http://en.wikipedia.org/wiki/Fooled_by_Randomness before you take a plunge.

solutionyogi• 17 years ago

If you know .NET, Do have a look at NinjaTrader and their forums. You will find quite a few folks who are in to automated trading.

I ventured into retail trading (manual) and lost around 5$. Email me if you are interested, I can give you more details (email address available in my profile).

Ask HN: Has anyone used machine learning to trade?

Related Stories