AI That Picks Stocks Better Than the Pros

http://www.technologyreview.com/blog/guest/25308/

by pier0 • 16 years ago

86 108 16 years ago

The research paper, which is a pretty good read, can be found here http://www.robschumaker.com/publications/IEEE%20Computer%20-... [pdf]

mkramlich• 16 years ago

here's one in Python:

import random

stocks = (ibm, apple, ...)

stock_to_buy = random.choice(stocks)

bmf• 16 years ago

nit picking, but many of the words listed (such as "smaller" and "crude") aren't verbs.

georgieporgie• 16 years ago

From the article:

> (In his work, 'verb' is a technical term, and does not exactly correspond with the conventional definition of the word.)

jrockway3• 16 years ago

I'd like to see what this looks like when run on bond derivatives from the early 2000s. I wonder if it would have been able to foresee the subprime crisis better than the big investment banks.

(My theory, as someone involved in the industry, is that emotion dictates 90% of whether or not someone is going to make a trade. This is why nobody noticed how bad the subprime bonds / CDOs were -- they bought them, so they must be good. I feel the same way about my telephone and keyboard ;)

b0b0b0b• 16 years ago

They don't detail the number of trades nor do they account for commissions or slippage.

larsberg• 16 years ago

And it's all simulated. Last I heard, placing orders affects the market, especially if you're working with more than a toy account.

joshu4• 16 years ago

It totally affects the market.

arthurdent2• 16 years ago

depending on the strategy, its possible to build reasonable simulations of the affect your trades will have on the market, though the data you need to do so is harder to get and a lot more expensive. but for a strategy like the one described it'd be reasonable to take the existing market and assume that you could pay offers, hit bids, and maybe build in a little slippage to get a reasonable simulation.

academics though, typically don't care as much about simulating for transaction costs because they're more interested in perceived mispricing in the marketplace as an interesting economic/intellectual phenomenae. Frequently academic papers will assume a 1-price world where the 1-price is midmarket and build the model off that.

If the market were indeed entirely efficient, then the described trade wouldn't be better than chance even with no transaction costs or bid-ask.

bh42• 16 years ago

Don't index funds and completely random selections ALSO outperform most professional stock pickers?

GFischer• 16 years ago

Here's an answer:

http://www.studyfinance.com/jfsd/pdffiles/v8n1/liang.pdf

Apparently the random stocks outperform the "pros" picks after 6 months (the pros have the edge in the short term)

Actual quote from the study: "Results suggest that the pros selection statistically outperforms the random selection only in the one-week period. Over a six-month holding period, the random stocks perform better than the pros recommendations."

rauljara2• 16 years ago

My understanding of why the stock market is supposed to be good for society is that it encourages investors to invest in companies that need more capital. It has the nice side effect of allowing people to buy high value assets that are likely to increase in value over time, letting people save for their retirements, while simultaneously providing capital to companies that may need it. I fail to understand how a computer that holds on to stocks for < 20 minutes forwards either of those goals. Maybe someone can explain why this is a good thing to me, but this really seems like a perversion of what free markets are supposed to be good for. And again, maybe someone can explain why this would be a bad idea, but I'd really like to see a system where if you buy a stock you are not allowed to sell it for a fixed period of time that would be measured in months, and not seconds.

hugh34• 16 years ago

Market making. Suppose I decide I want to buy 3000 shares of IBM right now. I'll have made the trade within minutes, because I'll probably have bought it from a computer which holds shares for twenty minutes. Four years later, when I want to sell, I can sell it within minutes too, because I'll be selling it to another computer that will probably sell 'em off to a series of other computers, before eventually winding up in the pocket of another long-term investor.

If long-term investors were the only ones buying and selling shares it would be much more difficult to buy and sell shares when you wanted to. Net result: shares are a less attractive investment, IPOs raise less, economy suffers.

solutionyogi• 16 years ago

I would call it 'Liquidity'. And there are instruments where you are locked in for certain period of time e.g. 10 year treasury bonds. Luckily, there is an after market for treasury bonds where you can sell them after buying well before the 10 years are over.

And what you are proposing is that we should only allow Investing and restrict/remove trading. The problem with this approach is that you lose your ability to find 'market value' for your investment on an everyday basis. Let's say we force everyone to hold stock for at least a month. What if I had bought BP stock just before the gulf oil spill news came out? I will be stuck and would have to incur a huge loss by the time I am allowed to sell my stock. Also, there are thousands of investors and they may need to liquidate their position at any time based on their own circumstances.

And finally, stock market does achieve their goal of allowing company to raise capital. Theoretically, it doesn't matter what happens to the stock price after company has issued their common stock. They got their capital and can do what they want with it. On practical basis, company do care about their stock price for numerous reasons which I won't get into right now.

Eliezer2• 16 years ago

Bull. Life would go on.

jfager6• 16 years ago

Automated trading is a tax on transactions, not a lubricant. The machines jump in front of trades between willing "real" buyers and sellers and screw both (or one or the other, depending on the transaction). There are entire companies that now exist solely to try to hide the large trades that actual long-term investors (pension funds, etc) need or want to make, so that the computers can't pick up the signal and pounce in between.

rauljara2• 16 years ago

Thank you. That was the explanation I was looking for.

My next question then, is whether this flexibility to sell instantaneously is worth the risk of near instantaneous drops of 700+ points in the dow, as happened a couple of weeks ago. My gut tells me that letting computers have a free run of it is not good. But I'm definitely open to being convinced otherwise. And I'm definitely open to a better solution than you have to hold on to stocks for a month or more.

skybrian2• 16 years ago

Why does any reasonable investor need to sell within minutes? What does this solve that a giant daily auction wouldn't solve?

It seems to me the only reason such speed is necessary is to beat everyone else, and that's only necessary because everyone else does it.

joe_the_user3• 16 years ago

I think we saw in a ah ... little ... incident some weeks ago that high frequency traders are rather problematic market makers (700 point drop, some stocks reduced to 0, etc, remember?).

jsn• 16 years ago

> maybe someone can explain why this would be a bad idea, but I'd really like to see a system where if you buy a stock you are not allowed to sell it for a fixed period of time that would be measured in months, and not seconds.

Um, sure. It would be a bad idea for me if I'm not allowed to sell the stock I want to sell as soon as I want it, and it would be a bad idea for somebody willing to buy from me, if they are not allowed to.

Oh, you're asking why it's a bad idea for you? I don't know, maybe it's not. Do you think I should be prohibited from the trades I like because you think it's a bad idea?

yosho• 16 years ago

a lot of it is to prevent arbitrage opportunities. If the euro and the US currency exchange rate were out of line with Euro to Yen and US to Yen, someone could make money on the arbitrage.

The stock market is therefore a perfect balance of supply and demand and any discrepancies are immediately rectified.

jcl• 16 years ago

I'm no expert, but I think the AI's strategy works because people read news articles about stocks and then buy and sell in large numbers, causing a stock to temporarily be valued higher or lower than it "should" be.

By taking out some of the profit of knee-jerk reactions to news, the AI encourages people to be more restrained in their reactions, to consider longer-term effects, which makes for a more stable market.

gxti3• 16 years ago

Another reason that computerized trading (and speculative trading as well) is theoretically good for the market is that it makes prices more informative. If a trader or computer can successfully predict how a price will move based only on past and present information, then there is an inherent market inefficiency -- the market price is supposed to already reflect all of that information, and only moves when new information becomes available. Of course in reality information does not spread that readily and traders are not completely accurate at estimating prices, so correcting those inaccuracies is quite profitable and helps the market. By selling when a stock is overvalued and buying when it is undervalued, regardless of how that determination was made, the price becomes closer to the fundamental value and thus allows long-term investors to buy and sell at a fair price.

CWuestefeld• 16 years ago

I think this is a really important point. Reacting to new information nearly instantaneously improves the efficiency of the market by factoring in all available knowledge in the blink of an eye.

This high speed, in conjunction with its distributed "cloud" nature, makes the market the most efficient means of allocating resources conceivable. In regulated industries -- say, healthcare (viz Medicare/Medicaid), media (viz FCC), etc. -- decisions take months or even years, and are based as much on lobbying and pandering as concrete knowledge. The markets, on the other hand, produce an efficient allocation in fleetingly small timeslices, and do so with complete objectivity. At these levels, greed even crowds out corruption.

I believe that those who object to the kind of automated trading described in the OP are dreaming that they could be the ones to score a windfall on some specialized knowledge, rather than have to play on the same playing field (scoured of arbitrage opportunities) that everyone else does.

alphaBetaGamma• 16 years ago

I think your understanding of the benefit of the stock market is flawed: the length of holding does not impact the fact that you are providing capital to companies.

The only time money goes from investors to companies is at public offerings (initial or otherwise). All other transactions in shares (be it by long term investors or by high frequency hedge funds) do not benefit the companies directly. But they do provide an indirect benefit: without the existence of these investors (i.e. the secondary market) people would be very reluctant to invest in an IPO as they would be no exit strategy. It is important for people buying in an IPO that the stock will be in demand in the future. The holding period of the people who will buy is irrelevant, the size and level of aggregate demand is not.

This being said, I am not arguing that have it is a good thing that the majority of volume is high frequency in nature.

URSpider94• 16 years ago

My understanding of why (the iPhone app store) is supposed to be good for society is that (it provides customers with many useful applications and games). It has the nice side effect of (putting money in the pockets of many small developers). I fail to understand how (1,000 different bodily-function simulation applications) forwards either of these goals. Maybe someone can explain why this is a good thing to me, but this really seems like a perversion of what (the app store) is supposed to be good for.

Markets exist because people want to be able to buy and sell things efficiently, in one place, and make money. They are not out to serve a charitable goal. Buy-and-hold investment is not somehow "better" or more moral than fast trading, both are just choices made by individuals, companies, or computer algorithms operating on their behalf.

Think of it this way -- there is a buyer or seller on each side of every transaction this algorithm does, who is willingly (even urgently) entering into that transaction.

arthurdent2• 16 years ago

That's not really accurate.

The stock market does not generally act as a mechanism to allow investors to funnel capital to companies. The stock market, as most people think about it, is really a secondary market for fractional ownership of companies.

The process by which that fractional ownership came to be owned by other people is the process by which companies are funded (companies go public and sell shares to the public. this is the major process by which the stock market helps companies raise capital (in exchange for equity)). So all the trading that happens intraday, or even holding some stock for years and years generally does not actually serve to invest in the company.

The question of whether this is a good thing is entirely orthogonal to the point of the article, which seems to be that reading news fast gives you information about markets.

While marketmaking is indeed a valuable market service as described by hugh3, the AI in question doesn't fill that role in any meaningful way (the AI stock picker seems to be a market taker -- it actually removes liquidity when it sees opportunities)

A defensible argument for algorithms like this is that they help maintain market efficiency/ensuring that risk in the marketplace is accurately priced (if such a thing is possible). The AI basically is incorporating news that the world knows and acting on it. If the AI didn't do it, then the markets would, but slower. The fact that its making money on 20 second timescales just means that its getting news 20 seconds faster than the rest of the market, which isn't particularly shocking because you might imagine that the rest of the world is responding without the benefit of an AI that reads/interprets articles in bulk.

BrentRitterbeck7• 16 years ago

One word: liquidity!

breck• 16 years ago

> Using data from five non-consecutive weeks in 2005, a period chosen for its lack of unusual stock market activity, here's how AZFinText performed versus funds that traded in the same securities (which were all chosen from the S&P 500):

This is silly. It's quite easy to generate an algorithm that would perform similarly. I wonder how it would have performed had it been run across the entirety of 2005 (or any year for that matter), and not just during cherry picked weeks.

hugh34• 16 years ago

Yes, I'd like to see it run across the entirety of the available data.

Also, if I came up with a method like this, and I actually had evidence that it worked, I'd be patenting it and selling it to an investment bank for very large sums of money before I published the details.

khafra• 16 years ago

Anthropic bias: The other thousand programmers that came up with such an algorithm used it or sold it to an investment bank; this is the first one that published instead.

Periodic• 16 years ago

I read the title as "AI that picks socks better than the pros". I was very intrigued. As I waited for the link to load I mused about what a damning report this must be for the fashion industry. Can a computer really out fashion the fashionistas? Alas, it was not so. The fashion elite are still unassailable in their fortress of subjectivity.

sp332• 16 years ago

>I mused about what a damning report this must be

I think you mean darning.

theli0nheart• 16 years ago

> Using data from five non-consecutive weeks in 2005, a period chosen for its lack of unusual stock market activity

Doesn't this completely invalidate the results of this test?

Compare with: I've developed a system for testing whether or not any given number is prime. Using data from the numbers 5, 7, 11, and 17, ...

wheaties• 16 years ago

You know, it makes MUCH better news if the software is a whizz-bang than if it lost you a crap-ton instead.

jgoewert• 16 years ago

That was my first thought as well.

Actually my first thought was: In the other consecutive weeks, the system lost its shirt and jumped out the window.... twice.

jfno67• 16 years ago

Also the effect of the system trading is not factored in. If low capitalization or low volume stock were used than the result would not have been as positive. You have to do the the trade otherwise you just get a nice system for collective2.com

aresant• 16 years ago

Catch 22 - if this system is popularized, it will change the way stocks are traded, and in the process will damage its own value by destroying the control index.

Eliezer2• 16 years ago

Shouldn't there be some way to flag stories as "headline is obviously false"?

rms• 16 years ago

I believe it has been discussed by pg as part of a proposed revamp to the flagging system. It'll happen eventually.

PassTheAmmo• 16 years ago

So if I made a system that traded automatically and I was convinced that it would beat the market, how would I go about connecting it to the stock exchange? What costs would there be?

desponsible2• 16 years ago

Something like this - http://www.interactivebrokers.com

MikeCapone• 16 years ago

I wonder how well covered their downside is. Will a black swan make it blow up spectacularly, negating any previous gains?

cameldrv• 16 years ago

These types of systems are now common in the market, and you need more sophisticated algorithms to make any money. They said that they picked 2005 because it had a lack of "unusual market activity." I bet they picked it because their system didn't work in later years, because there's too much competition to make money with their system now.

kurumo• 16 years ago

I work in this field. That is not the first paper written about this system; the previous was "Sentiment Analysis of Financial News Articles", Schumaker et. al. This is not the only academic project describing something like this, see papers by Mittermayer and Lavrenko for other examples. Short summary of the results is basically this: you would have to be insane to trade purely on output of a system like this. There are multiple issues with this, the primary being that most of these systems do not and cannot distinguish documents leading the trend from those lagging the trend. Furthermore, to actually be able to trade on the indicators derived from news you need to encode the market expectation into your model, which is a separate (and difficult) problem. The margins these systems produce in simulated trading are small, and for some reason nobody takes transaction costs into account. This is not to say that systems like this do not exist in the real world - they certainly do, but the people that build and/or use them generally will not discuss the details, for obvious reasons.

naner2• 16 years ago

Not really surprising. This been done before to some degree by Kurzweil and Schmidhuber and I'm sure others.

desponsible2• 16 years ago

Any pointers?

naner2• 16 years ago

Be really smart? No way in hell could I do this. You need a strong math, statistics, and machine learning background to begin with.

jheriko• 16 years ago

"Computer programs can gamble better than gamblers"

Sorry to say, but I'm completely unsurprised by this - statistical analysis is the most sensible approach to the problem. Gamblers tend to be less sensible and calling a gambler a stock market trading pro doesn't stop him being a gambler...

I'd also be wary of labeling pros in a system which will naturally produce "lucky" and "unlucky" individuals regardless of skill... at least some proportion of successful traders are just lucky. I can't measure it, but it seems very heavily implied...

Maven911• 16 years ago

I have a question for all of you...how do so many of you know a lot about financial markets and finance...I made an Ask HN about it:

http://news.ycombinator.com/item?id=1422453

AI That Picks Stocks Better Than the Pros

Related Stories