Financial Statement Analysis with Large Language Models

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4835311

by mellosouls • 2 years ago

573 206 2 years ago

papers.ssrn.com

davidw• 2 years ago

If that's a thing, does it become a thing learning how to write a "poisoned" statement that misleads an LLM but is still factual?

SirLJ• 2 years ago

It is already been done as quants analyze company statements for the last decade at least, counting positive and negative words, etc...

Drakim• 2 years ago

Maybe, but it sounds hard if there are multiple LLMs out there that people might use to analyze such text. Tricking multiple LLMs with a certain poisonous combination of words and phrases sounds a lot like having a file that hashes to the same hash from different hashing techniques. Theoretically possible but actually practically impossible.

infecto6• 2 years ago

As someone alluded to, the narrative that management drives has been examined and studied many times over. What is management saying, what are they not saying, what are they saying but not loudly, what did they say before that they no longer speak about. There are insights to glean but nothing that is giving you an unknown edge. Sentiment analysis and the like go back well into the late 80s, early 90s.

dpflan2• 2 years ago

I recalling seeing a LinkedIn post by Greg Diamos at Lamini, they shared analysis of earnings calls. There are links on HuggingFace and GitHub, here they are:

- https://huggingface.co/datasets/lamini/earnings-calls-qa

- https://huggingface.co/datasets/lamini/earnings-raw

- https://github.com/lamini-ai/lamini-earnings-calls/tree/main

deadmutex• 2 years ago

It would've been interesting to compare models with larger context windows, e.g. Gemini with 1m+ tokens and Claude Opus. Otherwise, the title maybe should've been Financial Statement Analysis with GPT-4.

infecto6• 2 years ago

The study only captured the financial statements. I am unsure what a larger context window would buy you.

foota3• 2 years ago

You could dump all of a companies financial statements together, or dump all of a companies competitors in with it, for one.

antimatter15• 2 years ago

Figure 3 on p.40 of the paper seems to show that their LLM based model does not statistically significantly outperform a 3 layer neural network using 59 variables from 1989.

  This figure compares the prediction performance of GPT and quantitative models based on machine learning. Stepwise Logistic follows Ou and Penman (1989)’s structure with their 59 financial predictors. ANN is a three-layer artificial neural network model using the same set of variables as in Ou and Penman (1989). GPT (with CoT) provides the model with financial statement information and detailed chain-of-thought prompts. We report average accuracy (the percentage of correct predictions out of total predictions) for each method (left) and F1 score (right). We obtain bootstrapped standard errors by randomly sampling 1,000 observations 1,000 times and include 95% confidence intervals.

infecto6• 2 years ago

Was going to point out the same. Glad to have the paper to read but I don't think the findings are significant.

foota3• 2 years ago

I agree this isn't earth shattering, but I think the benefit here is that it's a general solution instead of one trained on financial statements specifically.

flourpower4718• 2 years ago

Not to mention, as somebody who works in quant trading doing ml all day on this kind of data. That ann benchmark is nowhere near state of the art.

People didn't stop working on this in 1989 - they realised they can make lots of money doing it and do it privately.

bethekind• 2 years ago

Do you use llama 3 for your work?

bossyTeacher3• 2 years ago

People didn't stop working on this in 1989 - they realised they can make lots of money doing it and do it privately.

Mind elaborating?

benreesman• 2 years ago

I never traded consistently and successfully but I did do a startup with a seasoned quant trader with the ambition of using bigger models to generate novel alpha. We mopped the floor with the academics who publish but that is whiffle ball compared to a real prop outfit that lasts.

Not having made it big myself I obviously don’t know the meta these days, but last I had any inside baseball, the non-stationarity and friction just kill you on trying to get fancy as opposed to just nailing it on the fundamentals.

Extreme execution quality is a game, people make money in both traditional liquidity provision and agency execution by being fast as hell and managing risk well.

Individual signals that are individually somewhat mundane but composed well via straightforward linear-ish regressions is a game: people get (ever decaying) alpha out of bright ideas (and rotate new signals in).

And I’m sure that LLMs have started playing a role, there’s a legitimate capability increase in spite of the dubious production-worthiness.

But as a blind wager, I bet prop trading is about what it was 5 years ago on better gear: elite execution (no pun intended) on known-good ways to generate alpha.

tempodox2• 2 years ago

But I bet it uses way more energy.

richrichie2• 2 years ago

The infamous 1/N portfolio comparison is missing. 1/N puts to shame many strategies.

primitivesuave2• 2 years ago

The area where I see this making the most transformational change is by enabling average citizens to ask meaningful questions about the finances of their local government. In Cook County, Illinois, there are hundreds of local municipalities and elected authorities, all of which are producing monthly financial statements. There is not enough citizen oversight and rarely any media attention except in the most egregious cases (e.g. the recent drama in Dolton, IL, where the mayor is stealing millions in plain view of the citizens).

apwell232• 2 years ago

citizens to ask meaningful questions about the finances of their local government.

is there a demand for this. I live in cook country. I really don't want to ask these questions. Not sure what I get out of asking these questions other than anger and frustration.

Kon-Peki• 2 years ago

Municipal bond rating agencies should be a client of such data.

And if not the rating agencies, people who invest in municipal bonds.

m463• 2 years ago

if all the citizens can ask these questions, I think it will make a difference.

and of course, the follow-up questions. Like who.

vharuck• 2 years ago

Most people won't care most of the time. But if the local government cuts the budget for something you like and says "We couldn't find the money," you may care that year.

abdullahkhalids• 2 years ago

The citizens ask LLMs (or more advanced future AIs) to identify if government finances are being used efficiently, and if there is evidence of corruption.

The corrupt government officers then start using the AIs to try to cover up the evidence of their crimes in the financial statements. The AI possibly putting the skills of high-end and expensive human accountants (or better) into the hands of local governments.

Who wins this attrition war?

berkes2• 2 years ago

Who wins this attrition war?

The AI companies. Double. People paying to use their products. But mostly by gaining a lot of leverage and power.

airstrike• 2 years ago

The corrupt government officers then start using the AIs to try to cover up the evidence of their crimes in the financial statements.

There's a difference between an AI being able to answer questions and it helping cover up evidence, unless you mean "using the AIs for advice on how to cover up evidence"

ketzo• 2 years ago

Corrupt government officers are one thing. But there is a ton of completely well-meaning bureaucracy in the U.S. (and everywhere!) that could benefit from a huge, huge step change in "ability to comprehend".

Bad actors will always exist but I think there's a LOT of genuine good to be done here!

oceanplexian2• 2 years ago

The corrupt government officers then start using the AIs

You're making it way too complicated. The government will simply make AI illegal and claim it's for safety or something. The'll then use a bunch of scary words to demonize it, and their pals in the mainstream media will push it on low-information voters. California already has a bill in the works to do exactly this.

hnthrowaway6543• 2 years ago

Let's say LLMs work exactly as advertised in this case: you go into the LLM, say "find corruption in these financial reports", and it comes back with some info about the mayor spending millions on overpriced contracts with a company run by his brother. What then? You can post on Twitter, but unless you already have a following it's shouting into the void. You can go to your local newspapers, they'll probably ignore you; if they do pay attention, they'll write an article which gets a few hundred hits. If the mayor acknowledges it at all, they'll slam it as a political hit-piece, and that's the end of it. So your best chance is... hope really hard it goes viral, I guess?

This isn't meant to be overly negative, but exposing financial corruption is mostly about information control; I don't see how LLMs help much here. Even if/when you find slam-dunk evidence that corruption is occurring, it's generally very hard to provide evidence in a way that Joe Average can understand, and assuming you are a normal everyday citizen, it's extremely hard to get people to act.

As a prime example, this bit on the SF "alcohol rehab" program[0] went semi-viral earlier this week; there's no way to interpret $5 million/year spent on 55 clients as anything but "incompetence" at best and "grift and corruption" at worst. Yet there's no public outrage or people protesting on the streets of SF; it's already an afterthought in the minds of anyone who saw it. Is being able to query an LLM for this stuff going to make a difference?

[0] https://www.sfchronicle.com/politics/article/sf-free-alcohol...

beepbooptheory• 2 years ago

Are people supposed to be outraged that that is too little or too much money?

That's still cheaper than sending them to prison!

Scubabear68• 2 years ago

Oh yeah. This. I live in a tiny community it our district school board has a $54 million budget right now, and all the audits are rubber stamps and wink and nudge from the State. When residents try to dig in and complain about waste and fraud we are shrugged off.

primitivesuave2• 2 years ago

Beautifully stated. I can only speculate, but I'd say the reason it is this way is due to the collective apathy/cynicism toward government. We have collectively come to expect a certain level of corruption and influence peddling. We have a high tolerance for incompetence in carrying out government operations. Only the most egregious offenders are brought to the public's attention, and in an age of increasingly short attention spans, people have forgotten by the time elections roll around.

That is, if they vote in the first place - in that example I gave above of a corrupt mayor stealing millions (Tiffany Henyard of Dolton, IL), the voter turnout was only 15%.

mulmen• 2 years ago

Why would you report financial crimes to Twitter? If your LLM uncovers financial crimes you should contact regulators and prosecutors. They're both incentivized to do something about it.

tchalla• 2 years ago

It’s your assumption that the lack of oversight is because of too much information. How will you validate that hypothesis before you invest in a solution?

kenjackson• 2 years ago

I think this is in general one of the big wins with LLMs: Simple summarization. I first encountered it personally with medical lab reports. And as I noted in a past comment, GPT actually diagnosed an issue that the doctors and nurses missed in real-time as it was happening.

The ability to summarize and ask questions of arbitrarily complex texts is so far the best use case for LLMs -- and it's non-trivial. I'm ramping up a bunch of college intern devs and they're all using LLMs and the ramp up has been amazingly quick. The delta in ramp up speed between this and last summer is literally an order of magnitude difference and I think it is almost all LLM based.

myth_drannon• 2 years ago

that's what I did with my town financial report. Asked chatGPT to find irregularities. The response was very concerning, with multiple expenses that looked truly very suspicious (like planting a tree - 2000$). I would have gone berserk at the town council meeting if I was an activist citizen.

3abiton• 2 years ago

Then companies tends to utilize LLMs to maximize the confusion of a message to the shareholders, cat and mouse game.

rgmerk• 2 years ago

“There are hundreds of local municipalities and elected authorities.”

That’s the root of your problem. Too many governments, not enough attention available to keep them accountable.

anonu2• 2 years ago

The most transformational change is by enabling average citizens to ask meaningful questions about their finances.

Around half of adults in the US are financially illiterate.

chollida12• 2 years ago

So the history of this type of research as I know it was that we

- started to diff the executives statements from one quarter to another. Like engineering projects alot of this is pretty standard so the starting point is the last doc. Diffing allowed us to see what the executives added and thought was important and also showed what they removed. This worked well and for some things still does, this is what a warrant canary does, but stopped generating much alpha around 2010ish.

- simple sentiment. We started to count positive and negative words to build a poor mans sentiment analysis that could be done very quickly upon doc release to trade upon. worked great up until around 2013ish before it started to be gamed and even bankruptcy notices gave positive sentiment scores by this metric.

- sentiment models. Using proper models and not just positive and negative word counts we built sentiment models to read what the executives were saying. This worked well until about 2015/2016ish in my world view as by then executives carefully wrote out their remarks and had been coached to use only positive words. Worked until twitter killed the fire hose, and wasn't very reliable as reputable news accounts kept getting hacked. I remember i think AP new's account got hacked and reported a bombing at the white house that screwed up a few funds.

You also had Anne Hathaway news pushing up Berkshire Hathaway's share price type issues in this time period.

- there was a period here where we kept the same technology but used it everywhere from the twitter firehose to news articles to build a realtime sentiment model for companies and sectors. Not sure it generates much alpha due to garbage in, garbage out and data cleaning issues.

- LLMs, with about GPT2 we could build models to do the sentiment analysis for us, but they had to be built out of foundational models and trained inhouse due to context limitations. Again this has been gamed by executives so alot of the research that I know of is now targeted at ingesting the Financials of companies and being able to ask questions quickly without math and programming.

ie what are the top 5 firms in the consider discretionary space that are growing their earnings the fastest while not yet raising their dividends and whose share price hasn't kept up with their sectors average growth.

nebula8804• 2 years ago

I have no window into this world but I am curious if you know anything about the techniques that investors used to short or just analyze Tesla stock during the production hell of 2017-2020? It was an interesting window in ways that firms use to measure as much of the company as they can from the outside. In fact was there any other stock that was as heaving watched during that time?

Looking back at that era it seemed investors were too focused on the numbers and fundamentals, even setting up live feeds of the factories to count the number of cars coming out and thats the same feeling I get from your post. It seems like dumb analysis ie. analysis without much context.

We now know from the recent Isaacson biography what was happening on the other side. The shorts failed to measure the clever unorthodox ways that Musk and co would take to get the delivery numbers up. For example: The famous Tent. Musk used a loophole in CA laws to set up a giant tent in the parking lot and allowed him to boost the production by eliminating entire bottlenecks from the factory design. There is also just the religious like fervor with which the employees wanted to beat the shorts. I dont think this can be measured no? It helped to get them past the finish line.

refulgentis• 2 years ago

Markets aren't sports teams, i.e. bimodal camps with us vs. them drama. Twitter discussion of markets, maybe, but not markets.

I've been on both sides of this trade, regularly.

Bear thesis back then was same as now. In retrospect, I give it a few more credits because Elon says they were getting close to bankrupt while he was posting "bankwupt" memes and selling short shorts.

Being a pessimist, and putting your money where your mouth is in markets, is difficult because you have to be right and have the right timing.

helsinkiandrew• 2 years ago

Most companies aren’t obsessed enough with shortens to try and hide poor results from analysis that will be exposed in 3 months anyway. There’s always ways around them - number of cars registered, number of delivery trucks visiting, time for delivery on website, how much overtime is being worked etc.

nicce2• 2 years ago

What will happen if everyone starts using heavy statistical methods or LLMs to predict stocks prices? And buys stock based on them? Will it absolutely make everything unpredictable?

Edit: assuming that they initially provide good predictions

infecto6• 2 years ago

This has already been a thing since the late 80s.

nicce2• 2 years ago

It hasn't been accurate enough to be meaningful, nor enough data.

surfingdino2• 2 years ago

HFT guys won't touch GPT. The stakes are too high. If LLMs could give those guys an edge they'd be all over this tech.

berkes2• 2 years ago

Or rather: If LLMs could give those guys an edge, there's no way they'd share their edge-giving LLMs with anyone, least of all their competition and the plebs.

tempusalaria• 2 years ago

A number of quant firms are among the largest global consumers of commercial LLMs

The only area you absolutely can’t use LLMs is in sub ms latency

lucianbr• 2 years ago

Isn't it already unpredictable? That is why nobody outperforms indices. And utterly irrational, which is again both expected and seen. This must be why Tesla continues to have a huge market value - Elon knows how to excite the LLMs. :)

01100011• 2 years ago

Top story on HN because we all secretly think we can be the next Jim Simons when in reality we're a few months away from posting loss porn to /r/WSB.

If standardized LLM models are used to analyze statements, expect the statements to be massaged in ways that produce more favorable results from the LLM.

tempodox2• 2 years ago

Yep, Goodhart's law is universal.

bbor2• 2 years ago

I can’t wait until there’s warnings on stock market apps like cigarettes and lottery tickets. Well actually I guess there are no warnings on lotto tickets, probably for the exact same reason as why the government doesn’t protect people from being scammed by hedge funds with way more info than they have: the government needs that revenue.

mb5• 2 years ago

CFD trading apps have this in the UK (and possibly EU, but I can’t see whether ESMA kept the rule):

“78% of retail investor accounts lose money when trading CFDs with this provider.”

https://www.fca.org.uk/news/press-releases/fca-confirms-perm...

Of course, there are a lot more ways to lose money than CFDs.

doctoboggan2• 2 years ago

If this were to become widely used, I can imagine executives writing financial statements, running them through an LLM, and tweaking it until they get the highest predicted future outcome. This would make the measure almost immediately useless.

oceanplexian2• 2 years ago

This is already how it works. Have you listened to an earnings call? Especially companies like Tesla? They are a dog and pony show to sell investors on the stock.

doctoboggan2• 2 years ago

I am not saying executives aren't currently trying to game the system. I am saying currently the best they can do is estimate how thousands of analysts will respond. If LLM analysts become wide spread then they would be able to run simulations in a feedback loop until their report is optimized.

perkolator• 2 years ago

Anyone who has read many financial statements would understand this whole idea is already useless outside of financial statement education.

LLMs could help a person learn to understand financial statements better, that is it.

There are not all these hidden gems in financial statements though that are being currently missed that language models are going to unearth.

Financial statements are already intentionally vague and often intentionally misleading.

Corporate CEO/PR/Marketing are already the masters of writing many words while saying absolutely nothing.

rafaelero• 2 years ago

Still, if you use GPT-4 it gives you 60% of accuracy in predicting if it's going to go up and down, which is considerably better than median human forecasters. Stop being so dismissive and start reading the numbers.

jellicle• 2 years ago

People are going to lose a LOT of money using this when the LLM says "buy" and old-school humans who read the same statement say "sell".

But up until that day, it will probably be cheaper.

dangerwill• 2 years ago

I guess this makes sense. Because while there should be some noise from the text translation into the internal representation of the financial data once ingested into the model, the authors purposefully re-formatted all the reports to be formatted consistently. That then should allow the model to essentially do less of the LLM magic and more plain linear regression of the financial stats. And often past performance does have an impact on future performance, up to a point.

I wonder what the results would have been with still-anonymized but non-fully standardized statements.

Still though, impressive.

nvy• 2 years ago

You can scrape the filings from EDGAR, which presents the statements in standardized format.

repeekad• 2 years ago

fintool.com already has an LLM on top of EDGAR

nextworddev2• 2 years ago

To everyone thinking they can sell a LLM wrapper based on this - this is a very tough domain. You will soon run into data, distribution, and low demand. Funds that would actually use this are already using it.

hadlock• 2 years ago

Most every company I know has had a drag and drop PDF to LLM tool since Q1 2023. It's not a phenomenally difficult product to code up.

nextworddev2• 2 years ago

Not talking about a Chat-with-PDF solution. Talking about some large scale SEC filing analysis and visualization solution.

avidphantasm• 2 years ago

Great. Humans no longer need to cook the books and can claim plausible deniability. The only problem is the hallucination errors could go against you as well as for you.

caseyy3• 2 years ago

They may claim, but there is no such plausible deniability. Not for lawyers using AI hallucinations, not for Tesla drivers crashing into things with FSD, not for tax fraud. People are ultimately held responsible and accountable for the way they use AI.

einpoklum• 2 years ago

Let's just skip ahead to just implementing the script of Idiocracy verbatim and be done with it.

motoboi• 2 years ago

Please add "and the financial result of this analysis will count toward your annual bonus" to the prompt.

jrochkind1• 2 years ago

how do you distinguish enterprise-level question-answering from other kinds of question-answering?

dankai• 2 years ago

if you want to see successful "machine learning based financial statement analysis", check out my paper & thesis. its from 2019 and ranks #1 for the term on google and gs because it is the first paper that applies a range of machine learning methods to all the quantitative data in them instead of just doing nlp on the text. happy to answer questions

paper https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3520684

thesis https://ora.ox.ac.uk/objects/uuid:a0aa6a5a-cfa4-40c0-a34c-08...

caseyy3• 2 years ago

In this section, we aim to understand the sources of GPT’s predictive ability.

Oh boy... I wonder how a neural net trained with unsupervised learning has a predictive ability. I wonder where that comes from... Unfortunately, the article doesn't seem to reach a conclusion.

We implement the CoT prompt as follows. We instruct the model to take on the role of a financial analyst whose task is to perform financial statement analysis. The model is then instructed to (i) identify notable changes in certain financial statement items, and (ii) compute key financial ratios without explicitly limiting the set of ratios that need to be computed. When calculating the ratios, we prompt the model to state the formulae first, and then perform simple computations. The model is also instructed to (iii) provide economic interpretations of the computed ratios.

Who will tell them how an LLM works and that the neural net does not calculate anything? It only predicts the next token in a sentence of a calculation if it's been loss-minimized for that specific calculation.

It looks like these authors are discovering large language models as if they are some alien animal. When they are mathematically describable and really not so mysterious prediction machines.

At least the article is fairly benign. It's about the type of article that would pass as research in my MBA school as well... It doesn't reach any groundbreaking conclusions except to demonstrate that the guys have "probed" the model. Which I think is good. It's uninformed but not very misleading.

famouswaffles2• 2 years ago

Who will tell them how an LLM works and that the neural net does not calculate anything?

You don't understand LLMs as well as you think you do. Yes, the neural network calculates things.

It only predicts the next token in a sentence of a calculation if it's been loss-minimized for that specific calculation.

No that's not necessary at all.

https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mec...

https://cprimozic.net/blog/reverse-engineering-a-small-neura...

caseyy3• 2 years ago

I have heard of generalization vs memorization, but the article you shared is very high quality. Thank you.

I do not think that SOTA LLMs demonstrate grokking for most math problems. While I am a bit surprised to read how little training is necessary to achieve grokking in a toy setting (one specific math problem), the domain of all math problems is much larger. Also, the complexity of an applied mathematics problem is much higher than a simple mod problem. That seems to be what the author of the first article you quoted thinks as well.

Our public models fail in that large domain a lot. For example, with tasks like counting elements in a set (words in a paragraph). Not to mention that they fail in complex applied mathematics tasks. If they have been loss-minimized for that specific calculation to the point that they exhibit this phase change, then that would be an exception.

But in the financial statement analysis article, the author says explicitly that there isn't a limitation on the types of math problems they ask the model to perform. This is very, very irregular, and there are no guarantees that model has generalized them. In fact, it is much more likely that it hasn't, in my opinion.

In any case, thank you again for the article. It's just such a massive contrast with the MBA article above.

SCM-Enthusiast• 2 years ago

Phase changes and grokking make me nervious... It seems once you reach a certain threshold of training, you can continually "phase-change" and generate these emergent capabilities. This does not bode well for alignment.

Animats• 2 years ago

How do they get a LLM to do arithmetic?

JonChesterfield2• 2 years ago

By not worrying whether it got the answer right. As long as some numbers go in and some numbers come out, vibes are good enough.

mewpmewp2• 2 years ago

Not sure about them, but usually you would let LLM execute code to calculate it.

uncomplete• 2 years ago

Headline should be 'LLM can read (and understand) PDFs'

diziet• 2 years ago

The fact that the paper does not mention the word "hallucinations" in the full body text makes me think that the authors aren't fully familiar with the state of LLMs as of 2024.

thedudeabides5• 2 years ago

bait

worik• 2 years ago

I am disturbed to see so much enthusiasm here for "trading"

Markets matter, and some speculation is useful, but the purpose of markets is not speculation. Obviously.

If you want to make some money get trained and get a good salary. Save your money in safe assets

if you want to get supper rich be super creative, ensure going broke will only effect you (i.e. do not do this while supporting a family), and found a firm. You will likely fail, but there is a chance of super wealth and a bigger chance of a wild ride that will be good for you

Trading from the perspective of greed runs the risk of total destruction. Putting you in jail, maybe. Bankruptcy if not too unlucky. Many people out of work because of your misallocation, and if you do not care about that I'm not interested in you

The financial system is a zero sum game. (The economy in general is not) There is always someone cleverer and they likely do not care if they crush you. International finance is a snake pit

Friends, look after friends. Maximise happiness. Be honest, be ethical, be safe

Live long and prosper

Terretta• 2 years ago

the purpose of markets is not speculation

Arguably, the purpose of markets is part price discovery, part liquidity, and arguably mostly to support economic growth and stability by channeling funds from savers to those who can invest them productively.

If you want to make some money get trained and get a good salary. Save your money in safe assets

Sure, as long as you're not into dynamism.

latency-guy22• 2 years ago

You assert a lot of personal opinions. On top of bad "characterizations" of people who are not you.

I don't think this is a signature of a person being honest, or trustworthy.

flourpower4718• 2 years ago

>> If you want to make some money get trained and get a good salary. Save your money in safe assets.

You mean like the good salary you get working for a trading firm?

I'm not sure what this comment you made is meant to be, but it reads like a blend of somebody who's high and a tik tok wellness influencer.

Trading is not a zero sum game in the sense you intend to suggest it is. It is 0 sum only if all participants have the same trading horizon.

The pool grows in the same way because it is linked to the economy. The markets are a variety of players with different requirements.

Most transactions occur between parties who have different horizons. Yes the hft makes money over 5s and the pension fund loses it. But the pension fund is looking at the return over the next year, so the small loss to the hft is just a cost of acquisition.

latency-guy22• 2 years ago

It's a long winded insult assuaged by a meme phrase at the end to not appear as one, I wouldn't treat it as anything more, what you've said is pretty apt though.

NicoJuicy• 2 years ago

The S&P 500 is obviously not a zero sum game, proof me wrong over the last decades.

Additionally, a wage is the same amount increase per month, while stock is in percentages.

Can't beat percentages depending on your wage/buy-in.

lxgr• 2 years ago

Don't even know where to begin with this:

No, not all of finance is a zero-sum game. If you're connecting a buyer and seller that otherwise wouldn't have met, you provided value. Same for connecting them through time (in that you can e.g. help prevent somebody having to panic-sell their house from getting a suboptimal price).

Sure, there's speculation, nepotism, corruption; there are immoral and illegal market practices with no end, but you're making it sound like that's the entire purpose of finance, and not an undesirable byproduct.

Also, as if these only exist there, and not everywhere where there is power and money: Politics, business, even charity are not immune.

Starting a company is more ethical than trading – seriously? While there might be a general trend, can you think of no philanthropic traders and of no unethical founders (some of them in jail)?

If you want to make some money get trained and get a good salary. Save your money in safe assets

100% agreement on the first part. But if everybody invests their money in "safe assets", there is no capital for people to start companies other than banks. Is that desirable? And who even determines what a safe asset is? What about people that manage and allocate risk? That's a function of finance again!

Friends, look after friends. Maximise happiness. Be honest, be ethical, be safe

I agree, but this arguably has little to do with the remainder of your sweeping generalization.

TacticalCoder• 2 years ago

I am disturbed to see so much enthusiasm here for "trading"

It is lots of fun. Very mathy. A nerd's dream.

Then the data. Oh the amount of data. 34 Gbit/s to get the full US options feed last I checked (someone posted that here I think). Much of the rest is kiddie stuff compared to dealing with that.

People can lament has much as they want that it drains the great minds: it is fun.

I didn't invent that game. Don't blame the players.

DonsDiscountGas• 2 years ago

A key research design choice that we make is to not provide any textual information (e.g., Management Discussion and Analysis) that typically accompanies financial statements. While textual information is easy to integrate, our primary interest lies in understanding the LLMs’ ability to analyze and synthesize purely financial numbers. We use this setup to examine several research questions

Seems like an odd test for a large language model. There are tabular models out there

bbor2• 2 years ago

Oh damn, I thought this was about real finances, turns out it’s just part of that weird “property” thing they do in NYC. I hope someone’s working on feeding their Ledger files into a structured language model…

More substantively, LLMs are for linguistic tasks. That’s why I’m super super bullish (heh) on llms for decoding EEG data, and incredibly bearish on their ability to accurately model a corporation’s asset flow. I just don’t see how the confounding variables / motivating forces would be at all linguistic. This is basically using LLMs for super-advanced arithmetic

pfdietz• 2 years ago

This would be an excellent target for poisoning.

gptisking2• 2 years ago

this is a great paper

gptisking2• 2 years ago

great paper

ryzvonusef• 2 years ago

Here is a view of an accountant trying to use AI with a financial statement:

https://www.youtube.com/watch?v=VxxmzoZTRW4

there are other videos on their youtube channel on the more analytical aspect of it, I just decided to share the latest one.

It's been a hit and miss for now, depending on the model used (chatgpt/gemini/clause etc.) the results can vary somewhat.

blacktechnology• 2 years ago

Is there any product using this theroy?

rulalala• 2 years ago

This research and the comments mentioning the need of being fast as hell connect with the human capacity of using the information. Just a hypothetical point: at some point only trading engines driven by AI could make a timely use of this source.

bionhoward• 2 years ago

Just don’t train em on GME, gets tunnel vision on the meme price spikes and ignores / gets worse on other stocks. Pretty funny

anonu2• 2 years ago

From a first principles approach: it does not really make sense to use an LLM to do fundamental analysis directly. Maybe you can use an LLM to write some python code to do fundamental analysis. But skipping that model building step and just applying a language model to numbers does not make intuitive sense to me.

I am surprised at the results in the paper. The biggest red flag is that the researcher are not sure why there is predictive ability in LLMs. Maybe they didn't control for some lookahead bias.

gromneer• 2 years ago

I don't have a horse in the race. I don't even know what financial statement analysis is. But it worries me that a novel reliance on these models for traditionally skilled labor jobs will turn into a dependence. These models use the built up experience of human practitioners to achieve similar results. But if a dependence grows, then there will be no more skilled human practitioners to further develop improved skills and knowledge for these jobs. Calcifying the skills in time.

namaria• 2 years ago

That wouldn't be a problem if the models actually worked for that. We don't miss cobblers.

But these models only seem to perform these jobs on the surface. Enough that companies will try them and waste resources. This will just hurt the bottom line optimizer shops and boost the professionals doing quality work on the long run.

akrymski• 2 years ago

But GPT4 has been trained up to recent events, so you can't do rolling predictions using just historical data with it. If your LLM knows the future obviously it can predict well, even if the company name is hidden.

EternalFury• 2 years ago

So, LLMs can’t do basic arithmetic very well at all but people use them to analyze financial statements?

fudiso• 2 years ago

great paper.

Financial Statement Analysis with Large Language Models

Related Stories