How to get stock fundamentals data with Python

http://theautomatic.net/2020/05/05/how-to-download-fundamentals-data-with-python/

by atreadw • 6 years ago

129 49 6 years ago

pmiller25• 6 years ago

This should be fun to play with in a Jupyter notebook. :)

mmaunder2• 6 years ago

My sense is that the golden age of the home API trader is about to emerge and I'd love to hear how things are going for other tinkerers. From my side, I decided to dive back into math, which I've ended up enjoying so much that I haven't surfaced yet to put any of it to work.

jxm2624• 6 years ago

Been trading for roughly 3 years now. And yup, I'm exactly in the same boat, just recently dove back into filling all the holes i have in my knowledge (portions of math/stats)

mmaunder2• 6 years ago

Khan is so great for that. Sal's contribution to society is massive. And he's great to listen to. Even at 1.5X speed. :-)

Math is funny that way. Miss out on some fundamental aspect and it'll ruin you. And Khan explains some really basic concepts in ways that has given me new perspective or insight.

tel2• 6 years ago

How does one get started? I feel like I have a lot of the math background, but approaching putting this toward an actual market feels opaque.

modo_• 6 years ago

Do you think this is the beginning of the golden age for home API traders due to market conditions or the tooling that enables home traders to operate getting better and better?

otoburb• 6 years ago

Not the OP, but I think it's the latter (tooling getting better and easier) combined with lower (or close-to-zero) trade execution fees and more widely available information.

AQuantized• 6 years ago

I've been up to this recently. None of the maths I've used is very advanced yet. I've mostly been trying to use simple statistical tools like regression analysis as well as possible. I know a few people who have found employed in finance after having done graduate work in stochastic processes, but I'm not sure what the payoff would be for some of the more advanced stuff as a non-institutional trader.

jidiculous• 6 years ago

cool tool! fyi you should probably add a license of some kind in your github repo.

mattbillenstein• 6 years ago

I've been playing with Alpaca - they give you access to a lot of financial data through their apis without relying on scraping Yahoo...

chosenbreed37• 6 years ago

I didn't see much scraping on that site. It just looked like regular calls. Maybe the API is not as clean or consistent as it could be. But anyway I did a quick search for Alpaca. This the one you meant: https://alpaca.markets/?

lukasluke• 6 years ago

Here's an API guide for alpaca if you're keen: https://algotrading101.com/learn/alpaca-trading-api-guide/

Disclaimer: One of my writers wrote it for my blog

submeta• 6 years ago

Ahh, thw beauty and versatility of Python again. And the momentum doesn’t stop. Never regret jumping off of Perl, then Ruby ten years ago to fully commit to Python. I can see so many use-cases for it (with the rich ecosystem), on an almost daily basis.

anonu• 6 years ago

Python quickly falls over with large datasets. Its great as a "glue" language or for POCs for large data processing or handling jobs - but at some point you need to graduate to faster things...

pletnes2• 6 years ago

What’s «large»? I’ve set up multi-TB analyses without issue, using dask. I don’t know of a more productive language for such analyses, either. What is even better?

gdulli• 6 years ago

Language speed becomes mostly irrelevant compared to the strategy, framework, toolset, and dataset design for parallelization.

If you have a use case where a program in C runs in a reasonable time on your laptop but one in Python doesn't, that solution is only going to take you so far before you'll want to graduate from your laptop and take advantage of the Python ecosystem again in real big data contexts.

jacobr1• 6 years ago

Between things like pyspark or using a rdbms you can scale pretty far. Even with other languages like java this is the case. Once your dataset goes beyond a single machine you need some kind of data platform.

pmiller25• 6 years ago

You know, actual databases are something you can "glue" onto a Python infrastructure.

Scarbutt• 6 years ago

Any programming language with a http client and a sqlite driver won't have problems with this example. Although, this is so simple that even sqlite might be overkill for analyzing the data.

mendelbot• 6 years ago

anybody know of a good Python API for options prices? I don't think Alpaca has them

pathseeker• 6 years ago

In a similar boat. Would like historical bid/ask if possible and live bid/ask (not 15 min delay). Willing to pay if it's not hundreds per month.

TuringNYC3• 6 years ago

I got them from InteractiveBrokers a while back. It was definitely not hundreds per month, more like 15/mo

https://www.interactivebrokers.com/en/index.php?f=14193#coll...

Schweigi• 6 years ago

I bought historical option data in the past directly from CBOE [0]. Depending on what you need the prices are very reasonable. E.g. for a single symbol ~$70-$80 for a year depending on the interval you need. There is a big discount if you just buy all the symbols.

I cannot recommend the stock prices though. Somehow there were just a lot of mistakes and you can get them from other sources cheaper.

[0] https://datashop.cboe.com

jonknee• 6 years ago

TD's API supports option chains. They don't have a good first party Python library, but it's pretty straight forward.

https://developer.tdameritrade.com/option-chains/apis/get/ma...

IEX only does end of day pricing and even that is delayed until the next day.

JCBarry• 6 years ago

Check out Tradier, https://documentation.tradier.com

In full disclosure, I work at Tradier.

larrykubin• 6 years ago

For anyone interested in stock market API's and financial data, I started a YouTube channel on this topic last year which is starting to gain some traction:

https://www.youtube.com/parttimelarry

It covers popular services and libraries like Alpaca, Robinhood Private API, TD Ameritrade option prices, Yahoo Finance, TradingView alerts, backtesting, and a lot more.

TuringNYC3• 6 years ago

Curious if you have used your foundational data infra for trading live? Were you able to beat a benchmark (e.g., beta-adjusted index returns?)

I'm asking because I've used semi-pro and pro infra (paid services) and it was hard to get consistent returns. It was easy to get returns, but rarely beyond, say, S&P 500 returns (at which point, I might as well just invest in the S&P 500 index.)

rmrfstar2• 6 years ago

Yup.

What you really want to know is "how problematic is the noise in this data"?

One way to answer it is to create your own FF sorts [1] and regress your Yahoo-derived factor returns against Ken French's (which come from CRSP/Compustat). If the intercepts are small/statistically insignificant and the R2's reasonable, then you're all set.

The big problem you will hit is survivorship bias in Yahoo!. My own research suggests that the quality is perfectly acceptable, provided you back-fill an unbiased universe (e.g. Russell 3K) from another source.

I'm actually surprised you didn't outperform the market with the survivorship bias. Back test must have covered a very small time period.

[1] https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data...

albanberg• 6 years ago

Looks awesome, I subscribed!

madballster• 6 years ago

What are some good fundamental data providers for small-time amateur investors (non-professionals)? I am looking for 1. US + international stocks 2. deep history of all historical accounting records (15+ years) 3. api, ideally with python examples

So far I have only found https://eodhistoricaldata.com and haven't tested them yet, is there anything better?

bsamuels• 6 years ago

They're the only provider if you're not willing to spend several thousand per year. Beware that a lot of their info is scraped from other sources and does not have accuracy guarantees.

You can also scrape the fundamentals directly from the 10-K statements each company publishes, but it's very difficult to get a clean, consistent dataset out of it.

nojito2• 6 years ago

Almost no amateur investor data provider has data integrity guarantees.

You're almost always better off getting the data yourself and cleaning it.

longtermd• 6 years ago

To be honest, I'm surprised yahoo didn't ban web scraping in their terms of service.

PhantomGremlin• 6 years ago

Does anybody know anything else about this?

Is Yahoo cool with web scraping? Is it just something they tolerate? Are they just too incompetent to care?

duke_core• 6 years ago

For me I know about what this data stands for and what it represents, I just don't know what to do _after_ acquiring this sort of data. Any pointers on that? I live in a really poor country where I can't do algotrading anyways but the eventual goal is to venture into that field by moving abroad, any pointers on what I can build right now with free APIs or from data like this?

How to get stock fundamentals data with Python

Related Stories