Open Football Data

http://openfootball.github.io

by vinhnx • 12 years ago

157 66 12 years ago

openfootball.github.io

ntietz• 12 years ago

This is really cool! Does anyone know if there are similar datasets for other sports out there? Even less clean datasets, as long as they have permissive licensing to allow sanitation and republication.

isuraed• 12 years ago

I believe ESPN has NBA play by play data.

ugexe2• 12 years ago

ESPN is not going to license or allow you republication rights to their data. ESPN data is also extremely difficult to sanitize (play by play does not match up with box scores, is just plain wrong, different formats, terrible html).

cwyers3• 12 years ago

The gold standard for freely-available sports data is baseball, with the Retrosheet project:

http://retrosheet.org/

The license on the data is a pretty permissive one, simply requiring attribution of the data to the Retrosheet project. Software to process Retrosheet files is available, under the GPL:

http://chadwick.sourceforge.net/doc/index.html

bnycum• 12 years ago

I'll add this. Sean Lahman's Database is also widely used. Though it's mainly whole season statistics, not game by game. Along with post-season, all star games, schools, salaries.

http://www.seanlahman.com/baseball-archive/statistics/

Then of course MLB has a bunch of data here, mainly the PitchF/X data since 2008 is gathered from here.

http://gd2.mlb.com/components/game/mlb/

caseydurfee• 12 years ago

Awesome, thanks for this. Do you know if there are any good sources of historical betting line info for baseball? It would be fun to put together a quantopian-type site for sports betting.

cabbeer• 12 years ago

Does anyone know if this is available for (American)Football?

cwyers3• 12 years ago

The same project seems to offer NFL play-by-play, but only for two seasons:

https://github.com/opensport/american-football.db

The best public football repository I am aware of is this, though:

http://www.advancedfootballanalytics.com/2010/04/play-by-pla...

dirtestbird• 12 years ago

Ive used this https://github.com/BurntSushi/nfldb and this https://github.com/BurntSushi/nflgame to make this http://www.ffgraf.com/

burntsushi• 12 years ago

Author here! That's a pretty nice visualization :-)

I thought I'd just squeeze in a few words about nflgame/nfldb. Both offer access to the same stuff: play-by-play data back to 2009. Both can be used with live games so that they are updated in real time (well, at least as frequently as NFL.com).

nflgame is responsible for pulling the JSON data and provides some rudimentary searching features. But it's slow.

nfldb stores all this data for you in a relational database. It comes with a script that updates the database while games are playing so that you can get access to live data. (It will even migrate the database for you if I've made any changes to the schema.)

Here's a quick example that shows how to get all of Julian Edelman's touchdown plays from last season:

    import nfldb
    
    db = nfldb.connect()
    
    q = nfldb.Query(db)
    q.game(season_year=2013, season_type='Regular')
    q.player(full_name='Julian Edelman').play(offense_tds=1)
    for g in q.as_plays():
        print g

Easy as pie!

There's an extensive wiki (almost 20,000 words) with tons of examples and explanation: https://github.com/BurntSushi/nfldb/wiki

Other features: aggregating data, player meta data (college, height, weight, etc.) and fuzzy player name matching.

llimllib• 12 years ago

I did something pretty similar, but it seems definitely less comprehensive: https://github.com/llimllib/soccerdata/ . Will be using this, thanks!

abeisgreat• 12 years ago

I'm curious if this data is actually public domain. Where are they sourcing it from? Are they legally allowed to redistribute? Etc.

bronson2• 12 years ago

Why wouldn't they? It's just raw facts, presented in their own minimal style.

buro92• 12 years ago

In the UK this rule applies: http://www.ipo.gov.uk/types/copy/c-otherprotect/c-databaseri...

    For copyright protection to apply, the database must
    have originality in the selection or arrangement of
    the contents and for database right to apply, there
    must have been a substantial investment in obtaining,
    verifying or presenting its contents. It is possible
    that a database will satisfy both these requirements
    so that both copyright and database right apply.

They would have a "database right" if they had placed a person at each match to gather the data and verify it, as that is a substantial investment.

How they originally acquired the data is important and shouldn't be presumed.

However that doesn't stop you from implementing your own database and re-acquiring the facts in some trivial way. Just bear in mind that accessing historical data may breach someone else's database right.

Database rights are usually proven by fake data inserted into the database to catch people copying it.

For example you could argue that the Rare Record Price Guide ( http://www.rarerecordpriceguide.com/ ) is just a collection of facts, and decide to copy it... but you'll discover when sued, that a few of the bands in the guide are fictional and designed to demonstrate that the database is theirs, and that it's not trivial to acquire and verify the data.

joosters2• 12 years ago

There was a court case regarding "database rights" for horse racing data in the UK, where the British Horseracing Board sued William Hill (a bookmaker) and lost the case. The BHB basically wanted to charge for the right to publish the horse racing fields (i.e. details of each race, the runners, etc).

Details of the case law here: http://www.out-law.com/page-392 And of the court case: http://www.out-law.com/page-5055

The lawsuit backfired completely. The BHB had wanted to charge newspapers for publishing horse racing fixtures, but it inspired the newspapers to turn around and question why the BHB wasn't paying them for devoting pages to the sport.

(Looks like the case also involved football fixtures as well)

notahacker• 12 years ago

An entity called Football Dataco representing the English and Scottish professional leagues', claimed these leagues' fixture lists and results fell within the scope of their IP even if sourced independently, and they successfully extracted large licensing fees from bookmakers and publishers whilst sending everyone else c&d letters.

There's been quite a bit of legal back and forth

- http://www.football-dataco.com/ - http://www.bbc.co.uk/news/business-17218968 - http://www.twobirds.com/en/news/articles/2012/football-datac...

MisterBastahrd2• 12 years ago

Yeah, me too. The official statistics of sports leagues are rarely ever in the public domain. Official being the key word here.

ngoel36• 12 years ago

Is there anywhere to get real-time play-by-play data?

packetslave• 12 years ago

There are several, and you'll pay a lot of money for them.

thom• 12 years ago

That's true. There are some circumstances in which Opta let you do interesting things with non-realtime data though:

http://www.optasports.com/playground-section.aspx

maaaats• 12 years ago

Betradar / Sportradar has it.

ddispaltro2• 12 years ago

Is there an open database for horse racing?

phillc732• 12 years ago

That depends on your definition of open.

The short answer is no. I've searched long and hard, high and low, for free (beer) horse racing databases for UK/IRE and Australia. To a lesser extent I've searched for HK, FR and GER data. I'm yet to find anything that is comprehensive and no cost.

There's a couple that I do use for UK/IRE racing which cost in the region of £35-£45 per month for access. Betwise/Smartform provides an historical database in MySQL, and daily race card/results updates. UKHorseRacing.co.uk provides CVS files with historical race data, their ratings and race results. I take these CVS files, combine them into a SQLite database and interrogate with R.

A slightly longer answer is, sort of. The Betfair API is currently open access for non-commercial and low volume use (as far as I'm aware). This will allow you to retrieve basic racing data - the cards before that race with horse name, jockey, barrier etc and the race results post-race including the Betfair Starting Price. After interrogating the API, you'll need to obviously compile the data into your own database. A bit of work, but feasible. Betfair has a developer programme and their are API bindings available in a number of different languages. I use R (R package developed by Betwise mentioned above), but I know Python is available. One caveat to mention is that Betfair are upgrading their API, so this will obviously have an impact on existing programs using the old one.

If anyone else has additional information or could point me in the direction of something else "free" I'd appreciate it as well.

joosters2• 12 years ago

As well as Betfair's 'live' API, they also provide historical betting data at http://data.betfair.com/

It is free but you need an active account with them to download the CSV files.

pessimizer• 12 years ago

Interesting articles on the lack of one, ordered newer to older:

http://thorotrends.com/news-and-views/50-blog/117-release-th...

http://www.anddownthestretchtheycome.com/2012/1/14/2706205/t...

http://www.paulickreport.com/news/ray-s-paddock/free-our-sta...

ddispaltro2• 12 years ago

Kinda makes you want to disrupt that whole industry.

sourc32• 12 years ago

You, my friend, are the best. As a huge soccer fan and a developer, getting this sort of data is really hard unless you shell out hundreds of dollars a month.

Already thinking about the apps that will use this! Thank you.

sourc32• 12 years ago

And kudos for calling it football :)

leemcalilly• 12 years ago

Yes this is a stellar idea!

isaacremuant• 12 years ago

Agreed. Having this for the world cup will be awesome to try toying around with a couple of games or apps.

The more the better!

philtar• 12 years ago

Last I checked it was thousands of dollars a month.

Something around the tune of $25k a year. Anyone actually paying for this now and can provide pricing?

rurabe• 12 years ago

perfect timing. thanks!

redshirtrob• 12 years ago

This looks cool. I see Gold Cup and NA Champion's League repos. Is there a plan to add MLS data? I know some people who would be super excited to get baseball-reference.com level data for MLS.

chevreuil• 12 years ago

The data format bothers me. Why not use a standard one like JSON?

nathancahill• 12 years ago

The size of JSON files is huge compared to delimited data. Languages like Python make it equally easy to consume delimited data and JSON, so it shouldn't matter much.

MisterBastahrd2• 12 years ago

At work I built a system to consume feeds of numerous automotive dealer inventories and the easiest to work with is always comma delimited. There are some people out there who have no business building an XML document, and unfortunately I've had to build adapters for many of them. It takes me a few hours to get set up to consume a new CSV feed and a few days for XML, not counting mapping their industry / category / manufacturer data to ours.

shirkey• 12 years ago

Agreed -- looking at the player data[1], IMO the format type is unrecognizable:

  ## GK / Goalkeepers

  Kawashima|Eiji Kawashima,   20 Mar 1983
  Nishikawa|Shusaku Nishikawa,   18 Jun 1986
  Gonda|Shūichi Gonda,   3 Mar 1989

  ## DF / Defenders

  Inoha|Masahiko Inoha,   28 Aug 1985
  G. Sakai|Gōtoku Sakai,   14 Mar 1991
  Nagatomo|Yuto Nagatomo,   12 Sep 1986
  Uchida|Atsuto Uchida,   27 Mar 1988
  Konno|Yasuyuki Konno,   25 Jan 1983
  Kurihara|Yuzo Kurihara,   18 Sep 1983
  H. Sakai|Hiroki Sakai,   12 Apr 1990
  Yoshida|Maya Yoshida,   24 Aug 1988
  Masato Morishige,      21 May 1987   ## Japan F.C. Tokyo

Comments as a double-hash, key fields are either player last name or occasionally first initial-space-last name, then three different delimiters of pipe, then comma, then tab. Choosing either a consistently delimited format or a more verbose JSON/YAML structure with clear metadata would seem to be a better approach.

[1] https://github.com/openfootball/players/blob/master/asia/jp-...

wambotron• 12 years ago

They do have a json web service: http://footballdb.herokuapp.com/api/v1/event/en.2013_14/roun...

grahamel• 12 years ago

all the scores are null

how often is the feed updated?

keithxm23• 12 years ago

This is impressive for the amount of work put in to formatting the data and making it easy to use in different ways.

For details and advanced analytics though, this one is much better: https://github.com/soccermetrics/soccermetrics-client-py

toyg2• 12 years ago

Holy fullbacks, Batman! I didn't know about SoccerMetrics. The Python client seems to be just a wrapper for their REST api: http://soccermetrics.github.io/fmrd-summary-api/started.html

Lots of numbers to crunch there!

This is all very nice but it would be nicer if there was some sort of cheap software that amateur teams could use to gather and then analyze their own data. There's a massive market out there for this sort of thing, the football world is very conservative and tends to move slowly.

hhamil2• 12 years ago

Hi,

I'm the founder of Soccermetrics and the creator of the Soccermetrics API. Thanks for the attention.

I've had the site for a little over five years now. I had been working on data models and algorithms for analysis of soccer matches and thought I would have a go at creating a company out of it. I even applied to YC which was a bit of a laugh in retrospect :) Right now I have another job that pays the bills but there are a few projects I do on the side, the API being one of them.

The API is the latest iteration of my data models exposed to the world as well as my attempt to build as close to a REST API as I could. I don't claim perfection and I'm sure others will have their opinion on it, which I welcome.

I wrote the Python client which is as you say a wrapper over the API, which serves well as a starting point. If you have ideas on how to extend it, please fork and contribute.

There are a few software tools out there that do what you wish. Statzpack is one, SportyBird is another. I have my doubts about how big the market really is for this kind of service, but everyone is very early in this space.

veganarchocap• 12 years ago

Where's Derby County's stats?! Just kidding this looks great!

fiatjaf2• 12 years ago

I don't know from where did this came from, but I like open formats. From where do the data come?

fiatjaf2• 12 years ago

What about RSSSF? http://www.rsssf.com/

m0skit0• 12 years ago

Impressive, thanks for sharing!

fatihpense• 12 years ago

Also have a look at this one: http://www.football-data.co.uk/data.php

iamwithnail• 12 years ago

Yeah, it's pretty awesome. I built http://test.gmbl.io off that data set to learn to code. Good place to start.

Kickdex is also pretty awesome, they use the Opta data to produce real time indices for teams and players.

rpedela• 12 years ago

A very cool project, but I have one question/issue.

The data format seems to be a custom text format which admittedly I could be wrong about. Is it possible to use TSV or CSV instead since it would be infinitely more useful since it could be directly imported into relational databases, Excel, etc.

ins429• 12 years ago

awesome, exactly what I need

dalek2point3• 12 years ago

its a shame that this is not being done under the wikidata framework. those guys have been thinking about databases like this for a while, and can be reliably trusted to at least keep it up for a reasonable amount of time.

yitchelle• 12 years ago

This is an interesting lecture [1] at Linuxwochen Wien 2013 that focuses on the usage of football.db. More data should be put into public domain.

https://cfp.linuxwochen.at/en/lww2013/public/events/61

Open Football Data

Related Stories