Statistics for Hackers

https://speakerdeck.com/jakevdp/statistics-for-hackers

by tomaskazemekas • 11 years ago

341 54 11 years ago

speakerdeck.com

jrbapna• 11 years ago

Anybody have a link to the actual talk?

TwoFx• 11 years ago

There is none. https://twitter.com/jakevdp/status/644863206339379201

analognoise2• 11 years ago

Can we please stop putting "hack" on EVERYTHING?

It isn't a hack to present mathematics in an understandable way - it's a pedagogical improvement for introductory works, not a "hack".

anewhnaccount• 11 years ago

The hacking here refers to the approach of "just compute it" rather than using an analytical approach.

analognoise2• 11 years ago

Then it's even worse than "not hacking" - it's not useful.

Any idiot can run a simulation, or compute "something". "Something" is only useful in context.

"If you can write a for loop, you can do statistics!" Ugh. Can nobody read anymore? It has to be a "slide deck"?

Jesus, pick up a book and ask some goddamn questions if you're interested.

minimaxir6• 11 years ago

It's actually the opposite: the "hacking" is used to provide better data and analysis using programmatic techniques (bootstrapping, cross-validation) than just by taking a spot average in Excel as gospel.

I've seen many, many data scientists from startups write blog posts with skewed data without using such techniques and failing to identify the potential statistical problems.

rankam2• 11 years ago

In my opinion, the "for hackers" title is in reference to multiple books that have been released with the "X for hackers" that targets people with hacking skills but do not have a formal background in X.

Machine Learning for Hackers http://www.amazon.co.uk/Machine-Learning-Hackers-Drew-Conway...

Design for Hackers http://www.amazon.co.uk/Design-Hackers-Reverse-Engineering-B...

Bayesian Methods for Hackers http://www.amazon.co.uk/Bayesian-Methods-Hackers-Probabilist...

EDIT: I'm not the author, but you can find Bayesian Methods for Hackers (free, released by the author) at the link below. I think it's a great resource for anyone wanting to explore Bayesian methods using Python.

https://github.com/CamDavidsonPilon/Probabilistic-Programmin...

r3bl• 11 years ago

Isn't working around things to make them work for you exactly what the essence of hacking is?

Plus, doesn't every hacker start as a script kiddie? Using something others came up with usually sparks the imagination. The next step is usually using the scripts right, followed by using the scripts extensively and finding issues, followed by making your own scripts (or improving the existing ones) to avoid these issues.

This looks pretty hackish to me.

minimaxir6• 11 years ago

Actually the example code is pretty close to the code you would use for more formal analysis. (naturally, real-world data needs more cleaning)

texthompson3• 11 years ago

I don't think that the title or the content claimed that "presenting mathematics in an understandable way" is a hack. "Statistics for Hackers" sounds to me like hackers are the intended audience.

This method of presentation makes sense to me. Most statistics classes that I've experienced were taught from the point of view of abstract math. That's certainly one way to do it, but I knew a lot of people for whom that wasn't the optimal presentation strategy. Now that computing is cheap and there is a large audience of people with programming knowledge, I think that teaching statistics through examples of simulation, bootstrapping shuffling and cross validation is a great way to learn things.

jmpeax• 11 years ago

It will pass, just like when use of that ridiculous word "grok" passed.

mhartl• 11 years ago

"Physics for Poets" is physics presented in a way tailored to poets [1], not physics as done by poets—i.e., for poets, not by poets. Similarly, "Statistics for Hackers" is for hackers, not by hackers. Thus the focus on brute-force computational solutions via for loops, with which hackers would presumably be familiar.

[1]: Where poets is a possibly inaccurate metonym for "less mathematically sophisticated students"

minimaxir6• 11 years ago

The talk is very explicitly a tutorial for a few good statistical methods for those without a statistical background, using simple tools "hackers" are familiar with and are occasionally necessary, especially in the case of bootstrapping. This isn't hacking in the sense of "growth hacking," thankfully.

pavpanchekha2• 11 years ago

This talk was great, especially its focus on non-parametric techniques. A lot of people are mislead by the traditional statistics approach, where it looks like stats is about memorizing a relatively short list of (really scary) formulas that let you compute one or another thing. People do not think about how these formulas come about, and the simple ideas behind them.

On the other hand, I think there is not enough written about how to do statistics with real, large, complex data sets. I tried to write something like this (https://pavpanchekha.com/blog/stats1.html) but of course the difficulty is that investigating a complex dataset is by definition too complex to really fit into a blog post. In a large dataset, the difficulties are the how and when you make decisions about what to analyze.

minimaxir6• 11 years ago

the difficulty is that investigating a complex dataset is by definition too complex to really fit into a blog post

It's not complex, but it requires answering scaleability questions. How do you fit the data into memory? How do you even get big data?

I managed to analyze 5.6M Hacker News comments (http://minimaxir.com/2014/10/hn-comments-about-comments/ ) by writing a scraper which dumps everything into a PostgreSQL database (https://github.com/minimaxir/get-all-hacker-news-submissions... ) because it's way too big to fit in memory. Likewise, to visualize the recent NYC taxi data, (http://minimaxir.com/2015/08/nyc-map/ ), I just used BigQuery and saved myself a load of time downloading the data and setting things up.

But in many cases, data is not easily accessible, and not cheap to store and process. I want to switch to an actual big data workflow with Spark, but running the numbers with EC2 storage and processing, it won't be cheap and I may have to set up a Patreon to subsidize things.

MengerSponge• 11 years ago

This might be useful the next time you need to scope your project's memory requirements.

http://yourdatafitsinram.com/

calpaterson• 11 years ago

I think he was talking about the complexity of the analysis and not about the technical complexity of getting and storing the data.

rsy96• 11 years ago

Every textbook on math I have ever read always start with intuition behind the idea, and then present definitions and proofs. Only some reference books, intended for look ups only, only enumerate formulas. I never understand why anyone claims that they only learn math by memorizing formulas.

eachro• 11 years ago

People claim they only learn math by memorizing formulas because for most, that's all the math they see before getting irreversibly turned off by the subject. Most people learn math by memorizing plug and chug formulas in high school. Aside from 9th grade euclidean geometry, I can't recall an instance where I had to derive anything in my hs math courses. College math courses, of course, are usually much more rigorous.

wodenokoto• 11 years ago

Because they learn math in junior high where they really don't care about the subject, so it gets boiled down to "remember this to pass the exam", regardless of what the book says.

pavpanchekha2• 11 years ago

Take a look at what AP Stats teaches.

porter• 11 years ago

This needs to be a course. I'd pay for it.

int32• 11 years ago

What books would y'all recommend for someone who has taken a couple of college-level proof courses but never took anything on probability and statistics?

tel2• 11 years ago

What kind of stats work are you interested in?

int32• 11 years ago

I'd like to start with formalizing my knowledge of the basics -- hypothesis testing and confidence intervals. I do some performance tuning work, and it seems like a good idea to understand if my changes are making statistically significant improvements.

chestervonwinch• 11 years ago

I've said this elsewhere, but I recommend Casella & Berger's Statistical Inference. It will take you from probability theory to statistics... all the basics. I found it to be a very readable text, and it is used for many first graduate courses in stats - for me it was used for a 2 semester sequence first focusing on probability, then on statistics.

saboot• 11 years ago

I second this, also for teaching yourself there is a solutions manual available which is great for doing the exercises with.

stdbrouw7• 11 years ago

I like Casella and Berger, especially because it does so well at showing the connection between probability and statistics (first five chapters or so), but it should be noted that its approach is very different from the one in the slide deck above. Casella/Berger is purely frequentist rather than computational or Bayesian, and it spends a lot of time on likelihood theory and treats e.g. bootstrapping only very summarily.

texthompson3• 11 years ago

I'm a big fan of two books:

* The Elements of Statistical Learning, by Hastie, Tibshirani and Friedman (https://web.stanford.edu/~hastie/local.ftp/Springer/OLD/ESLI...). * Probability Theory: The Logic of Science, by ET Jaynes (http://bayes.wustl.edu/etj/prob/book.pdf)

Best of luck. I can see from your post that you're thinking about performance tuning, I'm assuming you mean of software. That's a nice area - the nice part is that compared to fields like medical genetics, data on performance of software is relatively cheap to get, so a lot of issues about small sample sizes are surmountable.

hartator• 11 years ago

Coin toss is a bad example to take as coins can't be loaded. However, very interesting.

texthompson3• 11 years ago

Coins can be biased, they just need to be bent so far out of shape that it's obvious. Here's a fun example:

https://izbicki.me/blog/how-to-create-an-unfair-coin-and-pro...

GFK_of_xmaspast• 11 years ago

Why would you possibly think that? ( see also: http://news.stanford.edu/pr/2004/diaconis-69.html )

ziles88• 11 years ago

The term hacker is just so overused at this point. Does it just mean someone who is good at tech in the Bay?

code_sterling• 11 years ago

How so? I see anyone who refuses to accept things as they are, coupled with an innate curiosity and a drive towards getting to the root of a problem; and finally changing the rules to make it fit their prerogative, a hacker.

George Bernard Shaw would refer to this individual as an unreasonable man, but I prefer hacker.

spenczar5• 11 years ago

Bootstrapping deserves to be much better known. The fact that it works is miraculous.

solomatov5• 11 years ago

It doesn't always work, it has its applicability but it's wide enough.

stdbrouw7• 11 years ago

"It doesn't always work" really is not a very useful thing to say unless you elaborate on when exactly it doesn't work and why.

Bootstrapping doesn't work for very low n (e.g. n=10) because the resamples are not smooth enough and it doesn't always work that well for estimating quantiles. But analytic methods fare pretty poorly in these circumstances too and in any case people are mostly interested in confidence intervals around the mean anyway.

stdbrouw7• 11 years ago

For people who are interested in learning more about simulation, permutation tests and bootstrapping, the three techniques that the slide deck discusses, check out Allen Downey's Think Stats: http://greenteapress.com/thinkstats2/.

It really is a pity that introductions to statistics spend so much time on analytic approximations and so little on the underlying concepts.

Statistics for Hackers

Related Stories