HN user

danielsf

81 karma

mdaniels.com

Posts5
Comments57
View on HN

Author here: we scraped every script website on the Internet. First we tried to normalize the dataset but only doing stats on the top 1,000 box office, but we were missing too many scripts. So we decided to go big and then display a cut of the data that's only films in the top 2,500 box office (we had about half of those).

We're aware of sampling error and the potential for cherry-picking, but also struggled to figure out what was a representative sample.

This was omitted from the piece, but we count words and then convert to lines at ~10 words/line. When pop culture talks about dialogue in film, we use lines. So that felt more natural than "# of words spoken."

nope. when things are in an inactive tab...browsers slow down the JS and it crawls to a halt. So I just decided to stop it all together. You could have it in another window in the background though!

Yea I didn't think anyone would read this Medium post and was lazy about the Trello board. Filling it out now :)

Author of the Medium post here...

This point comes up a lot...there's a tension in making the data say something interesting that will get traffic/spread, which might undermine the rigor that goes into real data-analysis/data science/academic work.

IMO, as long as we disclose the source and preface the biases/problems, I'm ok with data that isn't perfect (after all, there's no such thing as a perfect data set).

The lyrical analysis that I did for rappers would never work in academia...the data set wasn't strong enough. But, it was good enough for the Internet as a side-project, and I think that most readers understood the integrity issues with the data (which I also highlighted in the narrative).

But yea, really good points about journalistic standards for coders who write journalism-esque content.

author here...I don't have any services. what's so ad-driven about it? It was meant to be a statement about why I'm so passionate about the space right now :)

I kept this focused on vocab so that the data viz was very straightforward and easy to digest/draw insights from. I've had many requests for album sales to be added, and I plan to as soon as possible :)