var_explained

author here, let me know if you have any questions!

The first few graphs use the interactive [Stack Overflow Trends](http://insights.stackoverflow.com/trends) tool, which is all public data. This is useful since it lets readers modify the graphs (for example, to add a few more tags to the comparison).

The later graphs use data that's already not public (what tags users visit together, and what countries tags are visited from), so there's no reason not to use visits instead.

The graphs for visits over time do look very similar (in general question traffic by tag roughly matches questions asked, but as a slightly lagging indicator)

OP author here: the rivalries there are directional, "if you liked X, you probably disliked Y", but not vice versa. (One thing I find interesting is that most rivalries really are directional: you see git users disliking svn, but not svn users disliking git).

Perhaps there's a way I could make that clearer in the post!

OP author here; in another blog post I explored the growth in questions about some cross-platform solutions. Ionic (now Ionic2) has been growing quickly, and React Native is among the fastest growing tags on the site. So you may get your wish!

https://stackoverflow.blog/2017/05/16/exploring-state-mobile...

That was an early hypothesis I considered, but it doesn't fit the data. That would show up as a larger tabs/spaces salary gap for big companies, when in fact the gap is similar across all company sizes:

https://twitter.com/drob/status/875493967865008129

Perhaps it could play a role as a more indirect effect: people leaving Google/Microsoft/Apple/etc, starting new companies or joining senior positions at others, and spreading this particular practice.

This is not the reason for the correlation. The p-value of the tabs/spaces connection is about 10^-13 (one in 10 trillion), and it can be found separately within each (large) country.

This does not mean the effect isn't confounded with some other factor, but it does mean it's not a multiple hypothesis testing issue.

You can explore the code and regression yourself! Take a look: https://github.com/dgrtwo/tabs-spaces-post

I've got a followup coming about what words lead to upvotes, and rust features quite prominently there!

The latter is indeed what that graph is showing: what percentage of each language's users get stuck in Vim (or more precisely what % of their Vim visits are to that question). It's not being confused with the most used programming languages.

I've now blogged in more detail about building Tagger News- check it out here! https://news.ycombinator.com/item?id=14343854

If you keep adding "magic" and doing careful research on what magic works and what doesn't, you end up roughly with the modern field of machine learning.

Random forests are a method that's often effective in taking into account many interactions among high dimensional data.

You mean Naive Bayes? Because it can't account for interactions between the effects of multiple words.

Oh, that was silly of us not to use BigQuery! I was just able to use that download a full million stories (though we still would have had the rate-limiting step of downloading the articles).

During a hackathon it can be hard to tell when to keep searching for an easy solution like that, as opposed to going with something slow you know will work- sometimes it turns out to be a dead end.

Thanks for the recommendations!

One of the devs here.

1. That's the way we were thinking about it :)

2. Oh, excellent! We hadn't found that or we'd have used it, and we'll start working with it.

3. Tomorrow I'm going to blog about how we approached the machine learning. Short version; we manually came up with regular expressions to classify a training set based on titles. The idea is that when we experimented with manual annotations on titles, the vast majority of the time we were looking for only a few key words. There's no question that this adds biases and will not be entirely accurate, but manual inspection convinced us it was a good enough approach for our hackathon, and most of the articles we identified with the resulting algorithm would not have been found by the title regex alone.

You can see the table of regular expressions [here](https://github.com/dodger487/analyze_hn/blob/master/topics.c...) and a bunch of (pretty unstructured) analysis code [here](https://github.com/dodger487/analyze_hn/blob/master/hn-analy...).

Which language is that? Definitely true that some technologies like SAS and Mathematica have a lot of off-Stack Overflow activity, but curious what you're referring to.

That feature has been added in the last few hours :) https://insights.stackoverflow.com/trends?tags=python%2Cphp%...

I used ggplot2 and gganimate in R. The custom ggplot2 theme is called theme_roboto(): https://github.com/juliasilge/silgelib/blob/master/R/graphin...

That's right: Haskell is disproportionately used in the evenings, but it's not highly used at any time. You can see in the 4th graph that of the 250 languages considered, Haskell is one of the lowest traffic.

Course author here; I agree about most of the lessons being outdated in the last two years, and that R for Data Science is a great modern source.

I'm working with DataCamp to develop an R course that covers dplyr, tidyr, and other newer additions to the R language.

There are many developers looking for a good job. There are many companies having difficulty finding qualified developers.

Our goal is to solve those problems, and it is not a zero-sum game.

mikk14 is right; the connections were filtered out below a particular threshold (set that threshold too low and everything is connected to everything; set it too high and you miss meaningful connections).

But note that precisely because git/github is used in combination with almost all other tags, it doesn't have a high correlation with any particular technology. A correlation (roughly) means "If I know you use tag X, you're more likely to also use tag Y." But knowing someone uses git doesn't let you guess what other technologies you use, because as you note they can use almost anything.

In short, if you're connected to everything, that means you're correlated with nothing.

It's also the most common language on GitHub in terms of both # of repositories and total pushes: http://githut.info/. Why is it hard to believe?

Perl is indeed quite polarizing (about 30% disliked, a bit more disliked than .NET). It's just not mentioned often enough to show up on the graphs.

It's just a figure of speech, and catchier than 10K.

What's interesting is that XKCD shows replies that are even worse than your last reply- even suggesting Googling provides a course of action. XKCD's more like "My grandmother could read files faster than that."