GitHub Language Trends

http://redmonk.com/dberkholz/2014/05/02/github-language-trends-and-the-fragmenting-landscape

by spatten • 12 years ago

113 65 12 years ago

err4nt• 12 years ago

Very interesting graphs, I was surprised that CSS has had a recent uptick but as somebody who has specialized in responsive layout in the past two years I guess that uptick represents my life too.

I can't get enough language statistics on Github! I run 'gitinspector' on my web server to compute language stats in individual git repositories, but one thing I haven't been able to figure out it how to chart the language stats for one git repository over time in a branch.

Does anybody know how you can chart language use over time in one repository?

AYBABTME• 12 years ago

I have projects (markdown repos) marked as CSS projects for which I haven't written more than 10 lines of CSS. Bootstrap makes it appear so, I guess.

gexla• 12 years ago

I don't know this is a good indication of trends. As the article (somewhat shallowly) shows, there are stories behind of these graphs.

Ruby has probably just settled to a normal position post early adopter. Shows that Ruby is still strong.

Javascript is probably building the sorts of libraries that other languages already have. These guys have had a lot of work to do.

davidw• 12 years ago

Yes, Ruby was very much "overrepresented" in the early days of github, so the decline in its growth relative to other languages is fairly normal sounding, rather than indicative of its actual decline.

Also, Javascript is something that is going to get checked in to lots of web projects one way or another. I wonder if they weed out duplicate copies of, say, jQuery.

grey-area• 12 years ago

I wonder if they weed out duplicate copies of, say, jQuery.

These stats have not in my experience been very reliable at all, and tend to massively overcount javascript. I've had a few projects tagged 'javascript' when in fact they were web projects and just had some common js libraries like jquery included (ruby and golang projects).

coyotebush• 12 years ago

I wonder if they weed out duplicate copies of, say, jQuery.

Linguist tries to filter out things like that:

https://github.com/github/linguist/blob/master/lib/linguist/...

courtson5• 12 years ago

Ah, it is only on internet forums where someone can argue that a sharply downward trending graph shows that a language is 'still strong'.

davorb• 12 years ago

"GitHub is a specific community that’s grown very quickly since it launched [writeup]. It was not initially reflective of open source as a whole but rather centered around the Ruby on Rails community"

Github has moved out of a niche over the past five years, and the graph demonstrates that.

andreasvc• 12 years ago

Not really. It is universal that statistics can easily misrepresent if one doesn't take a critical look at choice of data and how it is presented. For example in this case the focus on new repos/issues/etc. is suspicious to me. It could very well be that we just see a rise in one-off Javascript uploads to Github here, instead of a comparison of sustained development on mature projects.

largehotcoffee• 12 years ago

Language detection is based on lines of code

Karunamon• 12 years ago

This is important to note. As a huge ruby fanboy, one of my favorite things about the language are the shortcuts and syntactic sugar that are provided to make things easier.

I saw the decline and got a bit worried, then came here and had those fears assuaged. I do wonder if the LOC difference has anything to do with people getting more familiar with the language and doing more things in less code.

rumcajz• 12 years ago

This looks like commercial ecosystem (Java, JavaScript) migrating to GitHub rather than hobbyist ecosystems (Ruby, Python, Perl) being on decline.

camus22• 12 years ago

say what? Ruby isnt for hobbyists, neither is Python(dont know Perl).But you just wanted to hear yourself say that didnt you?

threeseed2• 12 years ago

It isn't entirely untrue. Ruby/Python are more popular in the hobbyist space and less so in the enterprise. Java is the opposite.

It shouldn't be seen as a reflection of the platform.

The_Double• 12 years ago

I think a large part of the rise in Java code is because of a large amount of open source android projects. (applications and mods/forks of the operating system)

alexchamberlain• 12 years ago

What is the cause for the correlation between Java and JavaScript?

pcmonk• 12 years ago

Probably random noise. Or else because it kind of tracks the arc of github going mainstream, and Java and Javascript are very mainstream.

romanovcode• 12 years ago

Java: Rise of Android ~2009

Javascript: Random noise.

sheetjs• 12 years ago

FWIW many projects written in web languages like PHP seem to be treated as javascript by Github because they also have javascript code (which happens to be more significant / larger than the underlying code). It's unfortunate that there is no way to specify the primary language of a project.

The Github language system is also somewhat unpredictable: https://github.com/SheetJS/test_files seems to alternate between AppleScript and Shell with each commit (even if no .scpt or .sh file was changed or added)

flohofwoe• 12 years ago

Indeed, Github's language detection is absolutely useless, and a manual setting is very much needed. One of my projects is a header-only C++ lib, which is detected as 100% C, another one is a fairly typical C++11 codebase, which is detected as 50% C and 50% C++, third is a Python project but has some big example 3D asset files in three.js format which is then listed as 1.2% Python and 98.8% Javascript. So at least for my github projects, the language classification is completely off.

zongitsrinzler• 12 years ago

If a project mainly consists of JavaScript it should be a JavaScript project, no?

ubernostrum• 12 years ago

Well, to take an example: I work on the Mozilla Developer Network (MDN). MDN is mostly a wiki, and the underlying codebase for that is all in Python. There's also the Demo Studio, which lets people upload web technology demos, and that's Python too.

But the wiki also has an embedded script-ish language (letting people create macros/templates for specific purposes and re-use them across multiple articles). That's JavaScript-based, and so is implemented in node.js. And there's a WYSIWYG editor for the wiki pages, which is a couple of MDN-specific plugins we've developed, plus off-the-shelf components (the editor + jQuery, which we also use for a few other things on-site).

So MDN is a Python project with a couple JS utilities attached. And if you set up a local copy, you can pretty easily see that Python is by far the heaviest-used language. But due to the way GitHub counts and reports statistics, it shows up as a JavaScript project.

To pick on an easy example: the copy of jQuery we have in our repository weighs in at just over 9,000 lines. So just having jQuery means you need to write over 10k lines of code in order to get the "real" language of your project recognized.

thatthatis• 12 years ago

No. The way a lot of languages handle 3rd party dependencies is via a manifest file of some kind -- requirements.txt. These get installed on the system when deployed.

JavaScript, however, tends to require (or at the least follow the practice) of having a copy of your JS 3rd party dependencies in your repo.

So you may have 30,000 lines of python dependencies and 20,000 lines of JavaScript dependencies, but only the JS shows up in your project files to get counted.

This isn't even asking the question of if the new code in the project or the total enabling code is what should be counted. If we count total enabling code, do we count the Linux kernel implicitly too?

return0• 12 years ago

Most HTML projects would then be "Images". My plain html project is marked as 'CSS', my arduino project (which contains a python file, but no java) is marked as 'Java'; that's 2/5 errors.

The files that get changed/updated are the ones that matter. If most of your diffs are PHP, then you're mostly workign with PHP.

yaph• 12 years ago

You can avoid 3rd party libraries being counted in by following some conventions, see the patterns ignored by Github's linguist https://github.com/github/linguist/blob/master/lib/linguist/...

notwedtm• 12 years ago

Why not just hash popular files and don't count ones that match?

chimeracoder• 12 years ago

Until December 2013, if you included non-minified Bootstrap Javascript files in your project, Linguist (the library Github uses to detect languages) would count them towards the Javascript LOC count for your project[0].

I had a number of small web server projects that were incorrectly classified as >95% Javascript for this reason.

Fortunately this was fixed, but I imagine there are a number of other low-hanging fruit that are causing projects to be misclassified as well.

[0] https://github.com/github/linguist/pull/856

jcbrand• 12 years ago

One of their reasons for the growth of Javascript:

the JavaScript development philosophy that encourages bundling of dependencies in the same repo as the primary codebase

I wonder how much this is still an issue with the rise of npm, bower and other package managers.

mateuszf• 12 years ago

node_modules directory is usually added to .gitignore. I don't think it is usually commited by accident as it results in a huge file list in diffs and git status command.

iLoch• 12 years ago

That's what he's saying. While the author's statement may not necessarily be incorrect, the trend these days is to manage your JS (both server and client side code) with package managers like NPM and Bower.

PeterisP• 12 years ago

It would be interesting to see which languages are rising. In those stats, everything except Javascript seems to be declining, and the total relative decline is much larger than JS growth, so something must be growing - but what it is?

kyberias• 12 years ago

It would be much more useful to show the absolute numbers. Or provide the raw data for others to do more meaningful graphs.

arscan• 12 years ago

It depends on what story you are trying to tell... Absolute numbers might not be all that meaningful when you are talking about relative popularity. Or it might be. Data visualization is hard because you typically have to trade information density for comprehensibility.

I assume the data came from http://www.githubarchive.org/

louthy• 12 years ago

Not sure if it's reasonable to make this call, but it seems the dynamically typed languages have significantly higher percentage of issues overall.

kijin• 12 years ago

It could be that dynamic languages have a lower barrier of entry, which encourages many more people to fork the code, find bugs, and make pull requests. It's easy for a PHP guy to find and report bugs in a JavaScript project. It's not so easy for a JavaScript guy to find bugs in a C++ project.

chx3• 12 years ago

I find the complete lack of Go surprising.

threeseed2• 12 years ago

Go is a new language so not that surprising. It is also over represented here on HN. You don't see Go jobs in the wild for example.

d0ugie• 12 years ago

You don't see Go jobs in the wild..

Do you fellas expect that to change over the next few years? I want to learn a new language, Go seems alluring, though I'd like to put myself on the right side of history.

Matrixik• 12 years ago

Search here: http://www.golangprojects.com/

chx3• 12 years ago

I do not think Go as something that would go on itself as a job. Instead if you already have a site (like Drupal which is my primary skill) and you want to add a very performant REST interface then Go provides a pleasant way to write one.

pjmlp• 12 years ago

Except for HN, at the enterprise level, Red Monk main customers, no one cares.

pron• 12 years ago

GitHub is slowly starting to reflect the software world at large, although the true picture is Java and C leading by a huge, huge margin. I don't expect GitHub to ever fully reflect that, as most Java shops, and nearly all C shops, would never host their code on GitHub.

gurkendoktor2• 12 years ago

I think it's safe to say that most projects on github are libraries and snippets, not finished products. Not even Ruby shops push all of their actual work to github. But at least there is a culture of sharing libraries, which drives the Ruby % up, and I don't see this happening a lot with C.

m_ram• 12 years ago

Violating all expectations and trends, new Java users on GitHub even grew as a percentage of overall new users, while everything else went downhill. This further supports the assertion that GitHub is reaching the enterprise.

This is more likely to be due to the rise of Android since 2009.

gurkendoktor2• 12 years ago

I would also expect a few CS students to push their homework to github nowadays, and most homework is Java.

skrebbel• 12 years ago

The last 2 graphs look like they were drawn by someone who hates colour blind people.

platz• 12 years ago

2010-2011 appears to be the "year of the great inflection point"

peteretep• 12 years ago

Presumably the big Perl spike is the migration of CPAN modules over to Github en-masse.

dham• 12 years ago

Just need to have a dropdown of the primary language like Bitbucket does. Probably have different results.

pecanpieyw• 12 years ago

Github should really just enable the developers to specify the language of their repo by themselves. Bitbucket get this right in the first place. Auto detection for language sounds cool, while doesn't work for most of the web project.

hcarvalhoalves• 12 years ago

Javascript is really hard to measure I believe, the numbers are always skewed because of 3 things:

1. A lot of repositories include 3rd party libraries.

2. A lot of software includes a web interface, even if the backend language is something else, but the LOC for Javascript can be equal or even higher because of 1.

3. JSON is counted as Javascript sometimes

jhcnt2• 12 years ago

In my experience, GitHub's so-called "detection" gets it wrong far more often than right. It's worse than useless -- it's misleading.

lkrubner• 12 years ago

To my mind, the big trend is toward polygot programming, which perhaps reveals what a transitional and perhaps revolutionary time this is in the world of computer programming. This paragraph struck me as the most important:

"Almost every language shows a long-term downhill trend... My initial guess is that users of languages below the top 12 are growing in share to counterbalance the decreases here. It’s also possible that GitHub may leave some users unclassified, which would tend to lower everything else’s proportion over time."

GitHub Language Trends

Related Stories