Don't Enforce R as a Standard

http://timotheepoisot.fr/2015/04/02/do-not-enforce-R/

by doctoboggan • 11 years ago

166 88 11 years ago

zzleeper2• 11 years ago

These were the same types of people that said to me "don't use R, no one uses it" several years ago.

I know there must be a balance between bleeding-edge and stability, but if in research you cannot use the new tools, then there will not be any progress at all.

gerty• 11 years ago

That's true. In my field, there's plenty of Matlab and it's arguably not even the best tool for the job: someone started using it, people improved on the work and now, no one cares about rewriting that.

weinzierl2• 11 years ago

"...it can be written in lisp or cobold for all I care"

The typo is too good. Cobold is the sprite that plays its tricks on us even 56 years after its first appearance.

rcthompson5• 11 years ago

I assumed it was intentional.

weinzierl2• 11 years ago

That makes it even better.

guelo• 11 years ago

My non-academist opinion is that the biggest problem in science is that there is little incentive for scientists to reproduce each other's results. Anything that makes it easier for other labs to rerun an experiment is a good thing for your field overall. The goal of a paper should be to communicate as plainly as possible how to reproduce the experiment. That probably means a good-enough standard programming language that everyone learns in school and uses in their work.

sz4kerto• 11 years ago

Look at the Reproducibility Project: https://osf.io/ezcuj/

Gimpei• 11 years ago

I agree, unless it's something that can't be done well in the current language. Or, if like me, you're an economist and the default language used by academics is horrible, horrible Stata, which needs to be replaced by R.

zzleeper2• 11 years ago

For me, both are horrible. I use Stata a lot and thought a lot about switching to R (there is even a nice tutorial by Matthieu Gomez at http://www.princeton.edu/~mattg/statar/ ).

However, I believe that moving to R is not the answer, as you will still be riddled with a lot of the problems that you faced in Stata such as a quirky syntax (except backticks. I hate those backticks). Maybe Python will catch up (I doubt it, what we need is a DSL, not something too general that requires loong commands to do a simple regress). My best bet would be Julia, but it's still a long way to go regarding things like missing values.

grayclhn5• 11 years ago

If you're a macroeconomist, the default is Matlab. I'm not sure which is worse.

locokoko• 11 years ago

Well I think it should communicate how to reproduce the experiment in a lab with the same or similar setup, not how to reproduce it with all kinds of different setups. If a lab does not have the equipment to reproduce a certain experiment, it's not the problem of the original authors.

Also, I think you should choose equipment which best suits the experiment, not other people.

claytonjy2• 11 years ago

If a lab does not have the equipment to reproduce a certain experiment, it's not the problem of the original authors.

While I think that's true, a paper can potentially be much more impactful if its' results can be easily understood by others, and isn't that really the point of science? In that sense, it's better for the author to stay away from niche methods and tools unless absolutely necessary.

aet• 11 years ago

I agree, but I still think the idea/algorithm/etc should be conveyed in sentences and math in the body of the paper. The code should purely be optional and not associated with the paper. I think the problem here is the author is referencing code in the paper, which puts the onus on the journal to evaluate the code. Juts put the code on your website. If people want it enough they will google you.

Mikeb852• 11 years ago

I think it's just that R has become the de-facto standard since the previous tools used were either terrible, or closed source and expensive.

And it's probably taken them so long to adjust to R, that they don't want to change again any time soon.

And you're right, the tool shouldn't matter, and I definitely agree with your stance on open source, but keep in mind those attitudes do exist. Speaking of attitudes concerning open source, here's one of my favourite blog posts about the subject: http://www.catuhe.com/post/Le-syndrome-du-Puppy.aspx

apalmer• 11 years ago

Why did R take over the world of statistics anyways? I remember about 8 years ago I had an interest in statistics. Everything was SAS, SPSS etc which i just didnt have the budget for... I happened to find R as it was a freely available for linux, impression I got was no one used it, well some limited use in academia, but no professional usage.

Fast forward it seems like the most widely used tool in computer data analysis. Did something fundamentally change?

stillsut• 11 years ago

Packages.

They can be built by a single prof/grad student.

They can be used by anyone based on R's package-manager GUI.

Research in most fields is going to move more quickly than any software team can add functionality to monolithic programs like SAS. Therefore, most functionality becomes available only in R. Users - people who want to run specialized-domain-function-X on their dataset are best suited by learning R, then using their colleagues packages.

rcthompson5• 11 years ago

I don't know much about SAS or SPSS, but do they really not have a usable package/module system for code reuse? Or is it just not as good as R's?

_deh3• 11 years ago

Agree entirely. Academics led and industry followed.

rcthompson5• 11 years ago

I think you answered your own question: "...SAS, SPSS etc which i just didnt have the budget for... I happened to find R as it was a freely available..."

Mikeb852• 11 years ago

People's attitudes towards open source changed, no doubt thanks to vendors like IBM, Oracle and others, who embraced R (and open source as a whole).

Also, R Studio has made the entry barrier to R very low...

data_spy• 11 years ago

Agreed on RStudio being a factor. For me its all about packages and RStudio. I totally believe some other languages are better in general (Python), but RStudio makes a difference at work. As I can quickly show co-workers charts, data views and Shiny Apps.

mahmud• 11 years ago

1) It's free.

2) CRAN, nothing like it for SAS & SPSS.

3) Early adoption by book authors. A whole generation of data scientists were taught in R.

rcthompson5• 11 years ago

I remember back in college that Microsoft was pretty much giving away student editions of all their software. I think they realized that if students, who are invariably on a budget, couldn't get access to MS software, they'd learn to use the FOSS replacements instead, and by the time they graduated and entered the workforce they would know the free stuff better and start spreading it. I guess SAS and SPSS didn't have any similar kind of strategy.

_deh3• 11 years ago

To expand on the free point - it's not just that SAS and SPSS have a price tag. You're then supposed to pay for a load of bolt-on modules. The extreme case is if you want to do decision trees in SAS, they've made a commercial decision that you must be doing some kind of 'Enterprise-Grade Data Mining', and therefore need to pay their 5-figure price tag rather than their 4-figure price tag.

I think this has been a big barrier to any sort of package ecosystem to rival CRAN. Even if your entire target audience has SAS or SPSS, if your new package depended on one or more of those modules, it's going to cost some of your prospective users hard cash to run it. Whereas R just installs the dependencies for you.

grayclhn5• 11 years ago

1) compatibility with S-Plus, which was widely used by statisticians for long before R

2) developing new statistical methods and other innovations (knitr, etc) is much easier in R or S-Plus than SAS, SPSS, etc. So R's improved a lot faster than the others.

rm9992• 11 years ago

The biggest boon for R was the rise of "data science". Data scientists have actually existed for decades, but they had different titles (quants, actuaries, quantitative researchers, etc) and came from fields that preferred enterprise software for various reasons.

Data scientists, on the other hand, largely rose from tech companies that hired software engineers and preferred open-source software. When I started in the field ~10 years ago there were very few reasonable open-source options outside of R. Matlab and SAS were prohibitively expensive, Octave was too immature, Python didn't have basic functionality like data frames. In short, it was our only viable option, so we used it. A lot of people started using it for the same reasons, so R's library support became the best. This is what keeps me semi-locked into R - I'll probably eventually move to Python or Julia, but R hit the critical mass first and in a huge way.

grayclhn5• 11 years ago

R's library support predates your timeline.

bbgm• 11 years ago

In the world of Bioinformatics, R has been the tool of choice for a long time due to packages like Bioconductor [1], which has been around since 2001.

----

1. http://www.bioconductor.org/

x0x04• 11 years ago

R won because of

1 - very high usability, particularly (contrast to python) for users who aren't developers, don't think like developers, and don't want to be developers

2 - very high expressibility

3 - it's free, so you can install it on lab computers and your laptop and everywhere else without forking out tons of money (see the price of matlab)

4 - a generate of grad students used it and it, in turn, was taught to a generate of undergrads

5 - academic statisticians tend to like it and tend to implement stats techniques in it, so there's a wide body of shared code on cran implementing most analyses you can think of

ak217• 11 years ago

It's funny that you mention usability and expressibility (expressiveness?) as R's upside compared to Python. I've used both extensively, and in my experience, R is horrendously bad at both, and Python blows it out of the water.

davidmr• 11 years ago

I'm torn on this. I've spent nearly the last 20 years in some form or another in academic/research computing.

On the one hand, attitudes like the third reviewer's are a primary reason for the state of HPC today, where new advances in research are just as often finding a way to run 30 year old code on a modern supercomputer as they are writing new software to take advantage of the fantastic array of new hardware. The number of times you hear "sorry, we can't change that. Livermore wrote that 15 years ago, and nobody knows how to change it anymore" is enough to drive a rational person over the edge.

On the other hand, the state of academic peer review being what it is, I don't fault the reviewer for suggesting the paper be resubmitted using more common methodology. A conscientious reviewer has a lot of papers to review, and spends a good amount of time on the ones that aren't written by crackpots. While the author was of course fully in their rights (and may even have advanced the field) to use their own software written in a relatively uncommon language, for the results and methodology to be understood, that's asking a lot of a reviewer.

I'm genuinely unsure as to how I would have responded to the review request. I have much sympathy for both people involved.

Blahah3• 11 years ago

The correct response from the reviewer would have been "I am not capable of reviewing code written in Julia, therefore I must decline this review request" rather than "I am not capable of reviewing code written in Julia, therefore I must recommend rejecting the paper".

blumkvist3• 11 years ago

Are you some sort of supreme authority on such matters?

protomyth• 11 years ago

But if the paper is not about the code and the code was just supplemental (as the article states), why should the language of the code have any relevance? If the author had declined to submit code, the author's paper might have sailed through just fine.

Obi_Juan_Kenobi2• 11 years ago

Have you submitted to academic journals before?

Reviewer comments are not a list of absolute requirements; they're thoughts and suggestions that help both the authors and editors improve the manuscript. They will include a recommendation for publishing, and they can make that recommendation contingent on particular issues, but that's somewhat unusual.

It's not completely clear what's happening in this case, but I'm strongly inclined to think this was a 'minor comment' from a reviewer.

   The three reviews were helpful and constructive, but these two comments infuriated me.

I think the author is simply taking exception that these comments are prevalent attitudes, not that they were significantly contributing to the editorial decision.

It's very common to address reviewer comments without actually changing anything. Basically, you say you disagree for reasons X, Y, and Z. The editor can disagree, ask for further clarification from the reviewer, or simply accept the argument. Nothing is set in stone.

As for the implication that there are other reviewers waiting in the wings with suitable experience, good luck.

jdietrich• 11 years ago

To give the other side of the argument:

Most scientists have little or no quality training in software development, but scientific research is increasingly reliant on software. At present, software is a glaring black box in a great deal of research, because very few reviewers have the skills to thoroughly scrutinise it.

A software monoculture always has negative consequences, but fragmentation can be equally problematic in many cases. A reviewer saying "This would be better in R" usually means "I have no chance of understanding your code, because it's not in R". For better or worse, R is currently the statistical computing lingua franca in most fields.

I believe that the scientific method is in real trouble, due largely to the immense complexity of much modern-day research. Scientists have access to immensely powerful analytical and statistical tools, but most lack the training and infrastructural support to use them in a rigorous manner. Bad practices in software and statistics are the norm, rather than the exception; I'm sure most of this is just an honest shortcoming, but I'm equally sure that the lack of CS and stats experience amongst reviewers is a gift to would-be Bogdanovs and Obokatas.

Science has always been a collaborative effort, but I think most fields are in desperate need of greater support from computer scientists and statisticians. Ultimately I would like to see those professions become deeply integrated into all scientific fields, with the expectation that all papers should credit a statistician and (where applicable) a computer scientist. Likewise, editors and reviewers need far closer ties to CS and statistics professionals. Of course, these issues are intertwined with many other problems in funding and peer review.

Until then, the hegemony of R may simply be a price we have to pay for better research in the short-term. A software monoculture at least gives reviewers a fighting chance of spotting issues with software that might affect the validity of results.

rjurney8• 11 years ago

I think its more like, "I have no chance of understanding your code, BECAUSE IT'S IN R." The vast majority of R users and I bet 95% of ecologists, can't read and understand R modules. Its the PHP of Data Science.

The point, then is to supply code with simple instructions so that anyone can run it. Secondary goal: write it in a language a human can actually read.

If you're doing science and you want people to understand your code, you use Python, not R.

hackerews• 11 years ago

Option 2: Make an API with a simple and well-documented interface. Forget about the language BS.

hayd2• 11 years ago

Also, many R packages end up being written in cpp.

bachmeier2• 11 years ago

If you're doing science and you want people to understand your code, you use Python, not R.

Wouldn't that depend on which language "people" know?

Its the PHP of Data Science.

Some might disagree with that statement. Some might even argue that languages where for loops are idiomatic and strongly promoted by the BDFL are the PHP of Data Science.

deanjones• 11 years ago

I don't disagree, but PHP is not the canonical reference for code which cannot be understood, that would be Perl.

PHP is the canonical reference for code which can be understood but not interpreted, because you don't know the values of various obscure global flags.

x0x04• 11 years ago

Python is an ugly language for scientific work; any of R, matlab, or julia are far preferable for not massively obscuring the intent of your code with the programmatic hoops you have to jump through. People who are programmers I think tend to wildly underestimate the effort needed to learn and read python.

Further, the vast majority of R modules are at least partially implemented in R and tend to be very readable.

hadley3• 11 years ago

If you want people to understand your code, you need to treat it like writing: rewrite, rewrite, rewrite. The language itself is almost irrelevant: you can write horrible code in Python and beautiful code in R.

ericssmith• 11 years ago

"A software monoculture at least gives reviewers a fighting chance of spotting issues with software that might affect the validity of results."

If I understand it correctly, this observation shows just what a short distance we've come in programming languages and software development since the 1960s. Verifying the correctness of a program shouldn't depend on good naming or being able to execute it in your head.

IndianAstronaut2• 11 years ago

R is an amazing data language and ecosystem. There just is no parallel in terms of diversity of capability. Python is a second but not a close second.

Our battle should be more with SAS.

laurencerowe• 11 years ago

Science has always been a collaborative effort, but I think most fields are in desperate need of greater support from computer scientists and statisticians. Ultimately I would like to see those professions become deeply integrated into all scientific fields, with the expectation that all papers should credit a statistician and (where applicable) a computer scientist.

This is very true, but it will be difficult to realise within the university centred system of research composed of many small labs, none of which can really offer any sort of career path for non-academic professionals.

minimaxir• 11 years ago

The only reason I still use R is because of the hadley-verse packages. (Namely dplyr and ggplot2). R's base packages are incredibly terrible and counterintuitive, especially compared to Python's pandas and scipy.

Pinatubo• 11 years ago

I think the author is being overly sensitive here, and the reviewers are actually giving good advice.

Most methods papers in most fields are ignored by the vast majority of researchers. Just like the author doesn't have the time to rewrite his code in R, his prospective audience doesn't have the time to learn a new software package just to try out his proposed method.

If the author actually wants to change the way research is conducted in his field, he needs to make it easy for others to try his method out and possibly change the way they do research. As of today, that means an R package.

Or, the author can dig in his heels, refuse to write an R package, and be ignored. Maybe that isn't fair, but as I've learned through my own experiences, that's the way it is.

mattexx• 11 years ago

Peer review is a huge part of our culture here at Climate. It's often unreasonable to expect a single reviewer to review a complex paper, so we sometimes address this by assigning each reviewer a domain.

Something like:

Reviewer 1: Review scientific theory (domain expert)

Reviewer 2: Review scientific methods and conclusions (senior scientist)

Reviewer 3: Review code for scientific accuracy (senior programmer)

moron4hire• 11 years ago

Even the language we use to talk about arithmetic, algebra, and calculus had to--at one time--be agreed to and standardized, so that mathematicians could have a common ground on which to speak to each other about their ideas and how they reached them.

The program is not just "a tool". It's your proof. You may have written a proof in the paper, but that was just a practice run. It's the one you wrote in the code that is the real proof, because there is little garaunteeing that seasoned software developers write code that matches the spec, say nothing about" amateur" programmers in the sciences.

Now, that does not preclude innovation. People still invent new mathematical notation systems. It's usually with the express purpose of solving problems that can't easily or at all be stolved in current systems. It is the bleeding edge of the science of math. But physicists are still expected to use the math their colleagues will understand, or spend a lot of time explaining their new system (QM anyone?).

There is, of course, the issue that math is significantly less rigorous and specific than code. Math doesn't compile and doesn't run. Or rather, it gets compiled and ran by people. Part of avoiding new, arbitrary notations is so that your work is accessible for verification. That gives running code, especially coffee with a comprehensive set of unit tests, a distinct advantage over math. But that doesn't obviate the bed for verification.

If you are using any nontrivial piece of custom software in your science, it needs to be scrutinized just as much as the rest of the math in your paper--if not more so, since it does the actual work.

The first reviewer is almost certainly right. If they can't understand your code, then they aren't the equivalent of lay-users complaining about open source software. First of all, it is still an issue of much debate whether or not releasing source code to user's for software products is necessary. If you are writing code for your science, you have an absolute duty to open source that code (even if in a non-extendable way, we still need to see the code). You wouldn't make a claim without providing the math. And you wouldn't provide math other people couldn't understand, arguing "some math is better than no math".

I don't think that standard should be R, or that it necessarily has to always be the same thing. It can adapt over time. But until you manage to release a few papers on how Julia improves over R, in explicit detail... "when in Rome, do as the Romans."

bachmeier2• 11 years ago

Does this story tell us more about R as a standard, or more about the refereeing process? I've dealt with referees for a long time and I'd have to go with the latter. This is to me yet another case of a referee viewing his/her preferences as the only correct way to do things.

scottfr• 11 years ago

This is really just an example of Bike Shedding.

Reviewing research papers is really, really hard work to do well. It takes a massive time commitment to truly digest and understand novel work to the level where you can provide a quality critique and review.

It's much easier to throw out facile critiques like the ones these reviewers provided. It shows that you know your stuff and did your job as a reviewer, without requiring you to actually understand the authors' innovations or lack-there-of.

tmalsburg22• 11 years ago

You are implying that the rest of the review, which we haven't seen, was useless and shallow. However, the author himself said that the reviews were actually "helpful and constructive". I agree that cases of bike shedding exist, but there is no evidence that this was the case here.

tmalsburg22• 11 years ago

The author writes that the software wasn't the focus of the manuscript and that the reviewers should not have commented on the software at all but that it actually not true. If you look at the preprint manuscript, you'll find that the software is prominently advertised in the abstract of the paper. In this situation, it is perfectly reasonable for the reviewer to comment on the implementation of the software. And as much as I like Julia, I think he is right when he says that an R package would be more useful.

Don't Enforce R as a Standard

Related Stories