R vs. Python for Data Science

https://github.com/matloff/R-vs.-Python-for-Data-Science

by Rincevent • 7 years ago

60 73 7 years ago

pletnes• 7 years ago

There is always Rpy for calling R functions from python, if you have to write your program in python but need a stats function from R. I guess python is always the choice if you’re writing software, although for interactive use YMMV.

haddr• 7 years ago

One of the important part that's was not mentioned is the ability to deploy and operationalize the model. I think Python has a slight advantage in this area. Especially when focusing on operationalisation and integration with other systems and flows.

hjk052• 7 years ago

It sounds like you don’t actually know R. Deployment in R is a one-click thing. In python it’s a complex path through virtual environments multiple incompatible packaging tools and dependency managers, at times compiling some c dependencies yourself etc. anaconda originally gained popularity because it was the only way to get a student started with python-numpy-scipy that didn’t require extensive prior technical capabilities and tons of up front investment in reading through guides on how to configure and set everything up.

I mostly use python and prefer it to R, but putting things into production is not a strength of python and R wins that comparison a thousand times over.

tpetry• 7 years ago

Theres one reason i was learning some R: the charting capabilities of ggplot2 are awesome. I have never produced so good looking graphics before.

gloflo3• 7 years ago

Try https://plotnine.readthedocs.io/ or https://seaborn.pydata.org/

gerty• 7 years ago

ggplot2 is not just a package, it's an implementation of a grammar language and this is what makes it so hard to substitute. plotnine tries to mimic it and is good for easy stuff but still far behind what can be easily done with ggplot2.

Gatsky4• 7 years ago

Plotly is pretty good in python.

https://plot.ly/python/

randomvectors4• 7 years ago

Which other graphing tools are you comparing it to?

I think ggplot2 is a decent library, but in my opinion it gets more praise than it deserves.

gloflo3• 7 years ago

That's written by an opinionated R person/book-writer and stays on a very basic level of anecdotes and hearsay. Some comparisons are super short-sighted. For example the author seems to consider searching pypi for keywords are reasonable way of finding functions for both quite specific and totally unspecific terms ("spatial data"...).

You don't miss anything by skipping this.

hackerlurker• 7 years ago

Python is the second-best language for everything.

yesforwhat• 7 years ago

2nd best over x categories is pretty good.

jacobush3• 7 years ago

Where x is large. Which it is for Python. Which is probably why I use it so often.

agumonkey• 7 years ago

ultimate worse is better ? always be second

bionsystem2• 7 years ago

Does that makes C the 1st or the 3rd ?

bayesian_horse4• 7 years ago

The point is that there may be advantages to other languages in particular areas like games, web development, system programming, machine learning or the like, but those are not the same languages for each area.

Annatar3• 7 years ago

No, that would be AWK.

randomvectors4• 7 years ago

Except for working with data, which is what the original post is about.

IanCal• 7 years ago

"RStudio is to be commended for developing the reticulate package, to serve as a bridge between Python and R. It's an outstanding effort, and works well for pure computation. But as far as I can tell, it does not solve the knotty problems that arise in Python, e.g. virtual environments and the like."

I'm not sure I follow this, you can just set the interpreter.

I've started using rMarkdown more heavily, with reticulate & python for most data munging and r for plotting. Partly because I already know how to solve the problems I have in python more quickly than in R. The only thing I have against it at the moment is the debugging story isn't very nice by default, though I've not looked into how to improve this.

edit - if you've not looked into rmarkdown, I heartily recommend it. It is to me what the final output of notebooks should be. I can easily interleave code and descriptions, hide what I want, run it from scratch entirely as a default, and produce a range of outputs including interactive static webpages. Once web packaging is finally sorted, it'll be near perfect.

physicsguy• 7 years ago

I think it's a lot less common for authors of packages to put them on to PyPi vs CRAN (mostly due to the absolute mess of Python packaging) so it doesn't surprise me that his searches turned up not much there. On GitHub there are those packages:

https://github.com/search?q=poisson+regression

I've not used Rcpp, but Pybind11 is pretty mature, and works well, so I'm not sure why he's saying it's under development; by the same measure there was an update to Rcpp last week, so that is too. He mentions Cython which allows you to compile Python code, but my main use case for that is exactly what he says - wrapping up C/C++ libraries, which is very easy in it.

"Python is currently undergoing a transition from version 2.7 to 3.x. This will cause some disruption, but nothing too elaborate."

This is pretty out of date; in Jetbrain's survey, 84% of devs had transitioned.

geertj• 7 years ago

I’m going on a limb here and may get downvoted. But I’ll say it anyway. This battle is over, and Python is now the standard for data science, just like git is the standard for version control. The mindshare, number of tools, and number of people that know and use Python is an order of magnitude higher those for R. As a new data scientist it does not make sense to start with anything but Python.

Gatsky4• 7 years ago

This is manifestly untrue in some areas, for example bioinformatics.

shusson• 7 years ago

Hmm I got the feeling that in genomics things were moving towards python too e.g https://github.com/hail-is/hail.

deng2• 7 years ago

Well, I'd say you're on a machine learning limb there, and "data science" is - at least IMHO - much more than that. As the article says, statisticians usually much prefer R over Python.

bayesian_horse4• 7 years ago

My feeling is that the statistician's preference of R is an historic artifact from when Python lacked clear ways to do things R documents nicely. And it seems to be changing.

Annatar3• 7 years ago

That's subjective. I maintain R where I work and more and more people are coming to me wanting to use it. I also maintain Python and the uptake isn't even 1% of what the want and need for R is. And that's with me not advertising either.

ekianjo2• 7 years ago

I would say in the academic field it is far from being a clear cut at this stage and the growth of R packages is pretty stunning considering it is a very niche language.

usgroup• 7 years ago

It’s not about R, it’s about CRAN. It blows python as a tool for specialised statistics out of the water and hence R is more standard for natural sciences, actuarial uses, medical, etc.

R is a rubbish general purpose language and should not be used as a general purpose language. But it’s damn awesome for anything that considers matrices and vectors as the primary units of manipulation. Further, CRAN has everything ...

AstralStorm• 7 years ago

Because PyPi doesn't?

I'm being facetious here, PyPi is a few times bigger but less curated.

fantispug2• 7 years ago

This is missing the most important difference - deployability. R was built as a language to use interactively and does things like raise warnings for things that should be errors, requires an external package (packrat) for reproducible package management, and in general is foreign to most developers running operations. Python has good error handling, scripting and logging out of the box and managable package management, and is familiar to most developers and operations. Python has much better libraries for building general purpose tools (but fewer libraries for complex statistics).

I disagree with the "learning curve"; if you've learned other programming languages Python has a pretty simple and familiar core, and Pandas (while the API is an inconsistent mess) is well documented. Base R is quirky compared with modern programming languages, and the API is pretty inconsistent.

I also strongly disagree with the Tidyverse bashing. I'd say it has the shortest learning curve (especially for someone familiar with SQL), and is one of the main reasons I still use R today outside of deep learning - I find it much more friendly to work with than any alternative.

em500• 7 years ago

managable package management

Your joking right? Eggs, wheels, virtualenv, venv, pyenv, poetry, conda, python2.7, 3.4, 3.5, 3.6 (yes, we have all 4 versions installed in my current company's "production environment", not to mention 3 different versions of python 2.7 but no 3.7)...

sgt101• 7 years ago

I got put right off python years ago when I ran into egg dependency hell, I'm glad it's not just me.

fantispug2• 7 years ago

My experience is pip and virtualenv (or venv in 3.3+) works pretty well in both interactive and deployment contexts. My main gripe is Pip doesn't resolve conflicts between downstream dependencies (which I could only get a stable environment from by mapping dependencies and aggressive version locking). Conda has some advantages for certain use cases (handles libraries with installation dependencies well, curated repository), but I've never felt a need to use poetry or pyenv. Why do you have so many versions of python in production? I'd be surprised if now there are many packages that only work on specific versions (especially on 2.7 point releases).

I don't have much experience with packrat - but as opposed to pip it's another thing you need to discover and install. And so people don't do it by default when releasing code, and I've had to bisect versions of dependencies to get a working version of code. This can happen in Python too, but is rarer.

bionsystem2• 7 years ago

This is missing the most important difference - deployability.

I've deployed both R and Python for completely junior datascientists team, on top of a poorly managed infrastructure. I'd say they both have pros and cons and are actually both pretty bad. But R's packrat makes it slightly better than python. Python is a mess when you want to reproduce a working environment. Conda and pip both have huge issues. R's package management is pretty poor too with completely misleading errors, but at least it's unique and once you know your way around the most common errors you can build and run different projects quite consistently.

I've managed both RStudio+Shiny for R and Jupyter for python and overall my experience is better with the R stuff too. Things look a bit standardized while Jupyter needs tons of dependancies and (I felt) lacks a clear opinionated way of doing things.

I have 0 opinion on the actual languages though, as I'm not a developer.

bayesian_horse4• 7 years ago

At least in my experience it has been pretty simple to deploy Python software.

thom2• 7 years ago

I have found that deploying and maintaining RStudio Server has been an absolute breeze, whereas JupyterHub (we use the systemd spawner) is kind of a pain. That said, my worst nightmares are the crossovers _between_ R and Python - getting R code that interacts with Reticulate to work and perform well, especially with all the MKL threading options, has taken so much effort and compromise I'd almost ban one language or the other and live with an unhappy team of data scientists.

mekane8• 7 years ago

I have to agree - deploying Python code is horrible, unless the environment you're deploying to is super tightly locked down.

I had a colleague try to set me up with their R-studio project recently and we gave up because getting the packages installed was such a mess. So I'm not currently a huge fan of either. I don't do much data science or machine learning, but I work with and support people who do.

ekianjo2• 7 years ago

Deployability is more complex with R indeed but at the same time it is far from the pyenv or virtualenv complexity associated with Python. Both have their quirks and I dont think Python has peaked in how it deals with dependencies and reproducible environments.

techno_modus2• 7 years ago

R is also quite controversial because of its GPL license. I know that there are ways to overcome this issue but most decision makers do not want to risk when they see that a product has such a license.

Scarblac• 7 years ago

That's strange, is it ever relevant what the license of the language is? It hasn't stopped Linux.

deng2• 7 years ago

He couldn't find a nearest neighbor searcher in Python? There are several in scikit-learn and scipy has cKDTree. Those are really not hard to find.

Liquid_Fire• 7 years ago

Similarly,

For instance, though functions are objects in both languages, R takes that more seriously than does Python. Whenever I work in Python, I'm annoyed by the fact that I cannot print a function to the terminal, which I do a lot in R.

I assume "print a function to the terminal" means print its source? If that is the main complaint, it is available out of the box in IPython (%psource), which if you are doing data science you are probably using already.

dagw4• 7 years ago

There is also the inspect module that lets you retrieve both the source code and anything else you want to know about a function.

rat9988• 7 years ago

After reading this paragraph, I started seriously questioning the objectiveness of the comparison.

By contrast, just now I tried to find nearest-neighbor code for Python and at least with my cursory search, came up empty-handed; there was just one implementation that described itself as simple and straightforward, nothing fast.
The following searches in PyPI turned up nothing: log-linear model; Poisson regression; instrumental variables; spatial data; familywise error rate; etc.

This is not how you search for things. Usually I search on google for "poisson regression scipy" if I'm looking for poisson regression.

patrick54152• 7 years ago

How you search for things and You search for things are not the same thing.

gloflo3• 7 years ago

A basic level of adjusting to a language's jargon and ecosystem is mandatory. On the other side there is 'tidyverse', what ever that might be.

dagw4• 7 years ago

even "python poisson regression" will get you what you want within the top 5 links.

Macuyiko• 7 years ago

It's strange how many managers and project owners have asked me in the past whether to go for R or Python in their environment, as if the choice for a programming language will break or make your data science initiative. (Even more fun: it's not uncommon to find organizations where IT has finally accepted to provide Python, but without any access to a package repository, with some people being surprised that Python alone is not enough).

In any case, I've worked extensively in both environments and I don't think the author has considered every aspect. Below or my two cents.

- Elegance: slightly disagree. R might look more concise, but the language comes with many strange aspects (quoting, non standard evaluation) that can put a wig between novice and experienced team members. Python is more verbose, perhaps, but cleaner overall

- Learning curve: disagree. Even when working in R, modern practice would ask you to learn the tidyverse or data.table first instead of sticking with base R. Good tutorials are available for both

- Libraries: depends, the notion of "libraries" is too broad anyway, better to split it up according to the subcategories below. Both come with lots of packages, so I'd agree with it being a tie

- Statistics: agree with R. R is still the statisticians language, and many implementations of some more obscure techniques are only available in R. This being said, most ML shops today would be more interested in e.g. a good GBM implementation or deep learning rather than some robust statistics package. In R: think regression, ANOVA, significance tests, time series and niche subfields like bioengineering. In Python: think RF, GBM, t-SNE, deep learning

- Parallel computation: I'd say both are lacking, and you'd need to look more towards tooling such as Spark anyway. I'd also say out of memory computing becomes your first concern more often. Dask and Pandas on Ray are very nice on Python

- Foreign interface: kind of disagree. I think Python has matured better here

- Object oriented programming: disagree. The problem with R is in fact that is has about 4 (or more) OOP ways

- Interop: agree that you should avoid it, at the moment, it will only make deployment more cumbersome

Some other concerns I'd consider.

- Pipeline approach to ML ("model dev / model run"): better in Python. E.g. the clear approach of scikit learn to consider both preprocessing as the model itself as part of the fit-transform-predict pipeline with clear methods is way better than R. I've seen many novice R users fall into the trap of preprocessing a data set before splitting in train/test, for example. This has been one of the biggest drivers to push me towards Python coming from R. Most established libraries in Python commit to a shared, best-practice way of thinking whereas every package in R seems to come with its own ideas in terms of pipeline and usage

- Deployment: also a win for Python. Better package management / reproducibility, though it is possible in R as well

- Data exploration: I find this easier in R. Packages like dplyr help a lot here. Pandas' API is somewhat cumbersome

- Charts / visualizations: ggplot2 in R is still a champion, though good dashboarding tools exist for Python as well. Still, I find this easier to use in R

- Spatial analysis: both come with very solid libraries, though I find whipping up a quick visualization easier in R

- Deep learning: clear win for Python. Tensorflow, PyTorch and even Keras are not fun to use in R

- Reports authoring: possible in both, though R's markdown functionality combined with RStudio is fantastic. Nevertheless, Jupyter notebooks can be made to act as a reporting tool for both languages

patrick54152• 7 years ago

Data exploration: I find this easier in R. Packages like dplyr help a lot here. Pandas' API is somewhat cumbersome

I can’t claim to have tried everything out there, but so far it’s been my experience that matlab beats the socks off everything else when it comes to data exploration. I’m mostly looking at time series like data (but mostly not statistics). The ability to do things like click on a datapoint and export it and it’s index back to your workspace sound trivial. But in practice it’s a huge convenience and I’ve been unable to find a plotting package for python or Julia etc that can do things like this.

Tazinho2• 7 years ago

I like your comparison. So it sounds that one may draw the conclusion that it doesn‘t matter that much which language one chooses. In the end both languages provide typically required functionalities for data science and it’s probably better to find out for yourself which language feels more intuitive/effective to use than listen to an overheated discussion on this topic.

kome2• 7 years ago

Stata and SPSS are so much easier and fast...

jacobush3• 7 years ago

SPSS is pretty expensive

kome2• 7 years ago

true, SPSS is way too expensive. This will ultimately kill it. But big corps like IBM just don't care.

And it's a pity. Like a lot of "old" technology and "old" languages, also SPSS is far easier to use for normal people. Asl usual, IBM is just wasting potential.

MayeulC• 7 years ago

For what it's worth, I think GNU Octave could put up a good fight here as well.

I'd be interested in seeing it included in the comparison, though (I'm afraid it would lose on multiple points if only due to lack of funding, but it is very usable).

I currently have a fairly big project written in Octave, but will likely rewrite it in python for maintainability (and would rewrite it again in something else if it grew too much for Python).

there's also Sage, which is an interesting contender, but I do not know enough about it to know how it compares. Arguably, though, Sage and Octave are more geared towards numerical computing than data science, and I think that's where they shine. So, depending on your data and the processing you need, those could be more adequate.

dagw4• 7 years ago

Arguably, though, Sage and Octave are more geared towards numerical computing than data science

Sage is not really aimed at numerical computing, even though it can be used for that. It's primary use case is more towards computational algebra and number theory (and related areas) and is generally more focused on features needed by researchers and academics.

wtdata• 7 years ago

Sincerely, the present question for me is if I should start investing heavily in Julia. Eventually Python (like every other language) will be superseded by some new and better suited language for a set of problems. The question is if it will be Julia now, or some other in the future.

williamstein• 7 years ago

I remember thinking very hard about exactly this question in 2004, and landing on Python (as the foundation for Sage). It can be unclear how LONG until a language will be superseded.

bayesian_horse4• 7 years ago

Scipy.spatial has everything I ever needed in terms of spatial lookups.

There are lots of regression options. Scipy, Scikit learn, Pymc3, PyStan.

Metaprogramming in Python is easier than in R, and arguably more predictable and consistent.

teekert• 7 years ago

I'm a Python user (and I bet the author is primarily an R user), and probably I am biased but these two conclusions I find strange:

* R has Better statistical correctness based on "some dude"?

* R has better OO programming because you can print functions to the command line?

In my workplace R and Python are both well represented and I always hear from the R users that there is no "real support" for classes in R as there is in Python and that they miss it. I can't judge for myself though.

thom2• 7 years ago

I think on the whole we're going to care less and less about the distinction between language bindings in data science as time goes on. If something like Apache Arrow takes off and we end up with a decent standard representation for dataframes (either in memory or distributed), and most of the heavyweight processing (e.g. XGBoost, TensorFlow etc) is written in C/C++ anyway, then I don't massively care what languages people are using to express themselves (and I personally think tidyverse on the ingestion side and ggplot2 on the output side win here).

randomvectors4• 7 years ago

Having learned and used both, I disagree with most of his points:

- Learning curve

- Machine Learning

- Parallel computation

- C/C++ interface

- Object orientation

All of these are wins for Python, some of them like the learning curve are wins by a huge margin. I should probably do a point by point rebuttal later but so many of his points are incorrect and/or poorly justified.

Gatsky4• 7 years ago

Software engineers doing data science like Python. Everyone else doing data science likes R.

Annatar3• 7 years ago

So true. Business users love R and most of those using it worked in academia previously.

rizoic• 7 years ago

In the course of work I end up using both R and Python. I think both have their own use cases. Some of the observations I have had are:-

- The tidyverse ecosystem had given a huge boost to R. It had brought intuitiveness and consistency to R which was much required especially if you are a programmer coming from other languages. Also there are other ecosystems like Bioconductor which are also very mature.

- Rstudio and especially Rmarkdown notebooks are much better for reproducible analysis than Jupyter.

- It is very difficult though to develop standalone tools with R. For example it doesn't have a good argument parser.

Python

- The language is much more intuitive and more ideal for developing standalone tools.

- The ecosystem is many cases very fragmented though with a lot of libraries doing similar things.

- It lacks a good plotting system. Matplotlib is very powerful but has a very steep learning curve. In comparison ggplot2 in R is very intuitive.

randomvectors4• 7 years ago

Tidyverse is so overrated that I don't even know where to start... sure dplyr is nice to use if you're working interactively and ggplot2 has good first principles as the basis of its design, but that's about as far as the tidyverse niceness goes. The apis, the documentation, the inconsistencies (on a language level, between tidy packages and within each package), the problems with backwards compatibility, the evaluation (standard vs non-standard vs tidyeval) - it's a big mess.

If you're writing code that's deployed in any way, it's best to avoid the tidyverse as much as possible. This is also acknowledged to some extent by the main developer - https://www.tidyverse.org/articles/2018/06/tidyverse-not-for...

Tazinho2• 7 years ago

The statement in the link means that one should avoid to include the term tidyverse as a pkg dependency. Instead one should name the specific pkgs from the tidyverse individually. This makes sense, as the tidyverse is a collection of pkgs and referring to it as a whole just blows up the pkg. However, there is no indication to not use tidyverse pkgs (dplyr, ggplot2, stringr,...) in packages or production.

Petefine• 7 years ago

For me the readability of tidyverse code is crucial. I like pandas and use it daily, but often it requires deciphering to understand what is happening, especially regarding indexing. But tidyverse code can be easily read, and that has been a big help in enabling collaberation amongst our data team.

jhanschoo• 7 years ago

Python has just one OOP paradigm. In R, you have your choice of several, though some may debate that this is a good thing.
Given R's magic metaprogramming features (code that produces code), computer scientists ought to be drooling over R.

The benefit of this... is very debatable.

scottlocklin• 7 years ago

Funny, he's dead right about tidyverse, but I've learned to ignore Hadley's packages for not playing well with ESS. Snake case is moronic anyway.

Someone noted below that python is better for deploys: true. R is better for interactive use, and has a better package universe, though quality control for packages is vastly lower than something like scikit learn. R also completely dominates in classical stats, which is generally bread and butter compared to having the latest goofy neural thing. Assuming you actually do data science.

R vs. Python for Data Science

Related Stories