Starting data analysis with R: Things I wish I'd been told

http://reganmian.net/blog/2014/10/14/starting-data-analysiswrangling-with-r-things-i-wish-id-been-told/

by ingve • 12 years ago

116 49 12 years ago

leeber• 12 years ago

He forgot the last one: "to use python instead"

That was my attempt at a joke. But seriously, I love python when it comes to manipulating data and doing anything statistical. You've got numpy, scipy, scikit-learn, etc.

vegabook3• 12 years ago

yes I have moved (back) to Python mainly because R is too slow when we get beyond a certain data size and the language is not powerful enough when data starts having to be moved around at scale. I have a 5-10 times speed improvement in native Python and another 30x more if I can vectorize things in Numpy. However a huge caveat is that R is much more succinct when it comes to exploratory analysis during what I call the "data rotation" phase because its vectorized nature is so much more efficient at selecting, reducing, cleaning and rotating data, than even Pandas can manage. It's irritating having to write list comprehensions constantly for what would often have been a ridiculously direct and efficient vectorized command in R. Moreover R's graphics leave matplotlib in the dust, though this advantage is eroding with the JS libraries taking over.

The other area where Python crushes R is if your data is live streaming. Here you inevitably need a full fledged programming language with proper asynchronous io capabilities and multithreading / multiprocessing that is not batch oriented.

infinite8s• 12 years ago

Can you give an example of the a somewhat complicated vectorized command in R that would require lots of list comps in Python?

Aqwis• 12 years ago

As a student of statistics, I'm kind of split on R. On one hand, it's just not a very well-designed language. The fact that it has three (!) independent object systems is a testament to this. On the other hand, as vegabook also mentions, working with vectors and matrices is just a lot more natural in R than in general-purpose languages like Python, because R's syntax has been built from the ground up to work with the kind of structures you usually work with in science.

I'm hoping Julia might become a good alternative to R and Python, but I can't see it catching on in the statistical community anytime soon given how many people are still using relics like SAS and Stata. The raw fact is that statisticians (considered as a group) just aren't very good at programming (and many older statisticians can't program at all), which means that a well-designed programming language may not necessarily be easy to use for a member of the statistical community used to point-and-click statistics suites.

hadley6• 12 years ago

I think R is a well-designed language. It definitely has its quirks (what language doesn't?), but by-and-large they are problems with the standard library, not the language. This is admittedly a subtle distinction, but it's much easier to fix problems with the standard library than it is with the language.

Three aspects of the language that make R particularly well suited for statistical programming are:

1) Missing values built in at a fundamental level.

2) Metaprogramming capabilities. The best way to solve many categories of data analysis problems is to design a domain specific language which allows you to easily combine independent pieces. R's incredible flexibility is great for this.

3) Fundamentally vectorised and functional. This allows you to elegantly express many common data analysis tasks.

tacos• 12 years ago

I know we don't reward snarky humor 'round these parts, but I was about to say the same thing. Python seems to own this space and the ecosystem around Python and math/stats/analysis is exceptionally healthy. If there's a specific place where R kicks ass please speak up -- it's fallen off my radar.

stdbrouw3• 12 years ago

R is in some ways more forgiving to newcomers. Sure, there's all sorts of weirdness around how vectors and matrices work, and don't get me started on the cryptic function naming, but (1) almost all batteries are included -- hardly ever a need to hunt around for packages, (2) RStudio is really nice, with graphics, a shell, a text editor, documentation etc. all in one place, (3) it's mature and well-tested.

I prefer Python myself, but after spending a couple of months with R I do understand why people like it.

(OTOH I'll be a happy person if I never ever have to work with SAS ever again.)

dthal• 12 years ago

I use both Python and R a fair bit. As a language, absolutely I prefer Python to R. However, I think there are two areas where R is better than Python and together, I think they add up to a durable advantage, at least for stats people. 1) Package support. Yes, Pandas and scikit-learn are good, but R still has a definite edge here. Here are three things I've needed lately where R has hands-down better code available: forecasting, frequent itemset mining, and network community detection. 2) Non-programming uses. There are a lot of tasks where you need a computer, but just to do one thing, a plot, calculate a statistic, ... stuff like that. R is better in that use case.

hadley6• 12 years ago

There are three areas where I think R is the clear winner:

1) An IDE for data analysis/programming: RStudio

2) Easy way to turn your analyses into reports: knitr

3) Easy way to turn your analyses into interactive webapps: shiny

(I also think R wins on visualisation and data manipulation, but I'm biased ;)

Gatsky4• 12 years ago

How do you feel about reproducible computing in python? R is very well set up to A) get it running on any platform easily B) report the crucial parts of the environment. I know that if I grab someone else's (published) code written in R, I'm pretty confident I can make it work. Part of this is the great package management through CRAN or Bioconductor, and also because often important reference data for bioinformatics is actually available through the package manager.

I haven't done much with Python, but I don't quite get the same feeling (happy to be told that the reality is otherwise!). For example, the opening line of the installation guide for Pandas doesn't inspire great confidence in me: "The easiest way for the majority of users to install pandas is to install it as part of the Anaconda distribution, a cross platform distribution for data analysis and scientific computing."[1] Do I really need to install the HDF5 package so I can split a concatenated variable into two columns??

[1] http://pandas.pydata.org/pandas-docs/stable/install.html

vegabook3• 12 years ago

Python's pip is pretty good though not quite as polished as CRAN. I have had few problems running complex code from third party sources, though one always has to be aware of the Python 2 v 3 "problem" (though it is diminishing now with most things available on 3). If you get pip up and running on a new Python installation you can avoid Anaconda/Canopy if you want a clean installation, and I have installed fairly complex Python setups in multiple locations without too much trouble. Let's be fair, R can also be tough if it calls a lot of third party libraries. Just try to get rJava working properly for example if the local R and Java installations are not both 32 or 64 bit. It can be a complete mess to disentangle this sort of stuff in R. Or for example running code that uses Cairo, on a mac. My experience is that Python's poor package management reputation is not really deserved anymore. Python's virtualenv also allows you hermetically to seal away an entire python environment, including its libraries, so that it will not conflict with other python environments that might have different versions of the interpreter and/or libraries. I am not aware of anything this robust in R.

Reproducible computing? The ipython notebook is awesome, though I am not sure if there is anything as good as knitr if your workflow is LaTeX oriented.

R "hands" will usually find Python a backward step when it comes to vectorized data manipulation, but its a forward leap if your data becomes too big or if you have to step out of the comfy environment of exploratory analysis into any form of (even trivial) production settings.

And no you definitely do not need HDF5 to effectively use Pandas.

deskamess• 12 years ago

I find knitr easy to use. They way it generates graphs and can output to pdf/html is really useful and is reproducible and easily shared. While essentially just markdown + R code the code can point to data sets instead of having it embedded. It has a good set of graphing libraries (ggplot2, etc) too. I can see how this could be the killer app that gets social science research papers written and produced in knitr. I always thought IPython would take this crown but R/knitr is looking good. Have not used Shiny yet

Edit:knitr not rdoc

jghn4• 12 years ago

The thing w/ reproducible research (I was an early BioC core member and have worked directly w/ its RR advocates) is that it requires having an exact set of R and packages. I know that BioC tries to do this (I wrote the original BioC package download script) but weird things can still happen. A few years ago I was tracing down a bug in some computational biologist's code that really traced down to some wacky version of a particular package which might be downloaded in the right circumstances.

In a previous life what we did was for every project you'd download a snapshot of an R environment, including all packages. That, and only that, was used for all computation for everything involving that project from start to finish. If Docker was around at the time, that's what we'd have used.

afarrell2• 12 years ago

You don't have to install the entirety of anaconda. You can install miniconda (from here: http://conda.pydata.org/miniconda.html) and then do `conda install $package_name` or, if like me you like to create separate environments for separate projects... `conda create --name $environment_name python; source activate $environment_name; conda install $package_name`

disclosure: I work on miniconda. I'm currently working on improving our developer experience. Complaints are welcome.

consz• 12 years ago

Totally agreed. I do model analysis on data sets with 200k-5m rows and anywhere from 500 to 20k columns. I originally started doing my work in R, but about two years ago, python started improving rapidly for heavy data analysis, and at the moment I'd say it's a clear winner.

IndianAstronaut• 12 years ago

For that kind of data or larger, I would avoid R and Python and move to writing my own algorithms or try out something for more heavy duty analysis such as Mahout or Spark. R and Python are still one box and memory constrained.

stdbrouw3• 12 years ago

Also, read "R language for programmers" http://www.johndcook.com/blog/r_language_for_programmers/

It explains a lot of the quirks.

hadley6• 12 years ago

Or try "Advanced R", http://adv-r.had.co.nz. It's my attempt to explain the language in such a way that programmers from other languages can appreciate the beauty and elegance behind the quirkiness.

earino2• 12 years ago

As a programmer, I can enthusiastically endorse this book. R "as a language" is hard to grok when you come to it with the mentality of "a programmer" due to it's history as an interactive environment for data analysis, and an interface to algorithms first, and a programming environment second.

Hadley's book is a solid introduction when you are trying to map concepts from other, more traditional, programming environments to the R way of thinking.

scishop• 12 years ago

Things that are likely to trip up new R programmers:

* Function arguments are always passed by value. Objects are copied if they are modified in a function.

* Function arguments are lazy evaluated.

* Watch out for automatic factor conversion when importing data. R will display your string data as text, but behind the scenes it will treat it as an integer.

* R is slow. Really, really slow. All your intensive calculations should be handled by libraries written in C, Fortran or some other compiled language. Your R code should be mostly for glueing things together.

jghn4• 12 years ago

1) can always use environments! :)

Okay, I'll stop at 1).

Having used R for over 15 years now, the one "feature" which still makes me shake my head is the partial evaluation of function arguments. If function foo has an argument 'blah' and you say foo(bl=5), it'll autocomplete that to 'blah' if there's nothing better to use.

The number of bugs I've found due to that ....

bachmeier• 12 years ago

In my experience teaching R, the single biggest issue is that it is a dynamic language that is in love with silent casting [or really returning objects with unexpected types]. Although other bugs might be more common, none will frustrate the user more than the type of a variable changing without warning, and most of the time the error message refers to something else.

A couple of examples are stripping the time series attributes of a ts object and default conversion of a row of a matrix to a vector. These are the cases where they come to my office and say they have no idea what's going on. After using R for a decade, I know the language well enough that these are about the only errors I get.

greenleafjacob• 12 years ago

Something I've found useful is the sqldf package [0], which lets you use data frames as tables in SQL with all the power of joins and so on.

[0] https://code.google.com/p/sqldf/

hadley6• 12 years ago

NB: sqldf works by copying your data frame into a temporary SQLite database, running the query and then copying the data back. So it's not that fast.

Instead, learn a native R package like dplyr or data.table that supports all the power of SQL, and is v. v. fast.

Gatsky4• 12 years ago

+1 for dplyr. Really helpful, responsive community around it as well.

ggrothendieck• 12 years ago

(!) sqldf works with H2, MySQL and PostgreSQL too - its not limited to SQLite. (2) Most alternatives to SQL available in R don't work well with complex multi-way joins. You wind up materializing the full outer join first which can be quite problematic. (3) sqldf is fast enough and R is slow enough that quite often sqldf is faster than base R so if R is fast enough for you then its likely that sqldf is too. (4) The speed is often the last thing you need to worry about. The ability to express a query in a familiar way (if you know SQL) is often the more important consideration than having to learn a new system.

spo81rty• 12 years ago

I just recently started playing with R via Azure's new Machine Learning service. From what I have seen I am really impressed. It helps with the input and outputs of the data and makes it easy to turn your process in to a web service that can be easily consumed. I had planned on setting up some linux boxes to host R, but now I can just use Azure ML and not have to jack with it.

psychometry• 12 years ago

Strange that someone who knows about plyr (and is clearly not a beginning R user) is still using "for" loops.

houshuang2• 12 years ago

There are lot's of ways of doing things in R, and I am always learning. I always use vectorized functions when I do something to every row in a data.frame, but when I want to do something to every column, like here, I sometimes use for. How would you use something like plyr here, and would it be as easy to understand?

  likertcat <- c("1"="Not at all", "2"="To a small extent", "3"="To some extent",
    "4"="To a moderate extent", "5"="To a large extent")
  
  for(e in names(db[,9:44])) {
    db[[e]] <- revalue(db[[e]], likertcat)
    db[[e]] <- ordered(db[[e]], levels= c("Not at all","To a small extent",
      "To some extent","To a moderate extent","To a large extent"))
  }

nograpes• 12 years ago

Like this:

  db<-data.frame(replicate(44,1:5))
  likertcat <- as.ordered(c("Not at all", "To a small extent", "To some extent",
                 "To a moderate extent", "To a large extent"))
  db[ ,9:44] <- lapply(db[,9:44], function(x) likertcat[x])

SixSigma• 12 years ago

As an aside, that is not a valid Likert set. It is not balanced around a central non-comittal value.

sytelus• 12 years ago

R is very acquired taste - or rather you struggle through until you can memorize all the gotchas. For people who are not doing doing data analysis as a full time and perhaps only job, remembering intricacies of R is going to be deal breaker. Personally I would shift towards Python unless there are exclusive packages in R that you have to have it - and that's becoming rarer by the dat. Combined with iPython Notebook and recent wave of migrations of packages to Python, I see little need to deal with R on regular basis.

micheljansen• 12 years ago

Thanks for introducing me to iPython Notebook. I had never heard of this, but it looks very promising. The browser-based interface does not look as polished as R's studio application and I'm a bit confused why this functionality is not in the Qt console, but I'll check it out nonetheless.

afarrell2• 12 years ago

Much of python's power for data science comes from the scipy/numpy libraries and the tools built on top of them. Unfortunately, installing these requires first installing a bunch of fortran libraries. Fortunately, There is a free distribution of python that comes with them and comes with a package manager you can use to install python libraries that have C extensions. You can also install R packages and use them from iPython notebook. https://store.continuum.io/cshop/anaconda/

Starting data analysis with R: Things I wish I'd been told

Related Stories