HN user
I am using R to analyse and document (knitr and latex) epidemiologic data which does not involve parsing a lot of text to extract my analysis data set. Data preparation for this type of research involves more combining data from different source tables, restructuring repeated measures, etc. I only know how to do that using R. Can Python be incorporated into the knitr literate programming framework and is it worth learning another language?
Can you elaborate on data.table being 'a game changer'. I am inclined to agree, but I'm am just starting to get a handle on it. I am still hesitant and switching between sqldf, reshape2, base::merge and data.table more than I would like. Do you think it could become a dominant method for data preparation?
Correction: I should have said 'literate programming' instead of 'reproducible research', since I'm not in a position to follow all components of RR.
Do you think the love it/hate it dichotomy over R for data 'munging' stems from different ways of thinking about data. I'm slowly getting comfortable in R since returning to work in a sort of freelance arrangement that makes me highly motivated to use free or affordable tools. I started out, however, in clinical epidemiology data analysis using MS Access and SAS. I still think of data in terms of rectangular data sets, RDBMS and sql. I have a hard time with vector and matrix related terminology. I think I'm going to end up using reshape2 and data.table a lot since sqldf is noticeably slower even with my small data sets (compared with web analytics, finance, etc). The problem with sqldf and variable names containing a dot is a real drag as I try to adopt good coding style. I am missing the clarity and familiarity of sql statements, though, as I try to find my new workflow in R. I hope a more unified approach to data munging emerges soon. BTW, I totally espouse the reproducible research (RR) method of documenting study design, analysis, interpretation... I am loving knitr and latex for RR so I can no longer imagine using different tools for data munging and analysis.