Ask HN: Show me how to perform raw data analysis more effectively
https://news.ycombinator.com/item?id=1383538I've been a ruby guy for a few years now. The elegance of the language is the biggest draw for me. Unfortunately, I feel like every time I pull a ton of data from sql, or a set of flat files in to memory that I'm plagued by GC collection pauses and that it takes _much_ longer than it should to process the data.
That said, I'm looking for a better language or at least a better framework for just doing exploratory data analysis. We've done some big data transformations with hadoop (pig) and post processing the output with a little bit of C. Those datasets were absolutely huge and the problem warranted using something a bit more heavyweight like that.
For the small / medium sized datasets I'm looking for something a bit more elegant. What good tradeoff between speed and elegance have you found? Got any supporting code samples to make your case?