HN user

mcroydon

36 karma
Posts0
Comments13
View on HN
No posts found.

Thanks for the tip, I didn't think to copy the data to ephemeral storage like that. That'll probably speed things up a lot.

I ended up splitting the data in to a relatively small number (~200) of ~30MB gzipped files in order to initially saturate the mappers and speed things up. If that's not necessary after moving to ephemeral storage that's fine by me!

The biggest thing I got from PCI was a basic discussion of a lot of algorithms that I was able to research further on my own. I had no idea that I wanted to know more about K-means clustering until I read the book.

If that's all you get out of it great, but there's a lot of interesting stuff in there, covered in enough detail to get you started, which is more than enough to make me happy.