Ask HN: Processing multiple GBs of data on local machine

https://news.ycombinator.com/item?id=29930771

by EdwardDiego • 5 years ago

3 6 5 years ago

news.ycombinator.com

I recall a few blog posts on this, but I'm struggling to find them now that I need them.

I have about 100GB of Log4j formatted logs to process to find a particular needle in the haystack, and am looking for a decent way to process those files locally without breaking out Spark in EMR etc.

I recall a few blog posts on this subject, but my search fu is letting me down. Is this ringing bells for anyone?

Thanks in advance :)

viraptor• 5 years ago

I know one related one, but can't remember how to find it either. (I found a response to a response to that post - https://news.ycombinator.com/item?id=8920194)

However, if you're just after a specific needle... why not grep? With 100GB on a local machine you'll either need to do pre-processing / filtering, or you'll be IO-limited. So instead of heavy tools, why not start with basics?

EdwardDiego2• 5 years ago

Cheers, I was most likely going to, but wanted to stand on the shoulders of giants, as it were, and see what had worked for others, but that gives me a rough time-frame to look around in.

Thanks!

Andys• 5 years ago

ripgrep ? It works on my computer at over 1GB per second

EdwardDiego2• 5 years ago

Thanks :)

speedgoose• 5 years ago

You could also use SQLite after some little parsing.

VirusNewbie• 5 years ago

Spark works locally and should exploit multicore. I'd use spark instead of writing my own multithreaded system. If you don't like spark there's also apache beam.

Ask HN: Processing multiple GBs of data on local machine

Related Stories