How to process a million songs in 20 minutes

http://musicmachinery.com/2011/09/04/how-to-process-a-million-songs-in-20-minutes/

by brianwhitman • 15 years ago

141 15 15 years ago

musicmachinery.com

revorad2• 15 years ago

Can you dynamically adjust the number of EC2 instances to optimise for processing time or price?

plamere2• 15 years ago

yes, the number and size of EC2 instances can be set by switches on the command line when you launch the job.

revorad2• 15 years ago

I saw that but I mean how do you know you need 100 instances? Is there a way to estimate the optimum number or at least set boundaries, such as max price or max processing time?

Do you get back info such as how long each instance took to do its job?

KirinDave• 15 years ago

So... MapReduce? Kinda figured that when you had 6 zeros after your first zero.

This looks like a fun project, but I can't help but feel like Hadoop experience reports are a little late to the party at this point. Is there anyone out there who doesn't immediately think MapReduce when they see numbers at scale like this? If anything, the tool is overused, not neglected.

KevinEldon• 15 years ago

MapReduce wasn't my first thought, but that has more to do w/ my inexperience than anything else.

The cost though... that was shocking! It comes out to about $42/month to host the data and a $2 per analysis run. That's very affordable if you're producing results someone might want to pay for.

slewis• 15 years ago

Its a well-written tutorial on using mapreduce to process a specific large music dataset. Now someone can jump right in and start writing their own processing code without worrying about all the boring details.

Maybe the headline is a bit misleading but I find the post interesting and valuable.

joebo• 15 years ago

It would be interesting to know how long it takes to run locally on a single instance

hogu• 15 years ago

agreed, actually, I want to know how long it takes to run on say... an i7, I've gotten pretty poor performance out of those amazon small instances when doing alot of cpu oriented tasks.

thirdhaf• 15 years ago

Let's pessimistically assume you have to read all information associated with each file from disk. I suspect this sort of query is IO bound since processing is minimal and assuming we can mostly avoid fragmentation on disk we could feasibly read data at 100MB/s with a consumer level magnetic drive. The article says the data set is 300GB so we could feasibly get this done in an hour.

joshu• 15 years ago

Yeah. That can't possibly be the right thing to do...

dvcat• 15 years ago

Can anyone clarify if song data is dense? If it is dense, I am not even sure if Mapreduce is the right paradigm to use mainly because you will eventually get to a situation where transfer time overwhelms compute time.

brianwhitman• 15 years ago

the EN song data is dense in the sense that there is far more "columns" than rows in almost any bulk analysis -- average song unpacks to ~2000 segments, each with ~30 coefficients + global features.

however, in paul's case here he's really just using MR as a quick way to do a parallel computation on many machines. There's no reduce step, it's just taking a single average from each individual song and not correlating anything or using any inter-song statistics.

How to process a million songs in 20 minutes

Related Stories