How good is Google’s Instant Mix?

http://musicmachinery.com/2011/05/14/how-good-is-googles-instant-mix/

by hillad • 15 years ago

57 29 15 years ago

kefs4• 15 years ago

So, Google should purchase The Echo Nest.

chapel• 15 years ago

He addresses this in his article, but you really can't evaluate competitors to your own service in such a manner. Not only were the testing criteria highly subjective, the fact that songs he considered 'WTF' in the lists were the negative marks.

I know music is a very subjective subject, I really think he could have been more objective about the whole thing. Maybe using other peoples music collections and getting their own personal opinions on which songs work and which songs don't. Add to that, have other people rate the playlists he generated, so it wasn't just his opinion.

Also, wtf about Genius only getting 10 marks against it for not doing Beatles playlists? I'm sorry, if you hold the criteria that any songs that are out of place are considered negative, no songs in a playlist should be worth 24 'WTF' points. Not that it matters in the comparison, and it was nowhere near close.

I don't have access to the beta, and at this point don't really care about it, but this just screams as self promotion. I think it would have been a lot more respectable if he had been more objective concerning the tests he used.

watty• 15 years ago

I agree music is subjective and his results are biased. However, it's safe to say Google's attempts are terrible.

brianwhitman4• 15 years ago

It is impossible to be objective about music. The point of his "WTF" test (not speaking for paul, but i do work with the guy) was to look at each song by each provider and ask "would most people agree this doesn't belong" erring very far on the generous inclusive side. You can definitely quibble about whether or not a particular song deserves a WTF mark but you can't argue that google's results are very terrible overall.

You couldn't do a test with other collections because the beta is very limited right now-- I can only think of a few people I know that have access. But I can concur that my results are as terrible as his were.

Yes, he works for EN (which he's very clear about) and yes it's a bit inside baseball showing a service that most people can't use (because they're not developers or customers) but you can scroll past our results & take it as a post titled "How is Apple so much better than Google at such a data driven task?"

(I would have given the lack of beatles on genius -24 too! I also found a couple EN clunkers that I'd give a WTF to, but I'm a notorious jerk for those things, ask anyone i work with :)

kefs4• 15 years ago

> you can scroll past our results & take it as a post titled "How is Apple so much better than Google at such a data driven task?"

Sure, but that wouldn't exactly be fair seeing as Genius is 3+ years mature, and Google Music+Instant Mix is in beta and less than a week old. OP even seems to think so...

> The last time I took a close look at iTunes Genius was 3 years ago. It was generating pretty poor recommendations.

omaranto• 15 years ago

Nitpick: an empty playlist has no songs out of place, so it should score 0 WTFs (I don't understand how you got length(empty)=24). Giving Apple some WTFs instead of giving it 0 is clearly more in the spirit of the test than following the criterion to the letter would have been.

spicyj• 15 years ago

The playlists have 25 songs each, one of which is the original, so that's up to 24 opportunities to make a mistake.

kelnos• 15 years ago

While that might make sense in using a logical definition of what a "WTF" is in this context, assigning 0 WTFs for the inability to generate a playlist is useless for the purposes of this comparison.

From my perspective, I'd say an empty playlist is worth 24 WTFs, in the sense of "WTF, I expected 24 similar songs and got 0!"

jonnathanson• 15 years ago

His criteria are subjective, yes, but isn't that the point? The point of music recommendation engines isn't to figure out the absolute "best" playlist based on a starter song. It's to figure out the best playlist for the individual user. If the user is asking "WTF?" about the songs on his list, then by definition, the engine has failed that user.

While the author could have been more objective about his criteria, ironically enough, I think he missed the more salient point that he raised by implication: that music engines should be mapping the user's behavior patterns vis-a-vis the songs in his collection, and not so much objective connections between songs. This is what Genius does and has always tried to do, and it's why Genius seems to work better for the author. Genius doesn't focus as heavily on attempts to forge objective links between songs, so much as it focuses on attempts to draw links in behavior patterns w/r/t songs by likeminded users.

When listening to songs in a collection, our brain maps out its own connections between songs, as reflected in the way we compile our own lists consciously or subconsciously. Sometimes those connections make objective sense (i.e., "I want to listen to '70s funk, so I'm going to pick ten '70s funk songs in a row."). Sometimes those connections make little objective sense (i.e., "I am listening to a track by Lady Gaga, and afterward, I feel like listening to a track by J.S. Bach."). A good mixing engine figures out the idiosyncracies and subjectivity of our brains, as reflected statistically by the choices we've made in the past.

nl2• 15 years ago

Paul has done a lot of work in the field of playlist evaluation.

I think that if you survey the researchers in the field, the "WTF test" would be considered fairly reasonable - especially for a quick-and-dirty evaluation. Can you point to any specific songs that he said were WTF's and you think aren't, or vice/versa? If not then it would appear to meet the objectivity criteria.

Using his own music collection might be slightly more suspect. Changing that might have flipped the outcome of iTunes vs EchoNest, but wouldn't have changed the real news here: Google does really, really badly.

Tyrannosaurs3• 15 years ago

I think you can argue the individual songs but the overall findings are sound - Google is poor, EchoNest is good, iTunes is good bar one song (which is suspicious). The methodology is a bit finger in the air but then so are the findings.

plamere• 15 years ago

Hi Chapel - you suggest I could have been more objective, perhaps using other people's music collections and getting their own personal opinions. That, of course, would still be a subjective evaluation. It would be better, of course, more opinions means more data. In fact, I welcome people to make the same evaluation with their own collections. Enroll them in all 3 systems, create some playlists and evaluate them. Since most people don't have access to Google Music yet, this is hard to do. Still, you can look at the playlists that I generated and make your own WTF opinions about them. Or better yet, count the WTFs in the playlist Google created during the Google I/O keynote. You can see it at 28:29 of this - http://www.youtube.com/watch?v=OxzucwjFEEs Here's a screencap. https://skitch.com/plamere/r9x2k/youtube-google-i-o-2011-key...

There's no objective evaluation of playlists. I've proposed a simple, subjective one that I think gets the job done. I'm happy to try other ones if you have something to suggest.

andrewflnr• 15 years ago

It would be interesting to run these songs through Pandora and compare the results.

MikeCapone2• 15 years ago

Could this be because this service is so new that they haven't yet had the benefit of gathering lots of data and stats from users?

lordlarm• 15 years ago

I too believe so.

Let Google build their product first, it is still a beta, for a reason. Music recommendation is close to impossible without lots of data - and it becomes unfair to compare Google Music with iTunes Genius, which has been on the market in 3 years.

brianwhitman4• 15 years ago

(again, i'm the co-founder, grain of salt time, but check it out yourself) but The Echo Nest makes amazing similarity judgments and playlists without any of the data that you think Google needs to wait for. We crawl the web and analyze the audio and figure out what people are saying about music and what it sounds like. We don't need usage data at all to make recommendations. No usage data went into the EN results in the OP.

Androsynth• 15 years ago

I have two thoughts on this: 1-As has been mentioned multiple times on HN recently, many of the best minds of our generation are working on the algorithms that go into products like genius. They are basically the same as recomendations, online advertisements, etc. Its not surprising that theyre really damn good at this point. 2-Google probably doesnt care about the product itself, more likely it just wants to mine the data that gets put into it. Like goog-411, the service is just a conduit to acquire data.

nl2• 15 years ago

2-Google probably doesnt care about the product itself, more likely it just wants to mine the data that gets put into it. Like goog-411, the service is just a conduit to acquire data.

To what end? Unless Google is trying to make sure that when its software becomes sentient and takes over the world it has good taste in music (heh) I can't see much use for the data beyond building a recommender.

bad_user• 15 years ago

Apple has data about purchased music from iTunes, with which it can do pretty damn good similarity scores. Google has no such data.

tintin• 15 years ago

I think it would be more interesting to recommend music by mood instead of genre.

Jazz doesn't mean a thing. When I'm happy I don't want to listen to down jazz although it's jazz. Maybe I would like some happy funk and metal as well.

Tyrannosaurs3• 15 years ago

Anyone else curious about why the Beatles returned nothing on iTunes?

[Note: All this applies to Beatles tracks ripped from CD rather than purchased via iTunes.]

I've just tried Genius (updated immediately before) on a selection of Beatles tracks in iTunes - Eleanor Rigby, Yellow Submarine, Lucy in the Sky with Diamonds, Sgt. Peppers Lonely Hearts Club Band and not one of them produces a playlist.

Even before they were selling Beatles tracks enough people will have ripped them from CD to have the data to produce playlists so it seems unlikely (but not impossible) that it's data related, which suggests that it's either an odd, very specific error or intentional.

What's stranger is that for some (but not all) of the tracks it will make Genius recommendations to buy...

Anyone any ideas? Some strange part of the Beatles licensing deal perhaps though that would be very odd as really who gains anything by that?

How good is Google’s Instant Mix?

Related Stories