Scala Data Pipelines for Music Recommendations at Spotify

http://www.slideshare.net/MrChrisJohnson/scala-data-pipelines-for-music-recommendations

by flying_whale • 12 years ago

93 10 12 years ago

ajones2• 12 years ago

Maybe it's just me, but it seems that several data engineering organizations are picking up Scala. Is there a reason for this outside of Slide 13 in the presentation that I am missing?

My team has plans of moving away from Scala and towards Python primarily because of the job market. There is a feeling that it will be significantly easier to find a good data engineer who uses Python than it will be to find the same who uses Scala.

lmm2• 12 years ago

There are certainly some cool big-data tools in Scala (personally I find Spark much more compelling than Scalding, but the fact that I'm even comparing two reasonable choices is more than you'd get with many languages), but I don't think that's what's driving adoption. My case for Scala would be that it's a great language to write and an even better one to read: even more expressive than Python but much safer to refactor, a type system powerful enough to encode most of the business-level constraints and easy enough that you'll actually use it, and it's easy to understand which parts of the code do what because the type system makes it possible to isolate and track effects in a reliable way. In all the companies I've seen adopting Scala it's been a bottom-up thing, less because it offers some particular killer feature than because developers want it and it makes them more productive.

Hiring experience with a specific language is of course easier with more popular languages. But unless you're hiring emergency consultants or something, it's more important to hire smart people who'll be able to learn whatever language comes up. And I suspect that while it'll be very easy to find someone who "knows Python", the talent level is... variable. Whereas people who know Scala, though harder to find, will be the kind of people who learn multiple languages, who enjoy learning new skills even when there's no obvious market for them - i.e. the kind of people who you want to hire.

ajones2• 12 years ago

Thanks for the response. I agree with your thoughts on hiring. I'm one of those guys who came in and learned Scala. The initial reason for my team using Scala was, "Why not?" Since that time, my colleague who hired me has left the company and I have been the lone engineer on the team.

We're currently looking to hire a second engineer and have also been rethinking the direction of the data engineering team. Changing the language to Python was a suggestion for the hiring factor and because it is the popular language for data engineering. Scala has been a fun language over the last year and a half and I wouldn't mind sticking with it because it has been fun. For that reason, I have spent time comparing the two languages for data engineering and haven't come to a consensus on which one is ideal for my team.

seanmcdirmid• 12 years ago

i.e. the kind of people who you want to hire.

Are they able to match salaries with Google? :)

It depends on the level of the developer you can find and are willing to pay for. Not every dev shop is able to hire top talent, simply because that talent is limited (by definition) and not always necessary.

Most shops that are adopting Scala are doing so because Java has been so stagnant and they need to stay on the JVM. The argument of Python vs. Scala is less clear.

stoplight• 12 years ago

and an even better one to read...and it's easy to understand which parts of the code do what

It's not always easy to understand (from Odersky himself https://gist.github.com/odersky/6b7c0eb4731058803dfd#file-fo...):

def toVector: Vector[B] = fold(Vector[B]())(_ :+ _)(_ ++ _)

To a Scala veteran, I'm sure that's easy to understand; to someone who's been learning the language (like me) it looks like gibberish.

I've also had to write code like this:

client.post(args).mapTo[Response].map(r => (r.success, r.serverException, r.unhandledException) match {

case (Some(response), None, None) => response

case (None, Some(serverEx), None) => throw serverEx

case (None, None, Some(unhandledEx)) => throw unhandledEx

})

simply because of the underlying api that other co-workers have built. When it takes you longer than 30 seconds to explain how a piece of code works to others, something is wrong.

All that being said, I do like the language; just not the compile times. It's also pretty close to ruby (with which I'm most familiar):

ruby:

numbers = [1, 2, 3, 4, 5]

numbers.select { |n| n >= 4 }

# [4, 5]

Scala:

val numbers = List(1, 2, 3, 4, 5)

numbers.filter(n => n >= 4)

// List(4, 5)

shanemhansen• 12 years ago

Many huge companies are already invested in java. Big data tools are overwhelmingly written in java. However java the language sucks and scala has a veneer of coolness that makes bigco developers feel like they are being innovative. (See walmart's recent use of scala for an example).

I'd focus on hiring the best data scientists you can, and let them retool if they feel the need.

jghn• 12 years ago

Depending on how folks are defining data engineering these days, my group may or may not be using Scala for that purpose. There's another group of computational biologists (read: data scientists who know biology) who come up with novel algorithms to gain new insight in our raw data and then our group takes those algorithms (among other things) and gets them working at massive scale in well vetted data pipelines. Our company refers to it as data engineering (as in "software engineering the data science") but I know that's not what everyone means when they say that.

The use of Scala predates me, it's been going on about 5 years although there's been a huge shift in that direction over the last year. This time last year I'd say it was about 50/50 Scala and Java and now almost all new code is written in Scala. I've been told that the initial "why" of scala was that it made it easy to write DSLs with which to write pipeline data flows w/o leaving the JVM. We have our own pipeline infrastructure written in house (in scala) although we're starting to move toward more OTS options (also in scala).

The surge over the last year has more to do with personnel changes within the group with more people joining who were more pro-Scala (such as myself), mainly for some combination of the functional trappings and the reduction of boilerplate and other nonsense from Java.

In terms of the job market, while we'd love to see a candidate who is a Scala wiz, we don't expect them to have an ounce of scala background. Personally I don't care if they have any java background either, although not everyone in my group agrees. I lean very heavily on the side of hiring smart people instead of hiring for buzzwords, assuming that we're not talking about contract positions.

FWIW the data scientists in our group are mostly split between R and Python, with some also writing a fair amount of scala. The latter has more to do with some of the in house pipeline management tools than any data science-y reasons.

mortenjorck• 12 years ago

All this work with matrix factorization and alternating least squares to find undiscovered gems a user might actually enjoy, and then they torpedo it with a simplistic and utterly useless "popular in your area" algorithm.

BhavdeepSethi• 12 years ago

Things to worry about? -Be patient with the compiler.

This should be number one!

Scala Data Pipelines for Music Recommendations at Spotify

Related Stories