How Kaggle Is Changing How We Work

http://www.theatlantic.com/technology/archive/2013/04/how-kaggle-is-changing-how-we-work/274908/

by antgoldbloom • 13 years ago

51 28 13 years ago

rm9993• 13 years ago

Kaggle is a great idea; I actually entered the industry partly through a data mining contest (pre-kaggle). But I think the article overstates Kaggle's influence on industry, at least at this point in time.

>the Kaggle ranking has become an essential metric in the world of data science. Employers like American Express and the New York Times have begun listing a Kaggle rank as an essential qualification in their help wanted ads for data scientists

No, it hasn't, and the nytimes job posting they link to doesn't list it as an "essential" qualification (they didn't link to the amex posting). I know many people with experience in the data science space and very few of us have taken part in a Kaggle competition. It's not that there is anything wrong with Kaggle, but the pay is low for the required effort to differentiate yourself. Many modeling competitions I've seen require an inefficient use of time in the "diminishing returns" part of the process, which means winning requires a lot of free time. I worked with a guy who won a couple prominent data modeling competitions, and frankly I thought he was a mediocre data scientist (but a very hard worker).

I sometimes wonder who takes part in the competitions; and then I remember myself from six years ago, applying for jobs and looking for a way to make my resume stand out.

achompas2• 13 years ago

Agreed. In conversations with data scientists looking to hire, we've never discussed Kaggle aside from "wow they reeeeeally focus on that evaluation metric."

In fact, our conversations quickly turn to the Netflix Prize, where first place won the competition with an algorithm that could not be ported to production, and we discuss how poorly these competitions map to reality.

None of the data scientists I know hire based on Kaggle score. Several don't even think positively of Kaggle.

tocomment• 13 years ago

Why couldn't it be ported to production?

reader5000• 13 years ago

I think Kaggle is great, but I don't really see the model of "work full time for months competing with 80,000 other people in hopes that your data model will turn out 0.001% more accurate than everybody else's [which at some point becomes largely a function of the initial seed on your RNG when you fit the model] in order to get a single pay check" to be a workable concept for work.

rm9993• 13 years ago

Yes, you succinctly made the same argument I was trying to make. I think Kaggle is pushing the industry to be more efficient, but in an ironically inefficient way.

F_J_H• 13 years ago

Don't overlook those who do it for reasons other than earning money.

Look at how many people solve complex crossword and sudoku puzzles everyday for free...

NewAccnt• 13 years ago

Just another exploitation of creative labor if you ask me. The people competing for prises may be doing it for fun, but as you can see in the industries using it, these are serious big money applications. Of course, nobody is blind to this and a majority of applicants are going to try to go for the cheapest and easiest solution for a shot at the gold.

paulgb• 13 years ago

It's interesting how differently the data science world has embraced this model vs. designers, of which a vocal set seem to scoff at any mention of 99designs. (I am currently competing in a Kaggle competition and the competitive factor is what makes it fun)

That said, I doubt more than a handful of people could make a living off competition winnings alone.

dbecker2• 13 years ago

Kaggle is pivoting, and they don't intend "making a living on Kaggle" to involve contest winnings. Instead, they are offering a platform for prospective employers can use to contract to highly ranked Kaggle members.

And those gigs pay hourly.

jurassic• 13 years ago

Good designers scoff at 99Designs because the expected payout is absurdly low for the amount of skill and effort usually required to win a "competition". The price structure undermines the pricing of struggling freelancers trying to break out in the low- to mid-tier end of the market. Bean-counting execs who don't care about design see no reason to pay $X000 or $XX000 to a pro designer with legitimate expenses when they can get several useable designs (I won't say "good") on spec for only $Y00. Nevermind that many of those crowdsourced designs come from kids and people overseas using pirated versions of Photoshop.

Besides the economic damage wrought by 99Designs, a lot of designers have gotten a sour taste in their mouth from the blatant copying/IP theft that often wins. If I come up with great idea A, and you do a nearly identical rendering of my idea but in what happens to be the buyer's favorite color.... who do you think will be selected as winner? Unfair outcomes are pretty common, especially because the people doing the choosing often wouldn't know a good design if it bit them in the face.

axusgrad• 13 years ago

There was a similar website, where customers posted a bounty for the cheapest travel itinerary meeting certain conditions, and anyone could submit a plan meeting those conditions, to try and win the bounty. Does anyone remember what it was called?

todsul• 13 years ago

Hi axusgrad, that's us, Flightfox. Like Kaggle and 99Designs, we're also Australian.

In reference to a few comments here about the inefficiency of crowdsourcing, we're moving more of the work to the back-end (i.e. after an "expert" is guaranteed the bounty). Instead of awarding an expert on Flightfox, you now "hire" an expert based on an initial "pitch".

So, instead of receiving work from let's say 99 experts, we're aiming to provide great results from only 3 experts. To do this, we're working on segmenting and profiling both customers and experts. Think Uber/Amazon rather than eBay.

billpaetzke• 13 years ago

http://flightfox.com

achompas2• 13 years ago

The article is sorely mistaken: a job posting that says "Kaggle participation will give you an advantage" does not mean Kaggle provides "essential metric[s]." That's just one example of the article overstating Kaggle's case.

Also love the irony of explaining the importance of a data science competition host by citing Tom Friedman, the King Of Generalizing Anecdotes.

jack_trades• 13 years ago

The cynic in me says, "How Kaggle wastes the time of many talented individuals while enabling corporations to give the finger to their staff."

I should start providing prizes for the best start-up. I'll give you a $20k prize and then turn around and sell it for $1M or more. You can put it on your resume. Win win.

EDIT: Sure, it's good for various things, but it is so detached from the reality that it's a bit out in the thicket. The cynic in me just doesn't get over the value handed over by competitors to the sponsors.

dj_axl• 13 years ago

> The cynic in me just doesn't get over the value handed over by competitors to the sponsors

I would think you don't have to hand over your algorithm, if you forego the prize money. Another way of looking at it... as alluded to by other posters, the "winning" data model may not be the best, so there might be less value handed over than you think.

dbecker2• 13 years ago

I wish there was a similar site where participants collaborate rather than compete to solve interesting data problems.

jph00• 13 years ago

A lot of commenters here seem to have missed this: "The really disruptive thing about Kaggle, though, comes through the company's new service, Kaggle Connect. Here, Kaggle acts as a match-maker, where customers with a specific problem can hire a specific data scientist well-suited to their problem; candidates are drawn the top tier of Kaggle participants: the top 1/2 of 1 percent, or about 500 data scientists."

The competitions are a good way to learn, practice, and get feedback on your methods. Kaggle Connect is where you can make a good living while doing a range of interesting work.

(I work for, and compete on Kaggle).

danbmil99• 13 years ago

I find irony in the cognitive dissonance betwixt these comments, and these: https://news.ycombinator.com/item?id=5540841

jmount• 13 years ago

Kaggle is pretty important. But concentrating only on accuracy tuning (ignoring data collection, data curation, and interviewing stakeholders to find real business needs) in machine learning is like celebrating only (premature) optimization in software engineering.

But don't get me wrong, lots of top notch results in the contests. It is just that it is testing only one facet of what is needed in a data scientist.

michaelochurch• 13 years ago

I'm not anti-Kaggle, but this is not the future.

Credible people will do one or two competitions because they care about the problem, and because they want to establish themselves well enough to get better jobs, etc. If it works for them, great. If it doesn't, they'll get bored and quit. In that case, the best people leave and you have a ghetto.

Right now, Kaggle makes sense because "data science" is still an ill-defined field but a lot of people want to get into it, and no one knows what it means or takes to get in, so people will try things out to see what happens.

If Kaggle wants to stay in play for the long term, they'll need to get really good at connecting talented people with very high-quality jobs.

There is something that I don't think all of the hiring-related startups get yet: as things are, there's such a shortage of quality jobs. That's a 5-year existential threat to the whole business model. What happens when people realize that these sexy startup jobs are just corporate jobs with better marketing? What happens when the dream dies? Right now, high-quality jobs are too rare for the hiring startups (unless the genuinely change the economy) to prevent people from getting just as disillusioned with these new services as they are with headhunters. Now, that's not because there's an intrinsic limit on interesting work (see: Lump of Labor fallacy) but we'd have to overthrow the management of a whole industry to change that.

Now, data science. It is attractive right now because it carries with it a promise of what software engineering was supposed to deliver but, for most people, doesn't: interesting work, implicit respect, autonomy. I feel like data scientist in many companies means "software engineer who gets dibs on the most interesting projects". I'm afraid that title inflation into the data science field might dilute that, however.

What we really need is to fire 90+ percent of software managers and trust engineers to pick their own projects and call some shots. I don't know how to turn that into a specific startup idea, but it will solve a lot of problems.

juskrey• 13 years ago

Big numbers of data crunchers in a big play. I wonder how many top scientists are themselves spurious, therefore, random leaders, since there are 85 thousands of them. Is Kaggle really a science place or a gamble? You know, we have seen things like that in any prediction market, e.g. financial.

stratosvoukel• 13 years ago

I cant see how Kaggle is good for the data scientist (but I can certainly see how it is good for the hiring companies). It seems like the design spec work. I find it horendous that people do work for a company and most of them never get paid for it. It is disgusting. The top #1 in Kaggle according to the article only got paid 6 times. Lets put it into perspective. Companies that wanted web developers stopped hiring them and started using a similar service, in which, everyone would create a version of the application, with only the frontend available for other contestants, in order to foster competition. Then the company chose the best one and only the winning team would get paid. This is capitalism at its worst, and long-term it is severely harmful. In a modern work-rights aware society this should be illegal, spec work that is... At least the communities should critic it in my opinion and not embrace it.

How Kaggle Is Changing How We Work

Related Stories