Two Kinds of Judgement

by dfranke • 19 years ago

37 31 19 years ago

danielha• 19 years ago

This was part of Paul's Stanford talk. It fits in nicely in regards to decisions from venture capitalists on investment and even Y Combinator on the founders program.

dfranke• 19 years ago

Experiment: let's see if this gets a higher score here or on Reddit: http://reddit.com/info/1idib/details

mattculbreth• 19 years ago

Oops, voted up on each. Do -1 (at least) on your count.

Sam_Odio• 19 years ago

Of course it will get a higher score on reddit, just because huge volume of traffic going to reddit.

But the scores themselves mean nothing, on absolute terms. What really matters is the article's relative rankings - which (roughly) determine the article's placement on the main page.

And that's what makes News.YC so appealing. Even though the community's smaller, we don't have to wade through pages of higher-ranked flickr photos to find the quality content.

Alex3917• 19 years ago

I'd downvote this just to be ironic, but in his infinite foresight pg removed the option.

nostrademons2• 19 years ago

That must be one of the things he can do to influence the outcome. ;-)

brlewis• 19 years ago

This is a really well-written essay, which actually surprises me a little given how timely it is to the YC application process. When did you start writing it?

gyro_robo3• 19 years ago

To me, it seems muddled; I don't see the usual thanks to the usual suspects so I assume PG single-foundered it instead of running it past RTFM et al.

pg4• 19 years ago

If you can point out a specific muddle I can fix, let me know.

pg4• 19 years ago

Yesterday afternoon. I was in the middle of turning that last Stanford talk into an essay. This was one of the points. During the talk I said this was important enough to be a separate essay, so I wrote one.

jason133• 19 years ago

I disagree with the writers assumption that the committees judgements are only inaccurate for the borderline cases. I think there are plenty of examples of committees judgements being wrong about candidates, that should have been viewed as outstanding.

danw• 19 years ago

They're not judging the candidate, they're judging the application. The candidate might be outstanding but if the application does not convey this then the committee won't know.

With any application the you write take a step back, leave it for a few days and then look at it as if it was writen by someone else. You'll quickly see where your assumptions lie and what you've missed.

jason133• 19 years ago

Yes, committees can do no wrong!!

pg4• 19 years ago

Where did I say that? All I said was that you can make mistakes in borderline cases at little cost.

nostrademons2• 19 years ago

"There will be a few stars who clearly should make the team, and many players who clearly shouldn't. The only place your judgement makes a difference is in the borderline cases."

Jason13 is saying that there are plenty of cases where the "stars" often appear mediocre until they hit stardom, at which point it's too late to bid for their talent. It certainly makes a difference whether your team includes the stars or not, and since they often won't be obvious at the initial selection...

vlad3• 19 years ago

Great essay.

I disagree that you can or should only protest those judgements about you specifically, but I do agree with the entire second half of your essay.

I agree with the idea behind the essay, that "If it doesn't matter to someone else whether they choose you, then the other party has the leverage. You want to put yourself in positions where you have the leverage." For example, if you apply to a college, and thousands of applicants are just like you, then you don't have leverage.

Your examples just ignore the fact that people get rejected from about 4 colleges a year, while they have a chance to correct their tests and homeworks about 1,000 times a semester.

This means that the numbers simply don't match up. If you appealed 1,000 college decisions, you would have a great chance to get into a lot of them. If you appealed 1,000 questions on a homework to one busy and egotistical professor, they would likely start avoiding you after the 10th question.

Therefore, people SHOULD appeal everything they want to, whether the judgement is specifically about them or not.

I saw that this was written originally as a post to a question about YC decisions. As far as advice about YC, I think the best explanation of why some people were rejected is that there is a hidden bias (though very well earned) to hang around with people you think you'll find interesting or ideas you think you'd love or have a lot to contribute to over the summer.

Goladus2• 19 years ago

I disagree you should appeal everything you feel is unjust. Sometimes it's appropriate, but it is difficult not to react with emotion when there is a sense of injustice and not all of those feelings lead you in the best direction.

If your partner breaks up with you, for example, appealing to 1000 different committees is only going to make things 1000 times worse for you.

You've got to decide whether the reward is worth the effort. Sometimes the reward doesn't make sense if you had to fight to get it. (eg, if you're not picked for a team then bitch your way in, there might be resentment that brings the whole team down such that you'd have been better off not bothering)

vlad3• 19 years ago

I did NOT say you should try to appeal everything you feel is unjust. I said let the person decide. I'm saying there should not be a blanket statement about NEVER appealing certain decisions. People should appeal for anything they WANT to (obviously after using common sense, not everything) especially if it's NOT personal, like college admissions.

I also did NOT say you should appeal to 1,000 different places. It was an example to point out that Paul was comparing apples to oranges.

And, by appeal, I don't think either I or Paul meant just words, but whatever it would be (retaking the physical, sending videos to a coach of your performance, sending a college a picture of a national award you just received in March after you were just rejected, etc.)

shiro2• 19 years ago

PG is just pointing out a general fact, which is well known to those who are constantly being judged. It doesn't change whether PG wrote this as a response to the post, or as a general advice. If you doubt that, ask anybody in the profession where being judged is in its nature (I sometimes audition as an actor so I think I know that business a bit.)

Your mentioning about hidden bias is somewhat true, except that it's not really hidden, but just implicit---in most of the judgement there are always some implicit criteria the choosers have, and good players know that. Assuming otherwise is too naive.

About appealing: As far as you're not in the few of the obvious top group or in the obvious fall-behind groups in terms of the primary factors, any arbitrary secondary factors can put you in either side. Those factors are so random and it's waste of time to appeal---sure, you may have some chance of the judgements reversed, but it is much wiser to spend that time to make yourself excel in the primary factors, so that next time such randomness will affect less to the outcome, because if you like to live a life taking challenges, you will be constantly judged, and you won't have time to deal with those randomness.

vlad3• 19 years ago

Great post. The point of my post is that I don't think there are two specific categories of judgement, where in one you appeal, and in the other, you don't. Reversing a rejection to 1 or 2 or 3 extra colleges, since you already paid the application fee, could be worth it. That's just one example. I'm not disagreeing with Paul' overall message, I just think the examples are very bad. I think sending a letter of appeal could be worth it in a lot of cases if the trade-off or risk-reward is worth it.

gyro_robo3• 19 years ago

There seems to be some confusion here with "the usual distribution of ability".

"There will be a few stars who clearly should make the team, and many players who clearly shouldn't. "

Rather, there are a few who clearly should, a few who clearly shouldn't, and many in-between.

Moreover, this assumes there is a quantity called "ability" and that the evaluator is accurately measuring it. I note the later downplaying of accuracy, but before that, it says:

"Probably the difference between them will be less than the measurement error."

How do you arrive at an estimate of "measurement error" in order to claim that? Did you first measure the evaluator's "measurement ability"? With what?

Even if we naively assume that we can order a group by "ability" with regard to start-ups, and that the difference between adjacent slots is small, it does not follow that your evaluation comes anywhere close to the presumed omniscient set-ordering.

The data you have is your self-selected subset, and even there you find that choice #30 is actually in the top few, which is a rather large measurement error; you might argue this doesn't matter in aggregate, but it still argues against any notion of small measurement error. If you can be off by almost your entire selection size then who knows what you missed in the numbers up to twice the chosen cut-off?

The statement "it matters least to judge accurately in precisely the cases where judgement has the most effect" seems self-contradictory. What I would guess you mean is that if someone is clearly good, they make the cut, and you don't need to scrutinize to find out how good; but it still matters that you put them on the right side of the cut!

It matters the most to judge accurately where judgement has the most effect -- given the nature of start-ups, your missed borderline case could be the one that would have been larger than all the rest put together. Remember your perceived borderline is not the same as the actual borderline. See Bessemer's anti-portfolio: http://www.bvp.com/port/anti.asp

The article seems to focus on accuracy initially, then semi-mea-culpa its way out of it later.

Reading hundreds of applications, your eyes glaze over, and you may only notice someone when they happen to push one of your buttons by chance.

It's fine to say you think you selected an "optimal subset" of people according to your checklist, biases, or opinions; but don't confuse that with "ability". You have an idea of what you want and pick according to resemblance to it: small teams that remind you of Viaweb's structure.

Using the game theory definitions, I don't find it discouraging that you chose an "optimal" set, which is clearly different in this case from "maximal" ;)

Just don't confuse maximal individual ability with optimal set selection, which is what you appear to do with your example of selecting players for a national team.

Tanenbaum would've failed Torvalds due to his own bias, which is clearly separate from Linus's ability.

wordsthatendinq• 19 years ago

I don't know the consequences of posting in a 5-day old thread, but I just felt like practicing my side interest in econ. My only experience with game theory was a half-term graduate course and I don't follow everything you're saying, but my gut feeling is that these definitions aren't important to the spirit of the article.

Suppose you're buying stocks; you choose a portfolio of stocks to maximize your return. Of course you'll make mistakes and pick bad stocks, but as long as you have enough good stocks in your portfolio, you'll still turn a profit (and companies won't come and argue "why didn't you buy my stock?"). The difference between YC and the stock market is that being chosen by YC can potentially increase your value; they are more like the coaches that train athletes. So whatever portfolio they pick will actually end up with a higher value than if it were just left alone. Also, it makes sense to evaluate people by things not so one-dimensional as "ability" (which by gyro_robo's argument we should maximize), otherwise we ignore other possible synergies between coach and trainee, such as similar personalities, that can cause the trainee to train more productively, independent of ability.

The problem with the stock market analogy is that all you need to do is to beat the market, and to do that you just need to perform better than average. Actually YC could do that if all they wanted was to make some money. Perhaps similarly, a sports coach could be lax about picking athletes if the second-best team was way behind. This is where I see the game theory coming in. If there was another team, similar to YC, that was similar in performance, then YC would have more of an incentive to minimize the measurement error. I'm not saying they don't already, I'm just saying it would make sense to do so.

Anyway, I don't know if this makes any sense but the essay is fun and probably a good piece to show to some younger people.

whacked_new• 19 years ago

This curiously reminds of "You and Your Research" posted here yesterday: http://www.paulgraham.com/hamming.html

The Nobel Prize committee, for instance, has to pick a handful prizewinners out of hundreds of first-class scientists. The ones that don't get picked are still first-class, but if the "Nobel Prize Effect" holds true, the ones that do receive the prize are the ones who don't sustain first-class work.

Of course this works differently for a sports team, because the feedback of your efforts are more or less immediate (from game winnings). But for things like startups then, if we really digest both these articles, I'd say the Nobel Prize analogy holds better. The ones who experience failure, or a lack of success, and obstinately jab at their problems, will still make a big kill. And the rejects are always far more numerous. For every Nobelist, there are probably a thousand others doing comparable work.

I once met a scientist who was considered Nobel-worthy. He came off mostly as a grumpy old man though, because he kept lamenting how his classmate got one but he didn't. If I was half as intelligent as him I would be doing cutting edge research as a hobby! Perhaps the world is better now that he didn't get the prize.

febeling• 19 years ago

I have clearly always somehow felt that it was like this, but I never really realized that there were these two distinct types of "judgement," or perhaps better (but uglier) types "being judged". It is certainly an important (however small) idea, and one without capitalism and freedom in general must really be hard to deal with. Put it into the "Zen of Business" teachings.

ced2• 19 years ago

I think that the distinction between the two kinds is rather artificial, from the applicant's perspective. Both involve estimators of your ability, be it your ability to take an exam, or your ability to write an application. Both are more-or-less correlated with something important (being good in math; being a good potential entrepreneur). And both estimators are rewarded for being good. Teachers get a raise, investors get money. Why should one rejection be personal, and the other not?

In fact, the incentive to be accurate seems much larger for investors than for teachers. Thus, I would be much more bummed about being ignored by a smart investor, than about being given an F by a 9-to-5 teacher.

Two Kinds of Judgement

Related Stories