Adventures in Cognitive Biases

http://web.mit.edu/~xiac/www/advbiases/

by srijan4 • 12 years ago

48 49 12 years ago

web.mit.edu

Mithaldu2• 12 years ago

I don't know where this is going, but so far it's cute.

There's one major gripe i have though, the UI doesn't let me accurately express my beliefs.

On the question "How many dollars does the average American spend at vending machines during a year?" i thought it might be somewhere around 100, between 10 and 1000. However i could only get it as far as 38 - 162, which didn't allow me to cover the right answer. Optimally though i'd like to be able to answer "i haven't got a friggin' clue".

On the other hand, i guess it demonstrates a massive cognitive bias on the side of the makers of this: Not everyone is confident in knowing the answer to something.

Difficulties on other questions:

  - cat sleep hours in a day: allowed me to set ranges like 0-48
  - degradation time of a plastic bag: did not allow me to set 200-5000

dthunt8• 12 years ago

Yeah, this is a fair criticism. There are explicit bounds encoded in each problem and when those conflict with your estimates, you will be frustrated.

Haven't got a frigging clue isn't a fair thing to say though. The whole point of the exercise is that most people have a significant misalignment going on between their modelling of reality and the track record of their modelling ability.

Exercises like figuring out how much milk a cow produces in a week or how fast a horse runs require you to consider what information you DO have suggests about upper and lower bounds and things like that, and then correctly realizing the confidence of your models is the other half of the equation.

You are pretty sure, I would imagine, that a cow doesn't produce 10,000 gallons of milk in a week. You are pretty sure that a 'heavy' user of a vending machine might use three a day and buy a bunch of snacks and a drink or two, and that 'heavy' users maximally represent Y% of the population, with high confidence.

You can use these intuitions to guide your estimates, and you do in fact do so on a regular basis. The exercise is calibrating those estimates against their empirical accuracy.

Mithaldu2• 12 years ago

The point here is: The website asked me about my beliefs. It did NOT ask me to work out the correct solution. And if i believe i cannot possibly give a useful estimate then that should be a viable answer i can give.

Your example is especially relevant since i don't even know what a gallon is. I guess it's maybe a liter or something? Who knows. Either way: Without actual research there is no more accurate modelling of my belief than "i don't know", period.

tempestn• 12 years ago

Similarly, some of the answers were unnecessarily vague. For instance, I actually happened to read just yesterday a blurb about racing horse speed, and I knew that while racehorses can reach around 30mph, the very fastest horses can apparently reach sprinting speeds (briefly) of around 50mph. So which answer does it want? (This was especially frustrating since for most of the other answers I had no friggin' clue...)

lurkinggrue2• 12 years ago

Yeah, many of the questions I would have just said "Not sure" if I was asked and then I would google the answer.

AdrianRossouw• 12 years ago

I think it's cute, but the overconfidence bias bit in the beginning is frustratingly geo-centric.

It's all general knowledge questions using imperial measurements, dollar spending power and american supermarket stats.

This needs a shrug button.

GhotiFish• 12 years ago

Yah, I feel like I'm being accused of being overconfident when my answer is "I don't know"

lurkinggrue2• 12 years ago

Exactly, I would have pressed the shrug button quite a bit here.

Oddly enough I did get the cat one spot on.

bluecalm• 12 years ago

It has potential but they need to fix two things:

-make standard units available: liters, kilograms, meters;

-allow users to express full range of my beliefs; I have no idea how many reincarnations of Dalai Lama there was I guessed 1000 and wanted to set a straight line from 1 to infinity (or something going down to 0 slowly). It didn't allow me to; it's not overconfidence from my side, it's a bug on your side :)

judk2• 12 years ago

Do you really have no idea?

Do you believe in a likely lower bound for the time gap between reincarnations? And an upper bound for when the first one was? Or do you believe that both of these could be arbitrarily small (large, resp.)?

ISeemToBeAVerb• 12 years ago

I'm not sure who this game is aimed at, but if you're trying to teach cognitive biases to people with no training in psychology you really need to add a tutorial to the beginning. I would have loved to go further with the game, but I became frustrated at the very first question when I was confronted with this:

"Drag down the circle so that your belief covers all answers that you think are reasonable. This diagram is your belief graph."

I have no idea what a "belief graph" is, or how to use one. I though I had answered the question by entering a number into the field, but then I'm presented with this strange graph.

Perhaps I'm in the minority around here when it comes to my knowledge of this domain, but if this is a game intended for laypeople, you need to be really clear and assume I know nothing at all about your methods and practices. Explain to me what a belief graph is and how to use it. Either have a pre-game tutorial, or use tool tips.

That's my 2 cents.

gravity135• 12 years ago

It becomes clear once you progress through a few of the examples what this means.

You don't need tutorials and tool-tips, you just need to be able to learn by doing.*

*and users who don't shy away on the first hint of confusion.

dthunt8• 12 years ago

Yes.

The problems absolutely illustrate the concept if you have a reliable overconfidence confidence bias. You do not need a background in psychology to understand "Hey, I feel 90% confidence, but am being surprised nearly 1 in 3 times. There's some kind of miscalibration going on there."

samatman• 12 years ago

The math here is problematic. I happened to know how many 'reincarnations' of the Dalai Lama there have been (hint: he's routinely referred to as the [redacted] Dalai Lama, much like Louis the Sun King gets a number). This gave me ten thousand points. Other guesses give me fractions of a point.

What am I supposed to learn from that?

edit: checking again, ten thousand, not a thousand. corrected.

scott_s2• 12 years ago

They asked how many "reincarnations" there have been of the Dalai Lama. I think, and maintain, that the answer is 0, as I don't think "reincarnation" is possible. Had they instead said, "How many people have there been that have been called 'The Dalai Lama'", I would have answered differently.

Since this is a game about cognitive biases, hosted at mit.edu, I maintained hope that they, too, would be this pedantic about semantics, but, sadly, no.

tzs• 12 years ago

If they had asked for an estimate of how many people died when the first Death Star was destroyed, would you answer be 0 because there was no actual Death Star?

scott_s2• 12 years ago

No, because such questions, to me, implicitly acknowledge we are talking about a fictional world. The question about the Dalai Lama, however, is a question about this world.

Another example that I would say 0 to is, how many times did Jesus Christ come back to life? But I would say 1 to, according to the Bible, how many times did Jesus Christ come back to life? The difference between Star Wars and the bible is that there are not large groups of people who insist that Star Wars is real, so I feel safe in assuming an implicit, "According to this fictional thing". I consider the "reincarnations of the Dalai Lama" to fall under the bible category.

PeterGriffin22• 12 years ago

I answered 0 gallons of milk, because I don't believe in cows.

Kidding aside, I get your frustration. Trivia questions by necessity have the format of "one short question" = "one short specific answer".

Real facts usually don't work this way. Historians argue or change the date of various historical events over time as more data emerges. The Solar System "lost" a planet few years ago, and so on. "Facts" are in constant flux.

There's also a complicated, detailed context, there are a lot of "depends" in a real answer. Say, "how fast a horse runs"... domesticated & trained or wild, what breed, on a road or on a meadow, with horseshoes or without, etc.

So when faced with trivia questions the last thing I could say is I feel "overconfident" answering. Especially when the answer is a number. The designated correct answer is just an arbitrary data point that the author stumbled upon earlier in a book or online and wrote down as the answer.

It's apocrypha masquerading as facts.

pdonis• 12 years ago

Since this is a game about cognitive biases, hosted at mit.edu, I maintained hope that they, too, would be this pedantic about semantics, but, sadly, no.

It's also very disappointing to have the site claim that there is one unique "right answer" to questions that are ambiguous. For example, the site claims that 30 mph is the "right answer" to "how fast can a race horse run", but Googling shows plenty of examples of race horses running 40 mph or faster. I think they meant something more like "what is the average speed of race horses when clocked during races", but that's not what the question said.

nemo• 12 years ago

Yeah, they worded the question poorly.

I answered 13 since the current Dalai Lama is the 14th and the first wasn't reborn. Also, to get technical he wasn't reincarnated, but reborn, which is an actual distinction Buddhists make.

MaysonL• 12 years ago

US cows produce twice as much milk as the average, I guess. And their assumption that everything is to be normally distributed is weird, and a distinct cognitive bias in and of itself.

nazgulnarsil• 12 years ago

yes, at the very least you should first move the circle vertically to define the bounds, then move the circle horizontally to create the shape of the distribution.

PeterGriffin22• 12 years ago

I think the interaction is very interesting and engaging, lots of potential there.

But the only thing I learned is to look up answers to trivia questions on Google, and to drag chart bars to match the text above.

Overconfidence isn't the problem people have when obviously faced with questions they don't know the answer to (I've never milked a cow, and I don't intend to), and the inferences the protagonist was making from his trip were very suspicious and arbitrary, so that was disengaging in a tale about using proper logic.

sitkack2• 12 years ago

Dude, you cheated.

dthunt8• 12 years ago

Not only cheated, but also destroyed an opportunity to learn something valuable.

GP can, however, take other calibration tests elsewhere.

Here is one: http://calibratedprobabilityassessment.org/

Don't cheat this time. There is a valuable lesson here.

clairity2• 12 years ago

while i honestly answered all of the questions in the story, i have to admit i did a view source on the codeword because i didn't have the patience to work out the problems in the google doc before seeing how the story ends (disappointingly, i might add).

dthunt8• 12 years ago

They're hypothetical inferences, but the downside to using actual realistic examples is that even when they're meticulously researched, people nitpick them to a degree that baffles me. Like, asserting "but the probability of the evidence is either 0 or 1, so it's clearly impossible to ever make any inferential reasoning work".

The examples here are well chosen to eliminate the usual attacks on inference. They're ODD. But they're not invalid.

wodenokoto• 12 years ago

I thought the test at the end was way too difficult given the simple introduction to calculating probabilities.

MacsHeadroom• 12 years ago

Yes, I came here to say the same thing.

dthunt8• 12 years ago

I am going to challenge that belief by asking you to do the following:

Try solving the blue and red pots problem, given only a single observation.

Now try again, this time with two observations.

alanfalcon• 12 years ago

I felt like the test at the end was more of a test of the adventure itself: if many people are off target, there's probably more that should be taught.

spokenn• 12 years ago

What do you think about Daniel Kahneman's 'Thinking Fast and Slow'?

jsweojtj• 12 years ago

as an astrophysicist, this one annoyed me: https://www.dropbox.com/s/udoug25cpd2zpiy/Screenshot%202014-...

gravity135• 12 years ago

Haha. I did the same exact thing.

kriro• 12 years ago

It was fun but the end was too much of a cliffhanger for my taste.

There should be a choice of metric/imperial and some of the questions were a little meh. I liked the Bayesian part.

nmac• 12 years ago

I think this is a very interesting game with an important message. However, the UX could be more engaging by having a more vertical leveling up structure. I felt as if I wanted to get to the next level but the onslaught of questions seemingly never ended- after a bit this becomes disengaging, specially when you aren't shown how many more units are needed to graduate.

dthunt8• 12 years ago

There is no graduation ceremony.

gravity135• 12 years ago

In case you're curious where it ends, you first go through the belief charts. Then you start working with a little probability calculator. It ends with an embedded google form with open-answer fields for responding to harder text questions.

gravity135• 12 years ago

I'd be interested to see the results of the first part of this. I've heard of those studies (perhaps they were just stories?) where groups of people collectively estimated things like number of jelly beans in a jar and were actually pretty accurate on average.

Obviously it's not going to be like that, because some of these things people can actually know - but I'm still curious.

dthunt8• 12 years ago

If you can find that, provide a link. I strongly doubt people are particularly good (without applying some sort of rigor) at estimating beans in jars.

geekingfrog• 12 years ago

This effect comes from the effect of regression to the mean. Some people will vastly overestimate, while some people will vatsly underestimate, thus cancelling out.

It holds as long as the people used are very diverse (not under the same cognitive bias) http://www.socialresearchmethods.net/kb/regrmean.php

clairity2• 12 years ago

the fascinating thing about those studies is the implication that many kinds of human knowledge are (normally?) distributed, and you can ask questions that's hard for an individual to get right but easy for a group (crowdsourcing). somewhat akin to how one brain cell doesn't contain knowledge, but a (large) collection of them do. it's like there's built-in fault tolerance. =)

rahimnathwani• 12 years ago

The overconfidence/underconfidence issue is similar to that expressed in the book 'How to measure anything'. The author claims to have trained people to become better estimators, by presenting them estimation challenges, and providing feedback about how often the actual value was within their stated range. IIRC people were asked to present a range for which they were 80% confident. If the answer was within the range too often (e.g. 90%) then their ranges were too wide, and if not often enough, they were overconfident. Over time, most subjects improved their calibration.

I started (yesterday) working on a simple web app to train people in this way. It's not yet ready to try out, but you can bookmark it here: https://github.com/rahimnathwani/measure-anything

judk2• 12 years ago

Arbitrary weird GUI control that doesn't work on mobile and doesn't have an accessible fallback mode, even though the information content can be easily expressed with short simple text: yep.

aristus• 12 years ago

The green cab question was changed from the original in a way that (I think) changes the logic.

http://heatherlench.com/wp-content/uploads/2008/07/bar-hille...

In the original, witnesses are only 80% accurate reporting color. In this version it's altered so that it's simply asserted that 80% of cabs in the neighborhood of the accident are green.

With that revision it seems to me that you would not update your priors but actually replace them. "The mix of cabs is 85/15. Oh wait, it's actually 20/80 the other way."

Edit: Ah, never mind. The paper gives both versions, specifically because the in second version you are supposed to throw away the prior.

Adventures in Cognitive Biases

Related Stories