The reviewer’s fallacy: when critics aren’t critical enough

http://www.slate.com/articles/arts/books/2018/01/the_reviewer_s_fallacy_when_critics_aren_t_critical_enough.html

by wormold • 9 years ago

84 42 9 years ago

I think Ash Williams (Bruce Campbell) said it best in season 2 of Ash Vs. the Evil Dead, something along the lines of, "critics have shitty taste."

I blame mostly the mindset that a job like that can create. A critic/reviewer is often pressured to agree with pop culture in order to have their opinions validated. Movies and music are a good example of this (cough Star Wars cough). Actually look at most critics vs audience aggregations on Rotten Tomatoes or the fact that very few outfits even write music reviews anymore beyond Pitchfork.

There are very few outliers in this case and are often viewed as malicious, for example Jim Sterling.

I personally never read reviews before I ingest some entertainment I have interest in. At most I read the closing paragraph of a review which is likely to contain no spoilers and sums up the critics opinion. Scores are irrelevant to me as well, I have my own scale.

Mediterraneo10• 9 years ago

very few outfits even write music reviews anymore beyond Pitchfork.

There’s still enough music-review sites out there that it can sometimes feel like a chore to keep up with them if you have varied tastes covered by each of them. Besides Pitchfork, look at The Quietus, Consequence of Sound, Tiny Mix Tapes, AV Club, Classics Today, La Folia, etc.

ClassyJacket• 9 years ago

very few outfits even write music reviews anymore beyond Pitchfork.

The is probably a pretty natural result of it now being so easy to simply listen to music you're curious about. Reading a review of an album will take as long as just going to YouTube or Spotify and listening to the first song.

zelos• 9 years ago

On the flip side, for most consumers there are probably several million albums/television series/songs/films a few clicks away. Choosing is still hard.

kenoyer130• 9 years ago

Roger Ebert said it was due to a Film Critic seeing 2-3 movies a Day. If you are seeing that many movies, you start to yearn for something different and challanging. If you see at most 1-3 movies a month on a good month, you tend to be ok with "just" entertaining movies. Also critics like to see movies that move the artform forward in some way. So while Bright on Netflix is a very servicable by the numbers movie, it is not really doing anything outside the box.

yesenadam2• 9 years ago

"Theodore Sturgeon, who once observed, “It can be argued that 90% of film, literature, consumer goods, etc. is crap.” The “It can be argued” part usually isn’t quoted, and the figure is very ballpark."

Actually it sounds like the author wasn't critical enough. They believed wikipedia. Strange. But I remembered reading about this recently, that Sturgeon used the word "crud", not crap.

The rest is pasted from wikiquote:

"I repeat Sturgeon's Revelation, which was wrung out of me after twenty years of wearying defense of science fiction against attacks of people who used the worst examples of the field for ammunition, and whose conclusion was that ninety percent of it is crud.

The Revelation: Ninety percent of everything is crud.

Corollary 1: The existence of immense quantities of trash in science fiction is admitted and it is regrettable; but it is no more unnatural than the existence of trash anywhere.

Corollary 2: The best science fiction is as good as the best fiction in any field. - Venture Science Fiction (March 1958)

The original expression of this has often been declared to have been "Sure, ninety percent of science fiction is crud. That's because ninety percent of everything is crud." According to Philip Klass Sturgeon made the remark during a talk at New York University around 1951. It has also commonly appeared in variant forms such as "Ninety percent of everything is crap" and is often referred to as "Sturgeon's Law" — though he himself gave that title to another phrase:

Sturgeon's Law originally was "Nothing is always absolutely so." The other thing was known as "Sturgeon's Revelation".

yesenadam2• 9 years ago

Wish I'd noticed before that about 90% of the sentences I quoted at the top are crap! They got the name right and the number.. apart from that, nothing.

j9461701• 9 years ago

An interesting article, most certainly.

I suppose the immediate response is the critic sphere is heavily self selected for people who really like books / movies / comics / etc. So what to most normal people would seem like flowery platitudes and a cultural aversion to tough talk might actually be a completely genuine love of the medium itself. If you just love to read, it's not vapid pleasantness to call every book fun and a joy - it's just how you genuinely feel.

I hate to sound like a philistine, but audience-critic discrepancies often occur when a work is less than pleasant to sit through, whether because of The Sorrow and the Pity–like length (a growing problem, pun intended) or grim subject matter. Take last year’s Best Picture winner, Moonlight, which has a 98 critics’ and 79 audience score on Rotten Tomatoes, and which I haven’t seen. The Rotten Tomatoes blurb calls it “The tender, heartbreaking story of a young man’s struggle to find himself, told across three defining chapters in his life as he experiences the ecstasy, pain, and beauty of falling in love, while grappling with his own sexuality.” I can get that at home.

I think this is another element of the critic sphere being very self selected. For whatever reason it selects for literary realist types who honestly care more about depressing stories featuring puppies with broken legs than any kind of sci-fi, fantasy, etc. genre film that's merely fun and not "meaningful". The critics aren't behaving fallaciously or malevolently, they as a group simply have genuinely different preferences for entertainment.

Here’s the heart of the problem: The set of critics’ and audiences’ interests do not perfectly overlap but rather form a Venn diagram.

Ultimately, this is the real problem whatever is causing the divergence in interests. It makes it hard to use critics, and harder to identify the genuine diamonds in the rough for you personally.

My personal solution is simply look to the average person directly. The internet is full of forums, and many of them discuss whatever medium you happen to care about, and those forums are invariably going to have plenty of threads along the lines of "What is your favorite lost gem?", "What movie do you think flew under the radar?" "What deserved more credit than it got?".

Fnoord• 9 years ago

Ultimately, this is the real problem whatever is causing the divergence in interests. It makes it hard to use critics, and harder to identify the genuine diamonds in the rough for you personally.
My personal solution is simply look to the average person directly.

IMDB is basically a layman's scale vote (from 1 to 10) while mentioning Metacritic's average critic review, whereas Metacritic contains both critic review and user review. You can pick either on Metacritic, but if you never care about critic review then perhaps IMDB is a better solution. I tend to use both since IMDB has more titles. Especially old, obscure, and international titles.

ebbv• 9 years ago

A simpler and obvious explanation for critical complacency; 90% of critics are also crap/crud.

stochastic_monk• 9 years ago

Or at least, the vast majority of published reviews are.

Recently, I was reading a review of Samuel Beckett [Nobel Prize in Literature 1969], from the NYTimes from 1958 [0], and at first I was shocked by how much higher the quality was than what one might read in the Times today.

Turns out that the author was a Stephen Spender, CBE, [1], a poet who was knighted for his contributions. It's not when or where something is written so much as by whom. You are exactly right that it's determined by who the critic is.

[0] http://www.nytimes.com/books/97/08/03/reviews/beckett-unnama...

[1] https://en.wikipedia.org/wiki/Stephen_Spender

pessimizer• 9 years ago

It might be simpler to say that the interaction between the reviewer's motivation to believe that 50% of everything is crap and the fact that 90% of everything is crap has emergent effects.

The major one is that bad qualities that randomly happened to occur more commonly in products that have previously made the 50% cut become misindicators of quality, distorting the evaluation of a constantly drifting subset of products that have or lack those misindicators. This occurs asymetrically (between misindicators of positive quality and misindicators of negative quality) because there are more products misidentified as being good, therefore it's relatively far more common for particular bad qualities to randomly be more associated with works above the median then the opposite. Works that fall below the median in a world of 90% crap would have very few good qualities at all.

duxup2• 9 years ago

Roger Ebert used to call things "a great entertainment". The author of the article uses that term too. When Roger said it I always understood it to mean that there was nothing special about the film.... but it was entertaining.

It felt like he was identifying a separate category of films. It's not really great among film, but it gets its own job done "entertainment", and in that way it is plenty good. It leaves room for praise for what it is, but also space for the few really "great" things.

lobster_johnson• 9 years ago

Ebert's reviewing philosophy was to review a film relative to its ambitions, genre and target demographic. If the film accomplished what it apparently set out to do, Ebert viewed it favourably. Many of his highest scores were for pure, disposable entertainment, as opposed to high-brow arthouse film such as Bergman or Antonioni.

SideburnsOfDoom• 9 years ago

review a film relative to its ambitions, genre and target demographic

So long as those assumptions are also stated in the review, that seems a good idea.

duxup2• 9 years ago

I felt like Roger s till held the disposable stuff to some level of ... reasonableness ... but yeah his system was relative to the film.

I liked it as you didn't have to slam some fun, but dumb, action flick to praise anther film, and yet you knew from Roger what you were in for.

jonny_eh• 9 years ago

The nice thing about movies, as opposed to video games and books, is that they don't take much time. A couple hours, and $20 or so, is often a fair trade in exchange for a dumb story filled with special effects and an excuse to go out.

hitekker• 9 years ago

Barely related: Rotten Tomatoes summaries feel like they've been paid for.

Sure, if a movie is rotten, they'll trash it. But when a movie is barely fresh, they seem to accord it more praise than the critics gave themselves. I don't have a particular instance in mind so I'd be curious if anyone else has noticed this.

cf498• 9 years ago

I dont think its that straight forward, but there have been instances lately where they behaved rather calculated

http://www.chicagotribune.com/entertainment/movies/ct-rotten...

indubitable• 9 years ago

Interesting point the article indirectly hits on:

- "Chummy logrollers—a perception heightened in the social media age. In a 2012 Slate piece called “Against Enthusiasm,” Jacob Silverman wrote, “if you spend time in the literary Twitter- or blogospheres, you’ll be positively besieged by amiability, by a relentless enthusiasm that might have you believing that all new books are wonderful and that every writer is every other writer’s biggest fan.”"

I've always assumed studios had simply managed to, through some means or another, 'corrupt' most reviewers in a process similar that happened to video games. I wonder how much of the problem might be that people are now just too close to the things, and in particular the people behind the things, that they're reviewing? It's far easier to be "critical" (read: truthful) about things and people you have no connection to. But in today's world of social media, reviewers and the creators of the products they're reviewing are going to be in near direct, if not indeed direct, communication. Knowing full well that speaking honestly about a product could hurt people you've formed relationships with is not something easy to do.

For megafilms like "The Last Jedi" (currently at 90% on rotten tomatoes contrasted against 49% for viewers), how many of the reviewers have at least one direct relationship to somebody involved with the project? It doesn't need to be corruption. Plain old reluctance to be honest to those close to us or do them soft favors with no expectation of direct personal benefit -- that basic aspect of human nature would go a long ways towards explaining the deterioration of honesty in reviews. Or taking it to another level, we often wear rose colored glasses towards those close to us. This entire process might be entirely subconscious.

iainmerrick3• 9 years ago

megafilms like "The Last Jedi" (currently at 90% on rotten tomatoes contrasted against 49% for viewers)

Be careful, The Last Jedi is definitely part of the "culture wars" effect somebody mentioned elsewhere. I don't think it's a very useful example for the effect this article is talking about (although it's very useful if you're interested in the culture wars).

If you actually read the Rotten Tomatoes reviews, many of the "viewers" are people complaining about gender representation. It's not a representative sample, it's just been flooded by "men's rights" activists.

Personally, I think the 90% is too high (I didn't like it very much) but the 49% is way low (most people I know liked it just fine, no quibbling).

tdb7893• 9 years ago

I looked at the reviews for The Last Jedi and I'm not seeing a ton of men's rights reviews (but they might be there just buried lower down). In my experience most people either fall into the the categories of thinking the movie was "fine" or "awful" and I think that because people with strong opinions review much more frequently it pulls the ratings down a lot.

Goladus2• 9 years ago

Be careful, The Last Jedi is definitely part of the "culture wars" effect somebody mentioned elsewhere.

I don't buy it. The Force Awakens is at 88% and culture warriors fought over that one, too. Both films have severe problems from a storytelling perspective, but the problems with The Last Jedi are much deeper, are of the sort more likely to turn off fans of the franchise as well as fans of The Force Awakens. There are plenty of culture-war-neutral analyses of The Last Jedi that absolutely eviscerate the film. For just one example: https://www.youtube.com/watch?v=vw7pcCj0ORk

While I can't speak to the 49% rating, where I'm seeing the major divide is between Star Wars fans who care about world building, character development, and a coherent story (historically important aspects of Star Wars); and fans who are content just to enjoy the emotional ride from one scene to the next while enjoying the nostalgia.

As from the culture wars perspective, it seems hard to deny that The Last Jedi takes an unnecessarily overt stance on the culture war, daring that sort of criticism. TLJ includes: a postmodern deconstruction of Luke Skywalker and the Jedi mythos, a purple-haired female admiral dominating the only talented white male on their side for displays of what they probably imagine to be toxic masculinity, and all the bad guys are white males with overt Nazi references. I mean, seriously, why can't they just make it Star Wars?

Take another Disney movie as a contrast: Moana. Moana is a hero's journey, just like the original Star Wars. Moana is about a girl character, her mentor is her grandmother, the story is set in the South Pacific and features non-European cultures and sources their mythology for the story. The twist at the end might be considered feminist. But nobody but serious cranks are complaining about it, because none of it gets in the way of the story and for the most part actually enhances it. Moana isn't made some kind of wunderkind. There's no artificially forced interaction between an SJW stand-in and a white male stand-in. There's no postmodern deconstruction of the myths or themes. Disney is just doing their usual thing of mining ancient myths and fairy tales and adapting them for entertaining mass consumption by modern audiences.

TheOtherHobbes• 9 years ago

The critics may genuinely have had a different view.

I loved TLJ. IMO it was a clever, challenging film - much more so than the usual generic superhero movie.

It had some flaws, but I didn't find it disappointing. But I can completely understand why it might infuriate and disappoint long-time fans.

Which view is correct?

The problem for critics and for creators is that there's no such thing as "the audience.' There are overlapping demographics with different expectations and different abilities to handle challenge and complexity.

If you make something too clever or challenging it slides off the bell curve and "the audience" - the median consumer - hates it. If you give the median consumer what they want and never challenge them, you get bubble gum and potboilers.

I used to know someone who made a nice living writing romantic fiction. It sold very well, because he was clever enough to hit the right tropes and write very consistently to the correct formulas - none of which is easy.

His fiction was well reviewed, but he hated it so much he had to give it up because he couldn't bear to do it any more.

Bottom line is there's a difference between product and art. Product is purely about sales to the middle of the bell curve. Art is about being original, challenging, and creative, and maybe even about having some fun while making it.

If product gets better consumer reviews - and it reliably does - does that make it "better", or not?

crondog3• 9 years ago

Is there a service like this: rate a bunch of media, then filter for critics that largely share your preferences, then subscribe to recommendations from just those critics? If there isn't, there totally should be.

jacobush3• 9 years ago

There is but it's from a limited selection of media and you can't know the identities of the reviewers and the set of reviewers changes all the time, as well as the set of available media. (I am talking of course, about Netflix.)

sgt101• 9 years ago

Also by consuming media preferences could/ should be changed, also new friends, change of life and political context... I don't think that jamming people in a bubble is a good plan

Vaskivo• 9 years ago

But... then aren't you closing yourself into your own filter bubble?

(It highly depends on your purpose. Do you want more movies/books like the ones you like? Or do you want to read great critique?)

Torgo2• 9 years ago

Finding a critic you like helps you cut through the chaff. There are more movies released per year than I care to see, so I rely on a trusted agent to narrow it down for me.

Bubbling is usually mentioned in the context of news, I think it's more important to avoid it there than in entertainment.

partycoder• 9 years ago

From the article:

Groupthink is a psychological phenomenon that occurs within a group of people in which the desire for harmony or conformity in the group results in an irrational or dysfunctional decision-making outcome. Group members try to minimize conflict and reach a consensus decision without critical evaluation of alternative viewpoints by actively suppressing dissenting viewpoints, and by isolating themselves from outside influences.

HarrietJones• 9 years ago

If there's more than 20 points between critic score and viewer score (in the viewers favor) on Rotten tomatoes, then the movie is usually good. There are exceptions to the rule, and recent inclusions in the culture wars (Justice League, Ghostbusters) can break this, but generally it's a good rule of thumb to follow. Reviewer scores come from people forced to watch hundreds of films, and as a consequence they reward poor quality idosyncrocy over good quality but predictable tropes. People crave variety, and when you review movies for a living, you lose touch of what variety is for normal people.

BosunoB• 9 years ago

This isn't a completely sufficient argument, though. Surely, any person into an artform will at some point be able to recognize the difference between what is good because it is good and what is good because it is different. A good critic would be able to differentiate these two things and give a review with some nuance, but it seems incomplete to say that the critic problem is mostly the fault of seeing way too much junk. Anybody who is actually in to anything has seen a ton of junk, but not everybody has lost their ability to clearly judge things.

pedz88• 9 years ago

You have touched on one of the main reasons we decided to launch www.bad-about.com. In a world full of positive hype from reviewers and sales spin from the companies themselves, we thought it would be helpful to collect and display JUST the criticisms to help balance it all out.

The reviewer’s fallacy: when critics aren’t critical enough

Related Stories