“Default Position Should Be Skepticism” and other advice for data journalists

https://www.propublica.org/nerds/hadley-wickham-your-default-position-should-be-skepticism-and-other-advice-for-data-journalists

by danso • 7 years ago

182 28 7 years ago

propublica.org

ndiscussion• 7 years ago

Oddly funny to me that there are data journalists out there not following the headline advice. The average person really is doomed.

yters• 7 years ago

Should I be skeptical about the default position?

Retric• 7 years ago

Yes, in that skepticism should not be taken to an extreme.

The problem with skepticism is it makes it harder to change your own opinions when they are incorrect. The goal is not to reinforce your own opinions, that’s going to happen either way. The goal is consider information that you would like to be true can also be false.

danso2• 7 years ago

Wickham doesn’t say to be “skeptical about the default position”, which implies being skeptical of the status quo. He seems to be saying be skeptical of what the data purports to be, until you understand its origin and collection process and etc.:

But until you’ve understood the process by which the data has been collected and gathered ... I think you should be very skeptical. Your default position should be skepticism.

This seems reasonable to me. Data is inherently a simplification of the reality it purports to observe and store. Not adequately researching and challenging the data is taking too much of a leap and faith.

Pinegulf• 7 years ago

Yes. If you figure out better position then it should become the new default. An interesting question would be if the default position depended on environmental factors.

3JPLW• 7 years ago

Same is true for any sort of science. We're trying to find some sort of a story, and thus we bias ourselves towards bugs that lead to a meaningful conclusion. Most experiments fail, but a bug can make them look like a success.

I find it especially important if you're applying supervised machine learning techniques. Far and away, the easiest sort of bug to make is mixing up training and testing and holdout data (or not doing it at all). It's such an easy bug to make because you're trying to improve the model accuracy... and a really easy way to do that is to train and test on the same data.

StanislavPetrov• 7 years ago

The "default position" should be skepticism for everyone, all the time - not just data journalists.

SketchySeaBeast2• 7 years ago

Easy to say, hard to practice. I often don't verify that the bag of rice I bought with a digital monetary transaction is rice until I use it for the first time, and I don't even check it for biological agents at that time!

rlucas• 7 years ago

Life Improvement Protocol:

Move as much as you can of your decision space into "default trust" environments.

Save the "default skepticism" environments for areas where your counterparties are particularly sketchy and/or the parts of your job that make you a professional risk-sniffer-outer (journalist, investor, cop, etc.).

Default skepticism is a hell of a bad way to go through life buying rice and greeting your coworkers in the morning.

BackBackBack3• 7 years ago

Default skepticism doesn't mean verify everything every time. It means that when confronted with a new claim, a claim you've not decided yet its validity, you should abstain from taking a position until you have enough evidence to accept or refute the claim.

In the case of your example, I gather you've bought rice before, you've probably bought at that store before, you've used money before and you've eaten food before. By the looks of it, you've probably cooked before with that same kind of rice. Hence you are not evaluating the rice and all its production line from scratch every time you buy the rice. You already evaluated that claim and you've found that the store and their products are acceptable to you and that is why you probably bought the rice.

In that case you abandoned long ago the default position because you already had the required evidence to take an action. Which is the only purpose of the default position.

The default position is not the forever position.

maxerickson• 7 years ago

Your sensory inputs provide a hell of lot of information about some rice.

I'd also argue that experience, in many locales, recommends being skeptical that there's anything wrong with food sold in a store.

Cypher• 7 years ago

Skepticism requires an unlimited amount of work for verifying a chain of evidence, if one piece is wrong then the whole body of data could be unless. Not easy for a data journalist to standup and get a new set of data when there's a time constraint.

Even an average person isn't even going to spend the morning verifying a weather report demanding more data for an accurate prediction. Instead they'll elect individuals which they trust. And it's unfortunate that they're either wrong, corrupted or exploited.

Even today I can't even watch my old favorite youtubers without being pitched some VPN, Web hosting or audiobook service. I feel the people I trust for entertainment have been compromised to say and act a certain way to make some corporate CEO happy who is manipulate numbers so some data journalist can come to some incorrect conclusion about the world and award him some approval rating.

danso2• 7 years ago

A lot of data journalism isn’t necessarily done on deadline, and when it involves public data (e.g. government records), oftentimes the resulting story is about how the data is wrong or insufficient.

munchbunny• 7 years ago

Skepticism requires an unlimited amount of work for verifying a chain of evidence, if one piece is wrong then the whole body of data could be unless. Not easy for a data journalist to standup and get a new set of data when there's a time constraint.

In theory, perhaps, but not in practice. In practice, the bar is usually at two standards:

1. at least two uncorrelated/independent sources of truth say the same thing (replication)

2. peer review turns up no clear flaws in methodology

Then your conclusions are good enough for most people. You will not be able to completely rule out the possibility of an incomplete understanding or a theory that is mostly right and a little wrong, but the vast majority of untrustworthy science in the news fails at basic diligence, not at "it looks right to the best of our knowledge but turned out wrong."

super_k_ro• 7 years ago

"the vast majority of untrustworthy science in the news fails at basic diligence"

I totally agree. I think it's due to our times, because many people just replace religious certainties (which seem to make a come-back this days btw) by scientific certitudes. Yet it's not how science works...

"it looks right to the best of our knowledge but turned out wrong."

It's the underlying of scientific process and it needs to be repeated until integrated :) With that in mind, you can hold opinions (because it's useful in life ^^) while managing to change your own mind if a better explanation appears (even if no one is immune to confirmation bias but we can try, can we ?)

BackBackBack3• 7 years ago

Skepticism by definition requires no amount of work. What you are describing is the action of taking a position, or wanting to take a position be it for or against.

Not easy for a data journalist to standup and get a new set of data when there's a time constraint.

Doing that is actively trying not to be a skeptic. A skeptic would wait for the party making the claim to present the set of data validating the position.

Now, I do not now if the default position in the media should be skepticism, but your misrepresentation of it makes me wonder if you are confusing what being a skeptic means and maybe that is why you find it so troublesome?

stillbourne4• 7 years ago

Skepticism is the application of the null hypothesis. Building from nothing other than observables we can establish a system of order where in testable statements can be made to assert logical arguments to establish a set a first order principles via a criteria on true or false evaluations. This is the foundation of science and mathematics. To state:

Skepticism requires an unlimited amount of work for verifying a chain of evidence

Is an equivocation that any statement that is attempted via conjecture and criticism is impossible. However, you are using a machine built entirely on those principles to communicate the idea that logical foundations are impossible. Congrats. You have won today's irony top score.

stillbourne4• 7 years ago

The default position should be skepticism for everyone. Including hacker news. I know I get constantly downvoted for skeptical view points here in the comments, I wish it were not so but quite a few people here engage in bullshit. Most commonly seen types of bullshit here is, ancap philosophy, cryptocurrency advocacy, diet fads, and chemophobia.

smt882• 7 years ago

Your impression, that HN is skeptic-hostile, is the opposite of my experience.

I browsed some of your comment history for evidence supporting your position.

I saw that your downvoted comments fit into buckets I often see downvoted (what I observe, not what I do myself):

- excessive profanity

- making a factual mistake

- saying something sarcastic

In the case of sarcasm, HN seems to dislike it either because the comment is taken sincerely or because it "doesn't add" to the conversation.

I didn't see any evidence that you were downvoted for skepticism.

stillbourne4• 7 years ago

I use profanity, I don't see it as excessive. "Bullshit" is probably my most often used obscenity but in the world of skepticism its used as an interjection when someone is dead wrong in the same way a religious person would blurt out "blasphemy" whenever someone takes the lords name in vein or such. Also, I do get downvoted for skepticism, not for factual mistakes. Show me where you believe I have made a factual mistake and I will show you the academic papers that back me up. Additionally, According to "The Relationship Between Profanity and Honesty" by Gilad Feldman et. al. https://journals.sagepub.com/doi/10.1177/1948550616681055 those who use profanity are typically more honest than those who don't. So why would hacker news be against profanity if it holds the skeptical view.

ThomPete• 7 years ago

Not just skepticism, but extreme skepticism. Literally assume what you are looking at is more likely anecdotal than solid evidence or patterns and even more likely politically biased.

TomMckenny• 7 years ago

You would have thought the meteor with Martian bacteria, the University of Utah's cold fusion and the OPERA team's superluminal neutrinos would have driven this lesson home by now.

distant_hat• 7 years ago

The problem is incentives. As long as they are incentivised to maximize click through rates, you'll get hyped up headlines. There really isn't that much exciting stuff happening every day in science, and even when there is, someone writing a clickbait headline will be better off than someone writing a measured one. An excellent Twitter handle in @justsaysinmice. https://twitter.com/justsaysinmice All it does is says 'in mice' for all studies where hyped up headline and sometimes the main body leave out the fact that the study is only in mice.

forgottenpass• 7 years ago

Journalists can't even competently review and summarize a published study.

The default position for data journalism should be: don't try, you fucking hacks.

“Default Position Should Be Skepticism” and other advice for data journalists

Related Stories