Guidelines on how to be a scientific sleuth released

56 2 1 year ago

From the project’s introduction:

"Anyone can do post-publication peer review. Anyone can be a steward of the scientific literature. Anyone can do forensic metascience. Anyone can sleuth. However, investigating the integrity of the published scientific literature often requires domain-specific knowledge that not everyone will have. This open source project is a collection of guides written and maintained by publication integrity experts to distribute this domain-specific knowledge so that others can participate in post-publication peer review."

btilly• 1 year ago

I consider this an extremely important project.

One of the most reliable cognitive biases that we have is that cognitive dissonance prevents us from taking feedback on that which is important to us. Instead we convince ourselves that we've become good at that which we care about, and reject all evidence that we need to improve. The more we care about it, the more we fall into this trap.

There are ways to reduce how often we fall into this trap. For example https://blog.codinghorror.com/the-ten-commandments-of-egoles... is a set of rules that many good programmers follow to avoid developing an ego about being a good programmer. This allows them to accept feedback from reality, and become better programmers over time. By contrast, programmers who think that they are good usually top out pretty quickly, and are usually pretty bad.

Unfortunately we always have a pull to believing that what we've done is valuable, and then falling into this trap. This causes entire fields of study, including scientific fields, to frequently fall into a self-reinforcing overconfidence in what they believe that they have shown. Which consistently results in missing their own errors, and rejecting contrary evidence.

Post-publication peer review by itself does not fix the problem. But it is a necessary feedback loop that is required for any meaningful fix. And the existence of post-publication peer review helps those not in the field to learn which fields should be trusted, and which not.

Therefore there is a lot of value in having more, and more effective, post-publication peer review. Even though we'll still have entire fields of science who are overconfident of their bad research.

Guidelines on how to be a scientific sleuth released

Related Stories