Medicine is plagued by untrustworthy clinical trials

https://www.nature.com/articles/d41586-023-02299-w

by headalgorithm • 3 years ago

333 271 3 years ago

it's 2023, we have the means to cheaply record and store audio and video evidence for basically any medical experiment. we can record every patient reaction and opinion without relying on the reasearchers' hearsay. we also have the means to store and distribute all the binary/textual raw data gathered throughout the experiments.

maybe as an intermediate step we could make available all the recordings to the peer reviewers and only offer the raw experimental data bundled in the paper publicly? maybe in the future we can have 1TB studies without breaking a sweat? maybe all the money we give to publishers can be spent on servers to archive all the primary data so at least we aren't simply filling the pockets of MBAs?

jamesdwilson• 3 years ago

how do you prevent cherry picking?

CrampusDestrus6• 3 years ago

that's such a vague question

for example, if you have 50 partecipants but only provide the multimedia evidence for 20 of them your study should be thrown out the window

pbmonster• 3 years ago

These are medical trials. How do you preserve the patients privacy in all of this?

Or do subjects need to wave all their doctor-patient privacy rights before joining any trial?

CrampusDestrus6• 3 years ago

If we discover that we can´t trust researchers then what else are we left with? Doctor-patient privacy works if the doctor is truthful in their reporting

Slaminerag2• 3 years ago

It's generally permitted to share de-identified patient data. As long as you're not sharing patients names, medical record numbers, birthdays, and a couple of other fields, you should be fine.

lost_tourist4• 3 years ago

Maybe we could do a double blind (including scientists) study where everyone waived their rights & are recorded then in another "typical" conditions study do none of that and compare the two and see which one seems to have the best, most accurate results.

WastingMyTime892• 3 years ago

maybe as an intermediate step we could make available all the recordings to the peer reviewers

The issue is clearly not the amount of data available to peer reviewers considering it's already easy to detect major flaws in a quarter of published peer reviewed research. The issue is that peer reviewers do a shoddy job which should surprise no one having ever published peer reviewed research.

And to be fair why should they do better? It's generally unpaid, it's poorly paid when it is paid and it's not particularly well considered.

sonicshadow6• 3 years ago

Sounds like a YC idea?

ttpphd• 3 years ago

LOL looks like someone has not had to get data collection protocols through IRB approval...

CrampusDestrus6• 3 years ago

if researchers are so untrustworthy then what's your solution?

Balgair2• 3 years ago

For a 'quick' overview into the mess that is IRBs, this book review is a good starting place: https://astralcodexten.substack.com/p/book-review-from-overs...

TL;DR: IRBs are a mess, hyper-individualized, and the problem ain't getting any better any time soon.

dennis_jeeves12• 3 years ago

it's 2023, we have the means to cheaply record and store audio and video evidence for basically any medical experiment. we can record every patient reaction and opinion without relying on the reasearchers' hearsay. we also have the means to store and distribute all the binary/textual raw data gathered throughout the experiments

This is a great approach IMO. Additionally skeptics (for example anti-vaxxers), should be physically present at the trials.

Eumenes• 3 years ago

The over prescription of statins is a great example of what pharma sponsored clinical trials result in

remote_phone• 3 years ago

This. Statins claimed to be free from adverse reactions, but it turns out that about 30% of the participants were taken out of their clinical study because of “non-compliance”. However, if you dig in further, the non-compliance was because of adverse reactions.

You can’t trust pharma companies if their data is secret.

TheBigSalad• 3 years ago

There must be more to it. Everyone who drops out is scrutinized.

anonuser1234562• 3 years ago

If you dig further into those adverse reactions, you’ll find they are approximately equal to adverse reactions of placebo.

nradov• 3 years ago

That is medical misinformation. No drug company ever claimed that their statins were free from adverse effects. There are many statins on the market now, and patients who experience bad side effects from one will often do well on another. Getting the treatment right is a trial and error process.

https://peterattiamd.com/why-a-recent-study-hasnt-shaken-my-...

epistasis2• 3 years ago

For your comment to make sense to me, you are saying that the pharma sponsored clinical trials are untrustworthy data?

I've not heard that claim but am interested. Overprescripton certainly doesn't require that the trials were bad in any way.

And I'm also curious about this idea of overprescription, because I hear it sometimes from extremely political people but have never heard it from scientists (and scientists are always trying to find some way to critique current practice, so that statins don't rise to that level is a surprise to me).

obblekk3• 3 years ago

There should be Nutrition Facts but for scientific trials. Independent agency just publishing quality assessments of the trial.

This should be an async non blocking evaluation. The statisticians who do it should be anonymous by default. There should be an appeals process for a scientist to explain why an unconventional new method is actually robust.

There should not be a single number published by this process, but rather a list of stats that speak to the overall quality of the trial on many dimensions (power, sources of bias, etc).

Only information that would not be the same on 99% of trials should be written on this label (no sec style everything is a risk word vomit disclosures).

There should not be a pre-emptive application for a label - it can only be gotten after paper submission to reduce gaming.

There should be an independent advisory org that scientists can literally call to ask for advice on structuring the trials. These calls must not be disclosed. Much like farmers can call the government to ask for help on xyz crop problem.

And these labels should never be used as the primary source of punishment. Any and all sanctions/penalties/dismissals must go through a new review process done by a different group.

Any scientist who gets a label in a particular year should be given a vote to review the review agency on several dimensions. These aggregate reviews should be published broadly but not trigger any automatic consequences.

Clear, accessible information is the basis for any self regulating human system. We need more of it in this field.

alphazard• 3 years ago

The success of the Nutrition Facts labeling does not get enough publicity.

Rather than outlawing certain ingredients, or creating some kind of health score which a product must be above, Nutrition Facts is a way for suppliers to attest to information about a product in way that is legally binding. If it isn't accurate the penalties are steep.

Consumers then have the information they need to vote with their wallets. Markets cannot function properly without symmetric information, and Nutrition Facts essentially creates a functioning market where one did not exist previously.

Any effort to regulate what's in food would probably be better spent expanding what must be in the Nutrition Facts label. I guess it's nice that we are finally getting around to banning artificial trans fats, but anyone who can read has been able to keep those out of their diet for years. The same can be said about the next bad ingredient, and the one after that.

taeric18• 3 years ago

This isn't without downsides, of course. The case of manufacturers adding allergens to food deliberately is alarming in its own way.

vxNsr• 3 years ago

Interestingly, there is some gaming of the main number everyone looks at on the nutrition facts chart: calories per serving.

All snacks aim to fall at or below a certain number the FDA (or some other agency) put out as being considered a snack. Planet Money did an episode on different M&M varieties having different total weights to account for their different calorie counts. So you get fewer by weight peanut butter M&Ms because they’re more calorie rich

mike_hearn6• 3 years ago

There's the Cochrane Collaboration. They don't tick off every item on your list but it's fairly close to what you're asking for. It's mentioned in the article as they do a lot of meta-studies. Unfortunately they only started trying to spot fraudulent RCTs in 2021. Also in recent times some people don't like them, because they did a big review of mask studies and found there was no reliable evidence that masks worked against COVID.

Cochrane (formerly known as the Cochrane Collaboration) is a British international charitable organisation formed to organise medical research findings to facilitate evidence-based choices about health interventions involving health professionals, patients and policy makers.[4][5] It includes 53 review groups that are based at research institutions worldwide. Cochrane has approximately 30,000 volunteer experts from around the world.[6]

The group conducts systematic reviews of health-care interventions and diagnostic tests and publishes them in the Cochrane Library.[7][4]

https://en.wikipedia.org/wiki/Cochrane_(organisation)

camelite• 3 years ago

"Many commentators have claimed that a recently-updated Cochrane Review shows that 'masks don't work', which is an inaccurate and misleading interpretation."

https://www.cochrane.org/news/statement-physical-interventio...

specialist4• 3 years ago

Also in recent times some people don't like them, because they did a big review of mask studies and found there was no reliable evidence that masks worked against COVID.

Oh.

I quickly found this:

"The new scientific review on masks and Covid isn’t what you think" Kelsey Piper

https://www.vox.com/future-perfect/2023/2/22/23609499/masks-...

Based on the criticisms, I expect Cochrane will revisit this topic.

Progress isn't a straight line.

renaudg3• 3 years ago

they did a big review of mask studies and found there was no reliable evidence that masks worked against COVID.

No, that was a misinterpretation of the review in the Covid-skeptic sphere. Cochrane have had to issue a statement to clarify : https://www.cochrane.org/news/statement-physical-interventio...

tl;dr : half of the people given masks in these studies didn't wear them consistently or at all, dragging efficacy results down.

p-e-w2• 3 years ago

There should be Nutrition Facts but for scientific trials.

No, there should be prison time for scientists who conduct unethical trials or publish fake results.

The public (and policy makers) place such immense trust in those people and what they publish that nothing less is even remotely adequate.

When someone puts arsenic in food, they go to prison – labeling the food with "contains arsenic" doesn't cut it.

Do this and watch science magically fix itself.

ekianjo• 3 years ago

Revoke their license and titles to start with so they cannot operate anymore in the field

twic4• 3 years ago

Which prison? As the article says:

Ultimately, a lingering question is — as with paper mills — why so many suspect RCTs are being produced in the first place. Mol, from his experiences investigating the Egyptian studies, blames lack of oversight and superficial assessments that promote academics on the basis of their number of publications, as well as the lack of stringent checks from institutions and journals on bad practices.

A substantial part of what's happening here is that first-world countries with generally good cultures of research integrity are basing medical policy on studies done in countries where the system encourages researchers to cheat. British and US authorities can't put Egyptian or Chinese researchers in prison, can they?

neaden• 3 years ago

Just to be clear, if you actually did this what you would see is stuff like Florida locking up every climatologist for doing "false science".

t0bia_s5• 3 years ago

- Do this and watch science magically fix itself.

Imagine same approach in politics. If some politics put lies in their speeches to manipulate with people, we should call them liers and put them in prison. But somehow it doesn't happen. Looks like society prefer conformity over responsibility.

jonlucc3• 3 years ago

Why would the statisticians be anonymous? I'm aware of at least a couple cases in which an independent set of statisticians were provided the data from a clinical trial specifically for a re-analysis. In one case, they showed some pretty concerning inconsistencies and the other confirmed no effect on the primary analysis, but suggested some sub-populations that might have shown an effect if a future study was properly powered. That follow-up clinical trial was just published showing pretty remarkable effect in the sub-population. I don't think there's reason to believe either independent analysis was anything other than independent.

There is already a mechanism for companies to submit questions to the FDA prior to clinical trial initiation. I'm not in these conversations, but I know the type of questions can be things like: would you accept this endpoint as a proxy for this indication, would you be satisfied with the effect size we expect, and are there other safety concerns you would expect us to evaluate other than those in our current plan. I assume EMA and other regulatory bodies have a similar process, but I'm not positive.

Disclaimer: I work in pharma, but pre-clinically. I am not involved in these clinical or regulatory issues.

obblekk3• 3 years ago

So junior scientists can be hired without them being concerned for future career prospects.

s1artibartfast2• 3 years ago

Im struggling to see the difference between this and the current FDA process, and think it is 90% the same.

Drugs have "prescribing information", referred to in industry as "labeling", which follows a consistent format containing safety, trial results, side effects, and mechanism of action.[1] I recommend people read them for drugs they take.

Third party analysis is the main difference here. In the current state, firms run the analysis for FDA review using standard practices, and must explain and get approval for any unconventional methods

There should not be a single number published by this process, but rather a list of stats that speak to the overall quality of the trial on many dimensions (power, sources of bias, etc).

Labeling contains many relevant numbers. Trial sizes, how many per arm, what was measured, and and final results. Maybe there could be some squishy qualitative summary, but that seems more risky. I would rather know that 1 out of 20 patients died than it got a "2" on the safety scale.

Only information that would not be the same on 99% of trials should be written on this label (no sec style everything is a risk word vomit disclosures).

Labeling contains drug specific information.

There should not be a pre-emptive application for a label - it can only be gotten after paper submission to reduce gaming.

Drug labeling requires pre-application and and a standard 12 month review period by the FDA prior approval

The FDA provides advice on structuring trials and acceptable design, size, power, endpoints. Firms do this by scheduling calls with FDA staticians and experts. [2]

And these labels should never be used as the primary source of punishment. Any and all sanctions/penalties/dismissals must go through a new review process done by a different group.

Maybe there is a difference here. I'm not sure what you mean by punishment? In the current system, The FDA can use the label as "punishment". The FDA may require addition "black box warnings" for drugs that are found to have serious side effects (e.g. high chance of death). They can also pull the label entirely, meaning the drug can not be sold.

This is basically how it works for medical device labeling in the EU. There are several "notified bodies" [3] which are private agencies to review the safety and efficacy. The firm then takes their mark of approval to the government agency.

https://www.accessdata.fda.gov/drugsatfda_docs/label/2014/20... https://www.fda.gov/media/72253/download https://climedo.de/en/blog/list-of-mdr-certified-notified-bo...

obblekk3• 3 years ago

The async thing means papers can be published without waiting for a gov agency to do its review. FDA takes the opposite approach.

Generally, reading a paper is much less risky to a person's health than taking a drug so the differences in review process add up meaningfully.

csours• 3 years ago

Yes, I have said this as "The UX of [medical] study papers is terrible". Some people do not agree, they think that it should not be made easier to understand, that non-experts cannot really understand medical studies, so they should not be more approachable. I think that's dead wrong.

light_hue_1• 3 years ago

This is a problem everywhere where the raw data isn't released (suitably anonymized).

In cognitive science, psychology, even computer science / ai / ml, business.

And the problem with rejected papers getting in somewhere else while being total garbage is pervasive. I've rejected a lot of papers because they were mathematically or statistically bogus only to see them get published elsewhere where reviewers were not so careful (a few times in Nature and Science).

We need an open science movement where you must release everything with your paper. The full pipeline to reconstruct every single result from the raw data. No hiding data. No hiding fmri scans. No "our code only runs on our machine". Etc.

v4dok• 3 years ago

I think value-based care is the only real incentive on this. Otherwise, there is simply no reason for anyone to care enough. Even the insurers, they found a way to make money by making sure their premiums factor in these things. In the expense of the patient. As long as the drug doesn't kill people, who cares if it works if I make money off it as a pharma? Unfortunately, value-based care can only be pushed top-down. Patients are not in a position of power against pharma companies on this matter.

derbOac• 3 years ago

I'm empathetic to what you're suggesting — I've published on open science and meta-science specifically, and think open data should be the default norm. The problem with clinical research, though, is that it starts running into conflicting considerations about patient and participant privacy. Even when people aren't patients per se, the focus often involves sensitive information.

You can just say "anonymize it" but that turns out to be more difficult than it seems initially, especially with many questions of interest.

Also, there's often too many opportunities to do science that is of real public benefit that comes with privacy expectations attached for all kinds of reasons. Cases where there is legitimate consented access but an expectation of privacy without data sharing.

People have tried to solve this problem in different ways (for example, methods where someone can analyze data without having access to it directly) and maybe those solutions will lead to a good resolution. But they often have problems of their own (overhead costs associated with providing anonymized remote data analysis), and don't solve all problems (guarantees of absolutely restricted access to personal data).

Spinnaker_2• 3 years ago

We wasted decades, billions of dollars, and countless promising careers due to bad and fraudulent research in areas such as Alzheimers.

Whatever the costs and challenges are, they are not nearly as high as maintaining the status quo.

egberts1• 3 years ago

I now know of a few more people who have completely lost both of their hearing after taking just one Wellburtin-class pill.

At the time, no mention was made in the pill’s warning pamphlet.

It is still difficult to secure a class-action suit in America.

Meanwhile, such quality of life would plummet into a silent world, even if one knew American Sign Language fluently beforehand, that tidbit can go against the victim in court.

appleflaxen• 3 years ago

If you read the article, the headline is wildly editorialized.

Whatever. Par for the course in 2023, right?

But this is Nature, a paragon of scientific literature, fueling the distrust of medicine.

There are great reasons to be skeptical of all trials and strengthen peer review and transparency, but this kind of headline is editorial malpractice, in my opinion.

taeric18• 3 years ago

Agreed that the headline makes it sound like a strong majority, when the article isn't nearly as strong on that.

Still, "Carlisle rejected every zombie trial, but by now, almost three years later, most have been published in other journals — sometimes with different data to those submitted with the manuscript he had seen. He is writing to journal editors to alert them, but expects that little will be done." is concerning. I'm almost afraid to know what the list of rejected papers covers.

peteradio4• 3 years ago

If a quarter of your body was covered in leaches would you consider yourself plagued by leaches?

twic4• 3 years ago

No, not really. The headline says "plagued with", the subhead says "in some fields, at least one-quarter of clinical trials might be problematic or even entirely made up", and the article substantiates that. One quarter of all trials is more than adequate to plague the whole enterprise, and there isn't a problem with this headline.

ethanbond• 3 years ago

Ah yes, the greatest of all weasel words: "problematic."

bowsamic4• 3 years ago

We should be distrustful of medicine, of all science in general. Ignoring flawed methodologies or inconclusive results just means it's no longer science, it's ideology. I'm a physicist though so perhaps I could be overly jaded about science and peer review compared to most scientists

dekhn5• 3 years ago

You simply can't apply the rules of publication in physics to medical biology research. Even highly quantitative biology is noticeably different in terms of standards of proof and quality of models.

jononomo2• 3 years ago

It has been since the introduction of peer review that science has taken a nose-dive. Peer-review is just such a terrible idea. All it does is bring politics and back-stabbing into science.

mjfl2• 3 years ago

It’s hard enough to run a clinical trial guys. It literally costs $100 million at a minimum, yet the requirements that make it cost this much are not enough. We are basically going to regulate new medicine out of existence.

WastingMyTime892• 3 years ago

Please read the article before commenting. The problem is not how hard it is to run clinical trial. It's that made up data is an endemic problem. It doesn't matter if clinical trials are hard or easy to organise when up to a quarter don't actually bother and just forge their results.

chaxor3• 3 years ago

But they have to make up that data! Because the work they are doing now was based on a other trial where they made up data, so you have to fix this data to match what was expected from the previous studies. Of course we need to protect their right to make up data. /s

droopyEyelids• 3 years ago

Hamilton Morris recently did a podcast that touched on some of this, I think it was https://www.patreon.com/posts/pod-78-legal-84786504

An interesting point he made was that in the wake of the Thalidomide (https://en.wikipedia.org/wiki/Thalidomide) scandal, the FDA started requiring drugs to be both safe _and effective_

https://www.fda.gov/about-fda/histories-product-regulation/p...

The 'effective' part has proven to be a big source of complexity in the following years, because while it's relatively easy to prove a drug is relatively safe, it's much more difficult and subjective to prove a drug is effective. That closes off a lot of areas of research and development.

The kicker is that Thalidomide was never sold in the USA to begin with.

Anyway, as any reader can imagine, there would be a lot of negative social outcomes to allowing the sale of ineffective drugs. There's a lot of trouble now with medical devices and drugs not being effective, even though we have the rule. I'm not against regulation, I think medical sales are a really complex issue and I don't know how to even judge where the right balance of safety/effectiveness and innovation/freedom could be.

throw9away63• 3 years ago

They kind of broke that when they allowed the approval of Aduhelm which is basically shown to be expensive and ineffective

dekhn5• 3 years ago

Thalidomide is sold in the USA as a treatment for several conditions- it's a highly effective drug and is mostly safe within the target population.

(I point this out because most people only tell the very first part of the thalidomide story).

sonicshadow6• 3 years ago

Yeah who needs truth and accuracy anyway? Think of the small businesses who will never be because they couldn’t get a simple drug on market with minimal testing

mjfl2• 3 years ago

You have to put $100 million into effective testing. Your comment is disingenuous.

lost_tourist4• 3 years ago

You should read the article, they aren't asking for more regulation, just more skepticism out of publishers and that scientists keep their raw data available for inspection by peers.

alexb_• 3 years ago

I'm not too knowledgeable when it comes to how scientific experiments/trials are done - are the people who collect the data, the people who interpret the data, and the people who fund/benefit from the data different parties? Or are they the same people?

WaitWaitWha• 3 years ago

In my experience, they are different (speaking as someone witnessing it from layman's perspective). Here is what I have seen in Phase II & III trials:

a) Pharma identities the type of patients they need (e.g., 25-50 female, not pregnant, with specific ailment if Phase III), specific tests, and measurements required throughout the study.

b) pharma contacts third party (3P) to manage study patients.

c) 3P has relationship with dozens or even hundreds of doctors' offices, knowing what office can fulfill the test & measure requirement, and has the potential trial patient pool.

d) 3P has existing contract with these doctors' & hospitals. They get patients onto the study. ( <--- #1 reason this is farmed out in my opinion)

e) Doctors & hospitals perform the study and collect the data.

f) doctors & hospitals pass the data to the 3P

g) 3P passes it to the pharma

h) repeat e) through g) as many times as the study requires it. This can be once, or many times over years.

i) pharma pays 3P, 3P pays doctors & hospitals, and they pay the trial patients - each taking their cut along the way.

There are variations on how this is done, sometimes no 3P, sometimes pharma will have their own pool of patients and 3P. Also this is a very rough flow as there are often checks, audits, and validations (should be) done during the study.

linuxftw8• 3 years ago

For vaccine manufacturers, it's all the same people. Even if you have whistleblowers come forward, they're ignored.

Here's one quick way to rig any clinical trial: Anyone and everyone that has any kind of negative reaction or health condition gets disenrolled. Since it's 'double blind' it appears on the surface level that there's no way to know who's in what group. Naturally, the end result is always the same: The test group had the same number of reactions as the control by the end of the study.

nocoolnametom2• 3 years ago

"Even if you have whistleblowers come forward, they're ignored."

Do you have any sources for this? I'm rather disbelieving of it, but would love to be proven wrong. I can't imagine that _some_ major news outlet wouldn't love to stick it to the status quo with a whistleblower, unless the "whisteblower" made false claims about their proximity in the company to the dangerous/illegal actions they are trying to bring attention to.

Balgair2• 3 years ago

Great question, and I'm unfortunately going to have to give the answer of 'it depends'.

Each study is different and therefore run differently. Many, many, factors determine how a study is run, analyzed, and published.

Most studies are very small, using only one site and a few volunteers. Most of these never see the light of day, as the results aren't publishable or are uninteresting. Think power law distributions with studies, not normal distributions. These studies are often so small that the collector and interpreter are the same person, typically they are also the grant writer and admin. If lucky, they may get some nurses or undergrads to help out. Again, I'd say this is ~80% of studies.

The really large studies that places like Pfizer run will separate out nearly all parts of a study. So consenters, nurses, intake, data admin, funding admins, stats guys, etc are all different people. These are very expensive studies to run so it's really only for FDA approval, not scientific inquiry and case studies.

Generally, most studies are very small and not publishable. They don't need to separate out everyone. Everyone kinda trusts that everyone else is doing their best. If something snazzy is found, then follow up studies will build on it's findings. Most of the time though, nothing is really found.

epicEHRsucks• 3 years ago

There unfortunately are too many perverse incentives that encourage fraudulent studies. Everyone should take findings "established in the literature" with a grain of salt. We should also incorporate more intuition and first-principles reasoning, i.e. obesity is a harmful state for humans even if 100 RCTs proved otherwise.

constantcrying2• 3 years ago

How can you do science if around a quarter of the data is just straight up noise? Even when analysing large amounts of studies the results becone contaminated very easily.

And why is it not standard practice to provide anonymous data or even publish the data? What reason exists for that? So that only the researchers them selves can analyze it?

thechao2• 3 years ago

The working stuff becomes part of a body of lore of "real science" that you soak up in the lab. Not a great method, for sure.

constantcrying2• 3 years ago

But the point of RCTs is that you can get unbiased, high quality results without having to rely on "lore" spreading among medical professionals about which treatments are effective and under which circumstances.

mike_hearn6• 3 years ago

Yes the problem is that the research system rewards publishing papers, but most of the work is collecting the data. So if you release your data then other groups can write papers based on your effort for far less cost. It's sort of analogous to the problem of open source business models in the software world: if company A writes the code and releases it for free and earns money from running a cloud service, and company B just offers a cloud service, then the second company can get much higher margins because they don't have to develop it.

Unfortunately it's not easy to see what the alternatives are, beyond simply not funding research through government/foundation grants. When science is paid for by companies you don't have this incentive issue because the research is judged based on (ultimately) whether it leads to successful products, not whether it leads to lots of papers getting published. You have other incentive issues of course.

twic4• 3 years ago

That's actually mentioned in the article, which is quite good and worth reading:

In 2016, the International Committee of Medical Journal Editors (ICMJE), an influential body that sets policy for many major medical titles, had proposed requiring mandatory data-sharing from RCTs. But it got pushback — including over perceived risks to the privacy of trial participants who might not have consented to their data being shared, and the availability of resources for archiving the data. As a result, in the latest update to its guidance, in 2017, it settled for merely encouraging data sharing and requiring statements about whether and where data would be shared.

jonlucc3• 3 years ago

I work in preclinical pharma research, so I have spent a good amount of time trying to recapitulate published data in animal models, so not quite clinical trial data. People who do this work learn how to evaluate trustworthiness. It can be as granular as "this lab is the only one publishing this kind of information, so we'll be skeptical" to a bit more broad "this class of drug isn't expected to have that biology" to "I trust this company over that institution". We route around the fact that some information isn't reliable, and that's not always because of dishonesty or fraud.

I'm not a clinician and don't deal with them regularly, but the impression I get is that new studies are published by researchers who have a lot of connections (dubbed thought leaders). They present at conferences. Other clinicians pick up the use case that matches their need (this patient has failed other therapies for this indication, let's try this new thing I'm now aware of). Then as experience grows, clinicians have more nuanced understanding of the use cases for that new information and its reliability. Frustratingly, this can take years, but that's bug that's also a feature.

evandijk70• 3 years ago

Privacy concerns are the most important reason. A more cynical reason might that the odds that a papers is deemed 'ok' is far larger if the raw/anonymous data is not provided. Remember, the authors of the suspect studies provided their data voluntarily, and in the end it only hurt their reputation/impact.

lost_tourist4• 3 years ago

Not a quarter of the data, quarter of the papers. There's a big difference there.

oldgradstudent9• 3 years ago

With the risk of starting a flame war, we recently had several well-publicised clinical trials that reported 95% efficacy of some certain modality. Yet, in reality, efficacy as defined in the trial turned out to be closer to 0%.

Instead of investigating what in the design and execution of the trial led to such a discrepancy, the problem was handled by denying there was a problem, changing the goalposts, reporting ad-hoc hypotheses as facts, silencing all critics, and forcing the public to take the modality anyway or lose jobs, school, and freedom of movement.

taeric18• 3 years ago

I can only guess you mean something regarding the pandemic? Do you have links to show things were "closer to 0%?" Sounds more than a touch outlandish. :(

I'm confident we will know even more about things as time goes on. I'm less confident on any nefarious motivations in most of it. Reality is that a lot of people died, and everyone was trying to gain control and an advantage over the situation. Mistakes were certainly made, but I am back to low confidence in thinking that everything was a mistake.

oldgradstudent9• 3 years ago

Do you have links to show things were "closer to 0%?" Sounds more than a touch outlandish. :(

Take the Pfizer vaccine. The clinical trial's main endpoint was ~95% efficacy in one thing and one thing only, prevention of symptomatic Covid.

Not reduction in mortality, not reduction in serious disease, not infection, and not spread. The only thing the trial tested and reported was prevention of symptomatic Covid. This is also the sole indication in the package insert as approved by the FDA.

In reality, everyone I know got vaccinated and got symptomatic Covd. I mean everyone, no exceptions. The situation is the similar in my entire country and around the world.

oldgradstudent9• 3 years ago

I'm confident we will know even more about things as time goes on. I'm less confident on any nefarious motivations in most of it. Reality is that a lot of people died, and everyone was trying to gain control and an advantage over the situation.

It's not a matter of being nefarious. They (CDC, FDA, health authorities all over the world) really though it was important, but they've used unacceptable means to enforce their beliefs.

Science dies when PR takes over reality.

If reality disagrees with the trial, you have to debug the rial, and find what in the design or exection went wrong.

Mistakes were certainly made, but I am back to low confidence in thinking that everything was a mistake.

Silencing critics by health authorities is not a mistake. It is an intentional act to enforce your views.

lost_tourist4• 3 years ago

You should post them so we can do a peer review of your statements :)

mydriasis2• 3 years ago

More and more it seems that science is being clouded by moneyed interests and greed. If we can't trust science, what can we trust?

thechao2• 3 years ago

This is just reporting bias. I've worked (on-and-off) in various scientific fields for ~30; it has always had its bad actors. (I even helped do the statistics for some!)

I'd urge you to consider following the situations: 1. Prescientific inquiry; and, 2. PreFDA food, drug, and medicine.

Both of those were orders-of-magnitude worse than what we have, now. Could we do better? Sure! Is it broken? No.

mydriasis2• 3 years ago

That brings a bit of hope!

LorenPechtel10• 3 years ago

Disagree. Broken isn't a binary--the current system is far better than what came before, but that doesn't mean there aren't serious flaws in the current system.

gigatexal• 3 years ago

In the age of QAnon and Alex Jones and science and vaccine deniers we can’t have things like clinical studies be corrupted. Ugh.

sonicshadow6• 3 years ago

Yeah not a great look, maybe science isn’t the truth after all

linuxftw8• 3 years ago

The 'vaccine deniers' have been calling the 'studies' and 'trials' trash for a long time. The evidence comes out that they're trash, and somehow the 'vaccine deniers' are still wrong?

They removed liability from manufacturers, and suddenly the CDC schedule exploded with new products. I mean, chicken pox has a vaccine now?

nocoolnametom2• 3 years ago

That which was asserted without evidence could correctly be dismissed without evidence. AT THE TIME the anti-vax crowd was basing their positions entirely upon anecdote, rumor, and often badly misread prepublication research and stats. Their methodology was inherently flawed. Even if the conclusions they came to have been "validated" their position was still built upon this same flimsy scaffolding. It's not like the "do your own research" blogs and videos somehow gathered the same evidence used by this paper. This also does not indicate that other positions held by the same crowd, which are similarly based upon "anecdata" and rumor, are somehow made more evident by this paper in Nature.

Dig1t• 3 years ago

I agree that the economic incentives changed, and it does seem somewhat suspicious.

I have a lot of work to do ahead of me, researching all of these vaccines for my kids. Makes my head hurt just thinking about it.

dablweb• 3 years ago

Ironic that nature publishes this despite being guilty of constantly promoting it.

jmpeax• 3 years ago

Raw data examined: Ok 56%, Flawed 18%, Zombie 26%

Raw data not available: Ok 97%, Flawed 2%, Zombie 1%

Perhaps it's not good to call an unknown as "Ok"? Maybe Carlisle should add his own paper to the mix?

twic4• 3 years ago

That's literally the point he's making! The terms "flawed" and "zombie" reflect positive identification of dodgy data, so of course when the data isn't available, they are less likely to apply, hence why:

This finding alarmed him, too: it suggested that, without access to the IPD — which journal editors usually don’t request and reviewers don’t see — even an experienced sleuth cannot spot hidden flaws.

gumby2• 3 years ago

Note that the article discusses research studies and not clinical trials for drug or device approval.

These research studies are important (look at how many were conducted on COVID-19 over the last few years) but are typically not held to a particularly high standard, as with most science. Which doesn’t excuse bad data or poor statistics (the latter supposedly supposed to be picked up in peer review).

piqufoh• 3 years ago

Hmm, I read the article as explicitly calling out "clinical trials" (as referenced in the title and abstract) and it makes no reference research studies. I don't understand the distinction between "research studies" and "clinical trials", surely all research studies where an RCT is performed with real patients and real drugs is a clinical trial?

gumby2• 3 years ago

I meant “trials for research studies” as opposed to “trials for drug or device approval.”

The amount of record keeping and oversight of a drug approval trial is enormous (and as a consequence insanely expensive) — data handling, having disjoint groups at each stage handling and analyzing data, etc and detailed records of every manufacturing step — think ISO9000 on steroids.

Nobody would bother to go to that effort for a scientific exploration, nor should they. So the bar is much lower.

I am making no excuse for shoddy science! But it is quite unlikely for a licensed drug.

gordian-not• 3 years ago

Masters and Phd requirements should include repeating another research and checking whether they can recreate the results

sealeck• 3 years ago

Unfortunately some of the experiments require a whole team to carry out and take a huge amount of time to set up as well as having to be carried out under very specific conditions.

Slaminerag2• 3 years ago

My institution's been included in studies primarily for patient access. If we're not included, then good luck getting enough patients, and even then it can take several years to enroll enough patients. Replicating such a trial would be near impossible.

freedude• 3 years ago

“I think journals should assume that all submitted papers are potentially flawed and editors should review individual patient data before publishing randomised controlled trials,” Carlisle wrote in his report.

This should be considered part of every journal's idea of due diligence and this shouldn't be a new idea. Shysters, con artists and snake oil salesmen have been around for a long time. The purpose of a Journal is to publish reliable information and weed out the garbage. How can you do that if you are not looking at the entire picture?

isaacremuant3• 3 years ago

No. no. This is not permitted. Trust the science or you're an antivaxer, Trumper, denier, racist, white supremacist.

Media and gov, or media at the behest of gov (as the twitter files prove, but it was obvious without them), censored as much as thet could, everything that went against what they wanted to push. It wasn't science but security/hygiene theater and it worked, because people did go along and did turn on their neighbours who opposed the measures.

But now we slowly get tidbits of things we can debate again ... Funny that, but no recognition that the entire lockdown, masks and vaccine mandate effort + economic destruction and theft (tax money to corps as "aid") was never a reasonable, logical or scientific response. It was an authoritarian and corrupt response of extreme Effectiveness and cynicism.

But here comes some guy to say "people have always quarantined in pandemics" or some other disingenuous claim that ignores the reality of what happened: healthy people denied basic human and constitutional rights.

LorenPechtel10• 3 years ago

The problem here is "healthy people". Covid's real key to success is the fact that most spread is presymptomatic. It's the apparently healthy people spreading it!

And note the lessons of history:

1) There will always be those who choose short term economic interests over safety when the threat isn't absolutely proven. They'll always close the barn door too late.

2) Places that take epidemic/pandemic threats seriously tend to fare better economically in the long run.

isaacremuant3• 3 years ago

No. The problem is that fundamentally, you and people like you decided that authortiarianism is ok if the gov says "things are scary" and everyone should suffer the consequences. No debate allowed.

The risk profile for people was always ridículous and you were never going to contain it once it was widespread, which it was, but somehow we believed in the rolling "it's just 2 weeks".

1) it wasn't short term. It wasn't safety. Short and long term you hurt the vulnerable and you helped the rich and powerful.

2) Rich places will keep being richer and poor places poorer, and when the powers that be decided that enough was enough, all the concerns of the hypochondriacs suddenly didn't matter.

You could tell, if you paid attention, that politicians weren't afraid after the initial surge, but wanted "the masses" to be. You even have definite proof in many places, one of them being UK and number 10. Can't link it now, sorry. But Google downing Street covid rules or something on that note and you can probably find a lot. It wasn't limited to the UK. It was everywhere you looked properly.

You were scammed and you either were well off and didn't mind that much or you want to pretend you weren't for your own mental health. Because the truth is a hard pill to swallow.

xhkkffbf• 3 years ago

Even if they aren't faked, the ability to shut down a trial that isn't delivering the right preliminary data is a big problem for society. Why should we trust these drugs?

throw9away63• 3 years ago

A lot of these studies are prerun in the 3rd world before the real one is done for credit. If sideffects show up the drug can sometimes be mixed with one that has the same known side effects to fool the studies.

Roark66• 3 years ago

They say "medicine". I would say science in general, perhaps we could generalise even further to "any human activity is full of unethical people trying to exploit it". But 25%!? That suggest there is a big problem with how we "do science". Unfortunately I have no solution to the problem. Publishing everything (including raw data) for every research would probably help somewhat, but only teams repeating experiments/trials would ensure it.

I wonder if we suddenly took 10% of all money spent on science (let's say in medicine) and instead of novel research we used it to redo randomly chosen previous research. Would we loose or gain in terms of new cures? And if we gained, what if we spent 15%, or 25%? That's a great idea for a scientific study to find a point of diminishing returns on "research verification".

Would someone please write a research grant request for this?

davidktr• 3 years ago

Of course there are big problems with how we do science. Much of it is garbage. Imagine most software was written by junior engineers, without any code review or input from seniors. That is today's science.

Most scientific legwork is done by absolute beginners, i.e., graduate students. They often lack a support structure to focus on what they have learned so far. Most of the world is not Oxbridge, MIT, Stanford.

Where are the beginners' supervisors, you might ask? Chasing the latest trend to secure funding. Pondering how their line of research can be formulated as buzzword-du-jour markov chain. Ass kissing the dean to get department funding.

Having worked in research for 15 years, I am certain about two things: (1) The scientific method yields better results than doing things freehand. (2) Randomly axing 50% of academia would improve the situation.

bobbylarrybobby• 3 years ago

That 10% investment would have a huge payoff too because it would shut down avenues that were only opened due to p-hacking long before they'd had a chance to seek further investment.

j-pb• 3 years ago

That and lawyer driven development.

I have multiple sclerosis, and I never know if a new medication came out because it's actually better or simply because they could get a new patent for a slightly modified molecule.

LorenPechtel10• 3 years ago

Disagree--you're talking about patent-driven development, not lawyer-driven development.

thenerdhead• 3 years ago

Another good reason why you should learn how to read studies:

https://peterattiamd.com/ns001/

https://biolayne.com/reps/how-to-read-research-a-biolayne-gu...

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7392212/

And as a bonus, read books by controversial figures who talk about these challenges through their own published studies and decided it isn't worth fighting.

What would be cool is to see "trust indicators" as part of a study's metadata as it is reviewed through time and continually shared by others. Could be a "study health score" or a checklist showing the study isn't biased heavily by sponsors, methods, or misleading language.

Especially given that medicine is becoming more personal with the advancements of AI and accessibility of tests, someone should be able to understand the health of a study at-a-glance reviewed by peer reviews rather than disclosed only by the authors.

randcraw• 3 years ago

A primary trust indicator of any research result is the reputation of the journal publishing the paper. Better journals demand more, not just in terms of the impact/insight of the results, but also the rigor of the experimental methodology used and how well the data was curated and the confounding variables identified and isolated.

And of course, the criteria for observational-based research differ a lot from mechanism-based, especially since the former can't control for nearly as many variables. The same goes for simulations or interpretive modeling, where experimentation is minimal.

Research studies differ quite a lot in how precisely the mechanism of action is identified, isolated, tested, and results interpreted. IMHO, blackening the trustworthiness of all of science is unhelpful, especially when some models and methods are surely more trustworthy and replicable than others. Better to identify and catalog specific sources of error (or imprecision) in order to remedy them than to just toss the baby.

darkclouds• 3 years ago

I'd trust the NHS if the National Institute of Clinical Excellence (NICE) actually published their minutes online. Most employees only know what they have been taught, so whilst I agree with the headline that some trial data is dodgy, it doesnt just end there.

There are multiple pathways to factor in, there is redundancy built in, ie secondary pathways, there is the fact that not all chemicals go where intended (best highlighted by radioactive isotopes). When looking at the history of patented medicine, this really started between WW1 and WW2, before WW1, most GP's prescribed what was found in the body on a like for like basis, and in some cases prescribed organs in various forms, like desiccated thyroid gland for thyroid related problems, for pernicious anaemia, raw liver used to be prescribed to women and so on, but that has its problems like contamination and diseases.

So from WW1/WW2 onwards the rise of patented medicine took hold, but the main problem with patented medicine is the human body hasnt evolved to use these new chemical compounds in the same way as unpatented chemicals which have been around for thousands of years. And todays GP's dont really highlight the side effects of the patented medicine, and because they simply dont ask what you have been eating and drinking etc, they have this hubris which sucker punches your trust and sucks you in like a black hole, until before you know it you are on half a dozen different drugs, your quality of life is going down the pan and you've been left a zombie wondering where did Hitler go wrong modelling the German state of the time on the British Empire? You Americans complaining about the cost of healthcare, should count yourself lucky those insurance companies are looking out for your long term interests and theirs!

Fortunately, hospitals in other countries publishes studies and as english seems to be the main language used for science, I have to tip my hat to the Chinese who are roaring up the charts in terms of investigating and publishing relevant studies that will complement a quality of life one hopes to achieve, and we cant forget Wikipedia, Pubmed and Google for connecting users with pertinent studies.

Saying that I do sometimes wonder if something like ChatGPT has written some studies due to the poor quality of english used, but generally they stand out like a sore thumb.

Anyway, does any know why there is a connection between MDMA, blood clotting and Manganese?

amai• 3 years ago

There should be a paperswithdata for medicine like there is a https://paperswithcode.com/ for data science.

jet_32951• 3 years ago

Have a look at Derek Lowe's excellent blog [0] to view the depths of malfeasance, if not outright fraud, in which "studies" are created.

[0] https://www.science.org/content/blog-post/fakin-it-modern-wa... is just his latest in a long string of well-documented posts.

OnlyMortal• 3 years ago

To be fair, science is now a “publish or be damned” business. So many papers are not worthy of publication and, frankly, are of common knowledge anyway.

Of course, this is driven by the money backing the research which, I’m sure, this is why clinical trials in the medical areas can be of a poor quality.

People want to keep their jobs.

tails4e• 3 years ago

Nutrition for me it a big issue. It seems like and ideal area for scientific and medical study to give us light, but seems so hard to get truly objective info. Fat is the devial/good/bad/OK, sugar is the devil/bad/ok, etc. Should I avoid white bread like a hole in the head, or is it fine? Too many agendas and not enough truth.

jononomo2• 3 years ago

I don't think it makes sense to publish the results of clinical trials and other scientific experiments until after they have been independently replicated.

Madmallard4• 3 years ago

This is entropy in action. We're not going to move forward as long as malicious behavior is rewarded with money in large-scale systems. I suspect in coming decades more incurable chronic illness will become rampant and polypharmacy will become more common and necessary for larger percentages of the population. It is really sad to see.

dennis_jeeves12• 3 years ago

I will start my own medical 'system' and be my own doctor. To hell with the mainstream and it's corrupt/stupid adherents.

pyuser583• 3 years ago

It doesn’t help that front-line doctors are expected to publish.

NoZebra120vClip• 3 years ago

It was interesting to find out that test subjects who pass away during clinical trials are excluded, erased, from the trial records. Their participation and data is not included in the final results, as if they were never in the trial to begin with.

Many pharmaceutical companies have policies which prohibit patients from participating in studies for anti-depressants if they have suicidal ideations. In practice, this significantly reduces the participation of any patient with depression.

Medicine is plagued by untrustworthy clinical trials

Related Stories