Alternative statistical method could improve clinical trials

https://phys.org/news/2021-12-alternative-statistical-method-clinical-trials.html

by FabHK • 5 years ago

33 9 5 years ago

FabHK• 5 years ago

The paper: https://www.pnas.org/content/118/49/e2105254118

Side note about an advertisement on the linked page. It's a google vpn ad and the behavior is new to me. On a mobile device while swipe scrolling, I scrolled on the ad itaelf. This resulted in the ad actually opening a new tab, even when I tried it multiple more times. I didn't tap on accident.

The ad is actually opening a new tab when you're just scrolling the page while touching the ad area and never actually tap it or intend to open the ad

spekcular• 5 years ago

I looked through the paper, and I don't understand the point.

The authors introduce a measure called the "fragility index" (rather, a modification of an existing fragility index) that's supposed to diagnose a lack of robustness in clinical trials satisfying the usual standards of p < 0.05.

First, the problem with clinical trial results failing to replicate or discover true effects goes far beyond the statistical methods used. For example, "virtually all major RCTs funded by an NIH institute (NHLBI) before 2000 were false positives. Once hypothesis preregistration is required in 2000, everything becomes a null." [0] Arguing about statistical methods is just rearranging deck chairs on the Titanic until all that institutional stuff is taken care of.

Continuing anyway: The introduced measure is extremely ad hoc and not supported by any theoretical framework (e.g. proofs of good asymptotic properties). I don't know how to interpret it, especially because (as the authors note) it can easily flag a study as "fragile" when it isn't. Say what you want about p-values, but they at least have mathematical guarantees when used properly.

Further, there's no discussion of how this compares to existing approaches. Why not just do careful Bayesian modeling? Or, why not try to learn something from the amazing success of machine learning researchers in predicting out of sample generalization (e.g. see [1])?

Maybe I'm missing something, but I really don't see what the contribution is here.

[0] https://twitter.com/paulnovosad/status/1427332860902584329 [1] http://jakewestfall.org/publications/Yarkoni_Westfall_choosi...

paviva2• 5 years ago

There is no point.

The fragility index is essentially a repackaged p-value, and serves only to introduce even more confusion on the top of this already often misunderstood metric. See very good discussion here: https://academic.oup.com/eurheartj/article/38/5/346/2422087

Also, trials are generally designed to recruit the minimal number of patients we can get away with, and this is done for good reason (cost, feasibility, and the ethical concern of "using" the lowest possible number of human beings, while allowing any beneficial treatment to benefit the most as soon as possible). If someone's then pointing out that this trial is "fragile" -- well, yes, it was designed so in advance! Would you propose to expose 100 more patients to an inferior treatment just to improve your fragility index?

esyir2• 5 years ago

Honestly? Yes. The issue I'd have with said minimums is that, at least from my experience in biomedical research is that they're often done with exceedingly poor statistical grounding and pathetic sample sizes. Admittedly, perhaps things are better in the clinical space, though my gut argues otherwise.

So, to re-answer your question

Would you propose to expose 100 more patients to an inferior treatment just to improve your fragility index

Absolutely, especially if the alternative is exposing a million more patients to a different, more expensive inferior treatment because the statistical analysis was garbage.

rscho• 5 years ago

The main point IMO is that 99.9% of active researchers in clinical medicine should never have been allowed to conduct research. I should know, I'm one of them. The state of the field is plainly abysmal and most clinical researchers don't know a mean from a median, yet happily design full-scale clinical trials.

stewbrew• 5 years ago

So they conduct a trial, ideally they pre-registered their trial, and nobody asked them about test power, beta, and type II error? Seriously?

Alternative statistical method could improve clinical trials

Related Stories