We're Incentivizing Bad Science

https://blogs.scientificamerican.com/observations/were-incentivizing-bad-science/

by slowhand09 • 7 years ago

341 180 7 years ago

blogs.scientificamerican.com

First issue: publications. I think this is not a big problem in the USA relative to China. Is there evidence of American scientists who have greatly benefited in reputation and prestige, despite doing very shoddy work? Many scientists have been rebuffed and rejected from grants/awards due to not having enough publications. American scientists, though, are more attuned to the quality and integrity of journals in which publications appear. Whether for tenure or grants, a Nature or Science or Cell paper will mean a LOT, and many scientists evaluating other scientists wouldn't care much about a journal publication with Impact Factor under 1.0.

Meanwhile in China, there are numerous article-factory-journals that are pay to publish, you can put your shoddy work in those and amp up your publication count easily. Surely, these exist in the USA, but are career scientists at major institutions utilizing these shady Chinese journals? There is evidence that some of these Chinese journals are publishing straight up BS, which is especially easy to do with data analysis where you could "clean" your data easily. Perhaps it is a cultural or political difference, but I don't see nearly as much rigorous self-reflection of Chinese scientists on this front?

Second issue, grants: publications are a small slice of this story. Science departments (not humanities) in universities are MAJOR revenue generators for the University. My university took 1/3 of your grant straight off the bat, to cover overheads like shiny facilities and administration and marketing. Meanwhile, the scientists themselves may make huge salary bonuses or advance their tech/staff substantially when they have substantial grants. So, getting a grant is great for you personally, and improves your chances of further grants.

So, is it really that surprising that there may be pressure to publish at all costs, to p-hack, to reach for those low-impact journals despite their lower reputation/impact, given both universities and scientists are BOTH massively financially benefited from this incrementalism? Does it really pay to reach for pie-in-the-sky, fundamental sea-changes in your field? It seems like a high variance, high risk strategy that only very bold, well-funded, devil-may-care scientists would employ.

rocketflumes• 7 years ago

I can't comment on China and other fields, but in the US for AI and robotics, in which I do research / publish in, there is definitely a growing trend of an over-emphasis on novelty research that disregards reproducibility in favor of "wow" factors and fancy demo videos. A lot of highly cited research papers/labs tend to be the most heavily promoted ones (on Twitter, in the press, etc), and they're not necessarily impactful/useful for the rest of the field.

K0SM0S• 7 years ago

Amidst this worrying trend of shallow research, begging for petty cred and grants more than knowledge, you have to wonder what has become of ethics, of integrity in science. Globalized academia needs some soul-searching, IMHO.

knzhou18• 7 years ago

Is there evidence of American scientists who have greatly benefited in reputation and prestige, despite doing very shoddy work?

This is the exact point of the replication crisis. Within social science, the answer is an absolute yes. That's why we have been having this conversation more and more over the past decade.

Meanwhile in China, there are numerous article-factory-journals that are pay to publish, you can put your shoddy work in those and amp up your publication count easily. Surely, these exist in the USA, but are career scientists at major institutions utilizing these shady Chinese journals?

In China, people know about scientific reputation perfectly well, which is why publication in Nature/Science/Cell ensures great financial reward.

There are shoddy journals everywhere, of course. Yes, top scientists at top institutions and rigorous fields in the US don't use them. The same is true for top scientists at top institutions in China. The average quality is probably lower (due to the inherent difficulties of doing cutting-edge science only two generations removed from mass starvation) but the principle is the same.

tnecniv• 7 years ago

Regarding your first point, the West may not be as bad a China, but there are some notably bad actors. Schon [1] fabricated results very impressive results at Bell Labs for years before getting caught. In fact, these results were so impressive that it shifted the focus of research groups in his field for years to try and get his methods to work. It was such a big scandal that my (and I assume other) universities added a research ethics requirement to their engineering PhD programs as reactionary measure.

[1] https://en.wikipedia.org/wiki/Schön_scandal

Vaslo• 7 years ago

I agree with this article and have always thought about this but never voiced it.

I think a few things could improve the quality and discovery of published papers:

-After so many publications, could it be mandatory for random samples in that author’s publication be tested? I get that there is a limit on resources, but some advanced undergrads could do this with guidance.

-I would love to see some version of a journal of failures. That is, well intentioned research that had poor outcomes. It was so frequently in Chemistry research that my compounds were useless or the methods to synthesize them did not work, and it would be helpful to document that. Unfortunately, there is no “Journal of Failed Chemistry.” Only research that ostensibly make a contribution with a clear outcome get published. So much time wasted experimenting where you could save another scientist time and encourage them down another path.

djtango2• 7 years ago

It was so frequently in Chemistry research that my compounds were useless or the methods to synthesize them did not work, and it would be helpful to document that.

This mirrors my experience where a lot of the post-docs carried war stories of things like lab/country-specific humidity playing a role in synthetic methods succeeding (or failing). There were a lot of dark arts/tricks of the trade that people carried around with them: stuff like going that extra mile to dry things of water super thoroughly (even if it was not mentioned in the paper we were referencing).

ISL• 7 years ago

Many "dark arts" aren't known as dark arts to the people who know them. They're just the things you always do, because if you don't do them, nothing works.

An approximate CS analog: Writing great commit messages, and using an SCM.

You don't have to do them, nobody writes it up because those who know regard it as trivial, but if you don't do them, almost nothing works. Everyone who knows what they're doing does it.

codegladiator• 7 years ago

Partly the issue is because of the naming... Like "poor outcomes" and "failed"...

Obviously there would be no awards for the poor outcomes and failed results and we would be back to square one where no one is incentivized to look at them positively.

tasogare• 7 years ago

What are you waiting for creating one? Motivate a few scientists to form a credible review comity, fill one paper [1] and voila you have the journal you dreamt about on a recognized and perennial web platform.

This is of course some work, but if you really want it to exists it’s better not to wait for someone else to do it. Also allowing anonymous authors would help I think (to avoid the name being associated with failure).

[1] https://www.openedition.org/15974?file=1

Edit: openedition.org is for humanities, but there probably is alternative for other fields.

MaxBarraclough• 7 years ago

What are you waiting for creating one?

Creating a journal is a tremendous task, and only highly respected researchers have any chance.

if you really want it to exists it’s better not to wait for someone else to do it

Again, most people simply cannot create a credible journal. Let's be realistic here.

wolfram74• 7 years ago

maybe a more serious version of the journal of irreproducible results? https://en.wikipedia.org/wiki/Journal_of_Irreproducible_Resu...

glofish8• 7 years ago

This is a variant of "the carrot and stick" approach that happens to be very short on the carrot and be very large on stick.

Very unlikely to be effective.

Time and again people think that when they have a problem it must mean the punishment was not severe enough. It won't work.

The problem is that good science is not rewarded enough. Nobody rewards you for having published reliable stuff five years ago.

weberc2• 7 years ago

Unless the GP edited, there's no suggestion of punishment in his comment.

cepp• 7 years ago

On failures: many years ago I stumbled across the idea of a `CV of Failures` where you list all of the accomplishments you missed/were rejected from, etc. I think it's a very humbling idea and can see some applicability here.

Example: https://www.princeton.edu/~joha/Johannes_Haushofer_CV_of_Fai...

bumby• 7 years ago

I recently noticed SURE journal, intent on publishing "unsurprising" economic articles.

I just wish there were more, or at least more journals rewarding sound methodology instead of unsound "results". Who knows how much effort is wasted reproducing sound, but unremarkable studies simply because they weren't published

Al-Khwarizmi2• 7 years ago

There are a bunch of journals like that. If you google "Journal of Negative Results", you will see several for different fields.

However, for some I know, it's really difficult to get in, because there are many more journals of positive than negative results, and there are probably many more negative results than positive... so your negative result had better be especially interesting.

SirLuxuryYacht• 7 years ago

How do you quantify rigor?

dredmorbius• 7 years ago

In mortises?

knzhou18• 7 years ago

I don't want to sound too triumphant, but over here in fundamental physics we have decades-old safeguards in place against all these problems.

The big experiments don't have publication bias: they proudly say exactly what they did, even if 90% of the time there are only negative results, because exclusions are important too. Experiments are inherently replicated, with multiple independent simultaneous experiments (LHC) or multiple independent analyses (EHT), data blinding throughout, and even occasionally a further layer of blinding using decoy signals (LIGO). The statistical standards for discovery are, in terms of p-values, about 10,000 times more stringent, and even still people are moving away from p-values entirely.

The resulting publications are put out for free, publicly, on the ArXiv. Later they are submitted to a relatively small family of low-cost journals, which everybody knows the reputations of.

Hopefully some of these lessons can be adapted to other fields.

SiempreViernes6• 7 years ago

cought 750 GeV cough... ahem. Let me sing you an OPERA about a how Andrei Linde was astonished by the biggest BICEP you've ever seen! It belonged to a LIBRA* that was faster than light and could fly trough walls!

* Couldn't get DAMA to work :(

knzhou18• 7 years ago

Sure, but there's a difference between the accuracy of speculative theory and experiment. Experiment is very reliable -- they never called the 750 GeV bump a discovery of anything, because it didn't meet the statistical standards. Even things that do meet the statistical standards are often not called discoveries, because the experimentalists think they are systematic errors at play, as we see with MiniBooNE and DAMA.

Speculative theory is not reliable, which is why all the papers on the 750 GeV bump were wrong. But it has never been reliable, because, well, it's speculative! For each new phenomenon there's probably only one right explanation, but far more than one paper. Being 99% wrong has been par for the course for almost a century.

oefrha7• 7 years ago

The high profile debacles were pretty quickly caught, so I’d say the system is working.

fungicide• 7 years ago

"Stanford Professor Andrei Linde celebrates physics breakthrough"

https://youtube.com/watch?v=ZlfIVEy_YOA

vadansky2• 7 years ago

While it's certainly more robust, it still has some issues. For example, I remember reading that because the initial results for the experiment measuring the size of the electron was too low, every experiment afterwards reported slightly bigger sizes until they reached the correct value.

Since they trusted the original result, if their results varied too much they discarded them. Ultimately they redid the experiment until they got something closer, and the only values that ended up getting published were results that were slightly bigger.

Correct me if I am wrong, as I read this story a while ago.

knzhou18• 7 years ago

That's an old Feynman story, but he was kind of bullshitting to make a point. If you actually look at the history of measurements, it doesn't look like that at all -- it instantly corrects to about the correct value right after Milliken and hovers around that value.

https://hsm.stackexchange.com/questions/264/timeline-of-meas...

astro123• 7 years ago

This is exactly the sort of thing that blinding is there to prevent. It is the same reason we in cosmology blind our data - to prevent us from biasing ourselves towards what other experiments have found.

Also, the charge of the electron experiments you are talking about happening over 100 years ago, so not that relevant to today :)

jsweojtj• 7 years ago

I think you're thinking of the oil-drop experiment for the charge of the electron that was done 110 years ago (1909). And the experiments in the subsequent decades. Feynman, in 1974, so 45 years ago, used it as an example of problems in science. So, in response to the original post saying that particle physics has been doing things more or less correctly for decades is not tarnished by your example.

More details here: https://en.wikipedia.org/wiki/Oil_drop_experiment#Millikan's...

johnminter• 7 years ago

This is where measurements by multiple groups come in. I would point out that in 2019 the basis for the SI base units were changed because of issues with the standards. The international community - lead primarialy by NIST and their counterpart agencies in other countries noted that, among other standards, that measurements of the "standard meter" bars were changing measureably and the replicates of the "official bars" differed from one another.

They proposed and agreed upon new "standards" based upon measurments with better precision and accuracy. See the Wikipedia summary [1]

[1] https://en.wikipedia.org/wiki/2019_redefinition_of_the_SI_ba...

im3w1l• 7 years ago

Seems like a bias-variance tradeoff. It's not obvious to me that what they did was bad.

oefrha7• 7 years ago

90% of the time there are only negative results

That’s an understatement.

knzhou18• 7 years ago

What people forget is that this was true in the old days as well. Even back in the 60s and 70s there were over ten accelerators throughout the world constantly doing science, and you can bet they were booked up with new speculative searches every month. The vast majority found nothing, and were forgotten. That's because there's a very finite number of things to discover in fundamental physics, which is of course what makes the field so exciting in the first place.

orbifold5• 7 years ago

CERN and the LHC are to me prime examples of bad science. You build huge machines at incredible cost, wasting hundreds of thousands of man hours of brilliant young people’s time. They are paid almost nothing (a checkout clerk at a Swiss supermarket makes more than a PhD student at CERN). Then you string along a subset of those for years, exploiting them for further cheap labor, somehow making them believe they are “lucky” to get that opportunity (a postdoc at CERN pays a fraction of what you can make at Google Zurich). In the time between experiments people only ever see simulated data, leading to a rude awakening when actual experimental data comes in (c.f. ALICE’s desaster of a analysis Pipeline). Then there is literally decades of over promising on ground breaking discoveries right around the corner (super symmetry, extra dimensions, dark matter). Defunding the super collider in the 90s in the US was probably one of the best science policy decisions they made.

whymauri• 7 years ago

I kinda wish this comment focused more on bad science rather than bad pay. If bad pay implied bad science then most of science is bad!

I suppose I'd be much more interested in learning about the latter half of your comment (specific scientific failures).

eganist• 7 years ago

Then there is literally decades of over promising on ground breaking discoveries right around the corner (super symmetry, extra dimensions, dark matter).

Every statement in your diatribe would seem to be predicated by the assumption that discoveries in theoretical physics (what you're complaining about) will result in revolutionary changes to your day-to-day experience (kinda i.e. applied physics) in a short enough span of time that you can actually enjoy them with an able body.

Probably not. Flight/space and semiconductors were notable exceptions, and they were largely driven by the century of hot and cold war that we know to be the 1900s.

Bad science is science that fundamentally doesn't yield a new understanding of the world, either by falsifying theories or confirming them. What you described is:

1. failure to meet exaggerated lay expectations, and

2. poor working conditions in the nonprofit space.

#2 (and possibly #1 depending on where the theoretical discoveries take us) can largely be resolved by pouring more money into science, not less. I'll point back to our century of war if you need past precedent.

Edit: removed a line that was unnecessarily insulting. Sorry about that.

knzhou18• 7 years ago

Nobody forces academics to work for low pay. I mean, that's what I'm doing right now and I certainly don't feel exploited.

Then there is literally decades of over promising on ground breaking discoveries right around the corner (super symmetry, extra dimensions, dark matter).

Suppose all TeV-scale colliders had been defunded in the 90s, including the LHC. Then we would have forever been stuck with the strong suspicion that a Higgs is there, but no proof, along with untested but plausible-sounding well-motivated speculation over what else could be. Is that not worse than actually knowing?

oefrha7• 7 years ago

They are paid almost nothing (a checkout clerk at a Swiss supermarket makes more than a PhD student at CERN). Then you string along a subset of those for years, exploiting them for further cheap labor, somehow making them believe they are “lucky” to get that opportunity (a postdoc at CERN pays a fraction of what you can make at Google Zurich).

That’s not unique to LHC, or CERN, or physics. It’s a general problem of academia, where PhDs and postdocs are paid a pittance compared to what they could otherwise earn in the industry. This problem is especially bad in high energy physics of course, since jobs are especially limited, and it’s the brightest people competing against each other, who could easily land jobs on Wall Street or Silicon Valley.

Then there is literally decades of over promising on ground breaking discoveries right around the corner (super symmetry, extra dimensions, dark matter).

Standard Model works exceedingly well at LHC. No one was actually sure about BSM (beyond Standard Model) so there was no “promise” really. Or the promise is: we may see something interesting, or we may disprove some otherwise interesting theories.

Defunding the super collider in the 90s in the US was probably one of the best science policy decisions they made.

Cancelling SSC was such a stupid waste of labor and money, it’s painful to see someone touting it as a triumph. Two words: defense budget. Enough said.

Disclosure: I worked for CMS for a while. (Not physically at CERN; was doing data analysis for CMS in the U.S.)

magicalhippo• 7 years ago

ALICE’s desaster of a analysis Pipeline

Any references to this? First I've heard of it.

dekhn3• 7 years ago

it is true that moving protobufs is more profitable than moving protons, but I think CERN represents a fundamentally interesting and important scientific endeavor.

glofish8• 7 years ago

Yeah but then you have different problems.

Hundreds of authors on each publication. Whose contributions are real and who is just cruising?

cortesoft• 7 years ago

Maybe this just shows that using authorship credit as a proxy for contribution is a bad system.

knzhou18• 7 years ago

I mean, can't you say that for any large group of people? You can't easily evaluate who's pulling the most weight at Google either. But that's not a problem, because the people at Google deciding promotions do know, and that's what matters. CERN works like that, as does experimental physics at large.

dguest2• 7 years ago

At the LHC the number is closer to 3500. I "cruise" on around a hundred papers each year: I haven't read them and in all likelihood never will. At the same time I worked really hard on one or two of them.

I have tried to figure out what would be required to remove myself from 98% of the papers we publish, but it turns out to be a lot of work (essentially you have to argue with the head of the experiment, who has other things to worry about).

dwaltrip• 7 years ago

Determining attribution for scientific contributions is a far less severe problem than having high-quality scientific contributions in the first place.

itronitron• 7 years ago

> Hopefully some of these lessons can be adapted to other fields

the primary lesson here is to get a commitment for decades of government funding

jfengel2• 7 years ago

Preferably in the billions of dollars. Nothing generates five sigma confidence levels like ten-figure budgets.

Schiphol• 7 years ago

The statistical standards for discovery are, in terms of p-values, about 10,000 times more stringent

But this has mostly to do with your subject matter being better behaved than others, no? The most profound changes and the healthiest research culture could not make 5 sigma a reasonable goal in psychology.

knzhou18• 7 years ago

Our statistical standards may be unrealistic for social science, but there was a petition a while ago to lower the cutoff from 0.05 to 0.001. I think that really would help.

Fomite4• 7 years ago

There's also a very healthy question of weather or not it's worth while. A p-value that's theoretical physics stringent will mean leaving a lot of cures on the shelf.

cde-v• 7 years ago

Market demands Bad Science, market supplies Bad Science. I do not see a problem here.

buboard3• 7 years ago

Most of it is being funded by public means. Industries such as pharma, which rely on that science did point out the replication crisis in medicine: https://en.wikipedia.org/wiki/Replication_crisis#In_medicine

glofish8• 7 years ago

A cynical view - alas it captures the truth.

It is very easy to point fingers, to blame it on funding, blame it on journals, blame it on media, but in the end:

- scientists decide who gets funded

- scientists decide who gets published

- scientists make exaggerated claims in the media

The source of the problem is the scientists, not understanding the damage they are doing to themselves.

I do foresee downvotes, because scientists do not like this idea at all :-)

majos2• 7 years ago

It's worth distinguishing between different types of bad science.

1. The first and most egregious type is outright fraud. This is when you intentionally manipulate or fake data. Everyone agrees this is bad, and honest actors are enough to prevent it. In some cases, other honest actors are sufficient to determine if the claims are fishy.

2. The second more subtle type is not paying attention to adaptivity. For example, maybe an investigator wants to look at the data before coming up with a hypothesis to test. This is dangerous because the investigator is already overfitting, so any p-values the investigator computes afterward do not mean what they're supposed to mean. This is less egregious because it's easy to do this just by not being careful or not knowing your statistics very well. A scientist can be honest, but imperfect, and do this. It's also not easy to sniff this out as a reviewer -- the scientist might just omit all the stuff that didn't work. But there appears to be growing awareness of this kind of problem.

3. The third, and hardest to solve, problem is not factoring in the whole population of experiments. This is where 100 labs independently try an idea and one of them gets a genuine (from their limited view) result with a genuine p-value. It's novel and that lab has (in the limited view) been careful about adaptivity and keeping their hypotheses carefully generated. Maybe they've even used carefully generated noise to ensure their conclusions generalize [1] (which would definitely cut down on this problem). So it's pretty much impossible for a reviewer to tell there's a problem, because they don't know about the 100 other people that tried this and failed because the randomness didn't go their way. Short of a public experiment registry, this one is hard to fix, especially because it may be that nobody's being malicious or ignorant.

[1] https://arxiv.org/abs/1411.2664

marcus_holmes• 7 years ago

> Short of a public experiment registry, this one is hard to fix, especially because it may be that nobody's being malicious or ignorant.

That seems like a really good idea, that would solve a lot of problems. Can we do that, please?

ars• 7 years ago

There is: https://clinicaltrials.gov/ I have a vague memory or reading somewhere that if you don't register your desired success condition in the trial ahead of time, even if you find something it's not considered a real result.

Fomite4• 7 years ago

For some studies this works really well - clinical trials, for example. But a lot of studies don't fit into a clean "Single experiment to measure a single outcome" paradigm, and pre-registration becomes somewhere between difficult and impossible.

kradroy• 7 years ago

My advisor was guilty of pushing me to do #2. "Your original hypothesis was disconfirmed? Find one that the data does not disconfirm and that'll lead to a follow-up paper!" I dropped out of grad school because I assumed this was the norm.

Aperocky• 7 years ago

That may not be a wrong assumption.

unlinked_dll• 7 years ago

I feel like evidence dis-confirming a hypothesis is as valuable as confirming it. How many other grad students are going to attempt the same experiment/concept if null results are never published?

I wish there was more praise for negative results in publication, because whether confirmed or not, the knowledge has value.

bhk2• 7 years ago

I would think that #3 is actually easy to solve by increasing the statistical standards.

#1 and #2 are difficult to solve, given the near-exponential growth of funding for academic research.

https://www.researchgate.net/figure/Evolution-of-the-number-...

tikej3• 7 years ago

#1 and #2 are difficult to solve, given the near-exponential growth of funding for academic research.

Could you show any data that supports the claim about nearly exponential growth of funding for academic research?

AFAIK the funding, especially considering growing number of people in the field, gets rather worse than better.

Gatsky2• 7 years ago

Point 2 is fine if you perform orthogonal validation. In biology, it is not possible to formulate hypotheses prior to gathering data in all cases. What was the hypothesis for the human genome project? What hypothesis could you conceive before ‘looking’ at the genome?

syrrim• 7 years ago

I don't think people understand how bad the third one is. If 100 labs try an idea and only one publishes, then clearly we can get a spurious p-value of .01. But what if 10 labs try it, and 7 publish their non-significant results. Then, a meta-analysis takes the results of those 7 studies and combines them together. Here the meta-analysis can get a spurious p-value of .01 with an order of magnitude less original studies and with most of them publishing their results. The culture of selective publishing and of parading any significant result no matter how small the effect size is incompatible with correct science.

FabHK2• 7 years ago

How would a meta-analysis wrongly arrive at a spurious p=0.01 when 7 of 10 labs publish their (non-significant) results?

aaavl2821• 7 years ago

A good example of no. 2 happened last week. Biogen, a large biotech company, announced it was resurrecting a previously failed drug for Alzheimers based on a post hoc analysis. The drug has potential to be the biggest drug of all time, so the stakes are quite high for the company. But the data they have to support their product is quite inconclusive.

I wrote an analysis of their data here: https://www.baybridgebio.com/blog/aducanumab-analysis.html

neumann• 7 years ago

3. That's why you should not rely on p-values!

dr-detroit• 7 years ago

Applied research has always been in fashion. Theoretical is always rare and underfunded. Many books on this subject have been written I recommend "A University for the 21st Century"

People in glass offices run everything in 2019 and the more of an expert you become in your field the more professional managers will feel you are a pest to be silenced/hated/removed.

buboard3• 7 years ago

I think we also need to facilitate the work of reviewers. With so many papers and vast literature, reviewing is a PITA and errors keep slipping through the cracks. I d like to use a simple wiki of "argument refutations" , where we can look up previous objections to arguments being made. This is useful work that is currently being lost in the email archives of journal editors.

It would also help if we could open up science to people outside the academia, and begin the process of de-pedestalization of academia altogether, but not in an unregulated, completely flat way - academic discourse cannot be done in facebook. We know it has to happen and it will happen but we pretend the current situation can last forever. Academia is turning to a place that sells indulgences.

RcouF1uZ4gsC3• 7 years ago

From a previous HN comment https://news.ycombinator.com/item?id=14022158 by dasmoth

“If you pay a man a salary for doing research, he and you will want to have something to point to at the end of the year to show that the money has not been wasted. In promising work of the highest class, however, results do not come in this regular fashion, in fact years may pass without any tangible result being obtained, and the position of the paid worker would be very embarrassing and he would naturally take to work on a lower, or at any rate a different plane where he could be sure of getting year by year tangible results which would justify his salary. The position is this: You want one kind of research, but, if you pay a man to do it, it will drive him to research of a different kind. The only thing to do is to pay him for doing something else and give him enough leisure to do research for the love of it."

-- Attributed to J.J. Thomson (although I've not been able to turn up a definitive citation -- anyone know where it comes from?)

itcrowd2• 7 years ago

The quote is part of a speech given by J.J. Thomson in 1916. The speech is partially reproduced (including your part) in the book "The life of Sir J.J. Thomson, O. M. : sometime master of Trinity College Cambridge" by Lord Rayleigh (1942). The original book is on the Internet Archive [1]. The speech is given at pages 198-200 and your quote appears approximately halfway in the quoted part of the speech [see 2].

[1] Full biography: https://archive.org/details/b29932208

[2] Screenshot of relevant quote: https://imgur.com/a/qicgorD

RcouF1uZ4gsC3• 7 years ago

Thank you so much for finding this. I am so happy to know that this is a real quote and not a mis-attribution to a famous person.

"Not all famous quotes are by famous people"

- Abraham Lincoln

z3c0• 7 years ago

As someone who's currently trying to carry their company into data science (I know, we're late to the party, but homebuilding usually is), this is something that keeps me up at night. I've been hammering the idea into my superiors that if we're going to effectively administer a scientific process, we have to not only change how we manage expectations, but really, throw them out entirely. We have to see the general pursuit of knowledge as it's own ROI, because odds are that tangible value won't come from a data science team for a long time.

But having seen how poorly we adopted agile, I'm skeptical this will happen (we devolved to "waterfall w/sprints" due to not properly managing expectations with our users). If we can't get people to move away from hard deadlines for regular releases, how are we going to make them wait for a properly verified hypothesis?

One solution I've seen is organizationally "hiding" science teams from users, and only allowing a select few drive the direction of the team. But it still comes down to that select few properly managing everyone else's expectations.

gowld3• 7 years ago

Why is homebuilding company is the right place for basic research? That's an interesting choice to join the echelon of Xerox PARC, Bell Labs, Microsoft Research, and Google X.

gowld3• 7 years ago

This quote is better than the whole OP article.

However... A flaw in Thomson's idea is that some research is expensive, costing much more than leisure time.

Another approach is angel-investing / business-loan style: Give a large grant, wait a long time, and then call in the note -- to return research that justifies the investment, or repayment with interest.

Or X-prize style: Give outsize awards for outsize accomplishments, and let the researches take on the risk.

dnautics2• 7 years ago

Give a large grant, wait a long time, and then call in the note

And what are the consequences if the scientist has pittered that money away? How do you decide if a scientist/manager is even deserving of that level of confidence in the fist place

kbenson• 7 years ago

Taken to an extreme, this is UBI, and I think we would all benefit to some degree from a system like that if we could get from here to there. On the other hand, I imagine there's quite a bit of research that requires expensive tests and/or apparatus to get right, and I'm not sure how you feasibly come to a situation where you have a bunch of hobby physicists on staff and provide them with a particle accelerator, but just for funsies.

I think allowing people to explore their passions is essential to bringing new ideas into fields, and will likely bring a renaissance to some areas of study, and at least portions of other fields, but I'm at a loss as to how it can help at the forefront of fields that require a lot of investment. Rocketry, for another example. Can anyone make a realistic case that another 10,000, or even 100,000 passionate people could achieve what Space X has over the last few years? I don't doubt they come up with many or all of the same ideas, but the testing of those ideas requires a lot of money.

zdragnar• 7 years ago

There are plenty of people passionate about perpetual motion and chemtrails. Until we get to a point where resources are effectively infinite, we still need some form of prioritizing research funding.

Even then, playing kerbal space program does not make you a rocket scientist. It requires a certain dedication to master various physics and mathematics disciplines to contribute to even a part of one, which is itself a significant investment that you won't find possible with UBI alone; there will still need to be some form of dedicated funding through either specific government programs or commercial enterprise.

Salary also has a bit of a sticky effect; people change fields less than jobs, and spend more time in jobs than in funsies hobbies. A rocket designed by a committee of hobbyists will likely perform much like any other design-by-committee process or product.

jimhefferon• 7 years ago

UBI

What is that?

Fomite4• 7 years ago

There have been suggestions that doing science funding via lottery (with some caveats) would be more effective and more efficient than our current grant proposal based system.

lainga• 7 years ago

Well, taken to another extreme, it's Philip Glass working as a plumber and taxi driver (as he did) between compositions.

ROBERT Hughes, the Australian art critic, filmmaker and writer, wandered into the kitchen of his fashionable loft home in New York’s SoHo to see how the plumber was going, setting up his new dishwasher.

On his knees grappling with the machine, the plumber heard a noise and looked up.

Hughes gasped: “My god, you’re Philip Glass. I can’t believe it. What are you doing here?”

Glass, one of the world’s most famous composers, said afterwards: “It was obvious that I was installing his new dishwasher, and I told him I would soon be finished.”

“But you are an artist,” Hughes protested.

Glass said: “I explained that I was an artist but that I was sometimes a plumber as well, and that he should go away and let me finish.”

mjklin• 7 years ago

The Make-Work Principle: As long as you pay people to do something, they will find something to do.

(from the book The Truth About Everything by Matthew Stewart)

zmmmmm• 7 years ago

The only thing to do is to pay him for doing something else and give him enough leisure to do research for the love of it.

heh, am involved currently in exactly the process of constructing such an environment. And we are doing exactly that: there is a high value deliverable that is relatively easy and predictable to achieve (but requires such specialised knowledge that it cannot be attained any other way than through very rare and highly skilled people), and then to attract actual good people, the people themselves are free to take the rest of their job / time and apply it to solving whatever they see as the most valuable contribution they can make on any time scale that is relevant.

On the one hand you can look at it as an embarrassing political fig-leaf, on the other, you can actually see it as optimal. I acutally don't think completely detaching people from obligations to reality often works out to be ultimately optimal anyway. You need something to calibrate what you are doing against.

mlthoughts20184• 7 years ago

I’ve come full circle on this topic through grad school and dissertation and working in product companies.

I think “fundamental research” is not a thing, except possibly in mathematical theory. There is only incremental research + good salesmanship.

Research results should show up at a predictable regular pace, or else the result of that effort should be better & deeper explanation of a negative result which is also valuable.

If you’re sinking 2 years of R&D costs on something and you’re not getting the positive result you wanted and the negative results are not incrementally adding up to a clearer and clearer diff between the state you’re in and the state you’re trying to get to, then it is wasted money and the researcher isn’t being effective in their job.

I really think even difficult research tasks need to be rescoped and broken up into a series of incremental challenges, each of which has a known way to address it. You must treat it with reductionist dogma and eventually you’ll keep breaking it down into constituent parts that have known solutions until you hit on the novel problems to solve and it will be at a level of scope small enough that you can infer the solution from existing methods.

That’s all there is in the world. There’s no miracle cure for cancer or climate change or aging or social inequality. Let alone random business problems. There’s just a big bunch of little tiny problems with boring solutions that get all glued together into bigger messes that are hard to figure out. You can try smashing with a hammer or basically scaling the hammer up or down, that’s about it.

edits: fixed typos

ssivark• 7 years ago

That’s all there is in [the] world.

That seems like a blanket assertion. It certainly seems like the most "manageable" mode of operating, but looking back at human history it seems like the "big breakthroughs" rarely happened that way.

The model of scientific research as harvesting the tail events (low probability & massive payoff) seems quite incompatible with what you've said.

That said, you probably think the way you do because of your experiences. Can you articulate that better?

Research results should show up at a predictable regular pace, or else the result of that effort should be better & deeper explanation of a negative result which is also valuable.

To make a slightly more provocative claim, I think that this fetish for predictable/steady research progress is one of the primary causes of the problem discussed in the article. While papers can be generated steadily, piling on details doesn't necessarily make insight. (To quote Alan Kay: "A change of perspective is worth 80 IQ points")

PS: There's an apt SMBC comic (Aaargh, I'm unable to find right now!) where researchers keep digging a tunnel linearly, and declare the field dead, while there is a big gold mine slightly off to the side.

8bitsrule2• 7 years ago

A fine example of this is Jansky's discovery of stellar radio emissions in the early 1930s. He found them while he was working for Bell Labs. Neither Bell Labs (of all places) nor astronomers were interested in further investigation.

That discovery would lead to radio astronomy ... no thanks to established interests.

repolfx• 7 years ago

But Bell Labs is remembered for things like the transistor, which had huge impact.

There's a basic assumption you're making here and which underlies a lot of the writing on this topic - that it's desirable, even morally virtuous, for research funding to be disconnected from application.

Bell Labs funded the transistor and not radio astronomy because apart from making cool TV documentaries, radio astronomy isn't actually useful for much. If we knew how to travel faster than light and explore the universe it'd be extremely useful, but we don't, so learning things about what a remote corner of the galaxy looked like a few billion years ago is easily argued to be a rather absurd waste of limited research dollars.

It's exactly what this op-ed in the Scientific American is talking about: a research field optimised to produce papers independent of any concrete economic utility function. In a world where such things get funded, what exactly should scientists be measured by? They can't be measured by market success because nobody cares or has any use for their output: their work is pure academic navel/star-gazing. So they pretty much have to be measured by volume of output or respect of their peers, both of which are closed and circular systems of measurement.

In my view the right fix for the science crisis is not to pay scientists to research whatever the hell they like with no success measurement at all: that really is directly equivalent to just firing them all and putting them on social security (or "UBI" as HNers like to call it). The right thing to do would probably be to just slash academic funding dramatically and reduce corporation taxes so corporate research can be given more funding. The net result would still be a drop in the amount of science done, but as Bayer's study makes clear, "not enough science" is not the world's problem right now.

6gvONxR4sf7o• 7 years ago

This doesn't work for funding things besides the person's salary. If I want to buy a fancy new microscope or build a collider or even just buy materials, and I don't want to pay for it myself, I'm gonna need funding above and beyond "leisure." The LHC and ISS aren't leisure projects. Even equipment with a less insane scale is super expensive.

DayDollar• 7 years ago

Actually, i think some part this bad science and low innovation rate- is conciousness. Goverments everywhere realized, that unless they have total social control, people can not be trusted with exponential powerfull technology of all kinds. So, i predict, that unless we as humans are not totally surveilanced and controlled- we will not see great leaps of technology in the forseeable future.

rapjr9• 7 years ago

When I was doing laser spectroscopy research as an undergrad my prof said there was an informal physics "association" that had taken an oath not to develop anything like another atomic bomb. He asked if I wanted to join and I said yes. So I certainly see some basis for people being scared of fundamental advances in science and technology and wanting to control and place limits on what is developed. There have been a lot of advances in astrophysics (little potential harm to things here on Earth) and not much research in gravity (big consequences if we could actually control gravity.) That doesn't require "total social control", just a strong influence at key checkpoints (like funding). Whether there is an active effort to reduce investigation in specific areas of science might be discoverable by a meta-analysis of papers showing what areas of research tend to not get funding and see if there is a correlation with the potential for social disruption.

rwj• 7 years ago

There are other ways in which the incentives are misaligned. Granting agencies that prefer to fund researchers who nearly always "succeed" in proving their hypothesis will find that research proposals are low-risk and low-information. At in at least one sense, the ideal experiment is one with a 50/50 chance of failing.

Disconfirmation is also important.

klyrs2• 7 years ago

I've always found it distasteful that the Clay Math Prize offers a million dollar reward for proving that P=NP but nothing for revealing the truth should it be otherwise.

bonoboTP3• 7 years ago

I think you misunderstood something. The problem itself is sometimes referred to as the "P=NP" problem, but that's just a name, it is also called the "P vs. NP" problem. If you can prove either P!=NP or P==NP, you get the prize.

Can you give a reference to back up your interpretation?

majos2• 7 years ago

Your interpretation is correct. The official problem statement [1] is "does P = NP?", so either "P = NP" or "P != NP" is a valid answer.

[1] https://www.claymath.org/sites/default/files/pvsnp.pdf

klyrs2• 7 years ago

Can you give a reference to back up your interpretation?

Alas, no. I do recall being astonished at my claim, and then being convinced by a colleague (which was backed up by plain language on the CMI page, in my memory...) but now that I'm re-reading (current and archive.org'd) I cannot find such a thing. Disturbing. Yet relieving. Fuck my memory.

hardtke• 7 years ago

I've noticed a trend across many scientific fields where publications are basically irrelevant since nobody reads them. Status is conferred by high profile talks, which tend to be invited. The material discussed in high profile talks is eventually published in a journal or conference proceeding (in the case of CS) and will get citations, but ideas only spread if they are picked by the program committees. The academics who publish many low impact papers are not really relevant.

Fomite4• 7 years ago

This is not applicable to a lot of fields. Generally, only CS and ancillary fields are ones I've encountered where talks and conferences are the primary "currency". In biomedicine, talks will only follow after publications, and are often intentionally paired.

notadoc• 7 years ago

Not surprising when science has become a tool for political and industrial purposes.

glofish8• 7 years ago

The real problem with science is the same as in politics.

We, the other scientists (just like the voters) incentivize certain behaviors and with that favor a certain type of scientists (and politician) to prosper.

There are all kinds of scientists (and politicians) competing for your trust (and votes). There are good and bad among them. As long as we reward the bullshitters more, they are the ones that will outcompete the others.

All these rules and regulations that people propose are ineffectual, as long as a certain level of self-criticism is not being applied:

- Stop believing and propagating the bullshit even if it seems to support your preconceived notions (or even the truth). This is very hard to do in practice.

As sad as it sounds: the greatest enemy of good science are the other scientists.

feels like some sort of prisoners's dilemma, it only works if all do it, otherwise, it is best to just not admit anything

leftyted• 7 years ago

A different framing of this is that politics is, more and more, intruding into science.

Paul Romer:

Politics does not lead to a broadly shared consensus. It has to yield a decision whether or not a consensus prevails. As a result, political institutions create incentives for participants to exaggerate disagreements between factions. Words that are evocative and ambiguous better serve factional interests than words that are analytical and precise.
Science is a process that does lead to a broadly shared consensus. It is arguably the only social process that does. Consensus forms around theoretical and empirical statements that are true. In making these statements, a combination of words from natural language and tightly linked symbols from the formal language of mathematics encourages the use of words that are analytical and precise.

from https://paulromer.net/mathiness/Mathiness.pdf

tobias3• 7 years ago

He also left as World Bank chief economist after he complained that the "Ease of doing Business" index suspiciously went down for Chile while a socialist party was in power.

Dowwie2• 7 years ago

Every time an arxiv paper gets published to HN, I wonder whether it's even worth reading until findings are peer reviewed..

enjoyyourlife• 7 years ago

*worth

Dowwie2• 7 years ago

thx

jplayer012• 7 years ago

At this point, what even is the point of good science if nobody believes it?

tasty_freeze• 7 years ago

If what you said was literally true, that nobody believes it, then there would be little point. But "nobody" is an exaggeration. Scientists are likely to believe it, or at least are in a much better position to evaluate the truth and utility of it, and build upon it.

When it comes to abstract science, eg, the ultimate origin of the universe, it doesn't materially affect anything if it is believed or not. But if someone produces a cheaper or longer lasting battery, the proof is in the pudding, and that basic research will have made a difference.

Then there is the category of hard science which is disbelieved because moneyed interests wish to discredit it, and/or it has become a political shibboleth to discredit the science. Those aren't due to bad science.

jplayer012• 7 years ago

The problem is that significant portions of the population refuse to believe anything that comes out of science. This directly affects what politics decides to do about real problems like climate change. This isn't some abstract issue, it's something that's affecting us on an everyday level because people are unwilling to listen to the evidence.

buboard3• 7 years ago

People are dying to believe in science , even when it fails repeatedly or drags on for decades - or else the BBC would have stopped reporting on new cancer cures, because people have had enough. They re moslty disbelieving of big narratives built on the science and things that look like FUD, and even more when emotional argumentation blankets the scientific arguments.

stevenwliao• 7 years ago

We need to incentivize thorough reviews as well.

Al-Khwarizmi2• 7 years ago

I agree with everything except for the jab at (paid) open access, which looks quite gratuitous to me. It may be true that "authors are willing to pay more to get their articles published in more prestigious journals. So, the more exciting the findings a journal publishes, the more references, the higher the impact the journal, the more submissions they get, the more money they make". But this is also true of non-open-access journals. Journals live off their prestige, and before paid open access was a thing, publishers still wanted to have high-impact journals so that university libraries would subscribe to them. It was a very distorted market (as it is now), but a market nevertheless.

While I often bash for-profit journals for being parasites that do little actual work and profit from withholding access to science that should be public, and for this I would open a bottle of champagne if they disappeared, I don't think journals have much to do with this particular problem of incentivization of bad science. Journals just respond to the demand of publishing more, and shallower, papers. That demand comes from hypercompetitivity in academia, where researchers need to fight for scarce positions and scraps of funding, often paired with too much bureaucratization (selection processes that look at "objective" and "verifiable" metrics like number of papers published at a given impact factor quartile, etc., instead of just asking a bunch of neutral experts whether the person is doing good research, which may be more opaque but also much more meaningful).

As evidence that journals are not the problem in this particular case, in fields like machine learning, where publication happens mostly in arXiv and conferences that don't charge for publishing or reading papers, the problems pointed out in the post also exist. Published models that only beat previous ones because they were lucky with random seeds or data splits are widespread.

gowld3• 7 years ago

researchers need to fight for scarce positions and scraps of funding

there is quite a lot of funding

often paired with too much bureaucratization (selection processes that look at "objective" and "verifiable" metrics

this is the problem.

The whole OP reads like a bizarre hit piece on open access. How could scientists paying to publish their work incentivize them to publish more? How would spamming the world with more publications inflate a scientist's impact factor? (It wouldn't -- impact factor would be diluted by the spam)

It was always possible to self-publish and to cite self-published work, and even without journals, a modern scientist can publish on a free webhost for even cheaper than an open-access journal.

lutorm2• 7 years ago

How could scientists paying to publish their work incentivize them to publish more?

I assumed that the point was that the journal is incentivized to publish as many articles as possible, and hence to lower review standards.

ipsa• 7 years ago

Television incentivizes forgetable reality TV, the radio incentivizes meaningless poppy music, social media incentivizes bickering about the controversies of today.

But, nowadays, you can also use your TV to watch a French arthouse film, to go to Youtube and be recommended a Japanese jazz album from 1974, to join the conversation on Twitter and ask questions to leaders in their respective fields.

Now you can swim against the current: Force all these power - and money - hungry institutions to fundamentally change their tune. Or you can find one of the many new waves to surf. Life is good, science is good, progress is good. The choice, as a scientist, is up to you. Can't write one groundbreaking paper a year? Write two or three mediocre ones. No amount of foundational change is going to make you a groundbreaking scientist. And change the channel once in while: the world is only getting bigger and more connected.

Gatsky2• 7 years ago

Yes well every solution we come up with invariably favours well established scientists... how can a new scientist get anywhere if it takes 10 years to produce substantive work?

There is a conceit in the final paragraph, where it is implied that we are missing out on cures for diseases etc due to wasteful scientific endeavours. This is not necessarily true. There have been many successes in the current era of medical science. Generally these are driven by technological advances such as monoclonal antibodies or next-generation sequencing.

cbkeller2• 7 years ago

The article makes some fair points, but strangely seems to imply that open access is somehow opposed to rigorous peer review, which certainly leaves an odd taste in my mouth.

Of course, scientific publication is subjected to a high degree of quality control through the peer-review process, which despite the political and societal factors that are ineradicable parts of human interaction, is one of the “crown jewels” of scientific objectivity. However, this is changing. The very laudable goal of “open access journals” is to make sure that the public has free access to the scientific data that its tax dollars are used to generate.

xvilka• 7 years ago

Partially can be solved with technology - all data and programs should be self-contained, interactive, edition and modification process should be visible for general public, better connections between papers rather than current citation mechanism.

angry_octet• 7 years ago

A previous Great Leader decided that he would improve our research output in a simple way: we should stop doing humdrum work, and only do work which would succeed famously.

Of course, there is some difficulty in determining which work is going to be brilliant before it is done. But he decided that he could do that, seemingly based on how PR worthy the proposal was.

At any rate it did immense damage and set back deep research by years. Naturally he left when he could wangle a better job elsewhere.

rblion• 7 years ago

Self-regulation by scientists of decades and centuries past has created modern science with all its virtues and triumphs. However, much like the bankers of the early 21st century, we risk allowing new incentives to erode our self-regulation and skew our perceptions and behavior; similar to the risky loans underlying mortgage-backed securities, faulty scientific observations can form a bubble and an unstable edifice. As science is ultimately self-correcting, faulty conclusions are remedied with ongoing study, but this takes a great deal of time.
Unless and until leadership is taken at a structural and societal level to alter the incentive structure present, the current environment will continue to encourage and promote wasting of resources, squandering of research efforts and delaying of progress; such waste and delay is something that those suffering diseases for which we have inadequate therapy, and those suffering conditions for which we have inadequate technological remedies, can ill afford and should not be forced to endure.

I agree.

We're Incentivizing Bad Science

Related Stories