How a word sparked a four-year saga of climate fact-checking

https://theconversation.com/how-a-single-word-sparked-a-four-year-saga-of-climate-fact-checking-and-blog-backlash-62174

by nl • 10 years ago

61 59 10 years ago

wolfgke10• 10 years ago

I see a large problem in the fact that (as it seems) the original data and the computer code is not published, too. This is in my opinion a contradiction to the scientific principle that any result or conclusion has to be able to verified independently as much as possible. Though not publishing data and computer programs (and ideally a virtual machine for independent execution of the program code, too) is unluckily accepted practise in science.

If it were otherwise any critic of the result could reapply the computer program to the non-detrended data and see that the conclusion will not change. On the other hand, if the computer program and data is not published he has to trust the conclusions that the researchers came to. So the only way for an independent person to check the result is to find indirect evidence that the conclusion could be wrong. And finding out that a different data set than (wrongly) claimed was used is exactly such indirect evidence (and in my opinion nearly as far as one is able to if one doesn't have access to the used data and computer programs).

DasBlob• 10 years ago

While I certainly share your sentiment that both, data and analysis software, should be available, I do see a conflict of interest here that cannot be easily resolved. On the one hand, we do assert (wrongly, IMO) that competition is a necessary element to drive scientific innovation and consequently require scientific actors to adhere to economic principles, i.e. publish more and better-received papers. On the other hand, allowing researchers to keep their competitive advantage, and having exclusive access to research data certainly is, contradicts the principle of reproducability and unneccessarily restricts further research that could be done based on that data. I don't see a way to easily resolve that in the current scientific framework. One (less than optimal) approach, that is frequently taken in astronomy, is to allow exclusive access to telescope data to the members of the consortium or the principal investigators for a certain amount of time, after which it is released to public. As for the software, it's easier. if the analysis method is sufficiently well documented (and it should be in any publication), people with access to the original data can easily attempt to reproduce the results and either way they'll get a publication out of it. So there is no incentive to keep the analysis software but there is a huge incentive to keep the original data.

wolfgke10• 10 years ago

The solution should rather be that society (and publishers) does not consider papers (or more generally: texts) as research that don't also publish everything possible for independent verification (in this case the source code and raw data). If this is not the case, the text is just a plain marketing statement (in contrast to research) comparable to the marketing statements that are made everyday by lots of companies.

tremon• 10 years ago

both data and analysis software should be available

I think that goes too far. The data alone is sufficient for independent verification, I don't think we should mandate that scientists publish their tools. However, I would very much support scientists using open tools to begin with. But if the tool is closed, that should not invalidate any results in the paper.

ajford2• 10 years ago

I agree on the propriety of data for a limited time. I worked in the field of Radio Astronomy, and it was common to have a two year period where the data was private to the Primary Investigator (and co-investigators/consortium). I think this is a relatively acceptable approach, given how long it can take to process and fact-check data and papers.

I'm also for the publishing and tracking of code, as in many fields the data is becoming so vast that it's impossible to process or handle manually, and many pipelines and large scale systems are being built to handle this data. This inevitably means that a simple bug in code (or false/incorrect assumption by the author) can result in many lost hours and years of work.

But publishing code can also lead to biases, if there is an unintentional bias in released code or approaches that become popular within a given field.

In the field I worked in, there were two very common pipelines used by a huge majority of researchers, and I always wondered if there might be some unintentional selection effect being introduced. Well, it turns out someone else thought so too, so there's now an effort to rewrite one of these tools from the ground up without using any of the original code. It's only something like 60% complete right now, but hopefully soon it will be complete and we'll get to see if results compare, or if we spot any unintentional effects.

jackmott4• 10 years ago

any result or conclusion has to be able to verified independently as much as possible

Well, collecting the data yourself and writing the code yourself would be even more independent.

wolfgke10• 10 years ago

Well, collecting the data yourself and writing the code yourself would be even more independent.

I wrote "any conclusion". This does not preclude collecting the data or writing the code yourself (and indeed you should if you doubt, say, the data). But it implies that the code that was used to come to the conclusion must be available.

jkot4• 10 years ago

Meanwhile, our team received a flurry of hate mail and an onslaught of time-consuming Freedom of Information requests for access to our raw data and years of our emails, in search of ammunition to undermine and discredit our team and results.
The mammoth process involved three extra rounds of peer-review and four new peer-reviewers. From the original submission on 3 November, 2011, to the paper’s re-acceptance on 26 April, 2016, the manuscript was reviewed by seven reviewers and two editors, underwent nine rounds of revisions, and was assessed a total of 21 times – not to mention the countless rounds of internal revisions made by our research team and data contributors. One reviewer even commented that we had done “a commendable, perhaps bordering on an insane, amount of work”.

So one side you do this "mammoth" and "insane" task to review paper. On other side you block people from doing independent audit?

Climate studies decide how billions (if not trillions) of dollars are spend. People should be allowed to cross examine raw data and program source code.

lostlogin• 10 years ago

If its coming with hate mail and or has a generally offensive tone, I don't think it unreasonable to be unhappy. The author doesn't appear to have blocked access. One can cross examine without being a dick.

jkot4• 10 years ago

Read linked article, offensive language was used AFTER they refused to provide raw data. It took lawsuit to get them.

bertil2• 10 years ago

My understanding is that for climate science, a lot of effort is spent to debunk claims, by well-financed groups who will publish without independent peer-review but with a clear goal in mind. Allowing those groups to access the same data means that the researchers will have to spend a lot more effort explaining why those claims are unfounded. That means the researchers are not really encouraged into sharing the data.

There is a more general issue in science or any data-intensive practice (Google ranking is another example) where the argument “People should be allowed to cross examine raw data” fails to acknowledge that ‘People’ generally do not have the ability or know-how to, and those who can generally have interested financial backing. In many cases, it is positive (Open Source Map is a good example of that: my employer is interested in identifying, say, bike parking spots) but there are examples where de-anonymisation, competitive pressure, make “sharing data” a pragmatic question that needs more than principles.

Will more people who can contribute positively to climate science help if they had free access to those data? It is the case for astronomy; I am less convinced for climate science.

jkot4• 10 years ago

Paranoid talks about "well financed groups" have no place in science.

Climate science is way more important than astronomy. It should have much higher standards for transparency, not other way around.

wolfgke10• 10 years ago

Will more people who can contribute positively to climate science help if they had free access to those data?

I think if we only consider it as science if it can be verified independently, also the "opposite side" (your "well-financed groups") has to provide evidence of the same high standard, which would strongly increase the level on which the debate is pursued (i.e. the opposing side also cannot get away with outlandish "evidence").

jackmott4• 10 years ago

There has been enough raw data and code out there for an amateur to confirm that global warming is real for decades. I did it myself ~10 years ago. Here is a small subset of data and code that is public domain:

http://www.realclimate.org/index.php/data-sources/

dzdt• 10 years ago

Global warming denialists are attempting a denial-of-service attack against global warming science. The goal is to keep the science unsettled so no major policy changes are made while those making the most profits from climate-changing activities continue to do so.

titzer• 10 years ago

I was about to respond to your comment and argue that some of them generally have good intentions, and that your comment was paranoia about a conspiracy....but then I realized you pretty much nailed it. I used to doubt climate science...15 years ago. Now I don't.

socialist_coder3• 10 years ago

Yup; just like tobacco, acid raid, DDT, and the hole in the ozone layer. Very very sad. It shows you how powerful greed is, when people are willing to sacrifice humans lives and even the future of our planet just to maximize profits.

This is one of the main failures of our capitalist economy, IMO. The system needs a built in way to assign a value on sustainability and human well being, rather than just profits.

socialist_coder3• 10 years ago

See https://en.m.wikipedia.org/wiki/Merchants_of_Doubt for mor on that, if you are interested. Very good book.

SeanDav4• 10 years ago

No idea why this is a surprising, even controversial result. We are are on the warming side of a recent (geologically speaking) ice age (actually glacial period) and we expect long term temperatures to be trending ever upwards.

phaemon2• 10 years ago

No we're not. The complete opposite in fact: the planet would naturally be cooling now were it not for the increased amount of greenhouse gases in the atmosphere.

SeanDav4• 10 years ago

"Technically speaking, we’re living during an ‘Ice Age’ today – in the sense that we have a world with glaciers. However, the present time is a relatively warm phase within a period of geological time when glaciers and ice sheets have typically been larger and more extensive than now."

Taken from: http://www.rgs.org/OurWork/Schools/Teaching+resources/Key+St...

There are other sources, but this is one example that makes it clear we have been in a warming phase the last few thousand years.

frankmcsherry2• 10 years ago

I'm personally inclined to support the climate change folks, being a scientist and stuff, but it's a bit crap to claim that it was a one word typo when the original text apparently read:

For predictor selection, both proxy climate and instrumental data were linearly detrended over the 1921–1990 period to avoid inflating the correlation coefficient due to the presence of the global warming signal present in the observed temperature record. Only records that were significantly ([pre]p<0.05[/pre]) correlated with the detrended instrumental target over the 1921–1990 period were selected for analysis.

That isn't a typo, it is just being wrong. I'm glad they re-did the work to confirm it was still true, but it's hard to argue with folks who didn't think the claim was supported by the data.

dozzie2• 10 years ago

Well, what you expect from people who happen to turn around in their (unverifiable) models 180 degrees (Amazon jungle net produces oxygen! No, it's a net consumer!) and who routinely produce models that use some physical value (heat capacity in greenhosue effect), but fail to account for major substances influencing the value (water concentration in atmosphere)?

jackmott4• 10 years ago

I'm not an expert on the history of opinions on amazon oxygen, but

but fail to account for major substances influencing the value (water concentration in atmosphere)?

is absolutely false.

jasonellis• 10 years ago

The one word typo was in the code, not the paper. It was setting DETREND to FALSE instead of TRUE. Their assertions were based on thinking they were working with detrended data instead of the raw data that they actually used.

frankmcsherry2• 10 years ago

In which case, one shouldn't write

Instead of taking the easy way out and just correcting the single word in the page proof, we asked the publisher to put our paper on hold

barney54• 10 years ago

For another view of this issue: https://climateaudit.org/tag/gergis/

patrickg_zill• 10 years ago

General observation is that neither the mathematical models nor the data is perfect.

For example, oceanic thermocline data, from what I understand, is not really collected.

We have an imperfect model being fed incomplete data? Why be surprised at debate over it?

People laugh today at ether and phlogiston debates, but such acrimonious debate pushed forward science to the point where we had exactly the right answer.

sxcurry• 10 years ago

The problem is that we're not really having debate, as this article makes clear. We are having scientists attacked by 2 groups: climate deniers funded by large corporate masters, and random groups of crackpots. Real debate would be good but I think we're well past that point.

How a word sparked a four-year saga of climate fact-checking

Related Stories