Mirroring US climate data

http://www.climatemirror.org/

by pirocks • 9 years ago

448 162 9 years ago

trothamel• 9 years ago

Can I be the first to claim that the people running the mirror are changing the data for their own nefarious purposes?

Not that I believe this or anything, but come on - you know that accusation is going to go around. It might make sense to use a remote server to timestamp files as they're uploaded, or something like that.

nirav72• 9 years ago

When there is a broad consensus across the scientific community about climate change, does it even matter at this point if the data has been changed? The U.S isn't the only country that collects this data. So even if changed, it doesn't matter.

ted_dunning• 9 years ago

Actually it really does matter. The US collects far more of this data than anybody else and archives far more of it.

Losing that collection or archive would be a disaster for science and a loss for all of humanity. Destroying that data would be reminiscent of ISIS destroying relics in Palmyra except that destroying that data jeopardizes our future.

lkbm• 9 years ago

When there is a broad consensus across the scientific community about climate change, does it even matter at this point if the data has been changed?

There's more to climate science than the yes/no of whether climate change is a thing. We don't just need to know it's happening--we want to know details of how to model it, what different inputs have what affects, what the feedback loops are and how strong they are, how they and other effects interact...

The U.S isn't the only country that collects this data. So even if changed, it doesn't matter.

Many countries collect data, but presumably the sets of data collected differ, and more data is better. Losing a large subset of our climate data would hurt advances in climate modeling.

pdonis• 9 years ago

When there is a broad consensus across the scientific community about climate change, does it even matter at this point if the data has been changed?

It does if the data that was changed is the data that was used to support the consensus.

More generally, all data collected by research that is funded by taxpayers should be collected, cryptographically signed and verified, and archived where any taxpayer can see it. I have never understood why that is not already standard operating procedure.

gus_massa2• 9 years ago

Consensus without data is indistinguishable from religion.

battlebot5• 9 years ago

There was once "broad consensus" that the world was flat. I am not at all comfortable when there is so much consensus. There needs to be debate, disagreement, and healthy skepticism at all times.

dawnerd• 9 years ago

On the actual mirrors site you can see some have been signed and verified: https://climate.daknob.net/

Ultimately still comes down to trust. Still better to have a backup of data that MIGHT be poisoned vs not having it at all. If we have multiple copies from the original source we should be able to determine if somethings been messed with.

caseysoftware• 9 years ago

Before publishing, they could do a checksum, insert it into the blockchain and then voila. Easy to validate if the data has changed.

ajmurmann3• 9 years ago

I am not sure checksums and block chains are gonna convince those people.

esaym• 9 years ago

I would say the same thing. Do you think you will really convince a climate denier with this data that you copied off of the Internet that, was copied from a gov server, on to another server and then torrented and copied to another server and then downloaded onto you PC... doubt it.

tracker17• 9 years ago

It's worth noting that not everyone who rebuffs some of the proposed actions is a climate denier. There's still some question as to the extent of human involvement. That said, smog is enough for me to be in favor of working towards renewable, less polluting energy sources. In my lifetime I've gone from having allergy-like flare ups once every few years, to now 2-3 times a year, mostly seems to follow the pollution levels not just the pollen count.

By the same token, not every "green" program is a great thing... and even in treaties, exempting the worst actors (China, India) because they are often considered third world doesn't do a lot to help the global issues. Improved working requirements as a requisite for international trade deals would go a long way towards "cleaning up."

I bring this up, as I tend to see a lot of people who question the approach, intensity, and cost getting lumped in with climate deniers.

tukelully4• 9 years ago

How successful is the effort to convince clinate deniers as of right now? I see this as an archival effort. If the source is destroyed, this data, among everything else being archived, will still be around and publicly accessible by anyone who wants to use for any purpose. That is inherently important. That said, once the first part is well taken care of I'm a big advocate for building more refined systems for storing and verifying the information.

lawless123• 9 years ago

Do you think you will really convince a climate denier with this data

These people where convinced a pizza restaurant was the center of a global pedo ring based on nothing but deliberately misconstrued sentences they claim are code.

They'll believe whatever they want.

jefurii3• 9 years ago

I work for a digital archives project and we're concerned about similar issues of provenance. Our way of dealing with this has been to structure our archives as JSON-formatted text in Git repositories, with binaries managed by git-annex.

Git uses hashes for everything: files are placed in .git/objects/ by their hash, each commit lists the hashes of files in the working directory, and each commit points to the hash(es) of its parent(s)

Using Git it's possible to verify the integrity of the entire repository and its history, making it impossible to tamper with without leaving traces. It's also possible to have multiple copies (clones) of the data and to verify that they are exactly the same.

If you have a copy of such a collection you can compare it against a copy held by the originating institution. If that becomes impossible for some reason, then if you can track down any of the original files you could prove that that portion of the data is correct, which lends trust to the rest of the collection.

Of course, it's possible this would not convince a die-hard trump supporter. /s

laurent123456• 9 years ago

I think that's why they suggest to use Webrecorder, to "create verifiable web archive files".

anc842• 9 years ago

Run webrecorder yourself, MITM yourself, tamper however you like. Verifiability depends on a trusted authority first.

ocschwar• 9 years ago

No. The right wing troll community beat you to it months ago.

hackuser2• 9 years ago

We are working to set up a secure and completely anonymous file deposit system for the future.

Provide a mailing address to which to send hard drives.

sandebert• 9 years ago

Good idea, I posted it to their web form.

hackuser2• 9 years ago

It's great to see their efforts. What are you doing? What am I doing? Reading about it isn't nearly enough.

hunterwerlla2• 9 years ago

In torrent form: http://climatetorrent.com/ Torrents are a great way of maintaining data integrity and allowing future sharing easily.

dev1n• 9 years ago

Is it possible to add ftp://aftp.cmdl.noaa.gov to that torrent?

caterama• 9 years ago

Similar idea: http://academictorrents.com/

BHSPitMonkey• 9 years ago

Thanks, is this yours? It would be great if the homepage displayed a short description of the different torrents/datasets, their sizes/contents, and the number of seeders for each. That would greatly help me decide which one(s) to prioritize helping out with, as I don't have resources to keep a copy of it all.

hunterwerlla2• 9 years ago

I don't know who is behind this, I just saw it on /r/DataHoarder

danbruc9• 9 years ago

Having backups is obviously always a good idea but that aside, how likely is it, that mirroring this data is actually necessary? There are obviously at least some people believing that it is necessary but then there are always people convinced that the most silly things are necessary. I am really unable to tell whether this is more of an overreaction or an urgent necessity, especially as someone not living in the US. Is the future just [perceived as] so unpredictable, that better safe than sorry is the right mindset?

po2• 9 years ago

Besides redundancy, mirroring also helps spread the cost around. These groups tend to have pretty tight budgets.

jefurii3• 9 years ago

Lots of copies keeps stuff safe.

dylz• 9 years ago

Think of it this way: servers cost money, bandwidth power costs money.

When your budget is cut, servers will shut down at some point in the future. It isn't a "the gov says rm -rf everything"

danbruc9• 9 years ago

I totally see how data could become unavailable, I am more asking whether it just really became more likely to happen. It is certainly no mistake not to rely on single sources, but if that is the motivation, then there is nothing special about now and climate data and mirroring all kinds of data would have been as good an idea a decade ago as now.

crispinb• 9 years ago

If you only create redundancy when you are sure it's necessary, you're too late.

danbruc9• 9 years ago

Of course and I think having those mirrors is a good thing, stuff disappears from the internet all the time. But that is not the question I am asking. We could have started mirroring climate data and all kinds of other data last year or five years ago, there is always a risk of losing data.

But this has obviously been triggered by the Trump presidency and is focused on climate data which implies that people are now more concerned than before and especially about one specific kind of data. Now I would like to know whether or not this elevated level of concern is justified.

andygates2• 9 years ago

Ancient sysadmin cannot upvote this enough.

ajmurmann3• 9 years ago

Well the EPA had been asked by Trump's transition team to provide a list of employees who have attended conferences about climate change. See NPR story: http://www.npr.org/sections/thetwo-way/2016/12/09/505041927/...

They have also removed all mentions of climate change from the white House website and we now many Republicans don't believe in man made climate change. So I think this is an entirely reasonable fear. We are talking about a administration that thinks that getting coal out of the ground like it's 1910 is the future...

gus_massa2• 9 years ago

They have also removed all mentions of climate change from the white House website

They replaced the old website with a new website. The old website is stored in: https://obamawhitehouse.archives.gov/ This is usual. Obama toke the same measure. Here is the Bush website if you like it: https://georgewbush-whitehouse.archives.gov/

With the change, Trump (technical staff) removed many subject they disagree, and many subjects they agree with. If Obama had a page about how great are ginger hair people, it would have disappeared too.

paradite2• 9 years ago

I'd like to discuss a slightly different issue here.

Aren't there already a plethora of peer-reviewed journal articles on the issue of climate change? Why is it still necessary to store the presumably raw data?

Even my high school geography notes have an abundance of information on the issue of climate change (for and against). So I doubt if this is necessary at all.

untog2• 9 years ago

Why would you not? With storage costs being as minimal as they are I can't see a good argument not to store data that's under threat of deletion.

danbruc9• 9 years ago

For [climate] models for example.

JamisonM• 9 years ago

If some large number of those peer-reviewed journal articles' conclusions are (or appear to be) undermined by future discoveries having the raw data to confirm or refute such a development taking into consideration assumptions about the data that might have since changed or been proven incorrect seems like a pretty important thing.

What is in your high school geography notes was dictated in part by political decisions about school curriculum - you cannot assume that the future of even high school geography notes will match your experience.

tukelully4• 9 years ago

A contrived analogy for this might be something like "Why wouldn't you delete the data once you've rendered a graph of it in PNG?". Data is always important. Just as much so as it's original interpretation and constraints.

dagss• 9 years ago

That is a bit like asking what you need source code for after having compiled the binary program.

Just like you have new versions of a program building on changes to source code, you can run newer and improved analysis on the scientific data (whether to see if you get the same result, a new additional insight or whatever).

Journal papers are often no better than having the binary (or perhaps more accurately the stdout dump of running the binary...)

danbruc9• 9 years ago

Why was this comment by ajmurmann flagged? Up to now that seems the best evidence provided for answering my question even though I think it is relatively weak evidence and it lacks a source for the second paragraph.

Turing_Machine2• 9 years ago

I didn't flag it, but I would guess that this is part of the reason:

"They have also removed all mentions of climate change from the white House website "

As gus_massa points out above, the entire Obama White House website (not just the pages dealing with climate change) has been transferred to the National Archives (not "removed") and has been replaced with a completely new site.

While the web is still new enough that it's hard to argue tradition, Obama did the same thing to Bush II's site, and Bush II did the same thing to Clinton's site (Clinton was the first, so there's no data before that).

beatpanda• 9 years ago

"Lessons learned under former Canadian Prime Minister Stephen Harper, a Conservative who was accused of muzzling scientists during his term, initially energized a "race" to save the U.S. government's climate data. Activists also studied policy papers by members of Trump's transition team to get a sense of what might be at risk.

But what they have witnessed so far has been "unprecedented," Murphy said, and "much more intense and publicly celebrated" than the precedent set in Canada.

"The transition team and Trump administration are very upfront and promoting of this plan; it's not that this is surreptitious," Murphy said."

http://www.sciencemag.org/news/2017/01/trump-officials-suspe...

gillianlish• 9 years ago

This isn't about necessity. This is about sending a message.

Message to Scientists - we value your work

Message to the World - we are not all bad

Message to future generations - we are not hopeless

Message to Trump - good day sir

po2• 9 years ago

Awesome. Is there the same thing but for all public datasets? I saw recently someone showed a federal site had put up a (hastily hacked in) link to download all data on the day of the inauguration. I bet there are a lot of federal employees who would like a well-resourced site to upload to.

beams_of_light• 9 years ago

I'm sincerely dismayed that this is necessary.

general_ai2• 9 years ago

It's not.

chordatum• 9 years ago

Not yet.

r00fus• 9 years ago

The most worrying part is that this is a symptom of a much bigger problem. This is likely going to be one of the less egregious issues raised in the next 4 years.

koheripbal• 9 years ago

This is a great example of how fear of something possibly happening, gives a larger audience the illusion that it will happen.

Literally no one in the administration has even suggested, ever, that they'd delete climate data.

nandhp• 9 years ago

Is this project collaborating with Archive Team? They have quite a lot of experience with this sort of scraping+archiving project (e.g. Geocities, Google Reader, Coursera, etc.).

http://archiveteam.org/index.php?title=Main_Page

olivierlacan• 9 years ago

Feels like the most important things people with some spare cash can do to help this project is help them raise money to host the mirrored climate science data on online.net: https://www.generosity.com/volunteer-fundraising/climate-mir...

jszymborski2• 9 years ago

If you want to help towards archiving climate data and other gov't public datasets, join the ArchiveTeam effort. All you need is to run their ftp-gov-grab script and a bit of bandwidth. VPSs work very well for this sort of thing, so if you have a box lying around, just run the script.

More info at the wiki: http://archiveteam.org/index.php?title=Ftp-gov

Ftp-gov-grab: https://github.com/ArchiveTeam/ftp-gov-grab

Join us on IRC: #cheetoflee on freenode

kpcyrd• 9 years ago

Correction: it seems to be #cheetoflee on efnet.org

jszymborski2• 9 years ago

oops, thanks for that.

fsttx• 9 years ago

Another useful tactic is to archive news and commentary (left and right) about current political events, both for accountability and perhaps, to prevent the rewriting of history in the era of "alternative facts".

jl62• 9 years ago

And not just archive it, but sign it, time stamp it, verify it, etc.. I see little to prevent someone using ML to auto-generate archives worth of fake history. Imagine a purported DB dump of Reddit with billions of subtlely fraudulent comments scattered throughout.

rmc2• 9 years ago

Is there any good way to timestamp things like this?

DanBC• 9 years ago

Providing diffs would be useful. A bunch of places do silent edits, sometimes making significant changes to meaning of articles.

BBC does this, and it's freaking annoying.

lorenzhs• 9 years ago

There was a great talk about the datamining possibilities of such an archive at 33c3. English translation: https://www.youtube.com/watch?v=bYviBstTUwo

angry_octet2• 9 years ago

It is not sufficient to have a record of facts when others have alternative facts. To your signed record of facts they would just introduce alternate facts with alternative signatures and claim you had made it up. Normal people don't have the tools or experience to differentiate, and the alternative facts support a story they want to believe, rather than hard truths.

News media won't even mention your so-called 'real facts', facebook and twitter might not let others see them. Before you know it you'll be denounced by your neighbours, work colleagues and friends as a traitor, a madman. You'll lose your job, or your partner will. There may be a show trial and public denunciation. This has all happened before: the USSR, McCarthy.

If you don't want that you need an alternate means of distributing real facts, and countering fake ones. It has to be hosted so deeply in our tech infrastructure that it can't be firewalled or jammed. It needs to be accessible covertly. And it needs to record all the alternative facts too, because they like to change their facts.

Let's call it Samizdat.

wybiral3• 9 years ago

The US government can't make all of the real scientists out there just disappear. Climate models have an impact on many different fields of research and unless there's a total shutdown of science (which seems absurd) those people and facts will still be there.

There's a portion of the population who's wanted to deny climate change for a long time and I believe they'll accept the alt-facts to justify their position. But I'm skeptical about the efficacy of successfully silencing existing news groups and the majority of scientists.

Has anything like that ever been successful in the connected world? That being said, we do absolutely need to make protect our internet from censorship.

battlebot5• 9 years ago

You talk about facts like you don't live in a post-positivist world! You have no idea what you are talking about. "Facts" and "truth" are part of philosophy, but bear very little value in the real world. Just get used to it.

tostrstrudel• 9 years ago

I created a script that allows for periodic archiving of the text of the top articles on several large news sites. There's a configuration file that allows for selecting any RSS feed to pull from.

https://github.com/xtruan/newsbot

mynewtb• 9 years ago

This would be perfect for IPFS I think.

Tempest1981• 9 years ago

I heard that Trump has given NOAA 100 days to justify their existence, or face heavy cuts. Does anyone know the details?

Here is the closest article I could find: http://www.salon.com/2016/11/23/politicizing-climate-change-...

smarx007• 9 years ago

"Walker has hinted that what NASA loses in earth science research could be shifted over to NOAA."

polyfractal• 9 years ago

This gets thrown around a lot, but has two major problems. First, it's unlikely that comparable funding will also be transferred from NASA to NOAA. E.g. NASA will be told to stop earth science research and have it's budget cut appropriately. NOAA will be told to start earth science research, but without the requisite budget increase. So effectively, it kills earth science under the guise of "moving" it.

Second, NOAA is not operationally or logistically equipped to handle the level of earth science that NASA is performing. NOAA manages a few weather satellites, but has nowhere near the infrastructure that NASA has, and requires, to manage the existing earth science research. Even if you gave NOAA the appropriate money, they don't have the infra or engineers to keep everything flying.

Planetary Radio did a great episode on this topic, but I'm struggling to find it right now.

tracker17• 9 years ago

It seems like Trump has been doing similar to quite a few federal agencies. I'm concerned with some of the early moves, but it's going to be interesting, I do hope that some common sense prevails. For example, bowing out of TPP isn't so bad.

stevehawk2• 9 years ago

Bowing out of the TPP has been hilarious for me to watch. 6 months ago if Obama had done it he would have been praised on nerd sites. But he didn't and Trump did so now it's an awful idea. It's right up there with my fellow nerds demanding H1B visa reform, Net Neutrality, and subsidizing clean energy while also demanding free markets. I just hear Fezzik going "I don't think that means what you think it means..."

* note - i support bailing on TPP, H1B visa reform, and subsidizing clean energy

ams6110• 9 years ago

Why shouldn't government agencies have to justify their budgets and operations periodically?

burkaman3• 9 years ago

They should, so why not send out a blanket order for all agencies to do this? Why target only a few specific departments for justification and gag orders?

epistasis2• 9 years ago

How about when they submit a budget, rather than as a focused witch hunt?

Of course, its not like this political atmosphere would listen to any budget justifications. It's all theater now, it doesn't matter what actual justifications there are, what actual facts there are, or what things actually look like when you spend the time to dive into them. What matters these days is a small number of opinion-makers feel in their hearts, and what they tell their followers to feel with quick one-sentence justifications.

nikdaheratik2• 9 years ago

It's called a budget review. It's Congress's job to do this, not the Administration. And it seems like a total waste of time compared to all of the other more important issues the Administration should be covering, IMO.

rhodysurf• 9 years ago

How about that almost every weather service in the US depends on their existence?

battlebot5• 9 years ago

Exactly. The thing that everyone is shitting their pants over is a nothingburger. NOAA once had a verifiable mission. The problem with government agencies is mission creep--they all got on the "climate change" bandwagon because that's where the bucks were. That is now over.

XorNot• 9 years ago

It would be a good use of IPFS to mirror this data into that network.

philippnagel• 9 years ago

Might be a use case for https://datproject.org.

neuromantik8086• 9 years ago

Or git-annex / git LFS.

dyukqu2• 9 years ago

Slightly off-topic, but my guess is that the next 4 years will be the golden age of privacy (in general), file-sharing and maybe even distributed web & federated services - for the US at least.

gramstrong• 9 years ago

Do you mean the golden age of private infrastructure, or the golden age of folks combatting a non-private infrastructure? Because from my point of view, privacy in the US is only going to continue to get worse (and that's not even necessarily a dig at the current administration).

dyukqu2• 9 years ago

What I mean is (I guess) people will be seriously considering their privacy - especially online/digital one.

allemagne• 9 years ago

It's plausible. My sincere, dark hope is that a Trump administration will shock everyone awake in a big way to why broad surveillance is a terrible idea.

Unfortunately it will probably take a huge scandal that baldly and stupidly shows how unchecked those powers are and what can be possible with them. I think Trump is capable of that kind of misstep. Like say SNL writers were PMing each other a tasteless joke about assassinating Trump and the president tweets about it and orders their arrest.

In the end, something dramatic like that could actually have a really positive effect on the consciousness of the public. It could also have a really negative effect where people accept the new normal. It's hard to know.

cool_look2• 9 years ago

So now we have complete polarisation.

Left leaning people will not accept any science generated under a Right leaning government

and as we already know Right leaning people find it hard to trust the science coming from a Left leaning government.

fisherjeff• 9 years ago

Eh, no. This isn't about not accepting scientific output under a Republican president – science was well received under Bush, as I recall, and will continue to be under Trump.

This is about the fear of losing scientific data that has already been produced.

cool_look2• 9 years ago

On the surface it is about preventing deletion.

But the subtext is that any 'discoveries' or 'corrections' will be rejected out of hand.

The prime ones being the recent corrections to the land temperature record that cooled the 1930s and 1940s and heated 2009 onwards yielding a steady linear increase graphic.

When those corrections are placed under the microscope and likely reversed, what will the reaction be ? Scientific shills of the Republicans are making up Alternative Facts... here is our precious archived dataset that true believers can reference.

gglitch• 9 years ago

Is there a publicly-available blockchain specifically for checksums of journalism and research?

fenollp• 9 years ago

Alternative facts will push us to find a way to discontinue them.

What is left to do to have truly open computation? * somewhat-to completely free computational power?: CircleCI, Travis CI, whatever * open source repositories to run code from?: Github hooks and others * trusting trust? * open hardware? * decentralized websites?

godshatter2• 9 years ago

Although NoScript and uBlock Origin show nothing being blocked, I still end up with a blank page when trying to view this site. Maybe it's a Pale Moon thing.

intrasight• 9 years ago

Me too

notsrg• 9 years ago

Surprised there's no comment here regarding the claim that using Dropbox is anonymous...

cema• 9 years ago

As much as I maintain a skeptical attitude to all sides of this scientific (and political) issue, I absolutely support the effort to keep the data available to the public. Science cannot move forward without the data (and metadata) available for everyone. Even more generally, society would suffer if it is not available.

(When I say "metadata", I refer to the data sources, methods of the data acquisition and so on. When a scientific field is politicised it is not only hard to trust the models but even the data, so I think the metadata are very important, if they are available.)

angelohuang• 9 years ago

While I appreciate the sentiment, I think this whole notion of an AntiScience threat due to the new administration is a bit sensationalized.

Mirroring US climate data

Related Stories