Keeping computers from ending science's reproducibility

http://arstechnica.com/science/news/2010/01/keeping-computers-from-ending-sciences-reproducibility.ars?utm_source=rss&utm_medium=rss&utm_campaign=rss

by yummyfajitas • 17 years ago

101 62 17 years ago

arstechnica.com

jerf4• 17 years ago

I am often told that I should keep my nose out of other science domain's business because they know more than I do. However, I think when they start building their science on top of computers, I start getting a say again. Here's what concerns me about this increasing use of computers:

It seems like the vast bulk of these simulations are iterative, and therefore subject to mathematical chaos. How many of these researchers have any clue what door they are walking through? How many of them know what a strange attractor is? I'm sure the answer is non-zero; I'm equally sure the answer is nowhere near 100%.

Small errors cascade even if you consider a non-chaotic classical model. (That is, not that there is such a thing as an iterative model that is not potentially subject to chaos, but rather than even if you don't understand chaos you can see that small errors can cascade. Chaos just makes it worse, and weirder.) A simulation will have bugs like any other large problem. A non-programmer approaches bugs by banging on the program until it seems to generate expected results. (About 50-80% of programmers do that too.) Therefore, many of these simulations are simply reflections of the simulator's expected result, due to the effect of the researcher's selection mechanism running on the results of the simulations they run. How do we verify that this is not the primary factor in the result of the simulation? This need not be conscious. It need not be ideological, either; I can easily envision a simulation that "should" return a boring or trivial result being monkeyed with until it produces something "interesting", because the simulators think the boring result should not obtain.

A lot of algorithms you can use in these simulations are fundamentally unstable when used iteratively; some exacerbated by floating point errors, some mathematically unstable even with perfect real numbers. How many of these simulations use something unstable without even realizing it, given that it could take a professional mathematician to work out whether that's the case? Even algorithms thought to be stable and reliable can fall apart under pathological situations, and one of the odd things about mathematics is just how often you end up hitting those pathological situations when programming; far more often than it seems like should be the case.

In information theory terms, a simulation can not contain more information that the sum total of the input data and the content of the simulation algorithm. How many simulators understand the full implications of that statement? I sure don't understand the full implications of that, but what I do understand makes me pause a bit. Very simple simulations with rules that can be verified and initial data that is very solid I can deal with; for instance, I like the cell-automata based social theories that show the spread of information or political views or something, especially when it is clear the researchers understand that it's only an approximation. But as the initial data starts getting sketchy or the simulation grows enormous, I start getting nervous about the actual information content of the output. Just because the output appears to be information doesn't prove that it is. It is vitally necessary to be able to check the simulation against real data. For instance, physical simulations of, say, cars crashing can be verified. How many simulations can actually be verified, though? Frequently the reason computers were reached for in the first place is the inability to do the real experiment. Any simulation that can't be verified should be presumed worthless by default. How often does that happen? (It's 20-f'ing-10 and "the computer said it, it must be right" still runs rampant through our culture....)

And of course there's the whole reproducibility issue, where the absolute bare minimum for science would be to publish the full simulation program, all data, the necessary invocation and compile instructions to bring the two together, and all necessary information to understand the input and the output. Clearly, this is not something that fits in a journal paper, but how often does this happen at all?

No, I am not referring to any specific discipline here and in particular I'm not actually referring to climate science. I'm nervous about the whole movement towards simulations in general.

Note that I'm not reflexively against the idea. Meet these bars and I'm happy; give me enough data for reproducibility and verify that your simulation is in fact simulating something real and corresponds to reality and I am happy. (Many physical simulations fit in here.) But as more disciplines jump in I am concerned that these bars are not well understood, and I'm seeing ever more press releases about simulations that can't possibly meet these bars.

yummyfajitas5• 17 years ago

I am often told that I should keep my nose out of other science domain's business because they know more than I do.

I've only heard such statements coming out of a few fields: math education, labor economics, climate science and psychometrics of race/gender. You should ignore such statements; they are nothing more than an attempt to bully you into accepting received wisdom from activists with a PhD.

As an actual scientist (rather than a political activist with a PhD), I strongly encourage you to stick your nose into any or all of my fields (quantum mechanics, PDEs, medical imaging, complex analysis, prediction markets). If you come up with dumb ideas, I'll even explain why they are dumb, rather than just demanding that you leave things to the experts.

kasterma• 17 years ago

As another scientist I have to strongly second this. Some things might take real work on your part to understand though. Even within different fields of mathematics there are difficulties in understanding (complaint about this: http://www.math.rutgers.edu/~zeilberg/Opinion104.html).

btilly2• 17 years ago

Such statements also come out of biologists discussing evolution. This is not, however, evidence that they aren't really doing science. Instead it is evidence that they've been burned out explaining basics over and over again to Creationists and want to get on with their lives.

However some do take the energy out for those explanations. One of the results of their energy is http://www.talkorigins.org/.

Hopefully some day someone will take the energy to do the same with climate science. Because as much as there is a lot of politics there, there is some real science there as well.

Retric6• 17 years ago

Education is a lot more complex than most people understand it to be. Even within the field few people actually have any understanding of how the brain and learning interact. Everything from diet, to time of instruction, to setting, to materials, to methods, to testing is all important. A classic example of which is how do you build a useful test? Well if you want to know how much learning took place you need to test them both before and after instruction. But you also need to test people twice without instruction to see if the tests give information away. You need a mix of problem difficulty to find out if someone understands the basics and at the same time, to see if they understood the minor details. etc. You then need to look at the test not just in terms of overall score but also how well they did on each type of problem.

PS: Education might seem like a flaky field and much of the discussions are devoid of good science. At the same time there is a lot of great research that has been which has real and important implications.

timwiseman2• 17 years ago

I agree with the sentiment you express and thank you for pointing it out.

I do think that thinking outsiders should stay out is uncommon in the field of math though. I am currently a math grad student, and while I am only a grad student what I have seen of the mathematical community makes it seem like it is very open to cross fertilization from other sciences and from amateurs.

Admittedly he is historical, but Fermat (professional judge when he did most of his mathematical work) is revered and there are some more recent amateurs that have made genuine contributions. Much of mathematics is inspired by other fields such as physics.

I admit I know some professional mathematicians were tired of finding flaws in amateurs proofs of Fermat's Last Theorem and rebuting supposed refutations of Godel's diagonal proof that the reals are uncountable, but it is very different to be tired of dealing with a constant stream like that than to be hostile towards outsiders in general.

From my limited perspective, I think math is one of the fields most open to outsiders and amateurs coming in and making real contributions, both culturally and in the fact that we rarely need expensive experimental setups so the barrier of entry (in terms of money and equipment) is low.

sparky3• 17 years ago

To echo the meme around here, there appears to be an opportunity here for a startup :)

If you can get buy-in from the research community, even in one niche field, I could see great utility in a central escrow house of sorts, where people can post research software, datasets, etc. and others can download the stuff and validate the results only after agreeing in some reasonably binding way not to use that information to scoop or copy the original researcher.

Buy-in in this case means that everybody should have to do it to publish in a certain venue. You might even do something fancy like providing people access to a VPS which will run the server without allowing validators to scp everything back to their own machines. Of course this is impossible to do perfectly; once you allow people to view the data in a text editor or hex dump, it is all over in a strict sense (scripts + screen-scraping), but it is still a step up from just handing over the files, and keeps honest people honest.

You might also provide some anonymization/randomization service for datasets (for instance, by applying an unknown linear transform to absolute numbers in the case that the results don't change, similar to what was done here: http://googleresearch.blogspot.com/2010/01/google-cluster-da... .

I think there are 2 reasons why researchers can be reluctant to share:

1) They are afraid of getting scooped by somebody else leveraging all the effort they put into developing the tools and curating the data and leapfrogging them.

2) Their results are suspect.

Such an escrow house would ameliorate the concern behind 1) and help suss out 2) :)

Anyone else have thoughts on this?

stevenbedrick• 17 years ago

In biomedicine, the National Library of Medicine has been taking steps in this direction. There are repositories for sequence, protein structure, and microarray data, for example, and many journals require that relevant data be deposited in these repositories as a condition of publication.

Efforts such as the Science Commons are also thinking about/working towards these sorts of solutions.

carbocation• 17 years ago

Actually, the NIH has already done this with many recent grants. Everyone funded through certain grants has to make their data available in dbGAP (genotype and phenotype), sometimes even pre-publication (which is raising some feathers).

Point 1 will always be true when people share methods and/or data pre-publication. If point 2 cannot be discerned with currently available methods (and I would argue that it can), then we are in a world of trouble.

mattheww• 17 years ago

Your concerns are certainly legitimate, and I'm sure there are some situations where they apply. However, in many physical sciences (I would venture to say the vast majority), simulations are not chaotic. They are based on stochastic models that have converging behavior. So, in general, it really is fair for most people to rely on the stability of the simulations.

As for your other points, the issue of simulation accuracy is already taken very seriously in the areas where I've seen it used.

Let me give an example from my field (high energy physics). The pre-eminent event generator of choice is called PYTHIA. It incorporates as much known physics as possible, and is continually updated. There are groups that regularly convene to compare the distributions it produces to those seen in real experiments and adjust the parameters to improve the results. The code is open source, so you can readily make improvements and submit them to the maintainers for inclusion in official releases.

Now the primary uses of these simulations are to either tune your analysis to separate signal from background or calculate corrections for different kinds of detector acceptance issues. No reviewer would ever accept a publication that used simulation if the paper did not include clear evidence that the simulation was valid, usually in the form of a data/simulation comparison.

Regarding reproducabilility, I think you're a little off the mark. The idea is not that you should simply repeat my analysis to get the same answer (this important verification step should be, and usually is, present in all scientific groups). Ideally, you should collect your own data, make your own simulations, and do your own analysis. Then we should see if we got the same answer.

Edit: grammar

jerf4• 17 years ago

"As for your other points, the issue of simulation accuracy is already taken very seriously in the areas where I've seen it used.... my field (high energy physics)"

Bad example, inasmuch as it is too good. Particle physics has petabytes of data (exabytes yet?) to test against and is very connected to the real world. With that check you can't stray very far.

"Ideally, you should collect your own data, make your own simulations, and do your own analysis. Then we should see if we got the same answer."

I would say a precondition of that step is first that I can reproduce your work. If I can't even reproduce your computation it's not even worth trying to reproduce the eventual results, as your eventual results are too questionable to begin with, as far as I am concerned. If I can't reproduce the computation you might as well have just pulled them from your bum.

Retric6• 17 years ago

In information theory terms, a simulation can not contain more information that the sum total of the input data and the content of the simulation algorithm.

This is false outside of the most trivial definition, when you simulate evolutionary systems you can create systems far more complex than the algorithm used to generate them, the problem domain, or any other factor prior to running the simulation. That information is crated from your random input stream which does not contain information in the classical definition.

PS: As to the larger context of the discussion. From a simulation standpoint sending other scientist the code and having them run it is a bad idea. Reproducibility does not involve the lab that made the discovery sending the devices used to make the discovery to another lab. The goal is for someone to be able to see the same effect with the most independent setup possible. If scientist A says "building this a model with these assumptions > this result." Then another lab creates their own software which runs and finds either the same or a different result. Or, they can look at the assumptions or inputs and say this is bad data try again. Or, they can look at the code and say, here is a bug fix it and try again. But, those are three independent steps best carried out by diffrent groups.

jerf4• 17 years ago

"when you simulate evolutionary systems you can create systems far more complex than the algorithm used to generate them, the problem domain, or any other factor prior to running the simulation."

Possible. I have to admit I'd have to think more carefully about this, but I'm not sure it's a knockdown win either. In the real world, the extra information comes from the environment in its capacity as a selection mechanism. It is a large source of such information. Simulations tend to have a radically simplified selection mechanism, what with not containing reality and all. Just because something looks complicated and can't be effectively gzip'ed doesn't mean its actually high in information, it's tricky stuff. I'd be much more inclined to buy this, if evolutionary programming was much more impressive than it actually is and actually routinely pulled off programs that were clearly high in information value.

(I've actually spent a bit of time with evolutionary computation. It has one of the largest hype/reality ratios in computer science.)

"Reproducibility does not involve the lab that made the discovery sending the devices used to make the discovery to another lab."

This is an artifact of the fact that is impossible to do this with physical objects, so we've built the mechanics of science around not being able to do that. Don't mistake historical accidents with scientific imperative. While it is still absolutely desirable that simulations as independently-constructed as possible and give as similar a result as possible, if there was an equivalent in the physical world of being able to rigorously examine someone else's experiment we would be doing that. How long would Cold Fusion have lasted if that was available?

Science is inevitably a product of its environment. We don't demand particle-physics precision from the sociologists not because it is undesirable, but because it is impossible. We don't pass around experimental apparatus to each other for examination not because it is undesirable, but because it is impossible. It isn't impossible for simulations. We're allowed to tweak our procedure in response. Science is not a religion.

lutorm• 17 years ago

I certainly agree with you that there is a lot of publication of results generated with computers that is non-reproducible. However, I'm not sure publishing the complete codes that were used is necessary, as long as there is a detailed description of the algorithm used. In essence, I think it's essential that multiple people write their own codes and reproduce the result. The danger in publishing your code is that it is easier for someone to "reproduce" your study by just taking my code and running it, and there is a danger in having the community rely on a small number of codes.

The PYTHIA example someone posted is a good one: It's my understanding that basically everyone in high-energy physics uses it, and while that means it gets a lot of testing I'm not sure that many actually bother to look through the code in detail, because "it's what everybody uses, so if it had a bug it would have been found already".

The situation is similar, but not quite as bad, in astrophysical hydrodynamics simulations. Basically the entire field uses 6 different codes, but they do work on different principles, and the teams do conduct controlled tests of the agreement between them on specific problems.

About the chaos problem, I suspect that the problem is worse. My impression is that often there is not even a basic test that the results are numerically converged, regardless of esoteric strange attractors.

I'm curious, can you mention any specific cases that you are worried about wrt the chaotic behavior.

joe_the_user• 17 years ago

The danger in publishing your code is that it is easier for someone to "reproduce" your study by just taking my code and running it, and there is a danger in having the community rely on a small number of codes.

Having a small number of codes seems like an excellent situation. What you should have is large of tests and much test data to test the small number of codes.

Testing code in thorough manner is an important addition to looking at it.

jedbrown2• 17 years ago

I agree with many of your sentiments. Here is a view from "inside" (I do PDE solvers from a mathematics and computational physics perspective, have written open-source models, and contribute to a popular open-source solver library).

1. Very little of numerical analysis is about propagation of rounding errors. I think you underestimate the reliability of floating point numbers in this context. The challenge of numerical analysis is not in representing the continuous one-dimensional object (real numbers), but in representing an infinite dimensional space in finite dimensions. You might appreciate Trefethen's "The definition of numerical analysis" (http://www.comlab.ox.ac.uk/nick.trefethen/publication/PDF/19...).

2. We can prove convergence of iterative methods for some problems of practical interest. More frequently, we can prove that if the iterative method converges, then it converges to a meaningful solution. Such results are unavailable for some systems, and this is where we enter the domain of Verification (the code converges to exact continuum solutions at the appropriate rate) and Validation (the continuum equations approximate reality). V&V is a rapidly growing field which offers a number of mature tools, but is still under-appreciated in many disciplines. The fact that V&V is rarely taught at university, even to applied mathematics graduate students, let alone physicists, doesn't help.

3. It takes effort to turn a one-off code that only runs in your special environment into portable, distributable software. Funding agencies and institutions place little emphasis on this process, so it is regularly neglected. It is exacerbated by the fact that few physicists have any background or interest in software engineering, thus making it more challenging to release non-broken software.

yummyfajitas5• 17 years ago

4. It is often not worth the effort to turn a one-off code into distributable software. There is a good chance that only a small number of people care about the PDE you are trying to solve.

I contemplated making a distributable solver for the Schrodinger equation, but there was little interest.

A quote I've heard attributed to Cleve Moler (chief scientist at Matlab): "PDEs are a niche market."

DaniFong• 17 years ago

In general, science hasn't been as reproducible as one would like it to be, there's so much tacit knowledge that's necessary for experiments, many unknown variables, so many demonstrations are so complicated that hardly anyone even goes through the motions. In general there are many more people writing in detail than there are people read and understand, many more people focusing on some narrow branch of their discipline than there are people with genuinely strong interest and understanding of the fundamentals.

My point, primarily, is that the danger of people using tools that they don't understand is an old one; possibly worsened by the narrow focus of modern science, and how easy computing tools have made simulations. But it's been here all along.

As far as sticking your nose into sciences, your nose is invited into any domains of knowledge of mine, but do keep your voice down: there are too many people making too many bold pronouncements without real knowledge; persistent, diligent, and quiet questioning seem to be much more effective at getting the heart of the matter.

jcnnghm• 17 years ago

Even worse, having reviewed the code that leaked during Climategate, it's pretty clear that the researchers are not competent programmers. The revelation that was most off putting was the "commented out" code that injected arbitrary values that was "just used for testing". Testing is great, but it has no place in the production code. If the researchers method of testing is to write test code in the production code, run it once to make sure things "work", then comment it out, we certainly shouldn't be making any decisions based on predictions made by that code. Their testing framework is apparently comments in the code.

For a project I'm currently working on, I have a 5:1 test to code ratio, largely because I'll be processing credit cards. While my project will probably only ever deal with a few hundred thousand dollars in transactions, I need to be sure that those transactions will be properly handled. In comparison, these researchers are encouraging a trillion dollar piece of legislation, and their testing methodology is intuition and spaghetti code.

Further troubling is the assertion that this is science. As far as I can tell it isn't testable or repeatable. As I recall, that's a pretty big part of what science is.

snth• 17 years ago

Can you provide references that would allow a researcher to educate himself? I'm genuinely interested, and there seem to be lots of fields that use software in experimentation and simulation but where there isn't much emphasis or education on doing it properly.

cynlillbox• 17 years ago

jeff:

i'd like strongly to encourage you to offer your opinions on things on which you are nonexpert. having only experts allowed to opine, with experts also relaining monopoly on experts, leads to echo chambers.

however, you seem to be reacting to a different article entirely. nowhere did the article mention simulations. if you look at Stodden's CV (the woman whose talk motivated the article) she's a statistician. The need for code-sharing is in fields like climate and bioinformatics where people's analyses lead to changes in politics, policy, pharma/health, etc; not for people deciding whether lorenz goes chaotic at precisely the right bifurcation parameter value.

more and more and more science is becoming statistical and data-driven. chaotic dynamics, though very important, is not the issue here. it's analysis of complex data coming from real-world systems.

there are many interesting and wise comments in this thread but they are almost all about an article on simulations -- ie. not the article the OP points to.

earthboundkid2• 17 years ago

“It is vitally necessary to be able to check the simulation against real data.”

Actually, that’s not enough either. In spam training, you have to have a completely set of data that you use for the final test but never, ever for training. Why? Because otherwise your spam filter will learn how to properly classify every message you train it with and nothing else.

For simulated science, this means it’s not safe to say, “Well, we trained the simulation until it could reproduce 2000-2010 when we gave it 1990-2000 as input data.” If you do that, your predictions for 2010+ are probably going to be worthless, since your simulation is like a cheating student who only knows the right answers on old tests he stole, not how to pass any test in general.

jvdh• 17 years ago

A similar argument actually goes for experiments that are somehow affected by computer networks.

If scientists use grid-computing, cloud-computing, or just the plain regular Internet, there is no way to accurately reproduce results for distributed applications. Luckily, some researchers are aware of this and there are now some projects starting to make testbeds and infrastructures to make environments where experiments can be reliably reproduced.

sparky3• 17 years ago

Links, for the curious:

http://www.planet-lab.org

http://www.hpl.hp.com/open_innovation/cloud_collaboration/cl...

http://cloud.cs.illinois.edu/

http://www.cs.duke.edu/courses/spring08/cps214/project.html (Testbeds and Emulation section)

wmf• 17 years ago

PlanetLab and the clouds are great ways to do non-reproducible research, because the other users in the system (where sometimes "the system" is the public Internet) create interference. Reproducible networking research either has to be simulated or has to be run on an isolated testbed such as Emulab.

jwhite• 17 years ago

NICTA and WinLab have developed a management framework called OMF for specifying and running experiments on testbed networks. One of the main goals of the project is to improve reproducibility in networking research.

http://omf.mytestbed.net

(It is mainly written in Ruby)

m104• 17 years ago

My (biotech) employer largely solves this issue by keeping a copy of the formal research specs outside of software altogether. All validation documents and research data are kept in paper form (in addition to digital form) in such a way that future researchers or inspectors could take those documents and data and reconstruct the research.

It wasn't always this way, unfortunately. I've been involved with trying to glean some formulaic/methodological insights from spreadsheets and code and it's not always possible to reverse engineer the essential methodology or be sure that mistakes were avoided.

Proper scientific research practices are similar to proper data backup practices: the documentation (backup files) are important, but they don't matter if you can't have successful reproducibility (restoration).

bbgm• 17 years ago

In the life sciences provenance is as important as reproducibility, and you can use software systems to manage provenance. The paper requirement will go away over time. Even the FDA, which required paper in the past, is moving to an electronic model. But you do need to document methods somewhere and not just in code, a reference document which could be shared, even if the specific implementation is different

bioweek• 17 years ago

Speaking of reproducibility, my friend getting a PHD in finance told me he was writing a paper using the data from some brokerage. I asked if he would publish the data and he told me it's confidential.

I talked myself blue in the face trying to explain how science doesn't work if you don't give people enough information to reproduce your research! I couldn't get him to understand though. Arggh so frustrating.

mahmud• 17 years ago

High resolution tick data is usually NDA'ed; but anything slower than a few ticker per minute is free to publish, if you capture it yourself.

If data feeds are expensive, you can run trading software on your own machines and capture the ticks.

dantheman• 17 years ago

It depends what type of science you're doing, and what stage the science is in. When we are at the gathering data hypothesis building stage observations, and case studies are important and are often based on confidential data.

dagw• 17 years ago

Assuming the data was some sort of financial time series then that means lots of people already have the data (or something close enough to it). So even though he can't publish the data he can say what the data is and when and how it was collected and that should be enough for most people in the field reproduce the results.

andrewcooke2• 17 years ago

i had this problem many years ago (15?). at the time i was working as a postdoc, calculating the evolution of the ionizing background with redhshift from the inverse effect (lyman alpha clouds near quasars get fried by the quasar; the extent of this gives an indirect way to measure the ionizing background at that redshift).

i had a bunch of perl scripts (ah, those were the days) that mangled various files before feeding them into fortran least-squares stats code that took a day or so to run.

by the end, it was pretty much chaos. i was a self-taught programmer, these were probably the second or third "significant" programs i had ever written. nothing was documented, everything took so long that i couldn't check much... i had bugs, of course.

in the end i published. maybe 6 months later i got an email from someone in the states. they were trying to reproduce my results. in the end, they did (as far as i know).

so the system worked.

incidentally (perhaps the only useful point here) they must have used different data. that's something worth explaining in more detail - "my" data came from years of painstaking work by a bunch of people working for my thesis supervisor. yet 6 months later the results could be duplicated from a week or so of data from a much more powerful telescope (the keck). so data in research often aren't as critical as you might think. things progress at such a rate that even if you don't share data, it's trivial to reproduce just a short time later... (and i am pretty sure that this is true in gene sequencing, for example)

frossie2• 17 years ago

But this is a problem that can be tackled, and those who take it seriouslt already do so. For example, in our pipelines we use an infrastructure that always adds every command executed on the file, with every exact parameter, to the metadata of the file, starting from one canonical archived file - and hence, one can indeed reproduce manually the result of the pipeline given sufficient time and dedication. [Edit: we also write the git shar1]

The same way that say, biology labs have processes they engage in in order to convince us that their samples are not contaminated, we can have processes that lead to the reproducibility of data.

andrewcooke2• 17 years ago

[is that frossie frossie? hi!]

i'm not sure what problem you're talking about - someone reproduced the results with separate code and data, so what's to worry about? (i don't mean that because it was confirmed it was ok, but rather that if it had been wrong, we would have known in the end... after all, people make mistakes all the time - science is a collective enterprise that relies on many overlapping, interlocking pieces)

deckard2• 17 years ago

Here's a practical solution I have proposed in my community (autonomous robots):

1. package code and data into tarball or VCS repo.; 2. place package on long-lived website; 3. compute SHA1 hash or similar from package (if git is used, this is the revision ID, conveniently); 4. publish the URI and hash in any paper that makes claims based on that code or data;

5. as a reviewer, prefer papers that follow this method, all else being equal; 6. as an editor, suggest that submissions use this method.

(Edit 2: In case it's not obvious, the purpose of the hash is to allow users to be pretty confident that the code they downloaded is indeed exactly the code used in the paper. By putting the hash in the paper, I make this promise. If I want to make an improved version available, I just put it up at the same site, but I must make the exact original available and identifiable as such. This simple method of ensuring identifiability is our contribution.)

My group does this with every paper. I have a paper describing this method coauthored with a student under review now at a good journal, and I'm looking forward to seeing the response. (Edit 1: see link in comment below)

I'd also appreciate feedback from HN.

plinkplonk2• 17 years ago

"My group does this with every paper."

Any links?

deckard2• 17 years ago

Sure. We started doing this last summer, and this is the first paper accepted for publication that uses the method.

http://autonomy.cs.sfu.ca/doc/wawerla_icra10.pdf

and here's a draft of our paper on the methodology, with rationale.

Long paper: http://autonomy.cs.sfu.ca/doc/wawerla_submitted_2009.pdf

Original short workshop paper:

http://autonomy.cs.sfu.ca/doc/wawerla_rss09_workshop.pdf

For context, here's the lab's publication list:

http://autonomy.cs.sfu.ca/publications.html

We plan to continue this, even if the idea doesn't catch on.

abscondment• 17 years ago

Reminds me of an interesting Wired article from 2008: "The End of Theory: The Data Deluge Makes the Scientific Method Obsolete" http://www.wired.com/science/discoveries/magazine/16-07/pb_t...

earthboundkid2• 17 years ago

A fun article, but Peter Norvig of Google has said he feels like it severely misunderstands his work.

alex_stoddard• 17 years ago

As a working programmer in biological research science I would love to see a requirement that papers involving software be published in the "literate programming" paradigm. At the very least per reviewed publication must include all software be open source (a least in a loose sense). It is depressing how often a published result depends on custom closed source software.

ynniv• 17 years ago

I have a strong (probably unusual) standard when it comes to computer modeled science: if explanation requires software or data that I don't have access to, I completely disregard it. That might sound kind of crazy given the state of science and my chosen profession, but (maybe because of my chosen profession) I know that a complex system can tell you whatever you want it to tell you. It won't be the truth, but it will be "convincing" to most people. If the research really mattered, someone would reproduce it with independent software and data anyway.

I think this is a good solution to the problem. If most people only believed in reproducible science, there would be pressure on authors to use software and data that can be shared, or no publisher would carry their article. Seems easy to me.

khaless• 17 years ago

Hmm, from what I remember, Reproducibility became a big issue in SIGMOD. In 2008 they introduced some new requirements when submitting a paper.

http://behind-the-enemy-lines.blogspot.com/2007/08/experimen...

Keeping computers from ending science's reproducibility

Related Stories