Reproducibility: When the Reproduction Fails !

In a recent blog post:

http://blog.stodden.net/2011/03/19/a-case-study-in-the-need-for-open-data-and-code/

Victoria Stodden elaborates on a clear example of why reproducibility
must  be a practical requirement for scientific publications, and how
open-data an open-code are essential to make reproducibility verification
a practical reality.

This is the case of a team at MD Anderson led by Keith Baggerly, that went out an try to replicate the results of a  set of papers published by a research group at Duke University. To their surprise, the Baggerly team was unable to replicate the results disclosed in the original papers, and thanks to the fact that most of the data and software was publicly available, they were able to perform “Forensics” on the papers and discover that they were plagued with poor practices, inconsistent management of data, and unexplicable results.

This forensics work led to the termination of clinical trials at Duke last November and the resignation of Anil Potti.

Among the interesting aspects of this case, is the fact that the forensics team had to use the Freedom of Information Act (FOIA) in order to get an investigation report that Duke sent to the National Cancer Institute. Duke refuse to share that report with the forensic team under the argument that it was “confidential” information. However, given that NCI is a federal agency, is subject to comply with the Freedom of Information Act.

“The Importance of Reproducible Research in High Throughput Biology:
  Case Studies in Forensic Bioinformatics”
    Keith A. Baggerly,

A full video of the presentation is available here:

http://videolectures.net/cancerbioinformatics2010_baggerly_irrh/

This lecture is mandatory material for anyone who honestly cares about scientific research.

As good practitioners,
Baggerly’s team put online, publicly available, ALL THE MATERIALS
needed to replicate what they did. You can find them here:

http://bioinformatics.mdanderson.org/Supplements/ReproRsch-Chemo/

A good case of a group who practice what they preach, and teach by example.

Reports on bioinformatics work at MD Anderson are now required to be reported in Sweave (A combination of R and LaTeX) to ensure that the report are REPRODUCIBLE from the original data to the final results.

Memorable quotes from this talk:

• “We wrote a paper on this [the fact that the results of the original experiment couldn’t be reproduced], we first circulate it to a Biological Journal, and we got the comment back that: “We are sorry, this story seems TOO NEGATIVE, can you FIX that ?“.
• “Duke administrators accomplished something monumental:
  They triggered a public expression of outrage from biostatisticians”.
• “The most common mistakes, are simple ones”
  • Off-by-one in an data table / software
  • Mixing up the sample labels
  • Mixing up the gene lables…
  • The fun thing about simple mistakes is that if you see them, they are easy to fix. But if the documentation (the paper report) is poor, you will not see them.

  We suspect that the simple mistakes are more common that we would like to admit.

• Please “label the columns” and “provide code”.
• When I’m reviewing papers, I look and see
  • Do they tell me where the data is ?
  • Do they give me code ?
  • Do they have URLs, are the links live ?
• “Reconstructing this takes a lot of time. We estimate that this took us between 1500 and 2000 hours to figure out how all this work” (the effort of replicating a published paper, due to the lack of details in the processes used).
• “At some stage, not only us, but also others should be able to precisely reproduce what we did, in other words :if they start with the same numbers they should get the same results”

Useful Links

• Google group on Reproducible Research
  http://groups.google.com/group/reproducible-research
• http://reproducibleresearch.net/index.php/Main_Page

http://www.nature.com/nm/journal/v17/n1/full/nm0111-135.html

“We wish to retract this article because we have been unable to reproduce certain crucial experiments showing validation of signatures for predicting response to chemotherapies, including docetaxel and topotecan. Although we believe that the underlying approach to developing predictive signatures is valid, a corruption of several validation data sets precludes conclusions regarding these signatures. As these results are fundamental to the conclusions of the paper, we formally retract the paper. We deeply regret the impact of this action on the work of other investigators.”

http://www.nature.com/nm/journal/v12/n11/abs/nm1491.html