Take Nobody’s Word for It !

That was the rule at the core of the Scientific Revolution of the 1600’s

“…to withstand the domination of authority
and to verify all statements
by an appeal to facts determined by experiment…”

It is then quite sad to see that after four hundred years, in 2013 it is an extreme rarity to find Journals and Conferences that require the verification of reproducibility in the papers that they accept.

In the large majority of cases, this does not affect the public, given that so many articles are focused on small incremental improvements, and are mostly written solely to fulfil the productivity quotas defined in the rewards system of academia, and therefore have very limited or none social impact or relevance.

However, from time to time, some papers are used as the basis to define large scale social decisions, in areas such as healthcare and economic policy.

How unacceptable, the lack of reproducibility verification should be in academic publications, is illustrated by the recent finding on the flaws of the paper

Growth in the Time of Debt
by Carmen M. Reinhart and Kenneth S. Rogoff

This paper, published in 2010, has been the foundation for policies of economic austerity in the European Union and the United States.

The outrage on bad science and poor publishing practices is making the pages of the New York Times,
and getting the spotlight in the Colbert Report:

Recently, a student at the University of Massachusetts at Amherst attempted to replicate the study published by Rogoff and Reinhart and found a series of inconsistencies and flaws, that are described in detail here:

Does High Public Debt Consistently Stifle Economic Growth?
A Critique of Reinhart and Rogoff

Thomas Herdon, the graduate student, who uncovered the flaws of the paper, did so as part of his homework in an Econometrics class. His professors, clearly ones well-educated in the true practice of the Scientific Method, asked their students to replicate the results of a published paper, as a class project.

Herdon found:

“…that

• coding errors,
• selective exclusion of available data, and
•  unconventional weighting of summary statistics

lead to serious errors that inaccurately represent the relationship between public debt
and GDP growth among 20 advanced economies in the post-war period.”

and that,

“…when properly calculated, the average real GDP growth rate for countries carrying
a public-debt-to-GDP ratio of over 90 percent is actually 2.2 percent, not −0.1 percent
as published in Reinhart and Rogoff. That is, contrary to RR, average GDP growth
at public debt/GDP ratios over 90 percent is not dramatically different than when
debt/GDP ratios are lower.”

To Reinhard and Rogoff’s credit, they were kind enough to share with the UMass students the spreadsheet used to analyse their data. One has to wonder, on the other hand, why is it that the spreadsheet was not part of original publication in the first place, as one sees today in true scientific journals such as PLoS ONE, that indeed facilitate the verification of reproducibility.

As the conclusions of Rogoff and Reinhart’s paper, led to economic policies that trigger social unrest, and potentially result in large scale economic damages at international scale,… we must learn again to:

 Take Nobody’s Word for It !

It is time for “Publish or Perish” to go, and be replaced with:

Replicate or Perish !

Richard Feynman on the Scientific Method