What Scientific Journals should learn from Microsoft

Since, of course, all of you have carefully read the

          End User License Agreement (EULA) of your Microsoft Windows installation,

you certainly have come across the following text:

Section 9. MICROSOFT .NET BENCHMARK TESTING. The software includes one or more components of the .NET Framework (“.NET Components”). You may conduct internal benchmark testing of those components. You may disclose the results of any benchmark test of those components, provided that you comply with the conditions set forth at http://msdn.microsoft.com/en-us/library/ms973265.aspx.

Following that link, you quickly arrive at this, even more interesting, text:

Begin Quote:

You may conduct internal benchmark testing of the .NET Framework component of the OS Components (“.NET Component”).  You may disclose the results of any benchmark test of the .NET Component, provided that you comply with the following terms:

(1) You must disclose all the information necessary for replication of the tests…

including:

• Complete and accurate details of your benchmark testing methodology,
• The test scripts/cases,
• Tuning parameters applied,
• Hardware and software platforms tested,
• The name and version number of any third-party testing tool used to conduct the testing,
•  And complete source code for the benchmark suite/harness that is developed by or for you and used to test both the .NET Component and the competing implementation(s);

(2) You must disclose

• The date(s) that you conducted the benchmark tests,
• along with specific version information for all Microsoft software products tested, including the .NET Component;

(3) Your benchmark testing was performed using all performance tuning and best practice guidance set forth in the product documentation and/or on Microsoft’s support Web sites, and uses the latest updates, patches, and fixes available for the .NET Component and the relevant Microsoft operating system;

End Quote

Disclaimer: the original formatting of the text has been modified here in order to bring attention to the interesting components.

To summarize, when it comes to things that matter:

                         REPRODUCIBILITY IS A REQUIREMENT

and reproducibility can only be realized by making available ALL the elements that were used for running your tests, including:

• Source code
• Data
• Parameters

Here is, therefore, what Scientific Journals and Conferences can learn from Microsoft:

                         If it can’t be replicated,
                         It never happened!