Reproducibility Initiatives for PAW-ATM

Background

SC16 introduced the first reproducibility efforts for the SC conference series technical papers program. This initial effort followed the strategy of [Result and Artifact Review and Badging], and asked for volunteers to complete an appendix to their paper that described the details of their software environment and computational experiments to the extent that an independent person could replicate their results. In SC16, 9 authors submitted appendices.

The format changed slightly with SC17, with the introduction of [a second appendix] describing how computational results were obtained. In SC17, more than half of the accepted papers had submitted a reproducibility appendix.

Reproducibility at PAW-ATM

For PAW-ATM, we adopt the following approach following the model of SC19:

Artifact Description Appendix:

We will use the format of one of the SC19 appendix for PAW-ATM submissions. Authors will provide the completed appendix (at most 2 pages), along with their submission. These 2 appendix pages do not count towards the submission page limit. The basic idea of the appendix is to enable readers and/or reviewers to gain first-hand experience with the computational experiments described in the paper, as far as reasonably possible.

Notes: - A paper cannot be disqualified based on information provided or not provided in this appendix, nor if the appendix is not available. The availability and quality of an appendix can be used in ranking a paper. In particular, if two papers are of similar quality, the existence and quality of the appendices can be part of the evaluation process.

FAQ for authors

Q. Is the Artifact Description appendix required in order to submit a paper to PAW-ATM?

A. No. These appendices are not required. If you do not submit any appendix, it will not disqualify your submission. At the same time, if two papers are otherwise comparable in quality, the existence and quality of appendices can be a factor in ranking one paper over another.

Q. Do I need to make my software open source in order to complete the Artifacts Description appendix?

A. No. It is not required that you make any changes to your computing environment in order to complete the appendix. The Artifacts Description appendix is meant to provide information about the computing environment you used to produce your results, reducing barriers for future replication of your results. However, in order to be eligible for the ACM Artifacts Available badge (see below), your software must be downloadable by anyone without restriction.

Q. Who will review my appendices?

A. The Artifact Description and Computational Results Analysis appendices will be submitted at the same time as your paper and will be reviewed as part of the standard review process by the same reviewers who handle the rest of your paper.

Q. Does the Artifacts Description appendix really impact scientific reproducibility?

A. The Artifacts Description appendix is simply a description of the computing environment used to produce the results in a paper. By itself, this appendix does not directly improve scientific reproducibility. However, if this artifact is done well, it can be used by scientists (including the authors at a later date) to more easily replicate and build upon the results in the paper. Therefore, the Artifacts Description appendix can reduce barriers and costs of replicating published results. It is an important first step toward full scientific reproducibility.

FAQ for reviewers

Q. What are my responsibilities when reviewing a paper that has an Artifacts Description appendix?

A. Reviewers are expected to read any appendix that is submitted with a paper. The information in these appendices is often included in any well-written paper. By collecting this information in a uniform way, we hope that the review process will be more effective without requiring a significant new burden on reviewers.

Q. Must I confirm that links in an appendix work?

A. Confirming links in an appendix are not required of every reviewer. We are asking for volunteers from the review committee who would be willing to validate links provided in appendices.

Q. Must I evaluate the code and results associated with a Computational Results Analysis appendix?

A. Code and results evaluations associated with a Computational Results Analysis appendix are not required of every reviewer. We are asking for volunteers from the review committee who would be willing to evaluate code and results associated with these appendices.

Q. How may I use appendices as part of the review process?

A. The absence or poor quality of an appendix cannot be used to categorically disqualify a paper. Furthermore, inaccessibility of artifacts such as source code cannot be used to categorically disqualify a paper. However, the presence and quality of an appendix can be used as part of the review process. In particular, if two papers have similar ranking based on the manuscripts, the availability and quality of appendices can provide additional evaluation criteria.