For PAW-ATM we adopt the following approach following the model of SC18.
Artifact Description Appendix:
We will use the format of one of the SC18 appendices for PAW-ATM submissions. Authors will provide the completed appendix (at most 2 pages), along with their submission. These 2 appendix pages do not count towards the submission page limit. The basic idea of the appendix is to enable readers and/or reviewers to gain first-hand experience with the computational experiments described in the paper, as far as reasonably possible.
Notes: A paper will not be disqualified based on information provided or not provided in this appendix, nor if the appendix is not available. The availability and quality of an appendix will be used in ranking a paper. In particular, if two papers are of similar quality, the existence and quality of the appendices will be used to define the final ranking.
The format will be made available to prospective authors as a .tex template.Q. Is the Artifact Description appendix required in order to submit a paper to PAW-ATM?
A. No. These appendices are not required. If you do not submit any appendix, it will not automatically disqualify your submission. At the same time, if two papers are otherwise comparable in quality, the existence and quality of appendices will be a factor in ranking one paper over another.
Q. Do I need to make my software open source in order to complete the Artifacts Description appendix?
A. Not necessarily. It is not required that you make any changes to your computing environment in order to complete the appendix. The Artifacts Description appendix is meant to provide information about the computing environment you used to produce your results, reducing barriers for future replication of your results. Nevertheless, we strongly encourage authors to provide access to the software, and we will provide links to the authors’ website from the workshop site.
Q. Who will review my appendices?
A. The Artifact Description and Computational Results Analysis appendices will be submitted at the same time as your paper and will be reviewed as part of the standard review process by the same reviewers who handle the rest of your paper.
Q. Does the Artifacts Description appendix really impact scientific reproducibility?
A. The Artifacts Description appendix is simply a description of the computing environment used to produce the results in a paper. By itself, this appendix does not directly improve scientific reproducibility. However, if this artifact is done well, it can be used by scientists (including the authors at a later date) to more easily replicate and build upon the results in the paper. Therefore, the Artifacts Description appendix can reduce barriers and costs of replicating published results: it is an important first step toward full scientific reproducibility. We understand that in many cases a full reproduction of the results might be costly and/or impractical; in this situation, we would suggest for the authors to provide a set of reference cases that could be easily reproduced with (relatively) limited resources, but would enhance the reviewer's or reader's confidence in the computational results.
Q. What are my responsibilities when reviewing a paper that has an Artifacts Description appendix?
A. Reviewers are expected to read any appendix that is submitted with a paper. The information in these appendices is often included in any well-written paper. By collecting this information in a uniform way, we hope that the review process will be more effective without requiring a significant new burden on reviewers.
Q. Must I confirm that links in an appendix work?
A. Confirming links in an appendix are not required of every reviewer. We will coordinate the review process making sure that at least one reviewer evaluates the appendix for each paper.
Q. Must I evaluate the code and results associated with a Computational Results Analysis appendix?
A. Code and results evaluations associated with a Computational Results Analysis appendix are not required of every reviewer. We will coordinate the review process making sure that at least one reviewer evaluates the appendix for each paper.
Q. How may I use appendices as part of the review process?
A. The absence or poor quality of an appendix cannot be used to categorically disqualify a paper. Furthermore, inaccessibility of artifacts such as source code cannot be used to categorically disqualify a paper. However, the presence and quality of an appendix will be used as part of the review process. In particular, if two papers have similar ranking based on the manuscripts, the availability and quality of appendices can provide additional evaluation criteria.