The author-created artifacts relevant to this paper will receive an ACM "artifact available" badge <strong>only if</strong> they have been placed on a publicly accessible archival repository such as Zenodo, FigShare and Dryad. A DOI will be then assigned to their artifacts and must be provided in the Artifact Appendix!
<p>Note: publisher repositories, institutional repositories or open commercial repositories are acceptable only if they have a declared plan to enable permanent accessibility! Personal web pages, GitHub, GitLab, BitBucket, Google Drive and DropBox are not acceptable for this purpose!</p>
<p>Artifacts do not need to have been formally evaluated in order for an article to receive this badge. In addition, they need not be complete in the sense described above. They simply need to be relevant to the study and add value beyond the text in the article. Such artifacts could be something as simple as the data from which the figures are drawn, or as complex as a complete software system under study.</p>...
1. Publicly available
2. Not publicly available
...
Artifact publicly available?
<ul>
<li>Package complete?
<ul>
<li>All components relevant to evaluation are included in the package?</li>
</ul>
</li>
<li>Well documented?
<ul>
<li>Enough to understand, install and evaluate artifact?</li>
</ul>
</li>
<li>Exercisable?
<ul>
<li>Includes scripts and/or software to perform appropriate experiments and generate results?</li>
</ul>
</li>
<li>Consistent?
<ul>
<li>Artifacts are relevant to the associated paper and contribute in some inherent way to the generation of its main results?</li>
</ul>
</li>
</ul>
The artifacts associated with the paper will receive an "Artifacts Evaluated - Functional" badge <strong>only if</strong> they are found to be documented, consistent, complete, exercisable, and include appropriate evidence of verification and validation....
1. Exceeded expectations
2. Met expectations
3. Fell below expectations
...
Artifact functional?
Can this artifact be easily reused and customized? For example, can it be used on a different platform, with different benchmarks, data sets, compilers, tools, under different conditions and parameters, etc.? Is any portable workflow framework used for this purpose?
<p>The artifacts associated with the paper will receive an "Artifact Evaluated - Reusable" badge <strong>only if</strong> they are of a quality that significantly exceeds minimal functionality. That is, they have all the qualities of the Artifacts Evaluated - Functional level, but, in addition, they are very carefully documented and well-structured to the extent that reuse and repurposing are facilitated. In particular, norms and standards of the research community for artifacts of this type are strictly adhered to.</p>...
1. Exceeded expectations
2. Met expectations
3. Fell below expectations
...
Artifact customizable and reusable?
Can all main results from the paper be validated using provided artifacts?
<p>Report any unexpected artifact behavior (depends on the type of artifact such as unexpected output, scalability issues, crashes, performance variation, etc).</p>
<p>The artifacts associated with the paper will receive a "Results replicated" badge <strong>only if</strong> the main results of the paper have been obtained in a subsequent study by a person or team other than the authors, using, in part, artifacts provided by the author.</p>
<p>Note that variation of empirical and numerical results is tolerated. In fact it is often unavoidable in computer systems research - see "how to report and compare empirical results?" in <a href="http://ctuning.org/ae/faq.html">AE FAQ</a>!</p>...
1. Exceeded expectations
2. Met expectations
3. Fell below expectations
...
Results validated?
Was any portable workflow framework used (e.g. Collective Knowledge) to automate preparation and validation of experiments?
We promote the use of workflow frameworks to help evaluators quickly validate results in an automated and portable way. The idea is to automatically award "artifact reusable" badge when authors use portable and customizable workflow frameworks....
Portable workflow framework used?
Would you like to nominate this artifact for distinguished artifact award?
Artifact must be publicly available, functional, reproducible and easily customizable and reusable!...
1. Nominate for distinguished artifact
2. Do not nominate for distinguished artifact
...
Distinguished artifact?
Did artifacts and results match authors' description? Provide explanation of your scores and what can be improved during rebuttal....
Comments for authors
Comments for PC