Note that it is not obligatory to make artifacts publicly available!

Notes:

• ACM does not mandate the use of above repositories. However, publisher repositories, institutional repositories, or open commercial repositories are acceptable only if they have a declared plan to enable permanent accessibility! Personal web pages, GitHub, GitLab and BitBucket are not acceptable for this purpose.
• Artifacts do not need to have been formally evaluated in order for an article to receive this badge. In addition, they need not be complete in the sense described above. They simply need to be relevant to the study and add value beyond the text in the article. Such artifacts could be something as simple as the data from which the figures are drawn, or as complex as a complete software system under study.
• The authors can provide the DOI at the very end of the AE process and use GitHub or any other convenient way to access their artifacts during AE.

Note that proprietary artifacts need not be included. If they are required to exercise the package then this should be documented, along with instructions on how to obtain them. Proxies for proprietary data should be included so as to demonstrate the analysis.

We usually ask the authors to provide a small/sample data set to validate at least some results from the paper to make sure that their artifact is functional.

Can other users easily reuse and customize this artifact and experimental workflow? For example, can it be used on a different platform, with different benchmarks, data sets, compilers, tools, under different conditions and parameters, etc.?

Unfortunately, the current ACM criteria for awarding this badge are quite vague. We think that such badge should be awarded to artifacts which use any portable workflow framework such as CK with reusable automation "actions". You can find examples of reproduced papers with portable workflows and reusable actions here.

We usually ask the authors to demonstrate how their artifact can work on a different platform, in a different environment, with a different model/data set/software. We want to make sure that it is possible to reuse a given artifact in a different experimental setup.

Report any unexpected artifact behavior (depends on the type of artifact such as unexpected output, scalability issues, crashes, performance variation, etc).

Note that variation of empirical and numerical results is tolerated. In fact it is often unavoidable in computer systems research - see "how to report and compare empirical results" in the AE FAQ page, the SIGPLAN Empirical Evaluation Guidelines, and the NeurIPS reproducibility checklist.

Since it may take weeks and even months to rerun some complex experiments such as deep learning model training, we suggest to use a staged AE where we will first validate that artifacts are functional before the camera ready paper deadline, and then use a separate AE with the full validation of all experimental results with open reviewing and without strict deadlines. We successfully validated a similar approach at the ACM ASPLOS-ReQuEST'18 tournament (SW/HW co-design of Pareto-efficient deep learning) and ADAPT'16, and we saw similar initiatives at the NeurIPS conference.

We are also working with the community on an open-source technology to enable "live" papers where experimental results are continuously validated by the community (see the RPi crowd-tuning paper and the MLPerf/MobileNets crowd-benchmarking demo).

You can find examples of reproduced papers with portable workflows here.

See our related effort at the Supercomputing Student Cluster Competition: common artifact format, automation workflow, SCC'18 application workflow example.