We aim to make artifact submission as simple as possible. You just need to pack your artifact (code and data) using any publicly available tool you prefer. In some exceptional cases when rare hardware or proprietary software is used, you can arrange a remote access to machine with the pre-installed software.
Then, you need to prepare a small and informal Artifact Evaluation appendix using our AE LaTeX template (common for PPoPP, CGO and PACT conferences - see below) to explain evaluators what your artifacts are and how to use them (you will be allowed to add up to 2 pages of this Appendix to your final camera-ready paper). Please see this PPoPP'16 paper for the example of such AE appendix.
At least two reviewers will follow your guide to replicate your results (for example, exact output match) or reproduce them (for example, varying performance numbers or scalability on a different machine), and will then send you a report with the following overall assessment of your artifact based on our reviewing guidelines:
Since our eventual goal is to promote artifact validation and sharing (rather than naming and shaming problematic artifacts), you will be able to address raised issues during the rebuttal. Furthermore, we allow a small amount of communication between reviewers and authors whenever there are installation/usage problems. In such cases, AE chairs will serve as a proxy to avoid revealing reviewers' identity (the review is blind, i.e. your identity is known to reviewers since your paper is already accepted, but not vice versa).
From our past Artifact Evaluation experience, we have noticed that the most challenging part is to automate and customize experimental workflows. It is even worse, if you need to validate experiments using latest software environment and hardware (rather than quickly outdated VM and Docker images). Most of the time, some ad-hoc scripts are used to implement these workflows. They are very difficult to change and customize, particularly when an evaluator would like to try other compilers, libraries and data sets.
These problems motivated us to develop Collective Knowledge Framework (CK) -
a small, portable and open-source infrastructure and repository to help researchers quickly prototype and share their experimental
workflows with all related artifacts as reusable Python components with a unified JSON API and JSON meta description.
CK supports Linux, MacOS, Windows, Android and reduces the burden of researchers and evaluators by automatically detecting
and resolving all required software dependencies across diverse hardware,
unifying autotuning, statistical analysis and predictive analytics (via scikit-learn, R, etc),
and enabling interactive reports.
Please check out how ARM uses CK to crowdsource benchmarking of real workloads, General Motors to collaboratively optimize Caffe framework, and Imperial College (London) to crowdsource compiler bug detection (see PLDI'15 artifacts in the CK format). If you would like to share your artifacts in the reusable CK format, please check Getting Started Guide, CK portable workflows and list of already shared CK plugins.
PPoPP'15 article together with this LaTeX example. You can change \hspace and \raisebox parameters to better fit stamp to your paper.
We strongly encourage you to submit your AE appendix (up to 2 pages) as an auxiliary material for the Digital Library (while removing all unnecessary or confidential information) along with the final variant of your paper. This will help readers better understand what was evaluated.
Though you are not obliged to publicly release your artifacts (in fact, it is sometimes impossible due to various limitations), we also strongly encourage you to share them with the community (even if they are not open-source). You can release them as an auxiliary material in Digital Libraries together with your AE appendix or use your institutional repository and various public services for code and data sharing.
Even accepted artifacts may have some unforeseen behavior and limitations discovered during evaluation. Now you have a chance to add related notes to your paper as a future work (if you wish)..