This guide (V20151015) should help you describe and submit your artifacts for a review. It gradually evolves based on our past Artifact Evaluations and your feedback (see this presentation with an outcome of the past PPoPP/CGO'15 AE). It should also help you prepare your artifacts for a possible public release, if you plan to do so (for example as an auxiliary material in a Digital Library or on your personal web page).
We are trying to make artifact submission as simple as possible. You just need to pack your artifact (code and data) using any publicly avialable tool you prefer or arrange a remote access to machine with pre-installed software (exceptional cases when rare hardware or proprietary software is used).
Then, you need to prepare a small and informal guide for reviewers using our AE LaTeX template (see below) to explain what are your artifacts, how to access and use them, and what is the expected result (we currently discuss with the conference chairs and ACM to let you keep it as Appendix in your paper if your artifact passes evaluation).
At least two reviewers will follow your guide to replicate your results (for example, exact output match) or reproduce them (for example, varying performance numbers or scalability on a different machine), and will then send you a report with the following overall assessment of your artifact based on our reviewing guidelines:
Since our eventual goal is to promote artifact validation and sharing (rather than naming and shaming problematic artifacts), you will be able to address raised issues during the rebuttal. Furthermore, we allow a small amount of communication between reviewers and authors whenever there are installation/usage problems. In such cases, AE chairs will serve as a proxy to avoid revealing reviewers' identity (the review is blind, i.e. your identity is known to reviewers since your paper is already accepted, but not vice versa).
PPoPP'15 article together with this LaTeX example. You can change \hspace and \raisebox parameters to better fit stamp to your paper.
We strongly encourage you to submit your AE appendix as an auxiliary material for Digital Library (while removing all unnecessary or confidential information) along with the final variant of your paper. This will help readers better understand what was evaluated. We currently discuss procedures with ACM DL colleagues - please stay tuned!
Though you are not obliged to publicly release your artifacts (in fact, it is sometimes impossible due to various limitations), we also strongly encourage you to share them with the community (even if they are not open-source). You can release them as an auxiliary material in Digital Libraries together with your AE appendix or use your institutional repository and various public services for code and data sharing.
Even accepted artifacts may have some unforeseen behavior and limitations discovered during evaluation. Now you have a chance to add related notes to your paper as a future work (if you wish)..
Note, that we are developing open-source Collective Knowledge framework to help researchers share their artifacts as unified, reusable and customizable components together with experimental workflows and interactive articles. If you are interested to know more, please check some online examples and recent DATE'16 paper.