|High-end GPGPU card for the distinguished artifact||$500 for the top experimental workflow in CK format|
Artifact Evaluation for PPoPP 2017
Since our philosophy is that AE should act as a mechanism to help authors prepare their materials and replicate or reproduce experimental results, we spent one more week shepherding these artifacts. During this process we allowed back-and-forth anonymous communication between evaluators and authors to resolve concerns, documentation issues and bugs. At the same time, we also successfully tried an "open reviewing model", when we asked the community to publicly evaluate several artifacts already available at GitHub, GitLab and other project hosting services. This allowed us to find external reviewers who had access to very rare HPC servers or proprietary benchmarks and tools. With the help of such shepherding, a 100% success rate for all 27 artifacts was achieved, which reflects a significant achievement and effort by both authors and evaluators. We thank them all for their hard work!
All papers with evaluated artifacts received an AE seal and were allowed to add up to 2 pages of Artifact Appendix to let readers better understand what was evaluated and how.
Authors of accepted PPoPP 2017 papers will be invited to formally submit their supporting materials to the Artifact Evaluation process. The Artifact Evaluation process is run by a separate committee whose task is to reproduce (at least some) experiments and assess how the artifacts support the work described in the papers. This submission is voluntary and will not influence the final decision regarding the papers.
Papers that successfully go through the Artifact Evaluation process will receive a seal of approval printed on the papers themselves. Authors of such papers will have an option to include their Artifact Appendix to the final paper (up to 2 pages). Authors are also encouraged (though not obliged) to make these materials publicly available upon publication of the proceedings, by including them as "source materials" in the Digital Library.
To encourage reproducible experimentation and participation in artifact evaluation, NVIDIA will give a high-end GPGPU card for the highest ranked artifact! To promote sharing of artifacts and experimental workflows as reusable and customizable components cTuning foundation and dividiti will give $500 for the highest ranked experimental workflow implemented using Collective Knowledge framework.
If you have questions, comments and suggestions on how
to improve artifact submission, reviewing, customization
and reuse, please do not hesitate to get in touch with the AE steering committee!