Artifact Evaluation for Systems Conferences and Journals
We work with the community and ACM to improve methodology and tools for reproducible experimentation, artifact submission / reviewing and open challenges!
About Artifacts Committee Submission Reviewing FAQ Prior AE
Important dates:
Paper decision: 6 December 2017
Artifact submission: 12 December 2017
Technical clarification: 9-11 January 2018
Decision announced: 12 January 2018
Final paper: 15 January 2018
Public CGO-PPoPP AE discussion: 26 Feb 2018 (17:15)

PPoPP AE Chairs:
Malavika Samak
Grigori Fursin
(cTuning foundation / dividiti)

Grigori Fursin, Bruce Childers

Artifact Evaluation for PPoPP 2018

[ Back to PPoPP 2018 conference website ]

Artifact evaluation is over - see accepted artifacts here !

Authors of accepted PPoPP 2018 papers are invited to formally submit their supporting materials to the Artifact Evaluation process (AE). AE is run by a separate committee whose task is to reproduce (at least some) experiments and assess how submitted artifacts support the work described in the papers. This submission is voluntary and will not influence the final decision regarding the papers.

The papers that successfully go through AE will receive a set of ACM badges of approval printed on the papers themselves and available as meta information in the ACM Digital Library. Authors of such papers will have an option to include their Artifact Appendix to the final paper (up to 2 pages). Authors are also encouraged (though not obliged) to make these materials publicly available upon publication of the proceedings, by including them as "source materials" in the Digital Library.

If you have any questions, please check AE FAQs and do not hesitate to contact AE chairs and the steering committee or post your question to our LinkedIn group.

How to submit

Please prepare your artifacts for submission using the following guide. Then, register your submission at the PPoPP AE EasyChair website - you will be asked to submit your paper title, author list, artifact abstract, pdf of your paper with an appendix describing how to access and validate your artifacts, and possible conflicts of interests with AE members. Please, do not forget to provide a list of hardware, software, benchmark and data set dependencies in your artifact abstract - this is essential to find appropriate evaluators!

Note that we collaborate with ACM to unify packing and sharing of artifacts as reusable and customizable components with a unified JSON API with the help of the Collective Knowledge framework (see ACM announcement). You are not obliged to use CK and can write your own scripts to prepare and run experiments. However if you are interested to try it, cTuning foundation offers free help to convert your workflows to the CK format while reusing already existing artifacts (for example, see reusable AI artifacts) for your submission. Feel free to contact Grigori Fursin as soon as possible for more details. Please, check out a distinguished artifact from the joint PPoPP/CGO'17 AE which was shared using the CK framework: GitHub, PDF with appendix.

Reviewing process

Your artifacts will be reviewed according to the following guidelines. Artifacts receiving "met expectations" or above score will pass evaluation and will receive a stamp of approval. The highest ranked artifacts will receive prizes.

If you have questions, comments and suggestions on how to improve artifact submission, reviewing, customization and reuse, please do not hesitate to get in touch with AE chairs and/or steering committee!

Maintained by
cTuning foundation (non-profit R&D organization),
Grigori Fursin and volunteers!
Powered by Collective Knowledge
           Locations of visitors to this page