http://ctuning.org/cm/wiki/api.php?action=feedcontributions&feedformat=atom&user=GfursinCollective Mind v1 - User contributions [en]2026-04-28T17:41:03ZUser contributionsMediaWiki 1.25.1http://ctuning.org/cm/wiki/index.php?title=Reproducibility:AE&diff=874Reproducibility:AE2016-09-14T08:32:32Z<p>Gfursin: </p>
<hr />
<div>{{#externalredirect: http://cTuning.org/ae}}<br />
<br />
This page moved to [https://github.com/ctuning/ck/wiki/Enabling-open-science GitHub].<br />
<br />
<p style="text-align: center"><span style="font-size:x-large">'''<span style="font-family: tahoma,geneva,sans-serif">Artifact Evaluation in Computer Engineering</span>'''</span></p><p style="text-align: center"><span style="font-size:x-small">[ [[Reproducibility|back to main page]] ]</span></p><br />
Our mission is to push artifact evaluation to major conferences and journals while improving artifact sharing, reviewing and reusing process.<br />
<br />
We organized AE for the following events:<br />
<br />
*'''[[Reproducibility:AE:PPoPP2016|PPoPP'16]]'''<br />
*'''[[Reproducibility:AE:CGO2016|CGO'16]]'''<br />
*'''[http://www.adapt-workshop.org/2015/ ADAPT'15]'''<br />
*'''[[Reproducibility:AE:PPoPP2015|PPoPP'15]]'''<br />
*'''[[Reproducibility:AE:CGO2015|CGO'15]]'''<br />
*[http://www.adapt-workshop.org/2014 '''ADAPT'14''']</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility&diff=873Reproducibility2016-09-14T08:27:01Z<p>Gfursin: </p>
<hr />
<div>{{#externalredirect: https://github.com/ctuning/ck/wiki/Enabling-open-science}}<br />
<br />
This page moved to [https://github.com/ctuning/ck/wiki/Enabling-open-science GitHub].</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility&diff=872Reproducibility2016-09-13T13:50:51Z<p>Gfursin: Replaced content with "Moved to [https://github.com/ctuning/ck/wiki/Enabling-open-science GitHub]."</p>
<hr />
<div>Moved to [https://github.com/ctuning/ck/wiki/Enabling-open-science GitHub].</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Events&diff=871Reproducibility:Events2016-09-13T13:50:12Z<p>Gfursin: Replaced content with "Moved to [https://github.com/ctuning/ck/wiki/Enabling-open-science-events GitHub]."</p>
<hr />
<div>Moved to [https://github.com/ctuning/ck/wiki/Enabling-open-science-events GitHub].</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Jokes&diff=870Reproducibility:Jokes2016-09-13T13:44:27Z<p>Gfursin: </p>
<hr />
<div>Moved to [https://github.com/ctuning/ck/wiki/Enabling-open-science-jokes GitHub].</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Blogs&diff=869Reproducibility:Blogs2016-09-13T13:40:01Z<p>Gfursin: </p>
<hr />
<div>Moved to [https://github.com/ctuning/ck/wiki/Enabling-open-science-blogs GitHub].</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Articles&diff=868Reproducibility:Articles2016-09-13T13:36:21Z<p>Gfursin: Replaced content with "Moved to [https://github.com/ctuning/ck/wiki/Enabling-open-science-articles GitHub]."</p>
<hr />
<div>Moved to [https://github.com/ctuning/ck/wiki/Enabling-open-science-articles GitHub].</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Lectures&diff=867Reproducibility:Lectures2016-09-13T13:25:32Z<p>Gfursin: </p>
<hr />
<div>Moved to [https://github.com/ctuning/ck/wiki/Enabling-open-science-lectures GitHub].</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Repositories&diff=866Reproducibility:Repositories2016-09-13T13:22:25Z<p>Gfursin: Replaced content with "Moved to [https://github.com/ctuning/ck/wiki/Enabling-open-science-repos GitHub]."</p>
<hr />
<div>Moved to [https://github.com/ctuning/ck/wiki/Enabling-open-science-repos GitHub].</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Datasets&diff=865Reproducibility:Datasets2016-09-13T13:19:01Z<p>Gfursin: </p>
<hr />
<div>Moved to [https://github.com/ctuning/ck/wiki/Enabling-open-science-datasets GitHub].</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Initiatives&diff=864Reproducibility:Initiatives2016-09-13T13:10:39Z<p>Gfursin: Replaced content with "Moved to [https://github.com/ctuning/ck/wiki/Enabling-open-science-initiatives GitHub]."</p>
<hr />
<div>Moved to [https://github.com/ctuning/ck/wiki/Enabling-open-science-initiatives GitHub].</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Links&diff=863Reproducibility:Links2016-09-13T13:10:27Z<p>Gfursin: </p>
<hr />
<div>Moved to [https://github.com/ctuning/ck/wiki/Enabling-open-science-tools GitHub].</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Links&diff=862Reproducibility:Links2016-09-13T13:04:12Z<p>Gfursin: Replaced content with "Moved [https://github.com/ctuning/ck/wiki/Enabling-open-science-tools here]."</p>
<hr />
<div>Moved [https://github.com/ctuning/ck/wiki/Enabling-open-science-tools here].</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility&diff=861Reproducibility2016-04-13T09:09:00Z<p>Gfursin: /* Our events */</p>
<hr />
<div><center><br />
<span style="font-size:x-large"><span style="color: rgb(178, 34, 34)">Enabling collaborative and reproducible computer systems research with an open publication model</span></span><br />
</center><p style="text-align: center">[[File:Ae-stamp-ppopp2015.png|Ae-stamp-ppopp2015.png|link=http://cTuning.org/ae/ppopp2016.html]] [[File:Ae-stamp-cgo.png|Ae-stamp-cgo.png|link=http://cTuning.org/ae/cgo2016.html]] [[File:Logo-validated-by-the-community.png|Logo-validated-by-the-community.png|link=http://adapt-workshop.org/motivation2016.html]] &nbsp; &nbsp;&nbsp; [[File:CTuning foundation logo1.png|cTuning_foundation_logo1.png|link=http://cTuning.org]]&nbsp; &nbsp;&nbsp; [[File:dividiti.png|dividiti.png|link=http://dividiti.com]]</p><p style="text-align: center"></p><p style="text-align: center">This wiki is maintained by the [http://cTuning.org non-profit Tuning foundation].</p><br />
<br />
== News and upcoming events<br/> ==<br />
<br />
* We have released our new, small, open-source, BSD-licensed Collective Knowledge Framework (cTuning 4 aka CK) for collaborative and reproducible R&D: [http://github.com/ctuning/ck Sources at GitHub], [http://github.com/ctuning/ck/wiki documentation], [http://cknowledge.org/repo live repository to crowdsource experiments such as multi-objective program autotuning], [https://play.google.com/store/apps/details?id=openscience.crowdsource.experiments Android app to crowdsource experiments].<br />
*[http://drops.dagstuhl.de/opus/volltexte/2016/5762 Our Dagstuhl report on Artifact Evaluation]<br />
*[http://cTuning.org/ae Artifact Evaluation for computer systems' conferences, workshops and journals]<br />
*[http://cTuning.org/ae/ppopp2016.html PPoPP'16 artifact evaluation]<br />
*[http://cTuning.org/ae/cgo2016.html CGO'16 artifact evaluation]<br />
*[http://adapt-workshop.org ADAPT'16 @ HiPEAC'16] - successfuly featured [http://dl.acm.org/citation.cfm?id=2618142 our open publication model with community-driven reviewing, public Reddit-based discussions and artifact evaluation]<br />
*[http://www.dagstuhl.de/de/programm/kalender/semhp/?semnr=15452 Dagstuhl perspective workshop on artifact evaluation for conferences and journals]<br />
<br />
== Motivation<br/> ==<br />
<br />
Since 2006 we have been trying to solve problems with reproducibility of experimental results in computer engineering ''as a side effect'' of our [http://cTuning.org/project-milepost MILEPOST] , [http://cTuning.org cTuning.org], [http://c-mind.org Collective Mind] and [http://github.com/ctuning/ck Collective Knowledge] projects (speeding up optimization, benchmarking and co-design of computer systems using auto-tuning, big data, predictive analytics and crowdsourcing). <span style="font-size: small">We focus on the following technological and social aspects to enable collaborative, systematic and reproducible research and experimentation particularly related to benchmarking, optimization and co-design of faster, smaller, cheaper, more power efficient and reliable software and hardware:</span><br />
<br />
*developing public and open source repositories of knowledge (see our pilot live repository [[http://cknowledge.org/repo CK] and our vision papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]]);<br />
*developing [http://github.com/ctuning/ck collaborative research and experimentation infrastructure] that can share artifacts as reusable components together with the whole experimental setups (see our papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]];<br />
*evangelizing and enabling new open publication model for online workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*setting up and improving procedure for sharing and evaluating experimental results and all related material for workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*improving sharing, description of dependencies, and statistical reproducibility of experimental results and related material;<br />
*supporting and improving [http://cTuning.org/ae Artifact Evaluation] for major workshops and conferences including CGO and PPoPP.<br />
<br />
See our manifesto and history [http://cTuning.org/history.html here].<br />
<br />
== Our R&D ==<br />
<br />
Together with the community and [http://cTuning.org not-for-profit cTuning foundation] we are working on the following topics:<br />
<br />
*developing tools and methodology to capture, preserve, formalize, systematize, exchange and improve knowledge and experimental results including negative ones<br />
*describing and cataloging whole experimental setups with all related material including algorithms, benchmarks, codelets, datasets, tools, models and any other artifact<br />
*developing specification to preserve experiments including all software and hardware dependencies<br />
*dealing with variability and rising amount of experimental data using statistical analysis, data mining, predictive modeling and other techniques<br />
*developing new predictive analytics techniques to explore large design and optimization spaces<br />
*validating and verifying experimental results by the community<br />
*developing common research interfaces for existing or new tools<br />
*developing common experimental frameworks and repositories (enable automation, re-execution and sharing of experiments)<br />
*sharing rare hardware and computational resources for experimental validation<br />
*implementing previously published experimental scenarios (auto-tuning, run-time adaptation) using common infrastructure<br />
*implementing open access to publications and data (particularly discussing intellectual property IP and legal issues)<br />
*speeding up analysis of "big" experimental data<br />
*developing new (interactive) visualization techniques for "big" experimental data<br />
*enabling interactive articles<br />
<br />
== Our events<br/> ==<br />
<br />
*[http://cTuning.org/ae Artifact Evaluation procedures for computer systems conferences]<br />
*[http://cTuning.org/ae/ppopp2016.html PPoPP'16 artifact evaluation]<br />
*[http://cTuning.org/ae/cgo2016.html CGO'16 artifact evaluation]<br />
*[http://adapt-workshop.org/index2016.html ADAPT'15] @ HiPEAC'15 - workshop on adaptive self-tuning computer systems<br />
*[[Reproducibility:AE:PPoPP2015|PPoPP'15 artifact evaluation]]<br />
*[[Reproducibility:AE:CGO2015|CGO'15 artifact evaluation]]<br />
*[http://adapt-workshop.org/2015 ADAPT'15] @ HiPEAC'15 - workshop on adaptive self-tuning computer systems<br />
*[http://adapt-workshop.org/2014 ADAPT'14] @ HiPEAC'14 - workshop on adaptive self-tuning computer systems [ [http://adapt-workshop.org/2014/program.htm program and publications] ]<br />
*[http://www.occamportal.org/images/reproduce/TETC-SI-REPRODUCE.pdf Special journal issue] on Reproducible &nbsp;Research &nbsp;Methodologies at IEEE TETC<br />
*[http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14] @ PLDI'14<br />
*[http://www.occamportal.org/reproduce REPRODUCE'14] @ HPCA'14<br />
*[http://www.adapt-workshop.org/2014/program.htm ADAPT'14 panel] @ HiPEAC'14<br />
*[http://ctuning.org/making-computer-engineering-a-science-2013 HiPEAC'13 CSW thematic session] @ ACM ECRC "Making computer engineering a science"<br />
*[http://ctuning.org/hipeac3-thematic-session-2012-04 HiPEAC'12 CSW thematic session]<br />
*[http://exadapt.org/2012/program.html ASPLOS/EXADAPT'12 panel] @ ASPLOS'12<br />
*[http://ctuning.org/lab/education cTuning lectures (2008-2010)]<br />
*[http://hal.inria.fr/inria-00436029 GCC Summit'09 discussion]<br />
<br />
== Resources<br/> ==<br />
<br />
*[[Reproducibility:Links|Collection of related tools]]<br />
*[[Reproducibility:Initiatives|Collection of related initiatives]]<br />
*[[Reproducibility:Datasets|Collection of related benchmarks and data sets]]<br />
*[[Reproducibility:Repositories|Collection of public repositories]]<br />
*[[Reproducibility:Lectures|Collection of related lectures]]<br />
*[[Reproducibility:Articles|Collection of related articles]]<br />
*[[Reproducibility:Blogs|Collection of related blogs]]<br />
*[[Reproducibility:Jokes|Collection of jokes]]<br />
*[[Reproducibility:Events|Collection of related events]]<br />
<br />
== Discussions<br/> ==<br />
<br />
*[http://www.linkedin.com/groups/Reproducible-research-experimentation-in-computer-7433414 LinkedIn group on reproducible research]<br />
*[http://groups.google.com/group/collective-knowledge Main mailing list] (general collaborative and reproducible R&D in computer engineering)<br />
*[http://groups.google.com/group/ctuning-discussions cTuning foundation mailing list] (collaborative and reproducible hardware and software benchmarking, auto-tuning and co-design)<br />
<br />
== Follow us<br/> ==<br />
<br />
*[https://twitter.com/c_tuning cTuning foundation twitter]<br />
*[https://www.facebook.com/pages/CrowdTuning-Foundation/668405119902805 cTuning foundation facebook page] (recent)<br />
*[http://dividiti.blogspot.com dividiti blog]<br />
*[http://cknowledge.org/repo Collective Knowledge repository] (new)<br />
<br />
== Archive<br/> ==<br />
<br />
*[http://ctuning.org/wiki/index.php?title=Discussions:New_Publication_Model Outdated cTuning wiki page related to reproducible research and open publication model]<br />
*Outdated cTuning repository for program/processor performance/power/size optimization (2008-2010): [ [http://ctuning.org/wiki/index.php/CDatabase database], [http://ctuning.org/wiki/index.php?title=Special:CPredict web-service for online prediction of optimizations] ]<br />
*[http://c-mind.org/repo Collective Mind repository] (outdated)<br />
<br />
== Acknowledgments<br/> ==<br />
<br />
We would like to thank our colleagues from the [http://cTuning.org/lab/people cTuning foundation], [http://dividiti.com dividiti], [http://www.artifact-eval.org artifact-eval.org colleagues], [http://www.occamportal.org OCCAM project] for their help, feedback, participation and support.</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility&diff=860Reproducibility2016-04-13T09:06:09Z<p>Gfursin: </p>
<hr />
<div><center><br />
<span style="font-size:x-large"><span style="color: rgb(178, 34, 34)">Enabling collaborative and reproducible computer systems research with an open publication model</span></span><br />
</center><p style="text-align: center">[[File:Ae-stamp-ppopp2015.png|Ae-stamp-ppopp2015.png|link=http://cTuning.org/ae/ppopp2016.html]] [[File:Ae-stamp-cgo.png|Ae-stamp-cgo.png|link=http://cTuning.org/ae/cgo2016.html]] [[File:Logo-validated-by-the-community.png|Logo-validated-by-the-community.png|link=http://adapt-workshop.org/motivation2016.html]] &nbsp; &nbsp;&nbsp; [[File:CTuning foundation logo1.png|cTuning_foundation_logo1.png|link=http://cTuning.org]]&nbsp; &nbsp;&nbsp; [[File:dividiti.png|dividiti.png|link=http://dividiti.com]]</p><p style="text-align: center"></p><p style="text-align: center">This wiki is maintained by the [http://cTuning.org non-profit Tuning foundation].</p><br />
<br />
== News and upcoming events<br/> ==<br />
<br />
* We have released our new, small, open-source, BSD-licensed Collective Knowledge Framework (cTuning 4 aka CK) for collaborative and reproducible R&D: [http://github.com/ctuning/ck Sources at GitHub], [http://github.com/ctuning/ck/wiki documentation], [http://cknowledge.org/repo live repository to crowdsource experiments such as multi-objective program autotuning], [https://play.google.com/store/apps/details?id=openscience.crowdsource.experiments Android app to crowdsource experiments].<br />
*[http://drops.dagstuhl.de/opus/volltexte/2016/5762 Our Dagstuhl report on Artifact Evaluation]<br />
*[http://cTuning.org/ae Artifact Evaluation for computer systems' conferences, workshops and journals]<br />
*[http://cTuning.org/ae/ppopp2016.html PPoPP'16 artifact evaluation]<br />
*[http://cTuning.org/ae/cgo2016.html CGO'16 artifact evaluation]<br />
*[http://adapt-workshop.org ADAPT'16 @ HiPEAC'16] - successfuly featured [http://dl.acm.org/citation.cfm?id=2618142 our open publication model with community-driven reviewing, public Reddit-based discussions and artifact evaluation]<br />
*[http://www.dagstuhl.de/de/programm/kalender/semhp/?semnr=15452 Dagstuhl perspective workshop on artifact evaluation for conferences and journals]<br />
<br />
== Motivation<br/> ==<br />
<br />
Since 2006 we have been trying to solve problems with reproducibility of experimental results in computer engineering ''as a side effect'' of our [http://cTuning.org/project-milepost MILEPOST] , [http://cTuning.org cTuning.org], [http://c-mind.org Collective Mind] and [http://github.com/ctuning/ck Collective Knowledge] projects (speeding up optimization, benchmarking and co-design of computer systems using auto-tuning, big data, predictive analytics and crowdsourcing). <span style="font-size: small">We focus on the following technological and social aspects to enable collaborative, systematic and reproducible research and experimentation particularly related to benchmarking, optimization and co-design of faster, smaller, cheaper, more power efficient and reliable software and hardware:</span><br />
<br />
*developing public and open source repositories of knowledge (see our pilot live repository [[http://cknowledge.org/repo CK] and our vision papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]]);<br />
*developing [http://github.com/ctuning/ck collaborative research and experimentation infrastructure] that can share artifacts as reusable components together with the whole experimental setups (see our papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]];<br />
*evangelizing and enabling new open publication model for online workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*setting up and improving procedure for sharing and evaluating experimental results and all related material for workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*improving sharing, description of dependencies, and statistical reproducibility of experimental results and related material;<br />
*supporting and improving [http://cTuning.org/ae Artifact Evaluation] for major workshops and conferences including CGO and PPoPP.<br />
<br />
See our manifesto and history [http://cTuning.org/history.html here].<br />
<br />
== Our R&D ==<br />
<br />
Together with the community and [http://cTuning.org not-for-profit cTuning foundation] we are working on the following topics:<br />
<br />
*developing tools and methodology to capture, preserve, formalize, systematize, exchange and improve knowledge and experimental results including negative ones<br />
*describing and cataloging whole experimental setups with all related material including algorithms, benchmarks, codelets, datasets, tools, models and any other artifact<br />
*developing specification to preserve experiments including all software and hardware dependencies<br />
*dealing with variability and rising amount of experimental data using statistical analysis, data mining, predictive modeling and other techniques<br />
*developing new predictive analytics techniques to explore large design and optimization spaces<br />
*validating and verifying experimental results by the community<br />
*developing common research interfaces for existing or new tools<br />
*developing common experimental frameworks and repositories (enable automation, re-execution and sharing of experiments)<br />
*sharing rare hardware and computational resources for experimental validation<br />
*implementing previously published experimental scenarios (auto-tuning, run-time adaptation) using common infrastructure<br />
*implementing open access to publications and data (particularly discussing intellectual property IP and legal issues)<br />
*speeding up analysis of "big" experimental data<br />
*developing new (interactive) visualization techniques for "big" experimental data<br />
*enabling interactive articles<br />
<br />
== Our events<br/> ==<br />
<br />
*[[Reproducibility:AE:PPoPP2015|PPoPP'15 artifact evaluation]]<br />
*[[Reproducibility:AE:CGO2015|CGO'15 artifact evaluation]]<br />
*[http://adapt-workshop.org ADAPT'15] @ HiPEAC'15 - workshop on adaptive self-tuning computer systems<br />
*[http://adapt-workshop.org/2014 ADAPT'14] @ HiPEAC'14 - workshop on adaptive self-tuning computer systems [ [http://adapt-workshop.org/2014/program.htm program and publications] ]<br />
*[http://www.occamportal.org/images/reproduce/TETC-SI-REPRODUCE.pdf Special journal issue] on Reproducible &nbsp;Research &nbsp;Methodologies at IEEE TETC<br />
*[http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14] @ PLDI'14<br />
*[http://www.occamportal.org/reproduce REPRODUCE'14] @ HPCA'14<br />
*[http://www.adapt-workshop.org/2014/program.htm ADAPT'14 panel] @ HiPEAC'14<br />
*[http://ctuning.org/making-computer-engineering-a-science-2013 HiPEAC'13 CSW thematic session] @ ACM ECRC "Making computer engineering a science"<br />
*[http://ctuning.org/hipeac3-thematic-session-2012-04 HiPEAC'12 CSW thematic session]<br />
*[http://exadapt.org/2012/program.html ASPLOS/EXADAPT'12 panel] @ ASPLOS'12<br />
*[http://ctuning.org/lab/education cTuning lectures (2008-2010)]<br />
*[http://hal.inria.fr/inria-00436029 GCC Summit'09 discussion]<br />
<br />
== Resources<br/> ==<br />
<br />
*[[Reproducibility:Links|Collection of related tools]]<br />
*[[Reproducibility:Initiatives|Collection of related initiatives]]<br />
*[[Reproducibility:Datasets|Collection of related benchmarks and data sets]]<br />
*[[Reproducibility:Repositories|Collection of public repositories]]<br />
*[[Reproducibility:Lectures|Collection of related lectures]]<br />
*[[Reproducibility:Articles|Collection of related articles]]<br />
*[[Reproducibility:Blogs|Collection of related blogs]]<br />
*[[Reproducibility:Jokes|Collection of jokes]]<br />
*[[Reproducibility:Events|Collection of related events]]<br />
<br />
== Discussions<br/> ==<br />
<br />
*[http://www.linkedin.com/groups/Reproducible-research-experimentation-in-computer-7433414 LinkedIn group on reproducible research]<br />
*[http://groups.google.com/group/collective-knowledge Main mailing list] (general collaborative and reproducible R&D in computer engineering)<br />
*[http://groups.google.com/group/ctuning-discussions cTuning foundation mailing list] (collaborative and reproducible hardware and software benchmarking, auto-tuning and co-design)<br />
<br />
== Follow us<br/> ==<br />
<br />
*[https://twitter.com/c_tuning cTuning foundation twitter]<br />
*[https://www.facebook.com/pages/CrowdTuning-Foundation/668405119902805 cTuning foundation facebook page] (recent)<br />
*[http://dividiti.blogspot.com dividiti blog]<br />
*[http://cknowledge.org/repo Collective Knowledge repository] (new)<br />
<br />
== Archive<br/> ==<br />
<br />
*[http://ctuning.org/wiki/index.php?title=Discussions:New_Publication_Model Outdated cTuning wiki page related to reproducible research and open publication model]<br />
*Outdated cTuning repository for program/processor performance/power/size optimization (2008-2010): [ [http://ctuning.org/wiki/index.php/CDatabase database], [http://ctuning.org/wiki/index.php?title=Special:CPredict web-service for online prediction of optimizations] ]<br />
*[http://c-mind.org/repo Collective Mind repository] (outdated)<br />
<br />
== Acknowledgments<br/> ==<br />
<br />
We would like to thank our colleagues from the [http://cTuning.org/lab/people cTuning foundation], [http://dividiti.com dividiti], [http://www.artifact-eval.org artifact-eval.org colleagues], [http://www.occamportal.org OCCAM project] for their help, feedback, participation and support.</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Main_Page&diff=859Main Page2016-04-01T21:08:56Z<p>Gfursin: </p>
<hr />
<div><br />
<br />
<br />
<br />
<p style="text-align: center"><span style="font-size:x-large"><br />
'''''In September 2015, we have released a brand new Collective Knowledge framework for collaborative, systematic and reproducible computer system's research, and moved all further developments to GitHub: [http://github.com/ctuning/ck src], [http://github.com/ctuning/ck/wiki docs]'''''<br />
</span></p><br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
----<br />
<br />
'''This page is not updated since summer 2015 - see this [http://cTuning.org/reproducibility-wiki wiki] instead!'''''<br />
<br />
<br />
<br />
<p style="text-align: center"><span style="font-size:x-large">'''<span style="font-family: tahoma,geneva,sans-serif">Collective Mind<br/><span style="font-size:large">''towards collaborative, systematic and reproducible computer engineering''</span></span>'''</span></p><br />
{| border="0" cellpadding="10" cellspacing="1" width="1118"<br />
|- valign="top"<br />
| width="170" | <p style="text-align: center">[[File:Validate by community.png|Validate by community.png|link=http://cTuning.org/reproducibility]]</p><p style="text-align: center">[[File:CTuning foundation logo1.png|none|CTuning foundation logo1.png|link=http://cTuning.org]]</p><br />
| rowspan="1" | <br />
We are a group of researchers working with the community on a new methodology, infrastructure and repository to enable collaborative and reproducible research and experimentation in computer engineering as a side effect of our projects on combining performance/energy/size auto-tuning with run-time adaptation, crowdsourcing, big data and predictive analytics ('''see our [http://cTuning.org/history manifesto and history]'''). Our approach, in turn, helped to enable [http://c-mind.org/reproducibility new publication model] where all research material (code and data artifacts) is shared along with articles to be continuously discussed, validated and improved by the community! To evangelize this community-driven approach and set up an example, we started releasing all our benchmarks, data sets, predictive models and tools with unified interfaces since 2007 at [http://cTuning.org cTuning.org ]and later at [http://c-mind.org/repo c-mind.org/repo]. We use&nbsp; [http://adapt-workshop.org ADAPT workshop] on self-tuning computing systems to validate our new research and publication model. We hope that it can complement well recent academic initiatives on reproducible research at major conferences while focusing more on [http://c-mind.org/reproducibility technological aspects] of collaborative and reproducible research in computer engineering (rather than just sharing and validating artifacts).<br />
<p style="text-align: center">''This R&D is supported by [http://cTuning.org the cTuning foundation].''</p><br />
|}<br />
<br />
= Our long term vision<br/> =<br />
<br />
With the rapid advances in information technology and all other fields of science comes dramatic growth in the amount of processing data ("big data"). Scientists, engineers and students are drowning in experimental data and often have to divert their research path towards data management, mining, and visualization. Such approaches often require additional interdisciplinary skills including statistical analysis, machine learning, programming and parallelization, database management, and Internet technologies, which still few researchers have or can afford to learn in parallel with their main research work. Multiple frameworks, languages and public data repositories started appearing recently to enable collaborative data analysis and processing but they are often either covering very narrow research topics and too simplistic (just data and code sharing) or very formal and still require special programming skills often including Object Oriented Programming.<br />
<br />
Collective Mind technology (cM) attempts to fill in this gap by providing researchers and companies a simple, portable, technology-neutral and practically transparent way to gradually systematize and classify all their data, code and tools. Open source cM framework and repository fully relies on customizable public&nbsp; or private plugins (mostly written in python with support of any other language through OpenME interface) to gradually describe and classify similar data and code objects, or abstract interfaces of ever changing tools thus effectively protecting researchers' experimental setups. cM helps to easily preserve any complex research artifact (collection of files, benchmarks, codelets, datasets, tools, traces, models) with gradually and easily extensible JSON based meta description including classification, properties and either direct or semantic data connections. Furthermore, meta descriptions of all&nbsp; data can be transparently and easily indexed using third-party [http://www.elasticsearch.org ElasticSearch] enabling very fast and complex queries. At the same time, all research artifacts can be exposed to any public or workgroup user through unified web services to crowdsource experimentation, ranking, online learning and knowledge management.<br />
<p style="text-align: center">[[File:C-mind-picture.png]]</p><br />
cM uses agile top-down methodology originating from physics to represent any experimental scenario and gradually decompose it into connected plugins with associated data or compose it from already shared plugins similar to "research LEGO". Universal structure immediately enables replay mode for any experiment, thus making this framework suitable for recent projects on reproducibility of experimental results and new publication model where experiments and techniques are validated, ranked and improved by the community. For example, we easily moved all our past R&D on program and architecture multi-objective auto-tuning, co-design and dynamic adaptation to cM plugins and gradually make them available together with all research artifacts at [http://c-mind.org/repo http://c-mind.org/repo]. We hope that cM will be useful to a broad range of researchers and companies either as an open-source, community driven solution to systematize their research and experimentation, or possibly as an intermediate step before investing into more complex or commercial knowledge management systems.<br />
<br />
''Here is [[Reproducibility:Links|our list of links]] to initiatives, publications, tools and techniques related to collaborative and reproducible reserarch, experimentation and development in computer engineering.''<br />
<br />
==== Related Collective Mind publications and presentations<br/> ====<br />
<br />
*Our new open publication model proposal (2014) ([http://dl.acm.org/citation.cfm?id=2618142 ACM pdf], [http://arxiv.org/abs/1406.4020 arXiv pdf])- it summarizes our practical experience with sharing and reviewing experimental results and research artifacts since 2007; we plan to validate it at our [http://adapt-workshop.org ADAPT'15]<br />
*[http://hal.inria.fr/inria-00436029 GCC Summit publication (2009)] - introducing our vision on reproducible research and describing cTuning.org framework for collaborative and reproducible program and architecture analysis, optimization and co-design (all code and data for machine learning based compiler MILEPOST GCC have been publicly shared at [http://cTuning.org cTuning.org])<br />
*[http://arxiv.org/abs/1308.2410 INRIA/arXiv technical report (2013)] - introducing long term Collective Mind vision; considerably updated journal version will be available in Fall 2014<br />
*[http://c-mind.org/repo/?view_cid=shared1:dissemination.publication:530e5f456ea259de ACM TACO publication (2012)] - introducing crowdtuning (crowdsourcing auto-tuning)<br />
*[http://c-mind.org/repo/?view_cid=shared1:dissemination.publication:a31e374796869125 IJPP publication (2011)] - introducing machine learning based compiler, cTuning.org and reproducible R&D on program and architecture optimization<br />
*[http://www.slideshare.net/GrigoriFursin/presentation-fursin-hpsc2013fursin1 Long term vision slides] - "Systematizing tuning of computer systems using crowdsourcing and statistics"<br />
*[http://www.slideshare.net/GrigoriFursin/presentation-fursin-hpsc2013fursin2 cM basics slides] - "Collective Mind infrastructure and repository to crowdsource auto-tuning"<br />
<br />
= Public repository of knowledge<br/> =<br />
<br />
''Do not waste your research material - use Collective Mind Framework and Repository to describe, run and share your experiments with the community!''<br />
<br />
*[http://c-mind.org/repo Beta live Collective Mind repository] (3rd generation opened in 2013 substituting previous cTuning repository and infrastructure available since 2008) - we described and shared all our past research developments, codelets, benchmarks, data sets, models, statistical analysis, modeling and online learning plugins and tools to start top-down analysis and optimization of existing computer systems. We used it as the first practical example to motivate new publication model where all research artifacts are continuously shared, validated and improved by the community. After many years, it seems that community finally started moving in this direction and we even see some related initiatives in major conferences including OOPSLA and PLDI. We believe that our project and feedback from the community collected since 2006 is complementary and can help with various technological aspects of collaborative and reproducible research in computer engineering.<br />
<br />
= Common infrastructure and support tools =<br />
<br />
*[[Tools:CM|Collective Mind Infrastructure (cM)]] - plugin-based framework and repository for collaborative, systematic and reproducible research and experimentation<br />
**[[Tools:OpenME|OpenME]] - universal and simple event-based interface to "open up" black box applications and third-party tools such as GCC, LLVM and Open64 to be able to monitor, learn and predict any fine-grain optimization decision inside through external plugins<br />
**[[Tools:Alchemist|Alchemist]] - OpenME plugin to convert compilers into interactive analysis and optimization toolsets<br />
<br />
= Events =<br />
<br />
*[http://ctuning.org/cm/wiki/Events%3ATRUST2014 Workshop TRUST 2014 on reproducible research methodologies and new publication models] @ PLDI 2014 (Edinburgh, UK)<br />
*[http://www.occamportal.org/reproduce Workshop REPRODUCE 2014 on reproducible research methodologies and new publication models] @ HPCA 2014 (Orlando, Florida, USA)<br />
*[http://adapt-workshop.org/program.htm Panel on reproducible research methodologies and new publication models] at ADAPT 2014 @ HiPEAC 2014 (Vienna, Austria)<br />
*[http://www.hipeac.net/thematic-session/making-computer-engineering-science Thematic session on making computer engineering a science] @&nbsp; ACM ECRC 2013 / HiPEAC computing week 2013 (Paris, France)<br />
*[http://www.hipeac.net/thematic-session/collective-characterization-optimization-and-design-computer-systems Thematic session on collective characterization, optimization and design of computer systems] @ HiPEAC spring computing week 2012&nbsp; (Goteborg, Sweden)<br />
*[http://www.hipeac.net/conference/pisa/speedup Tutorial on Speedup-Test: Statistical Methodology to Evaluate Program Speedups and their Optimisation Techniques] @ HiPEAC 2010 (Pisa, Italy)<br />
*[http://c-mind.org/repo/?view_cid=77154d189d2e226c:0053bdf524fb9a58 Tutorial on cTuning tools for collaborative and reproducible program and architecture characterization and auto-tuning] @ HiPEAC computing systems week 2009 (Infineon, Munich, Germany)<br />
*[http://hal.inria.fr/inria-00436029 Public discussion on collaborative and reproducible analysis, design and optimization of computer systems] @ GCC Summit 2009 (Montreal, Canada)<br />
<br />
= Current customized usage scenarios<br/> =<br />
<br />
Designing novel many-core computer systems becomes intolerably complex, &nbsp;ad-hoc, costly and error prone due to limitations of available technology, enormous number of available design and optimization choices, and complex interactions between all software and hardware components. Empirical auto-tuning combined with run-time adaptation and machine learning has been demonstrating good potential to address above challenges for more than a decade but still far from the widespread production use due to unbearably long exploration and training times, ever changing tools and their interfaces, lack of a common experimental methodology, and lack of unified mechanisms for knowledge building and exchange apart from publications where reproducibility of results is often not even considered. Since 1993, we have spent more time on preparing and analyzing huge amount of heterogeneous experiments for self-tuning machine-learning based computer systems or trying to validate and reproduce others research results rather than on <span data-scayt_word="exending" data-scaytid="24">exending</span> our novel ideas.<br />
<br />
In 2007, we decided to start collaborative systematization and unification of design and optimization of computer systems combined with a [http://cTuning.org/cm-journal new publication model] where experimental results are validated by the community. One of the possible promising solutions is to combine public repository of knowledge with online auto-tuning, machine learning and <span data-scayt_word="crowdsourcing" data-scaytid="25">crowdsourcing</span> techniques where <span data-scayt_word="HiPEAC" data-scaytid="26">HiPEAC</span> and <span data-scayt_word="cTuning" data-scaytid="27">cTuning</span> communities already have a good practical experience. Such collaborative approach should allow community to continuously validate, systematize and improve collective knowledge about computer systems, and extrapolate it to build faster, more power efficient and reliable computer systems. It can also help to restore the attractiveness of computer engineering making it a more systematic and rigorous discipline rather than "hacking".<br />
<br />
We develop <span data-scayt_word="cTuning" data-scaytid="28">cTuning</span> collaborative research and development infrastructure and repository (current version is <span data-scayt_word="cTuning3" data-scaytid="30">cTuning3</span> aka Collective Mind) that enables:<br />
<br />
*gradual decomposition and <span data-scayt_word="parametrization" data-scaytid="32">parametrization</span> of complex computer systems and experiments into unified and inter-connected Collective Mind modules (components or plugins) with extensible meta-information<br />
*easy co-existance of multiple versions of tools and libraries<br />
*implementation of experimental pipelines with all related artifacts necessary for collaborative and reproducible research and experimentation<br />
*collection and sharing of statistics, benchmarks, <span data-scayt_word="codelets" data-scaytid="31">codelets</span>, tools, data sets and predictive models from the community<br />
*<span data-scayt_word="systematizaton" data-scaytid="34">systematizaton</span> of optimization, design space exploration and run-time adaptation techniques (co-design and auto-tuning)<br />
*collaborative evaluation and improvement of various data mining, classification and predictive modeling techniques for off-line and on-line auto-tuning<br />
*new publication model (workshops, conferences, journals) with validation of experimental results by the community<br />
<br />
Current <span data-scayt_word="cM" data-scaytid="35">cM</span> version includes public benchmarks, datasets, tools, techniques and some stats from past [http://cTuning.org/lab/people/gfursin <span data-scayt_word="Grigori" data-scaytid="36">Grigori</span> <span data-scayt_word="Fursin&#039;s" data-scaytid="39">Fursin's</span> research]:<br />
<br />
*support for most <span data-scayt_word="OSes" data-scaytid="40">OSes</span> and platforms (Linux, Android, Windows; servers, cloud nodes, mobiles, laptops, tablets, <span data-scayt_word="supercomputers" data-scaytid="41">supercomputers</span>)<br />
*multiple benchmarks (<span data-scayt_word="cBench" data-scaytid="43">cBench</span>, <span data-scayt_word="polybench" data-scaytid="45">polybench</span>, <span data-scayt_word="SPEC95" data-scaytid="46">SPEC95</span>,<span data-scayt_word="SPEC2000" data-scaytid="47">SPEC2000</span>,<span data-scayt_word="SPEC2006" data-scaytid="48">SPEC2006</span>,<span data-scayt_word="EEMBC" data-scaytid="49">EEMBC</span>,etc), hundreds of MILEPOST/CAPS <span data-scayt_word="codelets" data-scaytid="42">codelets</span>, <span data-scayt_word="thosands" data-scaytid="51">thosands</span> of <span data-scayt_word="cBench" data-scaytid="44">cBench</span> datasets<br />
*multiple compilers (<span data-scayt_word="GCC" data-scaytid="52">GCC</span>, <span data-scayt_word="LLVM" data-scaytid="53">LLVM</span>, <span data-scayt_word="Open64" data-scaytid="54">Open64</span>, <span data-scayt_word="PathScale" data-scaytid="55">PathScale</span>, Intel, IBM, <span data-scayt_word="PGI" data-scaytid="56">PGI</span>)<br />
*tools for program and architecture characterization (MILEPOST <span data-scayt_word="GCC" data-scaytid="57">GCC</span> for semantic features and code patterns; hardware counters for dynamic analysis)<br />
*plugins for powerful visualization and data export in various formats<br />
*experimental pipeline for universal program and architecture co-design, auto-tuning, performance/energy modeling and machine learning<br />
*<span data-scayt_word="OpenME" data-scaytid="59">OpenME</span> interface to instrument programs or statically enable adaptive binaries through <span data-scayt_word="multi-versioning" data-scaytid="62">multi-versioning</span> and decision trees for run-time adaptation/scheduling while easily mixing CPU/<span data-scayt_word="CUDA" data-scaytid="63">CUDA</span>/<span data-scayt_word="OpenCL" data-scaytid="64">OpenCL</span> <span data-scayt_word="codelets" data-scaytid="60">codelets</span> or any other heterogeneous programming models<br />
*plugins for online auto-tuning and performance model building<br />
*machine-learning enabled self-tuning <span data-scayt_word="cTuning" data-scaytid="66">cTuning</span> CC compiler that can wrap any existing compiler while using crowd-tuning and collective knowledge to continuously improve its own behavior<br />
*plugins for universal <span data-scayt_word="P2P" data-scaytid="69">P2P</span> data exchange through <span data-scayt_word="cM" data-scaytid="67">cM</span> web services<br />
*optimization statistics for various ARM, Intel and NVidia chips<br />
<br />
<br />
See [http://cTuning.org/reproducibility-wiki reproducibility wiki for further details].</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Links&diff=858Reproducibility:Links2015-10-27T11:54:02Z<p>Gfursin: </p>
<hr />
<div>[ '''''[[Reproducibility|Back to main page]]''''' ]<br />
<br />
=== '''Assorted (offline and online) tools'''<br/> ===<br />
<br />
*[https://www.docker.io Docker tool] - pack, ship and run applications as a lightweight container<br />
*[http://www.virtualbox.org VirtualBox] - create VM images<br />
*[http://www.occamportal.org OCCAM] - open curation for computer architecture modeling<br />
*[http://vida-nyu.github.io/reprozip ReproZip] - automatically packing experiments ([https://www.usenix.org/conference/tapp13/technical-sessions/presentation/chirigati related article])<br />
*[http://reproducible.io CARE tool] - Comprehensive Archiver for Reproducible Execution<br />
*[http://rr-project.org RR] - Mozilla project: records nondeterministic executions and debugs them deterministically<br />
*[http://www.pgbovine.net/cde.html CDE tool] - automatically create portable Linux applications with all dependencies<br />
*[http://ipython.org/notebook.html IPython Notebook] - a web-based interactive computational environment where you can combine code execution, text, mathematics, plots and rich media into a single document<br />
*[http://www.rstudio.com R-studio] - open source and enterprise-ready professional software for R<br />
*[https://www.codalab.org Codelab] - an experimental platform for collaboration and competition<br />
*[https://www.grid5000.fr Grid5000] - large-scale and versatile testbed for experiment-driven research in all areas of computer science, with a focus on parallel and distributed computing including Cloud, HPC and Big Data<br />
*[http://www.mygrid.org.uk MyGrid] - develops a suite of tools designed to "help e-Scientists get on with science and get on with scientists"<br />
*[http://figshare.com FigShare] - managing research in a cloud<br />
*[http://www.runmycode.org RunMyCode] - online workflows<br />
*[https://www.aptlab.net AptLab] - online workflows<br />
*[http://www.taverna.org.uk Taverna] - designing and executing workflows<br />
*[http://boinc.berkeley.edu BOINC] - open-source software for volunteer computing and grid computing<br />
*[https://mulcyber.toulouse.inra.fr/projects/ngspipelines NGS pipelines] - integrates pipelines and user interfaces to help biologists to analyse data outputed from biological applications such as RNAseq, sRNAseq, ChipSeq, BS-seq<br />
*[http://www.seek4science.org SEEK for Science] - finding, sharing and exchanging Data, Models, Simulations and Processes in Science<br />
*[http://www.cs.umd.edu/projects/skoll/Skoll/Home.html Skoll] - a process & Infrastructure for Distributed, continuous Quality assurance<br />
*[http://nepi.inria.fr NEPI] - simplifying network experimentation<br />
*[http://pgbovine.net/burrito.html Burrito] - rethinking the Electronic Lab Notebook<br />
*[http://orgmode.org Org mode] - keeping notes, maintaining TODO lists, planning projects, and authoring documents with a fast and effective plain-text system<br />
*[http://hal.inria.fr/inria-00436029 open source cTuning technology] - towards collaborative and reproducible performance autotuning via public repository of optimization knowledge, crowdsourcing, machine learning and collective intelligence (2006-cur.)<br />
<br />
=== '''Collective Knowledge Framework'''<br/> ===<br />
<br />
*[http://github.com/ctuning/ck Collective Knowledge Framework] - tiny, open source, python application (~200Kb) to help researchers decompose their ad-hoc experimental workflows, benchmarks, data sets, scripts and VM/Docker images into reusable and customizable components with Python wrappers and unified JSON API while using already installed and possibly proprietary tools, benchmarks and data sets:<br />
<br />
**[http://cknowledge.org/repo example of a CK-based interactive article with reproducible OpenCL autotuning experiments]<br />
**[http://github.com/ctuning/reproduce-clsmith PLDI'15 artifact converted to CK format]<br />
**[http://github.com/ctuning/reproduce-pamela-project Artifact repository for UK PAMELA project]<br />
**[http://github.com/ctuning/reproduce-carp-project Artifact repository for EU FP7 CARP project]<br />
**[http://github.com/ctuning/ctuning-programs Example of benchmarks shared in CK format]<br />
**[http://github.com/ctuning/ctuning-datasets-min Example of data sets shared in CK format]<br />
**[https://drive.google.com/folderview?id=0B-wXENVfIO82dzYwaUNIVElxaGc&usp=sharing Examples of CK repositories shared as zip files]<br />
**[http://github.com/ctuning/ck/wiki Full documentation]</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility&diff=857Reproducibility2015-10-27T11:49:36Z<p>Gfursin: </p>
<hr />
<div><center><br />
<span style="font-size:x-large"><span style="color: rgb(178, 34, 34)">Enabling collaborative and reproducible computer systems research with an open publication model</span></span><br />
</center><p style="text-align: center">[[File:Ae-stamp-ppopp2015.png|Ae-stamp-ppopp2015.png|link=http://cTuning.org/event/ae-ppopp2016]] [[File:Ae-stamp-cgo.png|Ae-stamp-cgo.png|link=http://cTuning.org/event/ae-cgo2016]] [[File:Logo-validated-by-the-community.png|Logo-validated-by-the-community.png|link=http://cTuning.org/reproducibility]] &nbsp; &nbsp;&nbsp; [[File:CTuning foundation logo1.png|cTuning_foundation_logo1.png|link=http://cTuning.org]]&nbsp; &nbsp;&nbsp; [[File:dividiti.png|dividiti.png|link=http://dividiti.com]]</p><p style="text-align: center"></p><p style="text-align: center">This wiki is maintained by the [http://cTuning.org non-profit Tuning foundation].</p><br />
== News and upcoming events<br/> ==<br />
<br />
* ''We have released our new, open-source, BSD-licensed Collective Knowledge Framework (cTuning 4 aka CK) for collaborative and reproducible R&D: [http://github.com/ctuning/ck GitHub] with an [http://github.com/ctuning/ck/wiki online documentation] and [http://cknowledge.org/repo live demo repository].''<br />
*''[http://cTuning.org/event/ae-ppopp2016 PPoPP'16 artifact evaluation]''<br />
*''[http://cTuning.org/event/ae-cgo2016 CGO'16 artifact evaluation]''<br />
*''[http://adapt-workshop.org ADAPT'16 @ HiPEAC'16] - features [http://dl.acm.org/citation.cfm?id=2618142 our open publication model with community-driven reviewing, reddit-based discussions and artifact evaluation]''<br />
*''[http://www.dagstuhl.de/de/programm/kalender/semhp/?semnr=15452 Dagstuhl perspective workshop on artifact evaluation for conferences and journals]''<br />
<br />
== Motivation<br/> ==<br />
<br />
Since 2006 we have been trying to solve problems with reproducibility of experimental results in computer engineering ''as a side effect'' of our [http://cTuning.org/project-milepost MILEPOST] , [http://cTuning.org cTuning.org], [http://c-mind.org Collective Mind] and [http://github.com/ctuning/ck Collective Knowledge] projects (speeding up optimization, benchmarking and co-design of computer systems using auto-tuning, big data, predictive analytics and crowdsourcing). <span style="font-size: small">We focus on the following technological and social aspects to enable collaborative, systematic and reproducible research and experimentation particularly related to benchmarking, optimization and co-design of faster, smaller, cheaper, more power efficient and reliable software and hardware:</span><br />
<br />
*developing public and open source Collective Mind repositories of knowledge (see our pilot live repository [[http://cknowledge.org/repo CK], [http://c-mind.org/repo cMind]] and our vision papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]]);<br />
*developing [http://github.com/ctuning/ck collaborative research and experimentation infrastructure] that can share artifacts as reusable components together with the whole experimental setups (see our papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]];<br />
*evangelizing and enabling new open publication model for online workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*setting up and improving procedure for sharing and evaluating experimental results and all related material for workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*improving sharing, description of dependencies, and statistical reproducibility of experimental results and related material;<br />
*supporting and improving [http://cTuning.org/ae Artifact Evaluation] for major workshops and conferences including CGO and PPoPP.<br />
<br />
See our manifesto and history [http://cTuning.org/history here].<br />
<br />
== Our R&D ==<br />
<br />
Together with the community and [http://cTuning.org not-for-profit cTuning foundation] we are working on the following topics:<br />
<br />
*developing tools and methodology to capture, preserve, formalize, systematize, exchange and improve knowledge and experimental results including negative ones<br />
*describing and cataloging whole experimental setups with all related material including algorithms, benchmarks, codelets, datasets, tools, models and any other artifact<br />
*developing specification to preserve experiments including all software and hardware dependencies<br />
*dealing with variability and rising amount of experimental data using statistical analysis, data mining, predictive modeling and other techniques<br />
*developing new predictive analytics techniques to explore large design and optimization spaces<br />
*validating and verifying experimental results by the community<br />
*developing common research interfaces for existing or new tools<br />
*developing common experimental frameworks and repositories (enable automation, re-execution and sharing of experiments)<br />
*sharing rare hardware and computational resources for experimental validation<br />
*implementing previously published experimental scenarios (auto-tuning, run-time adaptation) using common infrastructure<br />
*implementing open access to publications and data (particularly discussing intellectual property IP and legal issues)<br />
*speeding up analysis of "big" experimental data<br />
*developing new (interactive) visualization techniques for "big" experimental data<br />
*enabling interactive articles<br />
<br />
== Our events<br/> ==<br />
<br />
*[[Reproducibility:AE:PPoPP2015|PPoPP'15 artifact evaluation]]<br />
*[[Reproducibility:AE:CGO2015|CGO'15 artifact evaluation]]<br />
*[http://adapt-workshop.org ADAPT'15] @ HiPEAC'15 - workshop on adaptive self-tuning computer systems<br />
*[http://adapt-workshop.org/2014 ADAPT'14] @ HiPEAC'14 - workshop on adaptive self-tuning computer systems [ [http://adapt-workshop.org/2014/program.htm program and publications] ]<br />
*[http://www.occamportal.org/images/reproduce/TETC-SI-REPRODUCE.pdf Special journal issue] on Reproducible &nbsp;Research &nbsp;Methodologies at IEEE TETC<br />
*[http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14] @ PLDI'14<br />
*[http://www.occamportal.org/reproduce REPRODUCE'14] @ HPCA'14<br />
*[http://www.adapt-workshop.org/2014/program.htm ADAPT'14 panel] @ HiPEAC'14<br />
*[http://ctuning.org/making-computer-engineering-a-science-2013 HiPEAC'13 CSW thematic session] @ ACM ECRC "Making computer engineering a science"<br />
*[http://ctuning.org/hipeac3-thematic-session-2012-04 HiPEAC'12 CSW thematic session]<br />
*[http://exadapt.org/2012/program.html ASPLOS/EXADAPT'12 panel] @ ASPLOS'12<br />
*[http://ctuning.org/lab/education cTuning lectures (2008-2010)]<br />
*[http://hal.inria.fr/inria-00436029 GCC Summit'09 discussion]<br />
<br />
== Resources<br/> ==<br />
<br />
*[[Reproducibility:Links|Collection of related tools]]<br />
*[[Reproducibility:Initiatives|Collection of related initiatives]]<br />
*[[Reproducibility:Datasets|Collection of related benchmarks and data sets]]<br />
*[[Reproducibility:Repositories|Collection of public repositories]]<br />
*[[Reproducibility:Lectures|Collection of related lectures]]<br />
*[[Reproducibility:Articles|Collection of related articles]]<br />
*[[Reproducibility:Blogs|Collection of related blogs]]<br />
*[[Reproducibility:Jokes|Collection of jokes]]<br />
*[[Reproducibility:Events|Collection of related events]]<br />
<br />
== Discussions<br/> ==<br />
<br />
*[http://www.linkedin.com/groups/Reproducible-research-experimentation-in-computer-7433414 LinkedIn group on reproducible research]<br />
*[http://groups.google.com/group/collective-knowledge Main mailing list] (general collaborative and reproducible R&D in computer engineering)<br />
*[http://groups.google.com/group/ctuning-discussions cTuning foundation mailing list] (collaborative and reproducible hardware and software benchmarking, auto-tuning and co-design)<br />
<br />
== Follow us<br/> ==<br />
<br />
*[https://twitter.com/c_tuning cTuning foundation twitter]<br />
*[https://www.facebook.com/pages/CrowdTuning-Foundation/668405119902805 cTuning foundation facebook page] (recent)<br />
*[http://dividiti.blogspot.com dividiti blog]<br />
*[http://c-mind.org/repo Collective Mind repository] (outdated)<br />
*[http://cknowledge.org/repo Collective Knowledge repository] (new)<br />
<br />
== Archive<br/> ==<br />
<br />
*[http://ctuning.org/wiki/index.php?title=Discussions:New_Publication_Model Outdated cTuning wiki page related to reproducible research and open publication model]<br />
*Outdated cTuning repository for program/processor performance/power/size optimization (2008-2010): [ [http://ctuning.org/wiki/index.php/CDatabase database], [http://ctuning.org/wiki/index.php?title=Special:CPredict web-service for online prediction of optimizations] ]<br />
<br />
== Acknowledgments<br/> ==<br />
<br />
We would like to thank our colleagues from the [http://cTuning.org/lab/people cTuning foundation], [https://www.linkedin.com/company/dividiti dividiti], [http://www.artifact-eval.org artifact-eval.org], [http://www.occamportal.org OCCAM project] for their help, feedback, participation and support.</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:AE:PPoPP2016&diff=856Reproducibility:AE:PPoPP20162015-10-27T09:38:55Z<p>Gfursin: </p>
<hr />
<div>{{#externalredirect: http://cTuning.org/ae/ppopp2016.html}}<br />
<br />
<center><br />
{| border="0" cellpadding="15" cellspacing="1"<br />
|-<br />
| style="text-align: right width: 200px" | [[File:Ae-stamp-ppopp2015.png]]<br/><br />
| <p style="text-align: center"><span style="font-size:xx-large">'''<span style="font-family: tahoma,geneva,sans-serif">Artifact Evaluation for PPoPP'16</span>'''</span></p><p style="text-align: center"><span style="font-size:small">[ [http://conf.researchr.org/home/PPoPP-2016 Back to PPoPP'16 conference website] ]</span></p><p style="text-align: center"></p><p style="text-align: center"></p><p style="text-align: center"></p><br />
|}<br />
<br />
----<br />
<br />
{| style="width: 97%" border="0" cellpadding="20" cellspacing="1"<br />
|-<br />
| style="width: 32%" valign="top" | <div class="span4"><br />
<span style="font-size:large">'''<span style="font-family: tahoma,geneva,sans-serif">Important dates</span>'''</span><br />
<br />
*Paper decision: '''9 Nov 2015'''<br />
*Artifact submission: '''20 Nov 2015 AoE'''<br />
*Technical clarification: '''14-20 Dec 2015'''<br />
*Decision announced: '''22 Dec 2015'''<br />
*Public discussion: '''14 March 2016'''<br />
<br />
<span style="font-size:large">'''<span style="font-family: tahoma,geneva,sans-serif">Packaging guidelines</span>'''</span><br />
<br />
AE for CGO/PPoPP'15 used the following [http://www.artifact-eval.org/guidelines.html guidelines for artifacts] . We are preparing a revised version.<br />
<br />
<span style="font-size:large">'''<span style="font-family: tahoma,geneva,sans-serif">How to submit</span>'''</span><br />
<br />
Please read the [http://www.artifact-eval.org/guidelines.html guidelines] on what to submit. Please upload your submission to [https://easychair.org/conferences/?conf=aecgoppopp2016 EasyChair] (joint CGO/PPoPP submission).<br />
<br />
<span style="font-size:large">'''<span style="font-family: tahoma,geneva,sans-serif">Review process</span>'''</span><br />
<br />
The review process is described in detail [http://www.artifact-eval.org/review-process.html here]. We are preparing a revised version.<br />
</div><br />
| style="width: 33%" valign="top" | <br />
<span style="font-size:large">'''<span style="font-family: tahoma,geneva,sans-serif">Artifact Evaluation Committee (AEC)</span>'''</span><br />
<br />
'''Chairs:'''<br />
<br />
*[http://cTuning.org/lab/people/gfursin Grigori Fursin], cTuning foundation, France<br />
*[http://people.cs.pitt.edu/%7Echilders Bruce R. Childers], University of Pittsburgh, USA<br />
<br />
'''Committee (joint with CGO'16):'''<br />
<br />
*'''TBA'''<br />
<br />
<span style="font-size:large">'''<span style="font-family: tahoma,geneva,sans-serif">Prior Artifact Evaluation</span>'''</span><br />
<br />
Last year, PPoPP 2015 successfully ran [[Reproducibility:AE:PPoPP2015|artifact evaluation process]] for the first time. PPoPP 2016 continues this experiment.<br />
<br />
Artifacts have been already evaluated at several other conferences and workshops including recent [http://cTuning.org/event/ae-cgo2015 CGO 2015], [http://pldi14-aec.cs.brown.edu PLDI 2014], [http://2014.splashcon.org/track/splash2014-artifacts OOPSLA 2014] and [http://adapt-workshop.org/2014 ADAPT 2014]. Our eventual goal is to develop common evaluation methodology gradually and collaboratively.<br />
<br />
| style="width: 32%" valign="top" | <br />
<span style="font-size:large">'''<span style="font-family: tahoma,geneva,sans-serif">Discussions</span>'''</span><br />
<br />
We consider AE as a continuous learning curve. If you have questions or comments and suggestions on how to improve packaging and reviewing process, please get in touch!<br />
<br />
Our presenation about Artifact Evaluation Experience at CGO/PPoPP'15 is available online [http://www.slideshare.net/GrigoriFursin/presentation-fursin-aecgoppopp2015 here].<br />
<br />
|}<br />
</center></div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:AE:CGO2016&diff=855Reproducibility:AE:CGO20162015-10-27T09:35:22Z<p>Gfursin: </p>
<hr />
<div>{{#externalredirect: http://cTuning.org/ae/cgo2016.html}}<br />
<br />
<center><br />
{| border="0" cellpadding="15" cellspacing="1"<br />
|-<br />
| style="text-align: right width: 200px" | [[File:Ae-stamp-cgo.png]]<br/><br />
| <p style="text-align: center"><span style="font-size:xx-large">'''<span style="font-family: tahoma,geneva,sans-serif">Artifact Evaluation for CGO'16</span>'''</span></p><p style="text-align: center"><span style="font-size:small">[ [http://cgo.org/cgo2016 Back to CGO'16 conference website] ]</span></p><p style="text-align: center"></p><p style="text-align: center"></p><p style="text-align: center"></p><br />
|}<br />
<br />
----<br />
<br />
{| style="width: 97%" border="0" cellpadding="20" cellspacing="1"<br />
|-<br />
| style="width: 32%" valign="top" | <div class="span4"><br />
<span style="font-size:large">'''<span style="font-family: tahoma,geneva,sans-serif">Important dates</span>'''</span><br />
<br />
*Paper decision: '''8 Nov 2015'''<br />
*Artifact submission: '''20 Nov 2015 AoE'''<br />
*Technical clarification: '''14-20 Dec 2015'''<br />
*Decision announced: '''22 Dec 2015'''<br />
*Public discussion: '''14 March 2016'''<br />
<br />
<span style="font-size:large">'''<span style="font-family: tahoma,geneva,sans-serif">Packaging guidelines</span>'''</span><br />
<br />
AE for CGO/PPoPP'15 used the following [http://www.artifact-eval.org/guidelines.html guidelines for artifacts] . We are preparing a revised version.<br />
<br />
<span style="font-size:large">'''<span style="font-family: tahoma,geneva,sans-serif">How to submit</span>'''</span><br />
<br />
Please read the [http://www.artifact-eval.org/guidelines.html guidelines] on what to submit. Please upload your submission to [https://easychair.org/conferences/?conf=aecgoppopp2016 EasyChair] (joint CGO/PPoPP submission).<br />
<br />
<span style="font-size:large">'''<span style="font-family: tahoma,geneva,sans-serif">Review process</span>'''</span><br />
<br />
The review process is described in detail [http://www.artifact-eval.org/review-process.html here]. We are preparing a revised version.<br />
</div><br />
| style="width: 33%" valign="top" | <br />
<span style="font-size:large">'''<span style="font-family: tahoma,geneva,sans-serif">Artifact Evaluation Committee (AEC)</span>'''</span><br />
<br />
'''Chairs:'''<br />
<br />
*[http://cTuning.org/lab/people/gfursin Grigori Fursin], cTuning foundation, France<br />
*[http://people.cs.pitt.edu/%7Echilders Bruce R. Childers], University of Pittsburgh, USA<br />
<br />
'''Committee (joint with PPoPP'16):'''<br />
<br />
*'''TBA'''<br />
<br />
<span style="font-size:large">'''<span style="font-family: tahoma,geneva,sans-serif">Prior Artifact Evaluation</span>'''</span><br />
<br />
Last year, CGO 2015 successfully ran [[Reproducibility:AE:CGO2015|artifact evaluation process]] for the first time. CGO 2016 continues this experiment.<br />
<br />
Artifacts have been already evaluated at several other conferences and workshops including recent [http://cTuning.org/event/ae-ppopp2015 PPoPP 2015], [http://pldi14-aec.cs.brown.edu PLDI 2014], [http://2014.splashcon.org/track/splash2014-artifacts OOPSLA 2014] and [http://adapt-workshop.org/2014 ADAPT 2014]. Our eventual goal is to develop common evaluation methodology gradually and collaboratively.<br />
<br />
| style="width: 32%" valign="top" | <br />
<span style="font-size:large">'''<span style="font-family: tahoma,geneva,sans-serif">Discussions</span>'''</span><br />
<br />
We consider AE as a continuous learning curve. If you have questions or comments and suggestions on how to improve packaging and reviewing process, please get in touch!<br />
<br />
Our presenation about Artifact Evaluation Experience at CGO/PPoPP'15 is available online [http://www.slideshare.net/GrigoriFursin/presentation-fursin-aecgoppopp2015 here].<br />
<br />
|}<br />
</center></div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:AE:CGO2016&diff=854Reproducibility:AE:CGO20162015-10-27T09:34:52Z<p>Gfursin: </p>
<hr />
<div>#REDIRECT [http://cTuning.org/ae/cgo2016.html]<br />
<br />
<center><br />
{| border="0" cellpadding="15" cellspacing="1"<br />
|-<br />
| style="text-align: right width: 200px" | [[File:Ae-stamp-cgo.png]]<br/><br />
| <p style="text-align: center"><span style="font-size:xx-large">'''<span style="font-family: tahoma,geneva,sans-serif">Artifact Evaluation for CGO'16</span>'''</span></p><p style="text-align: center"><span style="font-size:small">[ [http://cgo.org/cgo2016 Back to CGO'16 conference website] ]</span></p><p style="text-align: center"></p><p style="text-align: center"></p><p style="text-align: center"></p><br />
|}<br />
<br />
----<br />
<br />
{| style="width: 97%" border="0" cellpadding="20" cellspacing="1"<br />
|-<br />
| style="width: 32%" valign="top" | <div class="span4"><br />
<span style="font-size:large">'''<span style="font-family: tahoma,geneva,sans-serif">Important dates</span>'''</span><br />
<br />
*Paper decision: '''8 Nov 2015'''<br />
*Artifact submission: '''20 Nov 2015 AoE'''<br />
*Technical clarification: '''14-20 Dec 2015'''<br />
*Decision announced: '''22 Dec 2015'''<br />
*Public discussion: '''14 March 2016'''<br />
<br />
<span style="font-size:large">'''<span style="font-family: tahoma,geneva,sans-serif">Packaging guidelines</span>'''</span><br />
<br />
AE for CGO/PPoPP'15 used the following [http://www.artifact-eval.org/guidelines.html guidelines for artifacts] . We are preparing a revised version.<br />
<br />
<span style="font-size:large">'''<span style="font-family: tahoma,geneva,sans-serif">How to submit</span>'''</span><br />
<br />
Please read the [http://www.artifact-eval.org/guidelines.html guidelines] on what to submit. Please upload your submission to [https://easychair.org/conferences/?conf=aecgoppopp2016 EasyChair] (joint CGO/PPoPP submission).<br />
<br />
<span style="font-size:large">'''<span style="font-family: tahoma,geneva,sans-serif">Review process</span>'''</span><br />
<br />
The review process is described in detail [http://www.artifact-eval.org/review-process.html here]. We are preparing a revised version.<br />
</div><br />
| style="width: 33%" valign="top" | <br />
<span style="font-size:large">'''<span style="font-family: tahoma,geneva,sans-serif">Artifact Evaluation Committee (AEC)</span>'''</span><br />
<br />
'''Chairs:'''<br />
<br />
*[http://cTuning.org/lab/people/gfursin Grigori Fursin], cTuning foundation, France<br />
*[http://people.cs.pitt.edu/%7Echilders Bruce R. Childers], University of Pittsburgh, USA<br />
<br />
'''Committee (joint with PPoPP'16):'''<br />
<br />
*'''TBA'''<br />
<br />
<span style="font-size:large">'''<span style="font-family: tahoma,geneva,sans-serif">Prior Artifact Evaluation</span>'''</span><br />
<br />
Last year, CGO 2015 successfully ran [[Reproducibility:AE:CGO2015|artifact evaluation process]] for the first time. CGO 2016 continues this experiment.<br />
<br />
Artifacts have been already evaluated at several other conferences and workshops including recent [http://cTuning.org/event/ae-ppopp2015 PPoPP 2015], [http://pldi14-aec.cs.brown.edu PLDI 2014], [http://2014.splashcon.org/track/splash2014-artifacts OOPSLA 2014] and [http://adapt-workshop.org/2014 ADAPT 2014]. Our eventual goal is to develop common evaluation methodology gradually and collaboratively.<br />
<br />
| style="width: 32%" valign="top" | <br />
<span style="font-size:large">'''<span style="font-family: tahoma,geneva,sans-serif">Discussions</span>'''</span><br />
<br />
We consider AE as a continuous learning curve. If you have questions or comments and suggestions on how to improve packaging and reviewing process, please get in touch!<br />
<br />
Our presenation about Artifact Evaluation Experience at CGO/PPoPP'15 is available online [http://www.slideshare.net/GrigoriFursin/presentation-fursin-aecgoppopp2015 here].<br />
<br />
|}<br />
</center></div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility&diff=853Reproducibility2015-10-26T11:05:16Z<p>Gfursin: </p>
<hr />
<div><center><br />
<span style="font-size:x-large"><span style="color: rgb(178, 34, 34)">Enabling collaborative and reproducible computer systems research with an open publication model</span></span><br />
</center><p style="text-align: center">[[File:Ae-stamp-ppopp2015.png|Ae-stamp-ppopp2015.png|link=http://cTuning.org/event/ae-ppopp2016]] [[File:Ae-stamp-cgo.png|Ae-stamp-cgo.png|link=http://cTuning.org/event/ae-cgo2016]] [[File:Logo-validated-by-the-community.png|Logo-validated-by-the-community.png|link=http://cTuning.org/reproducibility]] &nbsp; &nbsp;&nbsp; [[File:CTuning foundation logo1.png|cTuning_foundation_logo1.png|link=http://cTuning.org]]&nbsp; &nbsp;&nbsp; [[File:dividiti.png|dividiti.png|link=http://dividiti.com]]</p><p style="text-align: center"></p><p style="text-align: center">This wiki is maintained by the [http://cTuning.org non-profit Tuning foundation].</p><br />
== News and upcoming events<br/> ==<br />
<br />
* ''We have released our new, open-source, BSD-licensed Collective Knowledge Framework (cTuning 4 aka CK) for collaborative and reproducible R&D: [http://github.com/ctuning/ck GitHub] with an [http://github.com/ctuning/ck/wiki online documentation] and [http://cknowledge.org/repo live demo repository].''<br />
*''[http://cTuning.org/event/ae-ppopp2016 PPoPP'16 artifact evaluation]''<br />
*''[http://cTuning.org/event/ae-cgo2016 CGO'16 artifact evaluation]''<br />
*''[http://adapt-workshop.org ADAPT'16 @ HiPEAC'16] - features [http://dl.acm.org/citation.cfm?id=2618142 our open publication model with community-driven reviewing, reddit-based discussions and artifact evaluation]''<br />
*''[http://www.dagstuhl.de/de/programm/kalender/semhp/?semnr=15452 Dagstuhl perspective workshop on artifact evaluation for conferences and journals]''<br />
<br />
== Motivation<br/> ==<br />
<br />
Since 2006 we have been trying to solve problems with reproducibility of experimental results in computer engineering ''as a side effect'' of our [http://cTuning.org/project-milepost MILEPOST] , [http://cTuning.org cTuning.org], [http://c-mind.org Collective Mind] and [http://github.com/ctuning/ck Collective Knowledge] projects (speeding up optimization, benchmarking and co-design of computer systems using auto-tuning, big data, predictive analytics and crowdsourcing). <span style="font-size: small">We focus on the following technological and social aspects to enable collaborative, systematic and reproducible research and experimentation particularly related to benchmarking, optimization and co-design of faster, smaller, cheaper, more power efficient and reliable software and hardware:</span><br />
<br />
*developing public and open source Collective Mind repositories of knowledge (see our pilot live repository [[http://cknowledge.org/repo CK], [http://c-mind.org/repo cMind]] and our vision papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]]);<br />
*developing [http://github.com/ctuning/ck collaborative research and experimentation infrastructure] that can share artifacts as reusable components together with the whole experimental setups (see our papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]];<br />
*evangelizing and enabling new open publication model for online workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*setting up and improving procedure for sharing and evaluating experimental results and all related material for workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*improving sharing, description of dependencies, and statistical reproducibility of experimental results and related material;<br />
*supporting and improving [http://cTuning.org/ae Artifact Evaluation] for major workshops and conferences including CGO and PPoPP.<br />
<br />
See our manifesto and history [http://cTuning.org/history here].<br />
<br />
== Our R&D ==<br />
<br />
Together with the community and [http://cTuning.org not-for-profit cTuning foundation] we are working on the following topics:<br />
<br />
*developing tools and methodology to capture, preserve, formalize, systematize, exchange and improve knowledge and experimental results including negative ones<br />
*describing and cataloging whole experimental setups with all related material including algorithms, benchmarks, codelets, datasets, tools, models and any other artifact<br />
*developing specification to preserve experiments including all software and hardware dependencies<br />
*dealing with variability and rising amount of experimental data using statistical analysis, data mining, predictive modeling and other techniques<br />
*developing new predictive analytics techniques to explore large design and optimization spaces<br />
*validating and verifying experimental results by the community<br />
*developing common research interfaces for existing or new tools<br />
*developing common experimental frameworks and repositories (enable automation, re-execution and sharing of experiments)<br />
*sharing rare hardware and computational resources for experimental validation<br />
*implementing previously published experimental scenarios (auto-tuning, run-time adaptation) using common infrastructure<br />
*implementing open access to publications and data (particularly discussing intellectual property IP and legal issues)<br />
*speeding up analysis of "big" experimental data<br />
*developing new (interactive) visualization techniques for "big" experimental data<br />
*enabling interactive articles<br />
<br />
== Our events<br/> ==<br />
<br />
*[[Reproducibility:AE:PPoPP2015|PPoPP'15 artifact evaluation]]<br />
*[[Reproducibility:AE:CGO2015|CGO'15 artifact evaluation]]<br />
*[http://adapt-workshop.org ADAPT'15] @ HiPEAC'15 - workshop on adaptive self-tuning computer systems<br />
*[http://adapt-workshop.org/2014 ADAPT'14] @ HiPEAC'14 - workshop on adaptive self-tuning computer systems [ [http://adapt-workshop.org/2014/program.htm program and publications] ]<br />
*[http://www.occamportal.org/images/reproduce/TETC-SI-REPRODUCE.pdf Special journal issue] on Reproducible &nbsp;Research &nbsp;Methodologies at IEEE TETC<br />
*[http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14] @ PLDI'14<br />
*[http://www.occamportal.org/reproduce REPRODUCE'14] @ HPCA'14<br />
*[http://www.adapt-workshop.org/2014/program.htm ADAPT'14 panel] @ HiPEAC'14<br />
*[http://ctuning.org/making-computer-engineering-a-science-2013 HiPEAC'13 CSW thematic session] @ ACM ECRC "Making computer engineering a science"<br />
*[http://ctuning.org/hipeac3-thematic-session-2012-04 HiPEAC'12 CSW thematic session]<br />
*[http://exadapt.org/2012/program.html ASPLOS/EXADAPT'12 panel] @ ASPLOS'12<br />
*[http://ctuning.org/lab/education cTuning lectures (2008-2010)]<br />
*[http://hal.inria.fr/inria-00436029 GCC Summit'09 discussion]<br />
<br />
== Resources<br/> ==<br />
<br />
*[[Reproducibility:Initiatives|Collection of related initiatives]]<br />
*[[Reproducibility:Links|Collection of related tools]]<br />
*[[Reproducibility:Datasets|Collection of related benchmarks and data sets]]<br />
*[[Reproducibility:Repositories|Collection of public repositories]]<br />
*[[Reproducibility:Lectures|Collection of related lectures]]<br />
*[[Reproducibility:Articles|Collection of related articles]]<br />
*[[Reproducibility:Blogs|Collection of related blogs]]<br />
*[[Reproducibility:Jokes|Collection of jokes]]<br />
*[[Reproducibility:Events|Collection of related events]]<br />
<br />
== Discussions<br/> ==<br />
<br />
*[http://www.linkedin.com/groups/Reproducible-research-experimentation-in-computer-7433414 LinkedIn group on reproducible research]<br />
*[http://groups.google.com/group/collective-knowledge Main mailing list] (general collaborative and reproducible R&D in computer engineering)<br />
*[http://groups.google.com/group/ctuning-discussions cTuning foundation mailing list] (collaborative and reproducible hardware and software benchmarking, auto-tuning and co-design)<br />
<br />
== Follow us<br/> ==<br />
<br />
*[https://twitter.com/c_tuning cTuning foundation twitter]<br />
*[https://www.facebook.com/pages/CrowdTuning-Foundation/668405119902805 cTuning foundation facebook page] (recent)<br />
*[http://dividiti.blogspot.com dividiti blog]<br />
*[http://c-mind.org/repo Collective Mind repository] (outdated)<br />
*[http://cknowledge.org/repo Collective Knowledge repository] (new)<br />
<br />
== Archive<br/> ==<br />
<br />
*[http://ctuning.org/wiki/index.php?title=Discussions:New_Publication_Model Outdated cTuning wiki page related to reproducible research and open publication model]<br />
*Outdated cTuning repository for program/processor performance/power/size optimization (2008-2010): [ [http://ctuning.org/wiki/index.php/CDatabase database], [http://ctuning.org/wiki/index.php?title=Special:CPredict web-service for online prediction of optimizations] ]<br />
<br />
== Acknowledgments<br/> ==<br />
<br />
We would like to thank our colleagues from the [http://cTuning.org/lab/people cTuning foundation], [https://www.linkedin.com/company/dividiti dividiti], [http://www.artifact-eval.org artifact-eval.org], [http://www.occamportal.org OCCAM project] for their help, feedback, participation and support.</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility&diff=852Reproducibility2015-10-26T11:03:52Z<p>Gfursin: /* Motivation */</p>
<hr />
<div><center><br />
<span style="font-size:x-large"><span style="color: rgb(178, 34, 34)">Enabling collaborative and reproducible computer systems research with an open publication model</span></span><br />
</center><p style="text-align: center">[[File:Ae-stamp-ppopp2015.png|Ae-stamp-ppopp2015.png|link=http://cTuning.org/event/ae-ppopp2016]] [[File:Ae-stamp-cgo.png|Ae-stamp-cgo.png|link=http://cTuning.org/event/ae-cgo2016]] [[File:Logo-validated-by-the-community.png|Logo-validated-by-the-community.png|link=http://cTuning.org/reproducibility]] &nbsp; &nbsp;&nbsp; [[File:CTuning foundation logo1.png|cTuning_foundation_logo1.png|link=http://cTuning.org]]&nbsp; &nbsp;&nbsp; [[File:dividiti.png|dividiti.png|link=http://dividiti.com]]</p><p style="text-align: center"></p><p style="text-align: center">This wiki is maintained by the [http://cTuning.org non-profit Tuning foundation].</p><br />
== News and upcoming events<br/> ==<br />
<br />
* ''We have released our new, open-source, BSD-licensed Collective Knowledge Framework (cTuning 4 aka CK) for collaborative and reproducible R&D: [http://github.com/ctuning/ck GitHub] with an [http://github.com/ctuning/ck/wiki online documentation] and [http://cknowledge.org/repo live demo repository].''<br />
*''[http://cTuning.org/event/ae-ppopp2016 PPoPP'16 artifact evaluation]''<br />
*''[http://cTuning.org/event/ae-cgo2016 CGO'16 artifact evaluation]''<br />
*''[http://adapt-workshop.org ADAPT'16 @ HiPEAC'16] - features [http://dl.acm.org/citation.cfm?id=2618142 our open publication model with community-driven reviewing, reddit-based discussions and artifact evaluation]''<br />
*''[http://www.dagstuhl.de/de/programm/kalender/semhp/?semnr=15452 Dagstuhl perspective workshop on artifact evaluation for conferences and journals]''<br />
<br />
== Motivation<br/> ==<br />
<br />
Since 2006 we have been trying to solve problems with reproducibility of experimental results in computer engineering ''as a side effect'' of our [http://cTuning.org/project-milepost MILEPOST] , [http://cTuning.org cTuning.org], [http://c-mind.org Collective Mind] and [http://github.com/ctuning/ck Collective Knowledge] projects (speeding up optimization, benchmarking and co-design of computer systems using auto-tuning, big data, predictive analytics and crowdsourcing). <span style="font-size: small">We focus on the following technological and social aspects to enable collaborative, systematic and reproducible research and experimentation particularly related to benchmarking, optimization and co-design of faster, smaller, cheaper, more power efficient and reliable software and hardware:</span><br />
<br />
*developing public and open source Collective Mind repositories of knowledge (see our pilot live repository [[http://cknowledge.org/repo CK], [http://c-mind.org/repo cMind]] and our vision papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]]);<br />
*developing [http://github.com/ctuning/ck collaborative research and experimentation infrastructure] that can share artifacts as reusable components together with the whole experimental setups (see our papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]];<br />
*evangelizing and enabling new open publication model for online workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*setting up and improving procedure for sharing and evaluating experimental results and all related material for workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*improving sharing, description of dependencies, and statistical reproducibility of experimental results and related material;<br />
*supporting and improving [http://cTuning.org/ae Artifact Evaluation] for major workshops and conferences including CGO and PPoPP.<br />
<br />
See our manifesto and history [http://cTuning.org/history here].<br />
<br />
== Our R&D ==<br />
<br />
Together with the community and [http://cTuning.org not-for-profit cTuning foundation] we are working on the following topics:<br />
<br />
*developing tools and methodology to capture, preserve, formalize, systematize, exchange and improve knowledge and experimental results including negative ones<br />
*describing and cataloging whole experimental setups with all related material including algorithms, benchmarks, codelets, datasets, tools, models and any other artifact<br />
*developing specification to preserve experiments including all software and hardware dependencies<br />
*dealing with variability and rising amount of experimental data using statistical analysis, data mining, predictive modeling and other techniques<br />
*developing new predictive analytics techniques to explore large design and optimization spaces<br />
*validating and verifying experimental results by the community<br />
*developing common research interfaces for existing or new tools<br />
*developing common experimental frameworks and repositories (enable automation, re-execution and sharing of experiments)<br />
*sharing rare hardware and computational resources for experimental validation<br />
*implementing previously published experimental scenarios (auto-tuning, run-time adaptation) using common infrastructure<br />
*implementing open access to publications and data (particularly discussing intellectual property IP and legal issues)<br />
*speeding up analysis of "big" experimental data<br />
*developing new (interactive) visualization techniques for "big" experimental data<br />
*enabling interactive articles<br />
<br />
== Our events<br/> ==<br />
<br />
*[[Reproducibility:AE:PPoPP2015|PPoPP'15 artifact evaluation]]<br />
*[[Reproducibility:AE:CGO2015|CGO'15 artifact evaluation]]<br />
*[http://adapt-workshop.org ADAPT'15] @ HiPEAC'15 - workshop on adaptive self-tuning computer systems<br />
*[http://adapt-workshop.org/2014 ADAPT'14] @ HiPEAC'14 - workshop on adaptive self-tuning computer systems [ [http://adapt-workshop.org/2014/program.htm program and publications] ]<br />
*[http://www.occamportal.org/images/reproduce/TETC-SI-REPRODUCE.pdf Special journal issue] on Reproducible &nbsp;Research &nbsp;Methodologies at IEEE TETC<br />
*[http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14] @ PLDI'14<br />
*[http://www.occamportal.org/reproduce REPRODUCE'14] @ HPCA'14<br />
*[http://www.adapt-workshop.org/2014/program.htm ADAPT'14 panel] @ HiPEAC'14<br />
*[http://ctuning.org/making-computer-engineering-a-science-2013 HiPEAC'13 CSW thematic session] @ ACM ECRC "Making computer engineering a science"<br />
*[http://ctuning.org/hipeac3-thematic-session-2012-04 HiPEAC'12 CSW thematic session]<br />
*[http://exadapt.org/2012/program.html ASPLOS/EXADAPT'12 panel] @ ASPLOS'12<br />
*[http://ctuning.org/lab/education cTuning lectures (2008-2010)]<br />
*[http://hal.inria.fr/inria-00436029 GCC Summit'09 discussion]<br />
<br />
== Paper and artifact evaluation committee<br/> ==<br />
<br />
Rather than pre-selecting a dedicated committee for conferences, we select reviewers for reseach material (artifacts) and publications from a [http://cTuning.org/lab/people pool of our supporters] based on '''''submitted and publicly available''''' publications, their keywords and '''''public discussions''''' as described in our proposal [[http://arxiv.org/abs/1406.4020 arXiv]], [[http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]. Validated papers receive a stamp "Validated by the community". Artifacts can be shared along with publication in the ACM Digital LIbrary, HAL, Collective Mind Repository or any other public archive.<br />
<br />
As for the workshops, conferences and journals with the traditional publication model (CGO, PPoPP, PLDI), we select artifact evaluation committee (AEC) as described [[Reproducibility:AE|here]].<br />
<br />
== Packing and sharing research and experimental material<br/> ==<br />
<br />
Rather than enforcing specific procedure for packing, sharing and validation of experimental results, we allow authors of the accepted papers to include an archive with all related research material (using any [[Reproducibility:Links|publicly available tool]]) and ''readme.txt'' file describing how to validate their experiments. The main reason is the lack of a universally acceptable solution to pack and share experimental setups. For example, it is not always possible to use Virtual Machines and similar approaches for our research on performance/energy tuning or when some new hardware is being co-designed as we discuss in our proposal [[http://arxiv.org/abs/1406.4020 arXiv], [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]. Therefore, our current intention is to gradually and collaboratively find best procedure for packing using practical experience from our events such as [http://adapt-workshop.org ADAPT workshop] and from common discussions during [http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14 workshops]. See also nice guidelines for packing code and data along with publications [http://www.artifact-eval.org/guidelines.html here].<br />
<br />
== Validation<br/> ==<br />
<br />
After many years of evangelizing collaborative and reproducible research in computer engineering based on our practical experience, we finally start seeing some change in the mentality in academia, industry and funding agencies. Authors of two papers of our [http://adapt-workshop.org/2014 ADAPT'14 workshop] (out of nine accepted) agreed to have experimental results of their papers validated by volunteers. Note that rather than enforcing specific validation rules, we decided to ask authors to pack all their research artifacts as they wish (for example, using a shared virtual machine or as a standard archive) and describe their own validation procedure. Thanks to [http://cTuning.org/lab/people our volunteers], experiments from these papers have been validated, archives shared in [http://c-mind.org/repo our public repository] , and papers marked with a "validated by the community" stamp:<br />
<p style="text-align: center">[[File:Logo-validated-by-the-community.png|center|Logo-validated-by-the-community.png]]</p><br />
== Archive<br/> ==<br />
<br />
*[http://ctuning.org/wiki/index.php?title=Discussions:New_Publication_Model Outdated cTuning wiki page related to reproducible research and open publication model]<br />
*Outdated cTuning repository for program/processor performance/power/size optimization (2008-2010): [ [http://ctuning.org/wiki/index.php/CDatabase database], [http://ctuning.org/wiki/index.php?title=Special:CPredict web-service for online prediction of optimizations] ]<br />
<br />
== Resources<br/> ==<br />
<br />
*[[Reproducibility:Initiatives|Collection of related initiatives]]<br />
*[[Reproducibility:Links|Collection of related tools]]<br />
*[[Reproducibility:Datasets|Collection of related benchmarks and data sets]]<br />
*[[Reproducibility:Repositories|Collection of public repositories]]<br />
*[[Reproducibility:Lectures|Collection of related lectures]]<br />
*[[Reproducibility:Articles|Collection of related articles]]<br />
*[[Reproducibility:Blogs|Collection of related blogs]]<br />
*[[Reproducibility:Jokes|Collection of jokes]]<br />
*[[Reproducibility:Events|Collection of related events]]<br />
<br />
== Discussions<br/> ==<br />
<br />
*[http://www.linkedin.com/groups/Reproducible-research-experimentation-in-computer-7433414 LinkedIn group on reproducible research]<br />
*[http://groups.google.com/group/collective-knowledge Main mailing list] (general collaborative and reproducible R&D in computer engineering)<br />
*[http://groups.google.com/group/ctuning-discussions cTuning foundation mailing list] (collaborative and reproducible hardware and software benchmarking, auto-tuning and co-design)<br />
<br />
== Follow us<br/> ==<br />
<br />
*[https://twitter.com/c_tuning cTuning foundation twitter]<br />
*[https://www.facebook.com/pages/CrowdTuning-Foundation/668405119902805 cTuning foundation facebook page] (recent)<br />
*[http://dividiti.blogspot.com dividiti blog]<br />
*[http://c-mind.org/repo Collective Mind repository] (outdated)<br />
*[http://cknowledge.org/repo Collective Knowledge repository] (new)<br />
<br />
== Acknowledgments<br/> ==<br />
<br />
We would like to thank our colleagues from the [http://cTuning.org/lab/people cTuning foundation], [https://www.linkedin.com/company/dividiti dividiti], [http://www.artifact-eval.org artifact-eval.org], [http://www.occamportal.org OCCAM project] for their help, feedback, participation and support.</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility&diff=851Reproducibility2015-10-26T11:02:46Z<p>Gfursin: /* Motivation */</p>
<hr />
<div><center><br />
<span style="font-size:x-large"><span style="color: rgb(178, 34, 34)">Enabling collaborative and reproducible computer systems research with an open publication model</span></span><br />
</center><p style="text-align: center">[[File:Ae-stamp-ppopp2015.png|Ae-stamp-ppopp2015.png|link=http://cTuning.org/event/ae-ppopp2016]] [[File:Ae-stamp-cgo.png|Ae-stamp-cgo.png|link=http://cTuning.org/event/ae-cgo2016]] [[File:Logo-validated-by-the-community.png|Logo-validated-by-the-community.png|link=http://cTuning.org/reproducibility]] &nbsp; &nbsp;&nbsp; [[File:CTuning foundation logo1.png|cTuning_foundation_logo1.png|link=http://cTuning.org]]&nbsp; &nbsp;&nbsp; [[File:dividiti.png|dividiti.png|link=http://dividiti.com]]</p><p style="text-align: center"></p><p style="text-align: center">This wiki is maintained by the [http://cTuning.org non-profit Tuning foundation].</p><br />
== News and upcoming events<br/> ==<br />
<br />
* ''We have released our new, open-source, BSD-licensed Collective Knowledge Framework (cTuning 4 aka CK) for collaborative and reproducible R&D: [http://github.com/ctuning/ck GitHub] with an [http://github.com/ctuning/ck/wiki online documentation] and [http://cknowledge.org/repo live demo repository].''<br />
*''[http://cTuning.org/event/ae-ppopp2016 PPoPP'16 artifact evaluation]''<br />
*''[http://cTuning.org/event/ae-cgo2016 CGO'16 artifact evaluation]''<br />
*''[http://adapt-workshop.org ADAPT'16 @ HiPEAC'16] - features [http://dl.acm.org/citation.cfm?id=2618142 our open publication model with community-driven reviewing, reddit-based discussions and artifact evaluation]''<br />
*''[http://www.dagstuhl.de/de/programm/kalender/semhp/?semnr=15452 Dagstuhl perspective workshop on artifact evaluation for conferences and journals]''<br />
<br />
== Motivation<br/> ==<br />
<br />
Since 2006 we have been trying to solve problems with reproducibility of experimental results in computer engineering ''as a side effect'' of our [http://cTuning.org/project-milepost MILEPOST] , [http://cTuning.org cTuning.org], [http://c-mind.org Collective Mind] and [http://github.com/ctuning/ck Collective Knowledge] projects (speeding up optimization, benchmarking and co-design of computer systems using auto-tuning, big data, predictive analytics and crowdsourcing). <span style="font-size: small">We focus on the following technological and social aspects to enable collaborative, systematic and reproducible research and experimentation particularly related to benchmarking, optimization and co-design of faster, smaller, cheaper, more power efficient and reliable software and hardware:</span><br />
<br />
*developing public and open source Collective Mind repositories of knowledge (see our pilot live repository [[http://cknowledge.org/repo CK], [http://c-mind.org/repo cMind]] and our vision papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]]);<br />
*developing [http://github.com/ctuning/ck collaborative research and experimentation infrastructure] that can share artifacts as reusable components together with the whole experimental setups (see our papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]];<br />
*evangelizing and enabling new open publication model for online workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*setting up and improving procedure for sharing and evaluating experimental results and all related material for workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*improving sharing, description of dependencies, and statistical reproducibility of experimental results and related material.<br />
<br />
See our manifesto and history [http://cTuning.org/history here].<br />
<br />
== Our R&D ==<br />
<br />
Together with the community and [http://cTuning.org not-for-profit cTuning foundation] we are working on the following topics:<br />
<br />
*developing tools and methodology to capture, preserve, formalize, systematize, exchange and improve knowledge and experimental results including negative ones<br />
*describing and cataloging whole experimental setups with all related material including algorithms, benchmarks, codelets, datasets, tools, models and any other artifact<br />
*developing specification to preserve experiments including all software and hardware dependencies<br />
*dealing with variability and rising amount of experimental data using statistical analysis, data mining, predictive modeling and other techniques<br />
*developing new predictive analytics techniques to explore large design and optimization spaces<br />
*validating and verifying experimental results by the community<br />
*developing common research interfaces for existing or new tools<br />
*developing common experimental frameworks and repositories (enable automation, re-execution and sharing of experiments)<br />
*sharing rare hardware and computational resources for experimental validation<br />
*implementing previously published experimental scenarios (auto-tuning, run-time adaptation) using common infrastructure<br />
*implementing open access to publications and data (particularly discussing intellectual property IP and legal issues)<br />
*speeding up analysis of "big" experimental data<br />
*developing new (interactive) visualization techniques for "big" experimental data<br />
*enabling interactive articles<br />
<br />
== Our events<br/> ==<br />
<br />
*[[Reproducibility:AE:PPoPP2015|PPoPP'15 artifact evaluation]]<br />
*[[Reproducibility:AE:CGO2015|CGO'15 artifact evaluation]]<br />
*[http://adapt-workshop.org ADAPT'15] @ HiPEAC'15 - workshop on adaptive self-tuning computer systems<br />
*[http://adapt-workshop.org/2014 ADAPT'14] @ HiPEAC'14 - workshop on adaptive self-tuning computer systems [ [http://adapt-workshop.org/2014/program.htm program and publications] ]<br />
*[http://www.occamportal.org/images/reproduce/TETC-SI-REPRODUCE.pdf Special journal issue] on Reproducible &nbsp;Research &nbsp;Methodologies at IEEE TETC<br />
*[http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14] @ PLDI'14<br />
*[http://www.occamportal.org/reproduce REPRODUCE'14] @ HPCA'14<br />
*[http://www.adapt-workshop.org/2014/program.htm ADAPT'14 panel] @ HiPEAC'14<br />
*[http://ctuning.org/making-computer-engineering-a-science-2013 HiPEAC'13 CSW thematic session] @ ACM ECRC "Making computer engineering a science"<br />
*[http://ctuning.org/hipeac3-thematic-session-2012-04 HiPEAC'12 CSW thematic session]<br />
*[http://exadapt.org/2012/program.html ASPLOS/EXADAPT'12 panel] @ ASPLOS'12<br />
*[http://ctuning.org/lab/education cTuning lectures (2008-2010)]<br />
*[http://hal.inria.fr/inria-00436029 GCC Summit'09 discussion]<br />
<br />
== Paper and artifact evaluation committee<br/> ==<br />
<br />
Rather than pre-selecting a dedicated committee for conferences, we select reviewers for reseach material (artifacts) and publications from a [http://cTuning.org/lab/people pool of our supporters] based on '''''submitted and publicly available''''' publications, their keywords and '''''public discussions''''' as described in our proposal [[http://arxiv.org/abs/1406.4020 arXiv]], [[http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]. Validated papers receive a stamp "Validated by the community". Artifacts can be shared along with publication in the ACM Digital LIbrary, HAL, Collective Mind Repository or any other public archive.<br />
<br />
As for the workshops, conferences and journals with the traditional publication model (CGO, PPoPP, PLDI), we select artifact evaluation committee (AEC) as described [[Reproducibility:AE|here]].<br />
<br />
== Packing and sharing research and experimental material<br/> ==<br />
<br />
Rather than enforcing specific procedure for packing, sharing and validation of experimental results, we allow authors of the accepted papers to include an archive with all related research material (using any [[Reproducibility:Links|publicly available tool]]) and ''readme.txt'' file describing how to validate their experiments. The main reason is the lack of a universally acceptable solution to pack and share experimental setups. For example, it is not always possible to use Virtual Machines and similar approaches for our research on performance/energy tuning or when some new hardware is being co-designed as we discuss in our proposal [[http://arxiv.org/abs/1406.4020 arXiv], [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]. Therefore, our current intention is to gradually and collaboratively find best procedure for packing using practical experience from our events such as [http://adapt-workshop.org ADAPT workshop] and from common discussions during [http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14 workshops]. See also nice guidelines for packing code and data along with publications [http://www.artifact-eval.org/guidelines.html here].<br />
<br />
== Validation<br/> ==<br />
<br />
After many years of evangelizing collaborative and reproducible research in computer engineering based on our practical experience, we finally start seeing some change in the mentality in academia, industry and funding agencies. Authors of two papers of our [http://adapt-workshop.org/2014 ADAPT'14 workshop] (out of nine accepted) agreed to have experimental results of their papers validated by volunteers. Note that rather than enforcing specific validation rules, we decided to ask authors to pack all their research artifacts as they wish (for example, using a shared virtual machine or as a standard archive) and describe their own validation procedure. Thanks to [http://cTuning.org/lab/people our volunteers], experiments from these papers have been validated, archives shared in [http://c-mind.org/repo our public repository] , and papers marked with a "validated by the community" stamp:<br />
<p style="text-align: center">[[File:Logo-validated-by-the-community.png|center|Logo-validated-by-the-community.png]]</p><br />
== Archive<br/> ==<br />
<br />
*[http://ctuning.org/wiki/index.php?title=Discussions:New_Publication_Model Outdated cTuning wiki page related to reproducible research and open publication model]<br />
*Outdated cTuning repository for program/processor performance/power/size optimization (2008-2010): [ [http://ctuning.org/wiki/index.php/CDatabase database], [http://ctuning.org/wiki/index.php?title=Special:CPredict web-service for online prediction of optimizations] ]<br />
<br />
== Resources<br/> ==<br />
<br />
*[[Reproducibility:Initiatives|Collection of related initiatives]]<br />
*[[Reproducibility:Links|Collection of related tools]]<br />
*[[Reproducibility:Datasets|Collection of related benchmarks and data sets]]<br />
*[[Reproducibility:Repositories|Collection of public repositories]]<br />
*[[Reproducibility:Lectures|Collection of related lectures]]<br />
*[[Reproducibility:Articles|Collection of related articles]]<br />
*[[Reproducibility:Blogs|Collection of related blogs]]<br />
*[[Reproducibility:Jokes|Collection of jokes]]<br />
*[[Reproducibility:Events|Collection of related events]]<br />
<br />
== Discussions<br/> ==<br />
<br />
*[http://www.linkedin.com/groups/Reproducible-research-experimentation-in-computer-7433414 LinkedIn group on reproducible research]<br />
*[http://groups.google.com/group/collective-knowledge Main mailing list] (general collaborative and reproducible R&D in computer engineering)<br />
*[http://groups.google.com/group/ctuning-discussions cTuning foundation mailing list] (collaborative and reproducible hardware and software benchmarking, auto-tuning and co-design)<br />
<br />
== Follow us<br/> ==<br />
<br />
*[https://twitter.com/c_tuning cTuning foundation twitter]<br />
*[https://www.facebook.com/pages/CrowdTuning-Foundation/668405119902805 cTuning foundation facebook page] (recent)<br />
*[http://dividiti.blogspot.com dividiti blog]<br />
*[http://c-mind.org/repo Collective Mind repository] (outdated)<br />
*[http://cknowledge.org/repo Collective Knowledge repository] (new)<br />
<br />
== Acknowledgments<br/> ==<br />
<br />
We would like to thank our colleagues from the [http://cTuning.org/lab/people cTuning foundation], [https://www.linkedin.com/company/dividiti dividiti], [http://www.artifact-eval.org artifact-eval.org], [http://www.occamportal.org OCCAM project] for their help, feedback, participation and support.</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility&diff=850Reproducibility2015-10-26T11:01:50Z<p>Gfursin: /* News and upcoming events */</p>
<hr />
<div><center><br />
<span style="font-size:x-large"><span style="color: rgb(178, 34, 34)">Enabling collaborative and reproducible computer systems research with an open publication model</span></span><br />
</center><p style="text-align: center">[[File:Ae-stamp-ppopp2015.png|Ae-stamp-ppopp2015.png|link=http://cTuning.org/event/ae-ppopp2016]] [[File:Ae-stamp-cgo.png|Ae-stamp-cgo.png|link=http://cTuning.org/event/ae-cgo2016]] [[File:Logo-validated-by-the-community.png|Logo-validated-by-the-community.png|link=http://cTuning.org/reproducibility]] &nbsp; &nbsp;&nbsp; [[File:CTuning foundation logo1.png|cTuning_foundation_logo1.png|link=http://cTuning.org]]&nbsp; &nbsp;&nbsp; [[File:dividiti.png|dividiti.png|link=http://dividiti.com]]</p><p style="text-align: center"></p><p style="text-align: center">This wiki is maintained by the [http://cTuning.org non-profit Tuning foundation].</p><br />
== News and upcoming events<br/> ==<br />
<br />
* ''We have released our new, open-source, BSD-licensed Collective Knowledge Framework (cTuning 4 aka CK) for collaborative and reproducible R&D: [http://github.com/ctuning/ck GitHub] with an [http://github.com/ctuning/ck/wiki online documentation] and [http://cknowledge.org/repo live demo repository].''<br />
*''[http://cTuning.org/event/ae-ppopp2016 PPoPP'16 artifact evaluation]''<br />
*''[http://cTuning.org/event/ae-cgo2016 CGO'16 artifact evaluation]''<br />
*''[http://adapt-workshop.org ADAPT'16 @ HiPEAC'16] - features [http://dl.acm.org/citation.cfm?id=2618142 our open publication model with community-driven reviewing, reddit-based discussions and artifact evaluation]''<br />
*''[http://www.dagstuhl.de/de/programm/kalender/semhp/?semnr=15452 Dagstuhl perspective workshop on artifact evaluation for conferences and journals]''<br />
<br />
== Motivation<br/> ==<br />
<br />
Since 2006 we have been trying to solve problems with reproducibility of experimental results in computer engineering ''as a side effect'' of our [http://cTuning.org/project-milepost MILEPOST] , [http://cTuning.org cTuning.org], [http://c-mind.org Collective Mind] and [http://github.com/ctuning/ck Collective Knowledge] projects (speeding up optimization, benchmarking and co-design of computer systems using auto-tuning, big data, predictive analytics and crowdsourcing). <span style="font-size: small">We focus on the following technological and social aspects to enable collaborative, systematic and reproducible research and experimentation particularly related to benchmarking, optimization and co-design of faster, smaller, cheaper, more power efficient and reliable software and hardware:</span><br />
<br />
*developing public and open source Collective Mind repositories of knowledge (see our pilot live repository [[http://cknowledge.org/repo CK], [http://c-mind.org/repo cMind]] and our vision papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]]);<br />
*developing [http://github.com/ctuning/ck collaborative research and experimentation infrastructure] that can share artifacts as reusable components together with the whole experimental setups (see our papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]];<br />
*evangelizing and enabling new open publication model for online workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*setting up and improving procedure for sharing and evaluating experimental results and all related material for workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*improving sharing, description of dependencies, and statistical reproducibility of experimental results and related material.<br />
<br />
See our manifesto and history [http://cTuning.org/history here].<br />
<br />
=== Community-driven research and developments<br/> ===<br />
<br />
Together with the community and [http://cTuning.org not-for-profit cTuning foundation] we are working on the following topics:<br />
<br />
*developing tools and methodology to capture, preserve, formalize, systematize, exchange and improve knowledge and experimental results including negative ones<br />
*describing and cataloging whole experimental setups with all related material including algorithms, benchmarks, codelets, datasets, tools, models and any other artifact<br />
*developing specification to preserve experiments including all software and hardware dependencies<br />
*dealing with variability and rising amount of experimental data using statistical analysis, data mining, predictive modeling and other techniques<br />
*developing new predictive analytics techniques to explore large design and optimization spaces<br />
*validating and verifying experimental results by the community<br />
*developing common research interfaces for existing or new tools<br />
*developing common experimental frameworks and repositories (enable automation, re-execution and sharing of experiments)<br />
*sharing rare hardware and computational resources for experimental validation<br />
*implementing previously published experimental scenarios (auto-tuning, run-time adaptation) using common infrastructure<br />
*implementing open access to publications and data (particularly discussing intellectual property IP and legal issues)<br />
*speeding up analysis of "big" experimental data<br />
*developing new (interactive) visualization techniques for "big" experimental data<br />
*enabling interactive articles<br />
<br />
== Our events<br/> ==<br />
<br />
*[[Reproducibility:AE:PPoPP2015|PPoPP'15 artifact evaluation]]<br />
*[[Reproducibility:AE:CGO2015|CGO'15 artifact evaluation]]<br />
*[http://adapt-workshop.org ADAPT'15] @ HiPEAC'15 - workshop on adaptive self-tuning computer systems<br />
*[http://adapt-workshop.org/2014 ADAPT'14] @ HiPEAC'14 - workshop on adaptive self-tuning computer systems [ [http://adapt-workshop.org/2014/program.htm program and publications] ]<br />
*[http://www.occamportal.org/images/reproduce/TETC-SI-REPRODUCE.pdf Special journal issue] on Reproducible &nbsp;Research &nbsp;Methodologies at IEEE TETC<br />
*[http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14] @ PLDI'14<br />
*[http://www.occamportal.org/reproduce REPRODUCE'14] @ HPCA'14<br />
*[http://www.adapt-workshop.org/2014/program.htm ADAPT'14 panel] @ HiPEAC'14<br />
*[http://ctuning.org/making-computer-engineering-a-science-2013 HiPEAC'13 CSW thematic session] @ ACM ECRC "Making computer engineering a science"<br />
*[http://ctuning.org/hipeac3-thematic-session-2012-04 HiPEAC'12 CSW thematic session]<br />
*[http://exadapt.org/2012/program.html ASPLOS/EXADAPT'12 panel] @ ASPLOS'12<br />
*[http://ctuning.org/lab/education cTuning lectures (2008-2010)]<br />
*[http://hal.inria.fr/inria-00436029 GCC Summit'09 discussion]<br />
<br />
== Paper and artifact evaluation committee<br/> ==<br />
<br />
Rather than pre-selecting a dedicated committee for conferences, we select reviewers for reseach material (artifacts) and publications from a [http://cTuning.org/lab/people pool of our supporters] based on '''''submitted and publicly available''''' publications, their keywords and '''''public discussions''''' as described in our proposal [[http://arxiv.org/abs/1406.4020 arXiv]], [[http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]. Validated papers receive a stamp "Validated by the community". Artifacts can be shared along with publication in the ACM Digital LIbrary, HAL, Collective Mind Repository or any other public archive.<br />
<br />
As for the workshops, conferences and journals with the traditional publication model (CGO, PPoPP, PLDI), we select artifact evaluation committee (AEC) as described [[Reproducibility:AE|here]].<br />
<br />
== Packing and sharing research and experimental material<br/> ==<br />
<br />
Rather than enforcing specific procedure for packing, sharing and validation of experimental results, we allow authors of the accepted papers to include an archive with all related research material (using any [[Reproducibility:Links|publicly available tool]]) and ''readme.txt'' file describing how to validate their experiments. The main reason is the lack of a universally acceptable solution to pack and share experimental setups. For example, it is not always possible to use Virtual Machines and similar approaches for our research on performance/energy tuning or when some new hardware is being co-designed as we discuss in our proposal [[http://arxiv.org/abs/1406.4020 arXiv], [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]. Therefore, our current intention is to gradually and collaboratively find best procedure for packing using practical experience from our events such as [http://adapt-workshop.org ADAPT workshop] and from common discussions during [http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14 workshops]. See also nice guidelines for packing code and data along with publications [http://www.artifact-eval.org/guidelines.html here].<br />
<br />
== Validation<br/> ==<br />
<br />
After many years of evangelizing collaborative and reproducible research in computer engineering based on our practical experience, we finally start seeing some change in the mentality in academia, industry and funding agencies. Authors of two papers of our [http://adapt-workshop.org/2014 ADAPT'14 workshop] (out of nine accepted) agreed to have experimental results of their papers validated by volunteers. Note that rather than enforcing specific validation rules, we decided to ask authors to pack all their research artifacts as they wish (for example, using a shared virtual machine or as a standard archive) and describe their own validation procedure. Thanks to [http://cTuning.org/lab/people our volunteers], experiments from these papers have been validated, archives shared in [http://c-mind.org/repo our public repository] , and papers marked with a "validated by the community" stamp:<br />
<p style="text-align: center">[[File:Logo-validated-by-the-community.png|center|Logo-validated-by-the-community.png]]</p><br />
== Archive<br/> ==<br />
<br />
*[http://ctuning.org/wiki/index.php?title=Discussions:New_Publication_Model Outdated cTuning wiki page related to reproducible research and open publication model]<br />
*Outdated cTuning repository for program/processor performance/power/size optimization (2008-2010): [ [http://ctuning.org/wiki/index.php/CDatabase database], [http://ctuning.org/wiki/index.php?title=Special:CPredict web-service for online prediction of optimizations] ]<br />
<br />
== Resources<br/> ==<br />
<br />
*[[Reproducibility:Initiatives|Collection of related initiatives]]<br />
*[[Reproducibility:Links|Collection of related tools]]<br />
*[[Reproducibility:Datasets|Collection of related benchmarks and data sets]]<br />
*[[Reproducibility:Repositories|Collection of public repositories]]<br />
*[[Reproducibility:Lectures|Collection of related lectures]]<br />
*[[Reproducibility:Articles|Collection of related articles]]<br />
*[[Reproducibility:Blogs|Collection of related blogs]]<br />
*[[Reproducibility:Jokes|Collection of jokes]]<br />
*[[Reproducibility:Events|Collection of related events]]<br />
<br />
== Discussions<br/> ==<br />
<br />
*[http://www.linkedin.com/groups/Reproducible-research-experimentation-in-computer-7433414 LinkedIn group on reproducible research]<br />
*[http://groups.google.com/group/collective-knowledge Main mailing list] (general collaborative and reproducible R&D in computer engineering)<br />
*[http://groups.google.com/group/ctuning-discussions cTuning foundation mailing list] (collaborative and reproducible hardware and software benchmarking, auto-tuning and co-design)<br />
<br />
== Follow us<br/> ==<br />
<br />
*[https://twitter.com/c_tuning cTuning foundation twitter]<br />
*[https://www.facebook.com/pages/CrowdTuning-Foundation/668405119902805 cTuning foundation facebook page] (recent)<br />
*[http://dividiti.blogspot.com dividiti blog]<br />
*[http://c-mind.org/repo Collective Mind repository] (outdated)<br />
*[http://cknowledge.org/repo Collective Knowledge repository] (new)<br />
<br />
== Acknowledgments<br/> ==<br />
<br />
We would like to thank our colleagues from the [http://cTuning.org/lab/people cTuning foundation], [https://www.linkedin.com/company/dividiti dividiti], [http://www.artifact-eval.org artifact-eval.org], [http://www.occamportal.org OCCAM project] for their help, feedback, participation and support.</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility&diff=849Reproducibility2015-10-26T11:00:51Z<p>Gfursin: /* Our interdisciplinary events */</p>
<hr />
<div><center><br />
<span style="font-size:x-large"><span style="color: rgb(178, 34, 34)">Enabling collaborative and reproducible computer systems research with an open publication model</span></span><br />
</center><p style="text-align: center">[[File:Ae-stamp-ppopp2015.png|Ae-stamp-ppopp2015.png|link=http://cTuning.org/event/ae-ppopp2016]] [[File:Ae-stamp-cgo.png|Ae-stamp-cgo.png|link=http://cTuning.org/event/ae-cgo2016]] [[File:Logo-validated-by-the-community.png|Logo-validated-by-the-community.png|link=http://cTuning.org/reproducibility]] &nbsp; &nbsp;&nbsp; [[File:CTuning foundation logo1.png|cTuning_foundation_logo1.png|link=http://cTuning.org]]&nbsp; &nbsp;&nbsp; [[File:dividiti.png|dividiti.png|link=http://dividiti.com]]</p><p style="text-align: center"></p><p style="text-align: center">This wiki is maintained by the [http://cTuning.org non-profit Tuning foundation].</p><br />
== News and upcoming events<br/> ==<br />
<br />
*''Brand new, open-source, BSD-licensed Collective Knowledge SDK (cTuning 4 aka CK) for collaborative and reproducible R&D is now publicly available at [http://github.com/ctuning/ck GitHub] with an [http://github.com/ctuning/ck/wiki online documentation] and [http://cknowledge.org/repo live demo repository].''<br />
*''[http://cTuning.org/event/ae-ppopp2016 PPoPP'16 artifact evaluation]''<br />
*''[http://cTuning.org/event/ae-cgo2016 CGO'16 artifact evaluation]''<br />
*''[http://adapt-workshop.org ADAPT'16 @ HiPEAC'16] - features [http://dl.acm.org/citation.cfm?id=2618142 our open publication model with community-driven reviewing, reddit-based discussions and artifact evaluation]''<br />
*''[http://www.dagstuhl.de/de/programm/kalender/semhp/?semnr=15452 Dagstuhl perspective workshop on artifact evaluation for confernces and journals]''<br />
<br />
== Motivation<br/> ==<br />
<br />
Since 2006 we have been trying to solve problems with reproducibility of experimental results in computer engineering ''as a side effect'' of our [http://cTuning.org/project-milepost MILEPOST] , [http://cTuning.org cTuning.org], [http://c-mind.org Collective Mind] and [http://github.com/ctuning/ck Collective Knowledge] projects (speeding up optimization, benchmarking and co-design of computer systems using auto-tuning, big data, predictive analytics and crowdsourcing). <span style="font-size: small">We focus on the following technological and social aspects to enable collaborative, systematic and reproducible research and experimentation particularly related to benchmarking, optimization and co-design of faster, smaller, cheaper, more power efficient and reliable software and hardware:</span><br />
<br />
*developing public and open source Collective Mind repositories of knowledge (see our pilot live repository [[http://cknowledge.org/repo CK], [http://c-mind.org/repo cMind]] and our vision papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]]);<br />
*developing [http://github.com/ctuning/ck collaborative research and experimentation infrastructure] that can share artifacts as reusable components together with the whole experimental setups (see our papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]];<br />
*evangelizing and enabling new open publication model for online workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*setting up and improving procedure for sharing and evaluating experimental results and all related material for workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*improving sharing, description of dependencies, and statistical reproducibility of experimental results and related material.<br />
<br />
See our manifesto and history [http://cTuning.org/history here].<br />
<br />
=== Community-driven research and developments<br/> ===<br />
<br />
Together with the community and [http://cTuning.org not-for-profit cTuning foundation] we are working on the following topics:<br />
<br />
*developing tools and methodology to capture, preserve, formalize, systematize, exchange and improve knowledge and experimental results including negative ones<br />
*describing and cataloging whole experimental setups with all related material including algorithms, benchmarks, codelets, datasets, tools, models and any other artifact<br />
*developing specification to preserve experiments including all software and hardware dependencies<br />
*dealing with variability and rising amount of experimental data using statistical analysis, data mining, predictive modeling and other techniques<br />
*developing new predictive analytics techniques to explore large design and optimization spaces<br />
*validating and verifying experimental results by the community<br />
*developing common research interfaces for existing or new tools<br />
*developing common experimental frameworks and repositories (enable automation, re-execution and sharing of experiments)<br />
*sharing rare hardware and computational resources for experimental validation<br />
*implementing previously published experimental scenarios (auto-tuning, run-time adaptation) using common infrastructure<br />
*implementing open access to publications and data (particularly discussing intellectual property IP and legal issues)<br />
*speeding up analysis of "big" experimental data<br />
*developing new (interactive) visualization techniques for "big" experimental data<br />
*enabling interactive articles<br />
<br />
== Our events<br/> ==<br />
<br />
*[[Reproducibility:AE:PPoPP2015|PPoPP'15 artifact evaluation]]<br />
*[[Reproducibility:AE:CGO2015|CGO'15 artifact evaluation]]<br />
*[http://adapt-workshop.org ADAPT'15] @ HiPEAC'15 - workshop on adaptive self-tuning computer systems<br />
*[http://adapt-workshop.org/2014 ADAPT'14] @ HiPEAC'14 - workshop on adaptive self-tuning computer systems [ [http://adapt-workshop.org/2014/program.htm program and publications] ]<br />
*[http://www.occamportal.org/images/reproduce/TETC-SI-REPRODUCE.pdf Special journal issue] on Reproducible &nbsp;Research &nbsp;Methodologies at IEEE TETC<br />
*[http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14] @ PLDI'14<br />
*[http://www.occamportal.org/reproduce REPRODUCE'14] @ HPCA'14<br />
*[http://www.adapt-workshop.org/2014/program.htm ADAPT'14 panel] @ HiPEAC'14<br />
*[http://ctuning.org/making-computer-engineering-a-science-2013 HiPEAC'13 CSW thematic session] @ ACM ECRC "Making computer engineering a science"<br />
*[http://ctuning.org/hipeac3-thematic-session-2012-04 HiPEAC'12 CSW thematic session]<br />
*[http://exadapt.org/2012/program.html ASPLOS/EXADAPT'12 panel] @ ASPLOS'12<br />
*[http://ctuning.org/lab/education cTuning lectures (2008-2010)]<br />
*[http://hal.inria.fr/inria-00436029 GCC Summit'09 discussion]<br />
<br />
== Paper and artifact evaluation committee<br/> ==<br />
<br />
Rather than pre-selecting a dedicated committee for conferences, we select reviewers for reseach material (artifacts) and publications from a [http://cTuning.org/lab/people pool of our supporters] based on '''''submitted and publicly available''''' publications, their keywords and '''''public discussions''''' as described in our proposal [[http://arxiv.org/abs/1406.4020 arXiv]], [[http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]. Validated papers receive a stamp "Validated by the community". Artifacts can be shared along with publication in the ACM Digital LIbrary, HAL, Collective Mind Repository or any other public archive.<br />
<br />
As for the workshops, conferences and journals with the traditional publication model (CGO, PPoPP, PLDI), we select artifact evaluation committee (AEC) as described [[Reproducibility:AE|here]].<br />
<br />
== Packing and sharing research and experimental material<br/> ==<br />
<br />
Rather than enforcing specific procedure for packing, sharing and validation of experimental results, we allow authors of the accepted papers to include an archive with all related research material (using any [[Reproducibility:Links|publicly available tool]]) and ''readme.txt'' file describing how to validate their experiments. The main reason is the lack of a universally acceptable solution to pack and share experimental setups. For example, it is not always possible to use Virtual Machines and similar approaches for our research on performance/energy tuning or when some new hardware is being co-designed as we discuss in our proposal [[http://arxiv.org/abs/1406.4020 arXiv], [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]. Therefore, our current intention is to gradually and collaboratively find best procedure for packing using practical experience from our events such as [http://adapt-workshop.org ADAPT workshop] and from common discussions during [http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14 workshops]. See also nice guidelines for packing code and data along with publications [http://www.artifact-eval.org/guidelines.html here].<br />
<br />
== Validation<br/> ==<br />
<br />
After many years of evangelizing collaborative and reproducible research in computer engineering based on our practical experience, we finally start seeing some change in the mentality in academia, industry and funding agencies. Authors of two papers of our [http://adapt-workshop.org/2014 ADAPT'14 workshop] (out of nine accepted) agreed to have experimental results of their papers validated by volunteers. Note that rather than enforcing specific validation rules, we decided to ask authors to pack all their research artifacts as they wish (for example, using a shared virtual machine or as a standard archive) and describe their own validation procedure. Thanks to [http://cTuning.org/lab/people our volunteers], experiments from these papers have been validated, archives shared in [http://c-mind.org/repo our public repository] , and papers marked with a "validated by the community" stamp:<br />
<p style="text-align: center">[[File:Logo-validated-by-the-community.png|center|Logo-validated-by-the-community.png]]</p><br />
== Archive<br/> ==<br />
<br />
*[http://ctuning.org/wiki/index.php?title=Discussions:New_Publication_Model Outdated cTuning wiki page related to reproducible research and open publication model]<br />
*Outdated cTuning repository for program/processor performance/power/size optimization (2008-2010): [ [http://ctuning.org/wiki/index.php/CDatabase database], [http://ctuning.org/wiki/index.php?title=Special:CPredict web-service for online prediction of optimizations] ]<br />
<br />
== Resources<br/> ==<br />
<br />
*[[Reproducibility:Initiatives|Collection of related initiatives]]<br />
*[[Reproducibility:Links|Collection of related tools]]<br />
*[[Reproducibility:Datasets|Collection of related benchmarks and data sets]]<br />
*[[Reproducibility:Repositories|Collection of public repositories]]<br />
*[[Reproducibility:Lectures|Collection of related lectures]]<br />
*[[Reproducibility:Articles|Collection of related articles]]<br />
*[[Reproducibility:Blogs|Collection of related blogs]]<br />
*[[Reproducibility:Jokes|Collection of jokes]]<br />
*[[Reproducibility:Events|Collection of related events]]<br />
<br />
== Discussions<br/> ==<br />
<br />
*[http://www.linkedin.com/groups/Reproducible-research-experimentation-in-computer-7433414 LinkedIn group on reproducible research]<br />
*[http://groups.google.com/group/collective-knowledge Main mailing list] (general collaborative and reproducible R&D in computer engineering)<br />
*[http://groups.google.com/group/ctuning-discussions cTuning foundation mailing list] (collaborative and reproducible hardware and software benchmarking, auto-tuning and co-design)<br />
<br />
== Follow us<br/> ==<br />
<br />
*[https://twitter.com/c_tuning cTuning foundation twitter]<br />
*[https://www.facebook.com/pages/CrowdTuning-Foundation/668405119902805 cTuning foundation facebook page] (recent)<br />
*[http://dividiti.blogspot.com dividiti blog]<br />
*[http://c-mind.org/repo Collective Mind repository] (outdated)<br />
*[http://cknowledge.org/repo Collective Knowledge repository] (new)<br />
<br />
== Acknowledgments<br/> ==<br />
<br />
We would like to thank our colleagues from the [http://cTuning.org/lab/people cTuning foundation], [https://www.linkedin.com/company/dividiti dividiti], [http://www.artifact-eval.org artifact-eval.org], [http://www.occamportal.org OCCAM project] for their help, feedback, participation and support.</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility&diff=848Reproducibility2015-10-26T11:00:03Z<p>Gfursin: /* Follow us */</p>
<hr />
<div><center><br />
<span style="font-size:x-large"><span style="color: rgb(178, 34, 34)">Enabling collaborative and reproducible computer systems research with an open publication model</span></span><br />
</center><p style="text-align: center">[[File:Ae-stamp-ppopp2015.png|Ae-stamp-ppopp2015.png|link=http://cTuning.org/event/ae-ppopp2016]] [[File:Ae-stamp-cgo.png|Ae-stamp-cgo.png|link=http://cTuning.org/event/ae-cgo2016]] [[File:Logo-validated-by-the-community.png|Logo-validated-by-the-community.png|link=http://cTuning.org/reproducibility]] &nbsp; &nbsp;&nbsp; [[File:CTuning foundation logo1.png|cTuning_foundation_logo1.png|link=http://cTuning.org]]&nbsp; &nbsp;&nbsp; [[File:dividiti.png|dividiti.png|link=http://dividiti.com]]</p><p style="text-align: center"></p><p style="text-align: center">This wiki is maintained by the [http://cTuning.org non-profit Tuning foundation].</p><br />
== News and upcoming events<br/> ==<br />
<br />
*''Brand new, open-source, BSD-licensed Collective Knowledge SDK (cTuning 4 aka CK) for collaborative and reproducible R&D is now publicly available at [http://github.com/ctuning/ck GitHub] with an [http://github.com/ctuning/ck/wiki online documentation] and [http://cknowledge.org/repo live demo repository].''<br />
*''[http://cTuning.org/event/ae-ppopp2016 PPoPP'16 artifact evaluation]''<br />
*''[http://cTuning.org/event/ae-cgo2016 CGO'16 artifact evaluation]''<br />
*''[http://adapt-workshop.org ADAPT'16 @ HiPEAC'16] - features [http://dl.acm.org/citation.cfm?id=2618142 our open publication model with community-driven reviewing, reddit-based discussions and artifact evaluation]''<br />
*''[http://www.dagstuhl.de/de/programm/kalender/semhp/?semnr=15452 Dagstuhl perspective workshop on artifact evaluation for confernces and journals]''<br />
<br />
== Motivation<br/> ==<br />
<br />
Since 2006 we have been trying to solve problems with reproducibility of experimental results in computer engineering ''as a side effect'' of our [http://cTuning.org/project-milepost MILEPOST] , [http://cTuning.org cTuning.org], [http://c-mind.org Collective Mind] and [http://github.com/ctuning/ck Collective Knowledge] projects (speeding up optimization, benchmarking and co-design of computer systems using auto-tuning, big data, predictive analytics and crowdsourcing). <span style="font-size: small">We focus on the following technological and social aspects to enable collaborative, systematic and reproducible research and experimentation particularly related to benchmarking, optimization and co-design of faster, smaller, cheaper, more power efficient and reliable software and hardware:</span><br />
<br />
*developing public and open source Collective Mind repositories of knowledge (see our pilot live repository [[http://cknowledge.org/repo CK], [http://c-mind.org/repo cMind]] and our vision papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]]);<br />
*developing [http://github.com/ctuning/ck collaborative research and experimentation infrastructure] that can share artifacts as reusable components together with the whole experimental setups (see our papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]];<br />
*evangelizing and enabling new open publication model for online workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*setting up and improving procedure for sharing and evaluating experimental results and all related material for workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*improving sharing, description of dependencies, and statistical reproducibility of experimental results and related material.<br />
<br />
See our manifesto and history [http://cTuning.org/history here].<br />
<br />
=== Community-driven research and developments<br/> ===<br />
<br />
Together with the community and [http://cTuning.org not-for-profit cTuning foundation] we are working on the following topics:<br />
<br />
*developing tools and methodology to capture, preserve, formalize, systematize, exchange and improve knowledge and experimental results including negative ones<br />
*describing and cataloging whole experimental setups with all related material including algorithms, benchmarks, codelets, datasets, tools, models and any other artifact<br />
*developing specification to preserve experiments including all software and hardware dependencies<br />
*dealing with variability and rising amount of experimental data using statistical analysis, data mining, predictive modeling and other techniques<br />
*developing new predictive analytics techniques to explore large design and optimization spaces<br />
*validating and verifying experimental results by the community<br />
*developing common research interfaces for existing or new tools<br />
*developing common experimental frameworks and repositories (enable automation, re-execution and sharing of experiments)<br />
*sharing rare hardware and computational resources for experimental validation<br />
*implementing previously published experimental scenarios (auto-tuning, run-time adaptation) using common infrastructure<br />
*implementing open access to publications and data (particularly discussing intellectual property IP and legal issues)<br />
*speeding up analysis of "big" experimental data<br />
*developing new (interactive) visualization techniques for "big" experimental data<br />
*enabling interactive articles<br />
<br />
== Our interdisciplinary events<br/> ==<br />
<br />
==== Featuring new open publication model and validation of experimental results<br/> ====<br />
<br />
*[[Reproducibility:AE:PPoPP2015|PPoPP'15 artifact evaluation]]<br />
*[[Reproducibility:AE:CGO2015|CGO'15 artifact evaluation]]<br />
*[http://adapt-workshop.org ADAPT'15] @ HiPEAC'15 - workshop on adaptive self-tuning computer systems<br />
*[http://adapt-workshop.org/2014 ADAPT'14] @ HiPEAC'14 - workshop on adaptive self-tuning computer systems [ [http://adapt-workshop.org/2014/program.htm program and publications] ]<br />
<br />
==== Discussing technical aspects to enable reproducibility and open publication model<br/> ====<br />
<br />
*[http://www.occamportal.org/images/reproduce/TETC-SI-REPRODUCE.pdf Special journal issue] on Reproducible &nbsp;Research &nbsp;Methodologies at IEEE TETC<br />
*[http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14] @ PLDI'14<br />
*[http://www.occamportal.org/reproduce REPRODUCE'14] @ HPCA'14<br />
*[http://www.adapt-workshop.org/2014/program.htm ADAPT'14 panel] @ HiPEAC'14<br />
*[http://ctuning.org/making-computer-engineering-a-science-2013 HiPEAC'13 CSW thematic session] @ ACM ECRC "Making computer engineering a science"<br />
*[http://ctuning.org/hipeac3-thematic-session-2012-04 HiPEAC'12 CSW thematic session]<br />
*[http://exadapt.org/2012/program.html ASPLOS/EXADAPT'12 panel] @ ASPLOS'12<br />
*[http://ctuning.org/lab/education cTuning lectures (2008-2010)]<br />
*[http://hal.inria.fr/inria-00436029 GCC Summit'09 discussion]<br />
<br />
== Paper and artifact evaluation committee<br/> ==<br />
<br />
Rather than pre-selecting a dedicated committee for conferences, we select reviewers for reseach material (artifacts) and publications from a [http://cTuning.org/lab/people pool of our supporters] based on '''''submitted and publicly available''''' publications, their keywords and '''''public discussions''''' as described in our proposal [[http://arxiv.org/abs/1406.4020 arXiv]], [[http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]. Validated papers receive a stamp "Validated by the community". Artifacts can be shared along with publication in the ACM Digital LIbrary, HAL, Collective Mind Repository or any other public archive.<br />
<br />
As for the workshops, conferences and journals with the traditional publication model (CGO, PPoPP, PLDI), we select artifact evaluation committee (AEC) as described [[Reproducibility:AE|here]].<br />
<br />
== Packing and sharing research and experimental material<br/> ==<br />
<br />
Rather than enforcing specific procedure for packing, sharing and validation of experimental results, we allow authors of the accepted papers to include an archive with all related research material (using any [[Reproducibility:Links|publicly available tool]]) and ''readme.txt'' file describing how to validate their experiments. The main reason is the lack of a universally acceptable solution to pack and share experimental setups. For example, it is not always possible to use Virtual Machines and similar approaches for our research on performance/energy tuning or when some new hardware is being co-designed as we discuss in our proposal [[http://arxiv.org/abs/1406.4020 arXiv], [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]. Therefore, our current intention is to gradually and collaboratively find best procedure for packing using practical experience from our events such as [http://adapt-workshop.org ADAPT workshop] and from common discussions during [http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14 workshops]. See also nice guidelines for packing code and data along with publications [http://www.artifact-eval.org/guidelines.html here].<br />
<br />
== Validation<br/> ==<br />
<br />
After many years of evangelizing collaborative and reproducible research in computer engineering based on our practical experience, we finally start seeing some change in the mentality in academia, industry and funding agencies. Authors of two papers of our [http://adapt-workshop.org/2014 ADAPT'14 workshop] (out of nine accepted) agreed to have experimental results of their papers validated by volunteers. Note that rather than enforcing specific validation rules, we decided to ask authors to pack all their research artifacts as they wish (for example, using a shared virtual machine or as a standard archive) and describe their own validation procedure. Thanks to [http://cTuning.org/lab/people our volunteers], experiments from these papers have been validated, archives shared in [http://c-mind.org/repo our public repository] , and papers marked with a "validated by the community" stamp:<br />
<p style="text-align: center">[[File:Logo-validated-by-the-community.png|center|Logo-validated-by-the-community.png]]</p><br />
== Archive<br/> ==<br />
<br />
*[http://ctuning.org/wiki/index.php?title=Discussions:New_Publication_Model Outdated cTuning wiki page related to reproducible research and open publication model]<br />
*Outdated cTuning repository for program/processor performance/power/size optimization (2008-2010): [ [http://ctuning.org/wiki/index.php/CDatabase database], [http://ctuning.org/wiki/index.php?title=Special:CPredict web-service for online prediction of optimizations] ]<br />
<br />
== Resources<br/> ==<br />
<br />
*[[Reproducibility:Initiatives|Collection of related initiatives]]<br />
*[[Reproducibility:Links|Collection of related tools]]<br />
*[[Reproducibility:Datasets|Collection of related benchmarks and data sets]]<br />
*[[Reproducibility:Repositories|Collection of public repositories]]<br />
*[[Reproducibility:Lectures|Collection of related lectures]]<br />
*[[Reproducibility:Articles|Collection of related articles]]<br />
*[[Reproducibility:Blogs|Collection of related blogs]]<br />
*[[Reproducibility:Jokes|Collection of jokes]]<br />
*[[Reproducibility:Events|Collection of related events]]<br />
<br />
== Discussions<br/> ==<br />
<br />
*[http://www.linkedin.com/groups/Reproducible-research-experimentation-in-computer-7433414 LinkedIn group on reproducible research]<br />
*[http://groups.google.com/group/collective-knowledge Main mailing list] (general collaborative and reproducible R&D in computer engineering)<br />
*[http://groups.google.com/group/ctuning-discussions cTuning foundation mailing list] (collaborative and reproducible hardware and software benchmarking, auto-tuning and co-design)<br />
<br />
== Follow us<br/> ==<br />
<br />
*[https://twitter.com/c_tuning cTuning foundation twitter]<br />
*[https://www.facebook.com/pages/CrowdTuning-Foundation/668405119902805 cTuning foundation facebook page] (recent)<br />
*[http://dividiti.blogspot.com dividiti blog]<br />
*[http://c-mind.org/repo Collective Mind repository] (outdated)<br />
*[http://cknowledge.org/repo Collective Knowledge repository] (new)<br />
<br />
== Acknowledgments<br/> ==<br />
<br />
We would like to thank our colleagues from the [http://cTuning.org/lab/people cTuning foundation], [https://www.linkedin.com/company/dividiti dividiti], [http://www.artifact-eval.org artifact-eval.org], [http://www.occamportal.org OCCAM project] for their help, feedback, participation and support.</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility&diff=847Reproducibility2015-10-26T10:59:44Z<p>Gfursin: /* Acknowledgments */</p>
<hr />
<div><center><br />
<span style="font-size:x-large"><span style="color: rgb(178, 34, 34)">Enabling collaborative and reproducible computer systems research with an open publication model</span></span><br />
</center><p style="text-align: center">[[File:Ae-stamp-ppopp2015.png|Ae-stamp-ppopp2015.png|link=http://cTuning.org/event/ae-ppopp2016]] [[File:Ae-stamp-cgo.png|Ae-stamp-cgo.png|link=http://cTuning.org/event/ae-cgo2016]] [[File:Logo-validated-by-the-community.png|Logo-validated-by-the-community.png|link=http://cTuning.org/reproducibility]] &nbsp; &nbsp;&nbsp; [[File:CTuning foundation logo1.png|cTuning_foundation_logo1.png|link=http://cTuning.org]]&nbsp; &nbsp;&nbsp; [[File:dividiti.png|dividiti.png|link=http://dividiti.com]]</p><p style="text-align: center"></p><p style="text-align: center">This wiki is maintained by the [http://cTuning.org non-profit Tuning foundation].</p><br />
== News and upcoming events<br/> ==<br />
<br />
*''Brand new, open-source, BSD-licensed Collective Knowledge SDK (cTuning 4 aka CK) for collaborative and reproducible R&D is now publicly available at [http://github.com/ctuning/ck GitHub] with an [http://github.com/ctuning/ck/wiki online documentation] and [http://cknowledge.org/repo live demo repository].''<br />
*''[http://cTuning.org/event/ae-ppopp2016 PPoPP'16 artifact evaluation]''<br />
*''[http://cTuning.org/event/ae-cgo2016 CGO'16 artifact evaluation]''<br />
*''[http://adapt-workshop.org ADAPT'16 @ HiPEAC'16] - features [http://dl.acm.org/citation.cfm?id=2618142 our open publication model with community-driven reviewing, reddit-based discussions and artifact evaluation]''<br />
*''[http://www.dagstuhl.de/de/programm/kalender/semhp/?semnr=15452 Dagstuhl perspective workshop on artifact evaluation for confernces and journals]''<br />
<br />
== Motivation<br/> ==<br />
<br />
Since 2006 we have been trying to solve problems with reproducibility of experimental results in computer engineering ''as a side effect'' of our [http://cTuning.org/project-milepost MILEPOST] , [http://cTuning.org cTuning.org], [http://c-mind.org Collective Mind] and [http://github.com/ctuning/ck Collective Knowledge] projects (speeding up optimization, benchmarking and co-design of computer systems using auto-tuning, big data, predictive analytics and crowdsourcing). <span style="font-size: small">We focus on the following technological and social aspects to enable collaborative, systematic and reproducible research and experimentation particularly related to benchmarking, optimization and co-design of faster, smaller, cheaper, more power efficient and reliable software and hardware:</span><br />
<br />
*developing public and open source Collective Mind repositories of knowledge (see our pilot live repository [[http://cknowledge.org/repo CK], [http://c-mind.org/repo cMind]] and our vision papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]]);<br />
*developing [http://github.com/ctuning/ck collaborative research and experimentation infrastructure] that can share artifacts as reusable components together with the whole experimental setups (see our papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]];<br />
*evangelizing and enabling new open publication model for online workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*setting up and improving procedure for sharing and evaluating experimental results and all related material for workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*improving sharing, description of dependencies, and statistical reproducibility of experimental results and related material.<br />
<br />
See our manifesto and history [http://cTuning.org/history here].<br />
<br />
=== Community-driven research and developments<br/> ===<br />
<br />
Together with the community and [http://cTuning.org not-for-profit cTuning foundation] we are working on the following topics:<br />
<br />
*developing tools and methodology to capture, preserve, formalize, systematize, exchange and improve knowledge and experimental results including negative ones<br />
*describing and cataloging whole experimental setups with all related material including algorithms, benchmarks, codelets, datasets, tools, models and any other artifact<br />
*developing specification to preserve experiments including all software and hardware dependencies<br />
*dealing with variability and rising amount of experimental data using statistical analysis, data mining, predictive modeling and other techniques<br />
*developing new predictive analytics techniques to explore large design and optimization spaces<br />
*validating and verifying experimental results by the community<br />
*developing common research interfaces for existing or new tools<br />
*developing common experimental frameworks and repositories (enable automation, re-execution and sharing of experiments)<br />
*sharing rare hardware and computational resources for experimental validation<br />
*implementing previously published experimental scenarios (auto-tuning, run-time adaptation) using common infrastructure<br />
*implementing open access to publications and data (particularly discussing intellectual property IP and legal issues)<br />
*speeding up analysis of "big" experimental data<br />
*developing new (interactive) visualization techniques for "big" experimental data<br />
*enabling interactive articles<br />
<br />
== Our interdisciplinary events<br/> ==<br />
<br />
==== Featuring new open publication model and validation of experimental results<br/> ====<br />
<br />
*[[Reproducibility:AE:PPoPP2015|PPoPP'15 artifact evaluation]]<br />
*[[Reproducibility:AE:CGO2015|CGO'15 artifact evaluation]]<br />
*[http://adapt-workshop.org ADAPT'15] @ HiPEAC'15 - workshop on adaptive self-tuning computer systems<br />
*[http://adapt-workshop.org/2014 ADAPT'14] @ HiPEAC'14 - workshop on adaptive self-tuning computer systems [ [http://adapt-workshop.org/2014/program.htm program and publications] ]<br />
<br />
==== Discussing technical aspects to enable reproducibility and open publication model<br/> ====<br />
<br />
*[http://www.occamportal.org/images/reproduce/TETC-SI-REPRODUCE.pdf Special journal issue] on Reproducible &nbsp;Research &nbsp;Methodologies at IEEE TETC<br />
*[http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14] @ PLDI'14<br />
*[http://www.occamportal.org/reproduce REPRODUCE'14] @ HPCA'14<br />
*[http://www.adapt-workshop.org/2014/program.htm ADAPT'14 panel] @ HiPEAC'14<br />
*[http://ctuning.org/making-computer-engineering-a-science-2013 HiPEAC'13 CSW thematic session] @ ACM ECRC "Making computer engineering a science"<br />
*[http://ctuning.org/hipeac3-thematic-session-2012-04 HiPEAC'12 CSW thematic session]<br />
*[http://exadapt.org/2012/program.html ASPLOS/EXADAPT'12 panel] @ ASPLOS'12<br />
*[http://ctuning.org/lab/education cTuning lectures (2008-2010)]<br />
*[http://hal.inria.fr/inria-00436029 GCC Summit'09 discussion]<br />
<br />
== Paper and artifact evaluation committee<br/> ==<br />
<br />
Rather than pre-selecting a dedicated committee for conferences, we select reviewers for reseach material (artifacts) and publications from a [http://cTuning.org/lab/people pool of our supporters] based on '''''submitted and publicly available''''' publications, their keywords and '''''public discussions''''' as described in our proposal [[http://arxiv.org/abs/1406.4020 arXiv]], [[http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]. Validated papers receive a stamp "Validated by the community". Artifacts can be shared along with publication in the ACM Digital LIbrary, HAL, Collective Mind Repository or any other public archive.<br />
<br />
As for the workshops, conferences and journals with the traditional publication model (CGO, PPoPP, PLDI), we select artifact evaluation committee (AEC) as described [[Reproducibility:AE|here]].<br />
<br />
== Packing and sharing research and experimental material<br/> ==<br />
<br />
Rather than enforcing specific procedure for packing, sharing and validation of experimental results, we allow authors of the accepted papers to include an archive with all related research material (using any [[Reproducibility:Links|publicly available tool]]) and ''readme.txt'' file describing how to validate their experiments. The main reason is the lack of a universally acceptable solution to pack and share experimental setups. For example, it is not always possible to use Virtual Machines and similar approaches for our research on performance/energy tuning or when some new hardware is being co-designed as we discuss in our proposal [[http://arxiv.org/abs/1406.4020 arXiv], [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]. Therefore, our current intention is to gradually and collaboratively find best procedure for packing using practical experience from our events such as [http://adapt-workshop.org ADAPT workshop] and from common discussions during [http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14 workshops]. See also nice guidelines for packing code and data along with publications [http://www.artifact-eval.org/guidelines.html here].<br />
<br />
== Validation<br/> ==<br />
<br />
After many years of evangelizing collaborative and reproducible research in computer engineering based on our practical experience, we finally start seeing some change in the mentality in academia, industry and funding agencies. Authors of two papers of our [http://adapt-workshop.org/2014 ADAPT'14 workshop] (out of nine accepted) agreed to have experimental results of their papers validated by volunteers. Note that rather than enforcing specific validation rules, we decided to ask authors to pack all their research artifacts as they wish (for example, using a shared virtual machine or as a standard archive) and describe their own validation procedure. Thanks to [http://cTuning.org/lab/people our volunteers], experiments from these papers have been validated, archives shared in [http://c-mind.org/repo our public repository] , and papers marked with a "validated by the community" stamp:<br />
<p style="text-align: center">[[File:Logo-validated-by-the-community.png|center|Logo-validated-by-the-community.png]]</p><br />
== Archive<br/> ==<br />
<br />
*[http://ctuning.org/wiki/index.php?title=Discussions:New_Publication_Model Outdated cTuning wiki page related to reproducible research and open publication model]<br />
*Outdated cTuning repository for program/processor performance/power/size optimization (2008-2010): [ [http://ctuning.org/wiki/index.php/CDatabase database], [http://ctuning.org/wiki/index.php?title=Special:CPredict web-service for online prediction of optimizations] ]<br />
<br />
== Resources<br/> ==<br />
<br />
*[[Reproducibility:Initiatives|Collection of related initiatives]]<br />
*[[Reproducibility:Links|Collection of related tools]]<br />
*[[Reproducibility:Datasets|Collection of related benchmarks and data sets]]<br />
*[[Reproducibility:Repositories|Collection of public repositories]]<br />
*[[Reproducibility:Lectures|Collection of related lectures]]<br />
*[[Reproducibility:Articles|Collection of related articles]]<br />
*[[Reproducibility:Blogs|Collection of related blogs]]<br />
*[[Reproducibility:Jokes|Collection of jokes]]<br />
*[[Reproducibility:Events|Collection of related events]]<br />
<br />
== Discussions<br/> ==<br />
<br />
*[http://www.linkedin.com/groups/Reproducible-research-experimentation-in-computer-7433414 LinkedIn group on reproducible research]<br />
*[http://groups.google.com/group/collective-knowledge Main mailing list] (general collaborative and reproducible R&D in computer engineering)<br />
*[http://groups.google.com/group/ctuning-discussions cTuning foundation mailing list] (collaborative and reproducible hardware and software benchmarking, auto-tuning and co-design)<br />
<br />
== Follow us<br/> ==<br />
<br />
*[https://twitter.com/c_tuning cTuning foundation twitter]<br />
*[https://www.facebook.com/pages/CrowdTuning-Foundation/668405119902805 cTuning foundation facebook page] (recent)<br />
*[http://dividiti.blogspot.com dividiti blog]<br />
*[http://c-mind.org/repo Collective Mind repository] (outdated)<br />
*[http://cknowledge.org/repo Collective Knowledge repository] (new)</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility&diff=846Reproducibility2015-10-26T10:37:51Z<p>Gfursin: /* News and upcoming events */</p>
<hr />
<div><center><br />
<span style="font-size:x-large"><span style="color: rgb(178, 34, 34)">Enabling collaborative and reproducible computer systems research with an open publication model</span></span><br />
</center><p style="text-align: center">[[File:Ae-stamp-ppopp2015.png|Ae-stamp-ppopp2015.png|link=http://cTuning.org/event/ae-ppopp2016]] [[File:Ae-stamp-cgo.png|Ae-stamp-cgo.png|link=http://cTuning.org/event/ae-cgo2016]] [[File:Logo-validated-by-the-community.png|Logo-validated-by-the-community.png|link=http://cTuning.org/reproducibility]] &nbsp; &nbsp;&nbsp; [[File:CTuning foundation logo1.png|cTuning_foundation_logo1.png|link=http://cTuning.org]]&nbsp; &nbsp;&nbsp; [[File:dividiti.png|dividiti.png|link=http://dividiti.com]]</p><p style="text-align: center"></p><p style="text-align: center">This wiki is maintained by the [http://cTuning.org non-profit Tuning foundation].</p><br />
== News and upcoming events<br/> ==<br />
<br />
*''Brand new, open-source, BSD-licensed Collective Knowledge SDK (cTuning 4 aka CK) for collaborative and reproducible R&D is now publicly available at [http://github.com/ctuning/ck GitHub] with an [http://github.com/ctuning/ck/wiki online documentation] and [http://cknowledge.org/repo live demo repository].''<br />
*''[http://cTuning.org/event/ae-ppopp2016 PPoPP'16 artifact evaluation]''<br />
*''[http://cTuning.org/event/ae-cgo2016 CGO'16 artifact evaluation]''<br />
*''[http://adapt-workshop.org ADAPT'16 @ HiPEAC'16] - features [http://dl.acm.org/citation.cfm?id=2618142 our open publication model with community-driven reviewing, reddit-based discussions and artifact evaluation]''<br />
*''[http://www.dagstuhl.de/de/programm/kalender/semhp/?semnr=15452 Dagstuhl perspective workshop on artifact evaluation for confernces and journals]''<br />
<br />
== Motivation<br/> ==<br />
<br />
Since 2006 we have been trying to solve problems with reproducibility of experimental results in computer engineering ''as a side effect'' of our [http://cTuning.org/project-milepost MILEPOST] , [http://cTuning.org cTuning.org], [http://c-mind.org Collective Mind] and [http://github.com/ctuning/ck Collective Knowledge] projects (speeding up optimization, benchmarking and co-design of computer systems using auto-tuning, big data, predictive analytics and crowdsourcing). <span style="font-size: small">We focus on the following technological and social aspects to enable collaborative, systematic and reproducible research and experimentation particularly related to benchmarking, optimization and co-design of faster, smaller, cheaper, more power efficient and reliable software and hardware:</span><br />
<br />
*developing public and open source Collective Mind repositories of knowledge (see our pilot live repository [[http://cknowledge.org/repo CK], [http://c-mind.org/repo cMind]] and our vision papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]]);<br />
*developing [http://github.com/ctuning/ck collaborative research and experimentation infrastructure] that can share artifacts as reusable components together with the whole experimental setups (see our papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]];<br />
*evangelizing and enabling new open publication model for online workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*setting up and improving procedure for sharing and evaluating experimental results and all related material for workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*improving sharing, description of dependencies, and statistical reproducibility of experimental results and related material.<br />
<br />
See our manifesto and history [http://cTuning.org/history here].<br />
<br />
=== Community-driven research and developments<br/> ===<br />
<br />
Together with the community and [http://cTuning.org not-for-profit cTuning foundation] we are working on the following topics:<br />
<br />
*developing tools and methodology to capture, preserve, formalize, systematize, exchange and improve knowledge and experimental results including negative ones<br />
*describing and cataloging whole experimental setups with all related material including algorithms, benchmarks, codelets, datasets, tools, models and any other artifact<br />
*developing specification to preserve experiments including all software and hardware dependencies<br />
*dealing with variability and rising amount of experimental data using statistical analysis, data mining, predictive modeling and other techniques<br />
*developing new predictive analytics techniques to explore large design and optimization spaces<br />
*validating and verifying experimental results by the community<br />
*developing common research interfaces for existing or new tools<br />
*developing common experimental frameworks and repositories (enable automation, re-execution and sharing of experiments)<br />
*sharing rare hardware and computational resources for experimental validation<br />
*implementing previously published experimental scenarios (auto-tuning, run-time adaptation) using common infrastructure<br />
*implementing open access to publications and data (particularly discussing intellectual property IP and legal issues)<br />
*speeding up analysis of "big" experimental data<br />
*developing new (interactive) visualization techniques for "big" experimental data<br />
*enabling interactive articles<br />
<br />
== Our interdisciplinary events<br/> ==<br />
<br />
==== Featuring new open publication model and validation of experimental results<br/> ====<br />
<br />
*[[Reproducibility:AE:PPoPP2015|PPoPP'15 artifact evaluation]]<br />
*[[Reproducibility:AE:CGO2015|CGO'15 artifact evaluation]]<br />
*[http://adapt-workshop.org ADAPT'15] @ HiPEAC'15 - workshop on adaptive self-tuning computer systems<br />
*[http://adapt-workshop.org/2014 ADAPT'14] @ HiPEAC'14 - workshop on adaptive self-tuning computer systems [ [http://adapt-workshop.org/2014/program.htm program and publications] ]<br />
<br />
==== Discussing technical aspects to enable reproducibility and open publication model<br/> ====<br />
<br />
*[http://www.occamportal.org/images/reproduce/TETC-SI-REPRODUCE.pdf Special journal issue] on Reproducible &nbsp;Research &nbsp;Methodologies at IEEE TETC<br />
*[http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14] @ PLDI'14<br />
*[http://www.occamportal.org/reproduce REPRODUCE'14] @ HPCA'14<br />
*[http://www.adapt-workshop.org/2014/program.htm ADAPT'14 panel] @ HiPEAC'14<br />
*[http://ctuning.org/making-computer-engineering-a-science-2013 HiPEAC'13 CSW thematic session] @ ACM ECRC "Making computer engineering a science"<br />
*[http://ctuning.org/hipeac3-thematic-session-2012-04 HiPEAC'12 CSW thematic session]<br />
*[http://exadapt.org/2012/program.html ASPLOS/EXADAPT'12 panel] @ ASPLOS'12<br />
*[http://ctuning.org/lab/education cTuning lectures (2008-2010)]<br />
*[http://hal.inria.fr/inria-00436029 GCC Summit'09 discussion]<br />
<br />
== Acknowledgments<br/> ==<br />
<br />
We would like to thank our colleagues from the [http://cTuning.org/lab/people cTuning foundation], [https://www.linkedin.com/company/dividiti dividiti], [http://www.artifact-eval.org artifact-eval.org], [http://www.occamportal.org OCCAM project] for their help, feedback, participation and support.<br />
<br />
== Paper and artifact evaluation committee<br/> ==<br />
<br />
Rather than pre-selecting a dedicated committee for conferences, we select reviewers for reseach material (artifacts) and publications from a [http://cTuning.org/lab/people pool of our supporters] based on '''''submitted and publicly available''''' publications, their keywords and '''''public discussions''''' as described in our proposal [[http://arxiv.org/abs/1406.4020 arXiv]], [[http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]. Validated papers receive a stamp "Validated by the community". Artifacts can be shared along with publication in the ACM Digital LIbrary, HAL, Collective Mind Repository or any other public archive.<br />
<br />
As for the workshops, conferences and journals with the traditional publication model (CGO, PPoPP, PLDI), we select artifact evaluation committee (AEC) as described [[Reproducibility:AE|here]].<br />
<br />
== Packing and sharing research and experimental material<br/> ==<br />
<br />
Rather than enforcing specific procedure for packing, sharing and validation of experimental results, we allow authors of the accepted papers to include an archive with all related research material (using any [[Reproducibility:Links|publicly available tool]]) and ''readme.txt'' file describing how to validate their experiments. The main reason is the lack of a universally acceptable solution to pack and share experimental setups. For example, it is not always possible to use Virtual Machines and similar approaches for our research on performance/energy tuning or when some new hardware is being co-designed as we discuss in our proposal [[http://arxiv.org/abs/1406.4020 arXiv], [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]. Therefore, our current intention is to gradually and collaboratively find best procedure for packing using practical experience from our events such as [http://adapt-workshop.org ADAPT workshop] and from common discussions during [http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14 workshops]. See also nice guidelines for packing code and data along with publications [http://www.artifact-eval.org/guidelines.html here].<br />
<br />
== Validation<br/> ==<br />
<br />
After many years of evangelizing collaborative and reproducible research in computer engineering based on our practical experience, we finally start seeing some change in the mentality in academia, industry and funding agencies. Authors of two papers of our [http://adapt-workshop.org/2014 ADAPT'14 workshop] (out of nine accepted) agreed to have experimental results of their papers validated by volunteers. Note that rather than enforcing specific validation rules, we decided to ask authors to pack all their research artifacts as they wish (for example, using a shared virtual machine or as a standard archive) and describe their own validation procedure. Thanks to [http://cTuning.org/lab/people our volunteers], experiments from these papers have been validated, archives shared in [http://c-mind.org/repo our public repository] , and papers marked with a "validated by the community" stamp:<br />
<p style="text-align: center">[[File:Logo-validated-by-the-community.png|center|Logo-validated-by-the-community.png]]</p><br />
== Archive<br/> ==<br />
<br />
*[http://ctuning.org/wiki/index.php?title=Discussions:New_Publication_Model Outdated cTuning wiki page related to reproducible research and open publication model]<br />
*Outdated cTuning repository for program/processor performance/power/size optimization (2008-2010): [ [http://ctuning.org/wiki/index.php/CDatabase database], [http://ctuning.org/wiki/index.php?title=Special:CPredict web-service for online prediction of optimizations] ]<br />
<br />
== Resources<br/> ==<br />
<br />
*[[Reproducibility:Initiatives|Collection of related initiatives]]<br />
*[[Reproducibility:Links|Collection of related tools]]<br />
*[[Reproducibility:Datasets|Collection of related benchmarks and data sets]]<br />
*[[Reproducibility:Repositories|Collection of public repositories]]<br />
*[[Reproducibility:Lectures|Collection of related lectures]]<br />
*[[Reproducibility:Articles|Collection of related articles]]<br />
*[[Reproducibility:Blogs|Collection of related blogs]]<br />
*[[Reproducibility:Jokes|Collection of jokes]]<br />
*[[Reproducibility:Events|Collection of related events]]<br />
<br />
== Discussions<br/> ==<br />
<br />
*[http://www.linkedin.com/groups/Reproducible-research-experimentation-in-computer-7433414 LinkedIn group on reproducible research]<br />
*[http://groups.google.com/group/collective-knowledge Main mailing list] (general collaborative and reproducible R&D in computer engineering)<br />
*[http://groups.google.com/group/ctuning-discussions cTuning foundation mailing list] (collaborative and reproducible hardware and software benchmarking, auto-tuning and co-design)<br />
<br />
== Follow us<br/> ==<br />
<br />
*[https://twitter.com/c_tuning cTuning foundation twitter]<br />
*[https://www.facebook.com/pages/CrowdTuning-Foundation/668405119902805 cTuning foundation facebook page] (recent)<br />
*[http://dividiti.blogspot.com dividiti blog]<br />
*[http://c-mind.org/repo Collective Mind repository] (outdated)<br />
*[http://cknowledge.org/repo Collective Knowledge repository] (new)</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility&diff=845Reproducibility2015-10-26T10:00:07Z<p>Gfursin: </p>
<hr />
<div><center><br />
<span style="font-size:x-large"><span style="color: rgb(178, 34, 34)">Enabling collaborative and reproducible computer systems research with an open publication model</span></span><br />
</center><p style="text-align: center">[[File:Ae-stamp-ppopp2015.png|Ae-stamp-ppopp2015.png|link=http://cTuning.org/event/ae-ppopp2016]] [[File:Ae-stamp-cgo.png|Ae-stamp-cgo.png|link=http://cTuning.org/event/ae-cgo2016]] [[File:Logo-validated-by-the-community.png|Logo-validated-by-the-community.png|link=http://cTuning.org/reproducibility]] &nbsp; &nbsp;&nbsp; [[File:CTuning foundation logo1.png|cTuning_foundation_logo1.png|link=http://cTuning.org]]&nbsp; &nbsp;&nbsp; [[File:dividiti.png|dividiti.png|link=http://dividiti.com]]</p><p style="text-align: center"></p><p style="text-align: center">This wiki is maintained by the [http://cTuning.org non-profit Tuning foundation].</p><br />
== News and upcoming events<br/> ==<br />
<br />
*''Brand new, open-source, BSD-licensed Collective Knowledge SDK (cTuning 4 aka CK) for collaborative and reproducible R&D is now publicly available at [http://github.com/ctuning/ck GitHub] with an [http://github.com/ctuning/ck/wiki online documentation] and [http://cknowledge.org/repo live demo repository].''<br />
*''[http://adapt-workshop.org ADAPT'16 @ HiPEAC'16] - will feature [http://dl.acm.org/citation.cfm?id=2618142 our open publication model with community-driven reviewing, reddit-based discussions and artifact evaluation] ''. Paper submission deadline: '''9 October 2015'''<br />
*''[http://cTuning.org/event/ae-ppopp2016 PPoPP'16 artifact evaluation]''<br />
*''[http://cTuning.org/event/ae-cgo2016 CGO'16 artifact evaluation]''<br />
*''[http://www.dagstuhl.de/de/programm/kalender/semhp/?semnr=15452 Dagstuhl perspective workshop on artifact evaluation for confernces and journals]''<br />
<br />
== Motivation<br/> ==<br />
<br />
Since 2006 we have been trying to solve problems with reproducibility of experimental results in computer engineering ''as a side effect'' of our [http://cTuning.org/project-milepost MILEPOST] , [http://cTuning.org cTuning.org], [http://c-mind.org Collective Mind] and [http://github.com/ctuning/ck Collective Knowledge] projects (speeding up optimization, benchmarking and co-design of computer systems using auto-tuning, big data, predictive analytics and crowdsourcing). <span style="font-size: small">We focus on the following technological and social aspects to enable collaborative, systematic and reproducible research and experimentation particularly related to benchmarking, optimization and co-design of faster, smaller, cheaper, more power efficient and reliable software and hardware:</span><br />
<br />
*developing public and open source Collective Mind repositories of knowledge (see our pilot live repository [[http://cknowledge.org/repo CK], [http://c-mind.org/repo cMind]] and our vision papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]]);<br />
*developing [http://github.com/ctuning/ck collaborative research and experimentation infrastructure] that can share artifacts as reusable components together with the whole experimental setups (see our papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]];<br />
*evangelizing and enabling new open publication model for online workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*setting up and improving procedure for sharing and evaluating experimental results and all related material for workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*improving sharing, description of dependencies, and statistical reproducibility of experimental results and related material.<br />
<br />
See our manifesto and history [http://cTuning.org/history here].<br />
<br />
=== Community-driven research and developments<br/> ===<br />
<br />
Together with the community and [http://cTuning.org not-for-profit cTuning foundation] we are working on the following topics:<br />
<br />
*developing tools and methodology to capture, preserve, formalize, systematize, exchange and improve knowledge and experimental results including negative ones<br />
*describing and cataloging whole experimental setups with all related material including algorithms, benchmarks, codelets, datasets, tools, models and any other artifact<br />
*developing specification to preserve experiments including all software and hardware dependencies<br />
*dealing with variability and rising amount of experimental data using statistical analysis, data mining, predictive modeling and other techniques<br />
*developing new predictive analytics techniques to explore large design and optimization spaces<br />
*validating and verifying experimental results by the community<br />
*developing common research interfaces for existing or new tools<br />
*developing common experimental frameworks and repositories (enable automation, re-execution and sharing of experiments)<br />
*sharing rare hardware and computational resources for experimental validation<br />
*implementing previously published experimental scenarios (auto-tuning, run-time adaptation) using common infrastructure<br />
*implementing open access to publications and data (particularly discussing intellectual property IP and legal issues)<br />
*speeding up analysis of "big" experimental data<br />
*developing new (interactive) visualization techniques for "big" experimental data<br />
*enabling interactive articles<br />
<br />
== Our interdisciplinary events<br/> ==<br />
<br />
==== Featuring new open publication model and validation of experimental results<br/> ====<br />
<br />
*[[Reproducibility:AE:PPoPP2015|PPoPP'15 artifact evaluation]]<br />
*[[Reproducibility:AE:CGO2015|CGO'15 artifact evaluation]]<br />
*[http://adapt-workshop.org ADAPT'15] @ HiPEAC'15 - workshop on adaptive self-tuning computer systems<br />
*[http://adapt-workshop.org/2014 ADAPT'14] @ HiPEAC'14 - workshop on adaptive self-tuning computer systems [ [http://adapt-workshop.org/2014/program.htm program and publications] ]<br />
<br />
==== Discussing technical aspects to enable reproducibility and open publication model<br/> ====<br />
<br />
*[http://www.occamportal.org/images/reproduce/TETC-SI-REPRODUCE.pdf Special journal issue] on Reproducible &nbsp;Research &nbsp;Methodologies at IEEE TETC<br />
*[http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14] @ PLDI'14<br />
*[http://www.occamportal.org/reproduce REPRODUCE'14] @ HPCA'14<br />
*[http://www.adapt-workshop.org/2014/program.htm ADAPT'14 panel] @ HiPEAC'14<br />
*[http://ctuning.org/making-computer-engineering-a-science-2013 HiPEAC'13 CSW thematic session] @ ACM ECRC "Making computer engineering a science"<br />
*[http://ctuning.org/hipeac3-thematic-session-2012-04 HiPEAC'12 CSW thematic session]<br />
*[http://exadapt.org/2012/program.html ASPLOS/EXADAPT'12 panel] @ ASPLOS'12<br />
*[http://ctuning.org/lab/education cTuning lectures (2008-2010)]<br />
*[http://hal.inria.fr/inria-00436029 GCC Summit'09 discussion]<br />
<br />
== Acknowledgments<br/> ==<br />
<br />
We would like to thank our colleagues from the [http://cTuning.org/lab/people cTuning foundation], [https://www.linkedin.com/company/dividiti dividiti], [http://www.artifact-eval.org artifact-eval.org], [http://www.occamportal.org OCCAM project] for their help, feedback, participation and support.<br />
<br />
== Paper and artifact evaluation committee<br/> ==<br />
<br />
Rather than pre-selecting a dedicated committee for conferences, we select reviewers for reseach material (artifacts) and publications from a [http://cTuning.org/lab/people pool of our supporters] based on '''''submitted and publicly available''''' publications, their keywords and '''''public discussions''''' as described in our proposal [[http://arxiv.org/abs/1406.4020 arXiv]], [[http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]. Validated papers receive a stamp "Validated by the community". Artifacts can be shared along with publication in the ACM Digital LIbrary, HAL, Collective Mind Repository or any other public archive.<br />
<br />
As for the workshops, conferences and journals with the traditional publication model (CGO, PPoPP, PLDI), we select artifact evaluation committee (AEC) as described [[Reproducibility:AE|here]].<br />
<br />
== Packing and sharing research and experimental material<br/> ==<br />
<br />
Rather than enforcing specific procedure for packing, sharing and validation of experimental results, we allow authors of the accepted papers to include an archive with all related research material (using any [[Reproducibility:Links|publicly available tool]]) and ''readme.txt'' file describing how to validate their experiments. The main reason is the lack of a universally acceptable solution to pack and share experimental setups. For example, it is not always possible to use Virtual Machines and similar approaches for our research on performance/energy tuning or when some new hardware is being co-designed as we discuss in our proposal [[http://arxiv.org/abs/1406.4020 arXiv], [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]. Therefore, our current intention is to gradually and collaboratively find best procedure for packing using practical experience from our events such as [http://adapt-workshop.org ADAPT workshop] and from common discussions during [http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14 workshops]. See also nice guidelines for packing code and data along with publications [http://www.artifact-eval.org/guidelines.html here].<br />
<br />
== Validation<br/> ==<br />
<br />
After many years of evangelizing collaborative and reproducible research in computer engineering based on our practical experience, we finally start seeing some change in the mentality in academia, industry and funding agencies. Authors of two papers of our [http://adapt-workshop.org/2014 ADAPT'14 workshop] (out of nine accepted) agreed to have experimental results of their papers validated by volunteers. Note that rather than enforcing specific validation rules, we decided to ask authors to pack all their research artifacts as they wish (for example, using a shared virtual machine or as a standard archive) and describe their own validation procedure. Thanks to [http://cTuning.org/lab/people our volunteers], experiments from these papers have been validated, archives shared in [http://c-mind.org/repo our public repository] , and papers marked with a "validated by the community" stamp:<br />
<p style="text-align: center">[[File:Logo-validated-by-the-community.png|center|Logo-validated-by-the-community.png]]</p><br />
== Archive<br/> ==<br />
<br />
*[http://ctuning.org/wiki/index.php?title=Discussions:New_Publication_Model Outdated cTuning wiki page related to reproducible research and open publication model]<br />
*Outdated cTuning repository for program/processor performance/power/size optimization (2008-2010): [ [http://ctuning.org/wiki/index.php/CDatabase database], [http://ctuning.org/wiki/index.php?title=Special:CPredict web-service for online prediction of optimizations] ]<br />
<br />
== Resources<br/> ==<br />
<br />
*[[Reproducibility:Initiatives|Collection of related initiatives]]<br />
*[[Reproducibility:Links|Collection of related tools]]<br />
*[[Reproducibility:Datasets|Collection of related benchmarks and data sets]]<br />
*[[Reproducibility:Repositories|Collection of public repositories]]<br />
*[[Reproducibility:Lectures|Collection of related lectures]]<br />
*[[Reproducibility:Articles|Collection of related articles]]<br />
*[[Reproducibility:Blogs|Collection of related blogs]]<br />
*[[Reproducibility:Jokes|Collection of jokes]]<br />
*[[Reproducibility:Events|Collection of related events]]<br />
<br />
== Discussions<br/> ==<br />
<br />
*[http://www.linkedin.com/groups/Reproducible-research-experimentation-in-computer-7433414 LinkedIn group on reproducible research]<br />
*[http://groups.google.com/group/collective-knowledge Main mailing list] (general collaborative and reproducible R&D in computer engineering)<br />
*[http://groups.google.com/group/ctuning-discussions cTuning foundation mailing list] (collaborative and reproducible hardware and software benchmarking, auto-tuning and co-design)<br />
<br />
== Follow us<br/> ==<br />
<br />
*[https://twitter.com/c_tuning cTuning foundation twitter]<br />
*[https://www.facebook.com/pages/CrowdTuning-Foundation/668405119902805 cTuning foundation facebook page] (recent)<br />
*[http://dividiti.blogspot.com dividiti blog]<br />
*[http://c-mind.org/repo Collective Mind repository] (outdated)<br />
*[http://cknowledge.org/repo Collective Knowledge repository] (new)</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Links&diff=844Reproducibility:Links2015-10-26T09:34:11Z<p>Gfursin: </p>
<hr />
<div>[ '''''[[Reproducibility|Back to main page]]''''' ]<br />
<br />
=== '''Assorted (offline and online) tools'''<br/> ===<br />
<br />
*[https://www.docker.io Docker tool] - pack, ship and run applications as a lightweight container<br />
*[http://www.virtualbox.org VirtualBox] - create VM images<br />
*[http://www.occamportal.org OCCAM] - open curation for computer architecture modeling<br />
*[http://vida-nyu.github.io/reprozip ReproZip] - automatically packing experiments ([https://www.usenix.org/conference/tapp13/technical-sessions/presentation/chirigati related article])<br />
*[http://reproducible.io CARE tool] - Comprehensive Archiver for Reproducible Execution<br />
*[http://rr-project.org RR] - Mozilla project: records nondeterministic executions and debugs them deterministically<br />
*[http://www.pgbovine.net/cde.html CDE tool] - automatically create portable Linux applications with all dependencies<br />
*[http://ipython.org/notebook.html IPython Notebook] - a web-based interactive computational environment where you can combine code execution, text, mathematics, plots and rich media into a single document<br />
*[http://www.rstudio.com R-studio] - open source and enterprise-ready professional software for R<br />
*[https://www.codalab.org Codelab] - an experimental platform for collaboration and competition<br />
*[https://www.grid5000.fr Grid5000] - large-scale and versatile testbed for experiment-driven research in all areas of computer science, with a focus on parallel and distributed computing including Cloud, HPC and Big Data<br />
*[http://www.mygrid.org.uk MyGrid] - develops a suite of tools designed to "help e-Scientists get on with science and get on with scientists"<br />
*[http://figshare.com FigShare] - managing research in a cloud<br />
*[http://www.runmycode.org RunMyCode] - online workflows<br />
*[https://www.aptlab.net AptLab] - online workflows<br />
*[http://www.taverna.org.uk Taverna] - designing and executing workflows<br />
*[http://boinc.berkeley.edu BOINC] - open-source software for volunteer computing and grid computing<br />
*[https://mulcyber.toulouse.inra.fr/projects/ngspipelines NGS pipelines] - integrates pipelines and user interfaces to help biologists to analyse data outputed from biological applications such as RNAseq, sRNAseq, ChipSeq, BS-seq<br />
*[http://www.seek4science.org SEEK for Science] - finding, sharing and exchanging Data, Models, Simulations and Processes in Science<br />
*[http://www.cs.umd.edu/projects/skoll/Skoll/Home.html Skoll] - a process & Infrastructure for Distributed, continuous Quality assurance<br />
*[http://nepi.inria.fr NEPI] - simplifying network experimentation<br />
*[http://pgbovine.net/burrito.html Burrito] - rethinking the Electronic Lab Notebook<br />
*[http://orgmode.org Org mode] - keeping notes, maintaining TODO lists, planning projects, and authoring documents with a fast and effective plain-text system<br />
*[http://hal.inria.fr/inria-00436029 open source cTuning technology] - towards collaborative and reproducible performance autotuning via public repository of optimization knowledge, crowdsourcing, machine learning and collective intelligence (2006-cur.)<br />
<br />
=== '''Collective Knowledge Framework'''<br/> ===<br />
<br />
*[http://github.com/ctuning/ck Collective Knowledge Technology] - preserve (with distributed ID), organize, desribe, share and reuse your code and data via GitHub<br />
**[http://cknowledge.org/repo CK-based interactive article]<br />
**[http://github.com/ctuning/reproduce-clsmith PLDI'15 artifact converted to CK format]<br />
**[http://github.com/ctuning/reproduce-pamela-project Artifact repository for UK PAMELA project]<br />
**[http://github.com/ctuning/reproduce-carp-project Artifact repository for EU FP7 CARP project]<br />
**[http://github.com/ctuning/ctuning-programs Example of benchmarks shared in CK format]<br />
**[http://github.com/ctuning/ctuning-datasets-min Example of data sets shared in CK format]<br />
**[https://drive.google.com/folderview?id=0B-wXENVfIO82dzYwaUNIVElxaGc&usp=sharing Examples of CK repositories shared as zip files]</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Links&diff=843Reproducibility:Links2015-10-26T09:33:31Z<p>Gfursin: </p>
<hr />
<div>[ '''''[[Reproducibility|Back to main page]]''''' ]<br />
<br />
=== '''Assorted (offline and online) tools'''<br/> ===<br />
<br />
*[https://www.docker.io Docker tool] - pack, ship and run applications as a lightweight container<br />
*[http://www.virtualbox.org VirtualBox] - create VM images<br />
*[http://www.occamportal.org OCCAM] - open curation for computer architecture modeling<br />
*[http://vida-nyu.github.io/reprozip ReproZip] - automatically packing experiments ([https://www.usenix.org/conference/tapp13/technical-sessions/presentation/chirigati related article])<br />
*[http://reproducible.io CARE tool] - Comprehensive Archiver for Reproducible Execution<br />
*[http://rr-project.org RR] - Mozilla project: records nondeterministic executions and debugs them deterministically<br />
*[http://www.pgbovine.net/cde.html CDE tool] - automatically create portable Linux applications with all dependencies<br />
*[http://ipython.org/notebook.html IPython Notebook] - a web-based interactive computational environment where you can combine code execution, text, mathematics, plots and rich media into a single document<br />
*[http://www.rstudio.com R-studio] - open source and enterprise-ready professional software for R<br />
*[https://www.codalab.org Codelab] - an experimental platform for collaboration and competition<br />
*[https://www.grid5000.fr Grid5000] - large-scale and versatile testbed for experiment-driven research in all areas of computer science, with a focus on parallel and distributed computing including Cloud, HPC and Big Data<br />
*[http://www.mygrid.org.uk MyGrid] - develops a suite of tools designed to "help e-Scientists get on with science and get on with scientists"<br />
*[http://figshare.com FigShare] - managing research in a cloud<br />
*[http://www.runmycode.org RunMyCode] - online workflows<br />
*[https://www.aptlab.net AptLab] - online workflows<br />
*[http://www.taverna.org.uk Taverna] - designing and executing workflows<br />
*[http://boinc.berkeley.edu BOINC] - open-source software for volunteer computing and grid computing<br />
*[https://mulcyber.toulouse.inra.fr/projects/ngspipelines NGS pipelines] - integrates pipelines and user interfaces to help biologists to analyse data outputed from biological applications such as RNAseq, sRNAseq, ChipSeq, BS-seq<br />
*[http://www.seek4science.org SEEK for Science] - finding, sharing and exchanging Data, Models, Simulations and Processes in Science<br />
*[http://www.cs.umd.edu/projects/skoll/Skoll/Home.html Skoll] - a process & Infrastructure for Distributed, continuous Quality assurance<br />
*[http://nepi.inria.fr NEPI] - simplifying network experimentation<br />
*[http://pgbovine.net/burrito.html Burrito] - rethinking the Electronic Lab Notebook<br />
*[http://orgmode.org Org mode] - keeping notes, maintaining TODO lists, planning projects, and authoring documents with a fast and effective plain-text system<br />
*[http://hal.inria.fr/inria-00436029 open source cTuning technology] - towards collaborative and reproducible performance autotuning via public repository of optimization knowledge, crowdsourcing, machine learning and collective intelligence (2006-cur.)<br />
*[http://github.com/ctuning/ck Collective Knowledge Technology] - preserve (with distributed ID), organize, desribe, share and reuse your code and data via GitHub<br />
**[http://cknowledge.org/repo CK-based interactive article]<br />
**[http://github.com/ctuning/reproduce-clsmith PLDI'15 artifact converted to CK format]<br />
**[http://github.com/ctuning/reproduce-pamela-project Artifact repository for UK PAMELA project]<br />
**[http://github.com/ctuning/reproduce-carp-project Artifact repository for EU FP7 CARP project]<br />
**[http://github.com/ctuning/ctuning-programs Example of benchmarks shared in CK format]<br />
**[http://github.com/ctuning/ctuning-datasets-min Example of data sets shared in CK format]<br />
**[https://drive.google.com/folderview?id=0B-wXENVfIO82dzYwaUNIVElxaGc&usp=sharing Examples of CK repositories shared as zip files]</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Links&diff=842Reproducibility:Links2015-10-25T16:54:06Z<p>Gfursin: </p>
<hr />
<div>[ '''''[[Reproducibility|Back to main page]]''''' ]<br />
<br />
=== '''Assorted (offline and online) tools'''<br/> ===<br />
<br />
*[https://www.docker.io Docker tool] (pack, ship and run applications as a lightweight container)<br />
*[http://www.virtualbox.org] (create VM images)<br />
*[http://www.occamportal.org OCCAM] - open curation for computer architecture modeling<br />
*[http://vida-nyu.github.io/reprozip ReproZip] automatically packing experiments ([https://www.usenix.org/conference/tapp13/technical-sessions/presentation/chirigati related article])<br />
*[http://reproducible.io CARE tool from STMicroelectronics] (Comprehensive Archiver for Reproducible Execution)<br />
*[http://rr-project.org RR] (Mozilla project: records nondeterministic executions and debugs them deterministically)<br />
*[http://www.pgbovine.net/cde.html CDE tool] (automatically create portable Linux applications with all dependencies)<br />
*[http://ipython.org/notebook.html IPython Notebook] (a web-based interactive computational environment where you can combine code execution, text, mathematics, plots and rich media into a single document)<br />
*[http://www.rstudio.com R-studio] (Open source and enterprise-ready professional software for R)<br />
*[https://www.codalab.org Codelab] (an experimental platform for collaboration and competition)<br />
*[https://www.grid5000.fr Grid5000] large-scale and versatile testbed for experiment-driven research in all areas of computer science, with a focus on parallel and distributed computing including Cloud, HPC and Big Data<br />
*[http://www.mygrid.org.uk MyGrid] - develops a suite of tools designed to "help e-Scientists get on with science and get on with scientists"<br />
*[http://figshare.com FigShare] (managing research in a cloud)<br />
*[http://www.runmycode.org RunMyCode] (online workflows)<br />
*[https://www.aptlab.net AptLab] (online workflows)<br />
*[http://www.taverna.org.uk Taverna] (designing and executing workflows)<br />
*[http://boinc.berkeley.edu BOINC] (open-source software for volunteer computing and grid computing)<br />
*[https://mulcyber.toulouse.inra.fr/projects/ngspipelines NGS pipelines] (integrates pipelines and user interfaces to help biologists to analyse data outputed from biological applications such as RNAseq, sRNAseq, ChipSeq, BS-seq)<br />
*[http://www.seek4science.org SEEK for Science] - finding, sharing and exchanging Data, Models, Simulations and Processes in Science<br />
*[http://www.cs.umd.edu/projects/skoll/Skoll/Home.html Skoll] (A process & Infrastructure for Distributed, continuous Quality assurance)<br />
*[http://nepi.inria.fr NEPI] (Simplifying network experimentation)<br />
*[http://pgbovine.net/burrito.html Burrito] (Rethinking the Electronic Lab Notebook)<br/>[http://hal.inria.fr/inria-00436029 open source cTuning technology] (crowdsource auto-tuning and combine with machine learning using big data, predictive analytics and crowdsourcing) (2006-cur.)<br />
*[http://c-mind.org/repo Collective Mind technology] (towards collaborative, systematic and reproducible computer engineering using big data, predictive analytics and collective intelligence) (2011-2014)<br />
*[http://orgmode.org Org mode] - keeping notes, maintaining TODO lists, planning projects, and authoring documents with a fast and effective plain-text system<br />
*[http://github.com/ctuning/ck Collective Knowledge Technology] - preserve (with distributed ID), organize, desribe, share and reuse your code and data via GitHub<br />
**[http://cknowledge.org/repo CK-based interactive article]<br />
**[http://github.com/ctuning/reproduce-clsmith PLDI'15 artifact converted to CK format]<br />
**[http://github.com/ctuning/reproduce-pamela-project Artifact repository for UK PAMELA project]<br />
**[http://github.com/ctuning/reproduce-carp-project Artifact repository for EU FP7 CARP project]<br />
**[http://github.com/ctuning/ctuning-programs Example of benchmarks shared in CK format]<br />
**[http://github.com/ctuning/ctuning-datasets-min Example of data sets shared in CK format]<br />
**[https://drive.google.com/folderview?id=0B-wXENVfIO82dzYwaUNIVElxaGc&usp=sharing Examples of CK repositories shared as zip files]</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility&diff=841Reproducibility2015-10-25T16:51:43Z<p>Gfursin: /* Resources */</p>
<hr />
<div><center><br />
<span style="font-size:x-large"><span style="color: rgb(178, 34, 34)">Enabling collaborative, systematic and reproducible research and experimentation in computer engineering with an open publication model</span></span><br />
</center><p style="text-align: center">[[File:Ae-stamp-cgo.png|Ae-stamp-cgo.png|link=http://cTuning.org/event/ae-cgo2016]] [[File:Ae-stamp-ppopp2015.png|Ae-stamp-ppopp2015.png|link=http://cTuning.org/event/ae-ppopp2016]] [[File:Logo-validated-by-the-community.png|Logo-validated-by-the-community.png|link=http://cTuning.org/reproducibility]] &nbsp; &nbsp;&nbsp; [[File:CTuning foundation logo1.png|cTuning_foundation_logo1.png|link=http://cTuning.org]]&nbsp; &nbsp;&nbsp; [[File:dividiti.png|dividiti.png|link=http://dividiti.com]]</p><p style="text-align: center"></p><p style="text-align: center">This wiki is maintained by the [http://cTuning.org non-profit Tuning foundation].</p><br />
== News and upcoming events<br/> ==<br />
<br />
*''Brand new, open-source, BSD-licensed Collective Knowledge SDK (cTuning 4 aka CK) for collaborative and reproducible R&D is now publicly available at [http://github.com/ctuning/ck GitHub] with an [http://github.com/ctuning/ck/wiki online documentation] and [http://cknowledge.org/repo live demo repository].''<br />
*''[http://adapt-workshop.org ADAPT'16 @ HiPEAC'16] - will feature [http://dl.acm.org/citation.cfm?id=2618142 our open publication model with community-driven reviewing, reddit-based discussions and artifact evaluation] ''. Paper submission deadline: '''9 October 2015'''<br />
*''[http://cTuning.org/event/ae-ppopp2016 PPoPP'16 artifact evaluation]''<br />
*''[http://cTuning.org/event/ae-cgo2016 CGO'16 artifact evaluation]''<br />
*''[http://www.dagstuhl.de/de/programm/kalender/semhp/?semnr=15452 Dagstuhl perspective workshop on artifact evaluation for confernces and journals]''<br />
<br />
== Motivation<br/> ==<br />
<br />
Since 2006 we have been trying to solve problems with reproducibility of experimental results in computer engineering ''as a side effect'' of our [http://cTuning.org/project-milepost MILEPOST] , [http://cTuning.org cTuning.org], [http://c-mind.org Collective Mind] and [http://github.com/ctuning/ck Collective Knowledge] projects (speeding up optimization, benchmarking and co-design of computer systems using auto-tuning, big data, predictive analytics and crowdsourcing). <span style="font-size: small">We focus on the following technological and social aspects to enable collaborative, systematic and reproducible research and experimentation particularly related to benchmarking, optimization and co-design of faster, smaller, cheaper, more power efficient and reliable software and hardware:</span><br />
<br />
*developing public and open source Collective Mind repositories of knowledge (see our pilot live repository [[http://cknowledge.org/repo CK], [http://c-mind.org/repo cMind]] and our vision papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]]);<br />
*developing [http://github.com/ctuning/ck collaborative research and experimentation infrastructure] that can share artifacts as reusable components together with the whole experimental setups (see our papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]];<br />
*evangelizing and enabling new open publication model for online workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*setting up and improving procedure for sharing and evaluating experimental results and all related material for workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*improving sharing, description of dependencies, and statistical reproducibility of experimental results and related material.<br />
<br />
See our manifesto and history [http://cTuning.org/history here].<br />
<br />
=== Community-driven research and developments<br/> ===<br />
<br />
Together with the community and [http://cTuning.org not-for-profit cTuning foundation] we are working on the following topics:<br />
<br />
*developing tools and methodology to capture, preserve, formalize, systematize, exchange and improve knowledge and experimental results including negative ones<br />
*describing and cataloging whole experimental setups with all related material including algorithms, benchmarks, codelets, datasets, tools, models and any other artifact<br />
*developing specification to preserve experiments including all software and hardware dependencies<br />
*dealing with variability and rising amount of experimental data using statistical analysis, data mining, predictive modeling and other techniques<br />
*developing new predictive analytics techniques to explore large design and optimization spaces<br />
*validating and verifying experimental results by the community<br />
*developing common research interfaces for existing or new tools<br />
*developing common experimental frameworks and repositories (enable automation, re-execution and sharing of experiments)<br />
*sharing rare hardware and computational resources for experimental validation<br />
*implementing previously published experimental scenarios (auto-tuning, run-time adaptation) using common infrastructure<br />
*implementing open access to publications and data (particularly discussing intellectual property IP and legal issues)<br />
*speeding up analysis of "big" experimental data<br />
*developing new (interactive) visualization techniques for "big" experimental data<br />
*enabling interactive articles<br />
<br />
== Our interdisciplinary events<br/> ==<br />
<br />
==== Featuring new open publication model and validation of experimental results<br/> ====<br />
<br />
*[[Reproducibility:AE:PPoPP2015|PPoPP'15 artifact evaluation]]<br />
*[[Reproducibility:AE:CGO2015|CGO'15 artifact evaluation]]<br />
*[http://adapt-workshop.org ADAPT'15] @ HiPEAC'15 - workshop on adaptive self-tuning computer systems<br />
*[http://adapt-workshop.org/2014 ADAPT'14] @ HiPEAC'14 - workshop on adaptive self-tuning computer systems [ [http://adapt-workshop.org/2014/program.htm program and publications] ]<br />
<br />
==== Discussing technical aspects to enable reproducibility and open publication model<br/> ====<br />
<br />
*[http://www.occamportal.org/images/reproduce/TETC-SI-REPRODUCE.pdf Special journal issue] on Reproducible &nbsp;Research &nbsp;Methodologies at IEEE TETC<br />
*[http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14] @ PLDI'14<br />
*[http://www.occamportal.org/reproduce REPRODUCE'14] @ HPCA'14<br />
*[http://www.adapt-workshop.org/2014/program.htm ADAPT'14 panel] @ HiPEAC'14<br />
*[http://ctuning.org/making-computer-engineering-a-science-2013 HiPEAC'13 CSW thematic session] @ ACM ECRC "Making computer engineering a science"<br />
*[http://ctuning.org/hipeac3-thematic-session-2012-04 HiPEAC'12 CSW thematic session]<br />
*[http://exadapt.org/2012/program.html ASPLOS/EXADAPT'12 panel] @ ASPLOS'12<br />
*[http://ctuning.org/lab/education cTuning lectures (2008-2010)]<br />
*[http://hal.inria.fr/inria-00436029 GCC Summit'09 discussion]<br />
<br />
== Acknowledgments<br/> ==<br />
<br />
We would like to thank our colleagues from the [http://cTuning.org/lab/people cTuning foundation], [https://www.linkedin.com/company/dividiti dividiti], [http://www.artifact-eval.org artifact-eval.org], [http://www.occamportal.org OCCAM project] for their help, feedback, participation and support.<br />
<br />
== Paper and artifact evaluation committee<br/> ==<br />
<br />
Rather than pre-selecting a dedicated committee for conferences, we select reviewers for reseach material (artifacts) and publications from a [http://cTuning.org/lab/people pool of our supporters] based on '''''submitted and publicly available''''' publications, their keywords and '''''public discussions''''' as described in our proposal [[http://arxiv.org/abs/1406.4020 arXiv]], [[http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]. Validated papers receive a stamp "Validated by the community". Artifacts can be shared along with publication in the ACM Digital LIbrary, HAL, Collective Mind Repository or any other public archive.<br />
<br />
As for the workshops, conferences and journals with the traditional publication model (CGO, PPoPP, PLDI), we select artifact evaluation committee (AEC) as described [[Reproducibility:AE|here]].<br />
<br />
== Packing and sharing research and experimental material<br/> ==<br />
<br />
Rather than enforcing specific procedure for packing, sharing and validation of experimental results, we allow authors of the accepted papers to include an archive with all related research material (using any [[Reproducibility:Links|publicly available tool]]) and ''readme.txt'' file describing how to validate their experiments. The main reason is the lack of a universally acceptable solution to pack and share experimental setups. For example, it is not always possible to use Virtual Machines and similar approaches for our research on performance/energy tuning or when some new hardware is being co-designed as we discuss in our proposal [[http://arxiv.org/abs/1406.4020 arXiv], [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]. Therefore, our current intention is to gradually and collaboratively find best procedure for packing using practical experience from our events such as [http://adapt-workshop.org ADAPT workshop] and from common discussions during [http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14 workshops]. See also nice guidelines for packing code and data along with publications [http://www.artifact-eval.org/guidelines.html here].<br />
<br />
== Validation<br/> ==<br />
<br />
After many years of evangelizing collaborative and reproducible research in computer engineering based on our practical experience, we finally start seeing some change in the mentality in academia, industry and funding agencies. Authors of two papers of our [http://adapt-workshop.org/2014 ADAPT'14 workshop] (out of nine accepted) agreed to have experimental results of their papers validated by volunteers. Note that rather than enforcing specific validation rules, we decided to ask authors to pack all their research artifacts as they wish (for example, using a shared virtual machine or as a standard archive) and describe their own validation procedure. Thanks to [http://cTuning.org/lab/people our volunteers], experiments from these papers have been validated, archives shared in [http://c-mind.org/repo our public repository] , and papers marked with a "validated by the community" stamp:<br />
<p style="text-align: center">[[File:Logo-validated-by-the-community.png|center|Logo-validated-by-the-community.png]]</p><br />
== Archive<br/> ==<br />
<br />
*[http://ctuning.org/wiki/index.php?title=Discussions:New_Publication_Model Outdated cTuning wiki page related to reproducible research and open publication model]<br />
*Outdated cTuning repository for program/processor performance/power/size optimization (2008-2010): [ [http://ctuning.org/wiki/index.php/CDatabase database], [http://ctuning.org/wiki/index.php?title=Special:CPredict web-service for online prediction of optimizations] ]<br />
<br />
== Resources<br/> ==<br />
<br />
*[[Reproducibility:Initiatives|Collection of related initiatives]]<br />
*[[Reproducibility:Links|Collection of related tools]]<br />
*[[Reproducibility:Datasets|Collection of related benchmarks and data sets]]<br />
*[[Reproducibility:Repositories|Collection of public repositories]]<br />
*[[Reproducibility:Lectures|Collection of related lectures]]<br />
*[[Reproducibility:Articles|Collection of related articles]]<br />
*[[Reproducibility:Blogs|Collection of related blogs]]<br />
*[[Reproducibility:Jokes|Collection of jokes]]<br />
*[[Reproducibility:Events|Collection of related events]]<br />
<br />
== Discussions<br/> ==<br />
<br />
*[http://www.linkedin.com/groups/Reproducible-research-experimentation-in-computer-7433414 LinkedIn group on reproducible research]<br />
*[http://groups.google.com/group/collective-knowledge Main mailing list] (general collaborative and reproducible R&D in computer engineering)<br />
*[http://groups.google.com/group/ctuning-discussions cTuning foundation mailing list] (collaborative and reproducible hardware and software benchmarking, auto-tuning and co-design)<br />
<br />
== Follow us<br/> ==<br />
<br />
*[https://twitter.com/c_tuning cTuning foundation twitter]<br />
*[https://www.facebook.com/pages/CrowdTuning-Foundation/668405119902805 cTuning foundation facebook page] (recent)<br />
*[http://dividiti.blogspot.com dividiti blog]<br />
*[http://c-mind.org/repo Collective Mind repository] (outdated)<br />
*[http://cknowledge.org/repo Collective Knowledge repository] (new)</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Jokes&diff=840Reproducibility:Jokes2015-10-11T13:03:51Z<p>Gfursin: Created page with "[ '''''Back to main page''''' ] === '''Assorted jokes''<br/> === *[http://matt.might.net/articles/crapl CRAPL license] *[https://play.google.com/store/app..."</p>
<hr />
<div>[ '''''[[Reproducibility|Back to main page]]''''' ]<br />
<br />
=== '''Assorted jokes''<br/> ===<br />
*[http://matt.might.net/articles/crapl CRAPL license]<br />
*[https://play.google.com/store/apps/details?id=ctuning.com.researchpapertitlegenerator&hl=en Paper title generator (Android app)]</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility&diff=839Reproducibility2015-10-11T13:02:12Z<p>Gfursin: /* Resources */</p>
<hr />
<div><center><br />
<span style="font-size:x-large"><span style="color: rgb(178, 34, 34)">Enabling collaborative, systematic and reproducible research and experimentation in computer engineering with an open publication model</span></span><br />
</center><p style="text-align: center">[[File:Ae-stamp-cgo.png|Ae-stamp-cgo.png|link=http://cTuning.org/event/ae-cgo2016]] [[File:Ae-stamp-ppopp2015.png|Ae-stamp-ppopp2015.png|link=http://cTuning.org/event/ae-ppopp2016]] [[File:Logo-validated-by-the-community.png|Logo-validated-by-the-community.png|link=http://cTuning.org/reproducibility]] &nbsp; &nbsp;&nbsp; [[File:CTuning foundation logo1.png|cTuning_foundation_logo1.png|link=http://cTuning.org]]&nbsp; &nbsp;&nbsp; [[File:dividiti.png|dividiti.png|link=http://dividiti.com]]</p><p style="text-align: center"></p><p style="text-align: center">This wiki is maintained by the [http://cTuning.org non-profit Tuning foundation].</p><br />
== News and upcoming events<br/> ==<br />
<br />
*''Brand new, open-source, BSD-licensed Collective Knowledge SDK (cTuning 4 aka CK) for collaborative and reproducible R&D is now publicly available at [http://github.com/ctuning/ck GitHub] with an [http://github.com/ctuning/ck/wiki online documentation] and [http://cknowledge.org/repo live demo repository].''<br />
*''[http://adapt-workshop.org ADAPT'16 @ HiPEAC'16] - will feature [http://dl.acm.org/citation.cfm?id=2618142 our open publication model with community-driven reviewing, reddit-based discussions and artifact evaluation] ''. Paper submission deadline: '''9 October 2015'''<br />
*''[http://cTuning.org/event/ae-ppopp2016 PPoPP'16 artifact evaluation]''<br />
*''[http://cTuning.org/event/ae-cgo2016 CGO'16 artifact evaluation]''<br />
*''[http://www.dagstuhl.de/de/programm/kalender/semhp/?semnr=15452 Dagstuhl perspective workshop on artifact evaluation for confernces and journals]''<br />
<br />
== Motivation<br/> ==<br />
<br />
Since 2006 we have been trying to solve problems with reproducibility of experimental results in computer engineering ''as a side effect'' of our [http://cTuning.org/project-milepost MILEPOST] , [http://cTuning.org cTuning.org], [http://c-mind.org Collective Mind] and [http://github.com/ctuning/ck Collective Knowledge] projects (speeding up optimization, benchmarking and co-design of computer systems using auto-tuning, big data, predictive analytics and crowdsourcing). <span style="font-size: small">We focus on the following technological and social aspects to enable collaborative, systematic and reproducible research and experimentation particularly related to benchmarking, optimization and co-design of faster, smaller, cheaper, more power efficient and reliable software and hardware:</span><br />
<br />
*developing public and open source Collective Mind repositories of knowledge (see our pilot live repository [[http://cknowledge.org/repo CK], [http://c-mind.org/repo cMind]] and our vision papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]]);<br />
*developing [http://github.com/ctuning/ck collaborative research and experimentation infrastructure] that can share artifacts as reusable components together with the whole experimental setups (see our papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]];<br />
*evangelizing and enabling new open publication model for online workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*setting up and improving procedure for sharing and evaluating experimental results and all related material for workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*improving sharing, description of dependencies, and statistical reproducibility of experimental results and related material.<br />
<br />
See our manifesto and history [http://cTuning.org/history here].<br />
<br />
=== Community-driven research and developments<br/> ===<br />
<br />
Together with the community and [http://cTuning.org not-for-profit cTuning foundation] we are working on the following topics:<br />
<br />
*developing tools and methodology to capture, preserve, formalize, systematize, exchange and improve knowledge and experimental results including negative ones<br />
*describing and cataloging whole experimental setups with all related material including algorithms, benchmarks, codelets, datasets, tools, models and any other artifact<br />
*developing specification to preserve experiments including all software and hardware dependencies<br />
*dealing with variability and rising amount of experimental data using statistical analysis, data mining, predictive modeling and other techniques<br />
*developing new predictive analytics techniques to explore large design and optimization spaces<br />
*validating and verifying experimental results by the community<br />
*developing common research interfaces for existing or new tools<br />
*developing common experimental frameworks and repositories (enable automation, re-execution and sharing of experiments)<br />
*sharing rare hardware and computational resources for experimental validation<br />
*implementing previously published experimental scenarios (auto-tuning, run-time adaptation) using common infrastructure<br />
*implementing open access to publications and data (particularly discussing intellectual property IP and legal issues)<br />
*speeding up analysis of "big" experimental data<br />
*developing new (interactive) visualization techniques for "big" experimental data<br />
*enabling interactive articles<br />
<br />
== Our interdisciplinary events<br/> ==<br />
<br />
==== Featuring new open publication model and validation of experimental results<br/> ====<br />
<br />
*[[Reproducibility:AE:PPoPP2015|PPoPP'15 artifact evaluation]]<br />
*[[Reproducibility:AE:CGO2015|CGO'15 artifact evaluation]]<br />
*[http://adapt-workshop.org ADAPT'15] @ HiPEAC'15 - workshop on adaptive self-tuning computer systems<br />
*[http://adapt-workshop.org/2014 ADAPT'14] @ HiPEAC'14 - workshop on adaptive self-tuning computer systems [ [http://adapt-workshop.org/2014/program.htm program and publications] ]<br />
<br />
==== Discussing technical aspects to enable reproducibility and open publication model<br/> ====<br />
<br />
*[http://www.occamportal.org/images/reproduce/TETC-SI-REPRODUCE.pdf Special journal issue] on Reproducible &nbsp;Research &nbsp;Methodologies at IEEE TETC<br />
*[http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14] @ PLDI'14<br />
*[http://www.occamportal.org/reproduce REPRODUCE'14] @ HPCA'14<br />
*[http://www.adapt-workshop.org/2014/program.htm ADAPT'14 panel] @ HiPEAC'14<br />
*[http://ctuning.org/making-computer-engineering-a-science-2013 HiPEAC'13 CSW thematic session] @ ACM ECRC "Making computer engineering a science"<br />
*[http://ctuning.org/hipeac3-thematic-session-2012-04 HiPEAC'12 CSW thematic session]<br />
*[http://exadapt.org/2012/program.html ASPLOS/EXADAPT'12 panel] @ ASPLOS'12<br />
*[http://ctuning.org/lab/education cTuning lectures (2008-2010)]<br />
*[http://hal.inria.fr/inria-00436029 GCC Summit'09 discussion]<br />
<br />
== Acknowledgments<br/> ==<br />
<br />
We would like to thank our colleagues from the [http://cTuning.org/lab/people cTuning foundation], [https://www.linkedin.com/company/dividiti dividiti], [http://www.artifact-eval.org artifact-eval.org], [http://www.occamportal.org OCCAM project] for their help, feedback, participation and support.<br />
<br />
== Paper and artifact evaluation committee<br/> ==<br />
<br />
Rather than pre-selecting a dedicated committee for conferences, we select reviewers for reseach material (artifacts) and publications from a [http://cTuning.org/lab/people pool of our supporters] based on '''''submitted and publicly available''''' publications, their keywords and '''''public discussions''''' as described in our proposal [[http://arxiv.org/abs/1406.4020 arXiv]], [[http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]. Validated papers receive a stamp "Validated by the community". Artifacts can be shared along with publication in the ACM Digital LIbrary, HAL, Collective Mind Repository or any other public archive.<br />
<br />
As for the workshops, conferences and journals with the traditional publication model (CGO, PPoPP, PLDI), we select artifact evaluation committee (AEC) as described [[Reproducibility:AE|here]].<br />
<br />
== Packing and sharing research and experimental material<br/> ==<br />
<br />
Rather than enforcing specific procedure for packing, sharing and validation of experimental results, we allow authors of the accepted papers to include an archive with all related research material (using any [[Reproducibility:Links|publicly available tool]]) and ''readme.txt'' file describing how to validate their experiments. The main reason is the lack of a universally acceptable solution to pack and share experimental setups. For example, it is not always possible to use Virtual Machines and similar approaches for our research on performance/energy tuning or when some new hardware is being co-designed as we discuss in our proposal [[http://arxiv.org/abs/1406.4020 arXiv], [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]. Therefore, our current intention is to gradually and collaboratively find best procedure for packing using practical experience from our events such as [http://adapt-workshop.org ADAPT workshop] and from common discussions during [http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14 workshops]. See also nice guidelines for packing code and data along with publications [http://www.artifact-eval.org/guidelines.html here].<br />
<br />
== Validation<br/> ==<br />
<br />
After many years of evangelizing collaborative and reproducible research in computer engineering based on our practical experience, we finally start seeing some change in the mentality in academia, industry and funding agencies. Authors of two papers of our [http://adapt-workshop.org/2014 ADAPT'14 workshop] (out of nine accepted) agreed to have experimental results of their papers validated by volunteers. Note that rather than enforcing specific validation rules, we decided to ask authors to pack all their research artifacts as they wish (for example, using a shared virtual machine or as a standard archive) and describe their own validation procedure. Thanks to [http://cTuning.org/lab/people our volunteers], experiments from these papers have been validated, archives shared in [http://c-mind.org/repo our public repository] , and papers marked with a "validated by the community" stamp:<br />
<p style="text-align: center">[[File:Logo-validated-by-the-community.png|center|Logo-validated-by-the-community.png]]</p><br />
== Archive<br/> ==<br />
<br />
*[http://ctuning.org/wiki/index.php?title=Discussions:New_Publication_Model Outdated cTuning wiki page related to reproducible research and open publication model]<br />
*Outdated cTuning repository for program/processor performance/power/size optimization (2008-2010): [ [http://ctuning.org/wiki/index.php/CDatabase database], [http://ctuning.org/wiki/index.php?title=Special:CPredict web-service for online prediction of optimizations] ]<br />
<br />
== Resources<br/> ==<br />
<br />
*[[Reproducibility:Jokes|Collection of jokes]]<br />
*[[Reproducibility:Initiatives|Collection of related initiatives]]<br />
*[[Reproducibility:Links|Collection of related tools]]<br />
*[[Reproducibility:Datasets|Collection of related benchmarks and data sets]]<br />
*[[Reproducibility:Repositories|Collection of public repositories]]<br />
*[[Reproducibility:Lectures|Collection of related lectures]]<br />
*[[Reproducibility:Articles|Collection of related articles]]<br />
*[[Reproducibility:Blogs|Collection of related blogs]]<br />
*[[Reproducibility:Events|Collection of related events]]<br />
<br />
== Discussions<br/> ==<br />
<br />
*[http://www.linkedin.com/groups/Reproducible-research-experimentation-in-computer-7433414 LinkedIn group on reproducible research]<br />
*[http://groups.google.com/group/collective-knowledge Main mailing list] (general collaborative and reproducible R&D in computer engineering)<br />
*[http://groups.google.com/group/ctuning-discussions cTuning foundation mailing list] (collaborative and reproducible hardware and software benchmarking, auto-tuning and co-design)<br />
<br />
== Follow us<br/> ==<br />
<br />
*[https://twitter.com/c_tuning cTuning foundation twitter]<br />
*[https://www.facebook.com/pages/CrowdTuning-Foundation/668405119902805 cTuning foundation facebook page] (recent)<br />
*[http://dividiti.blogspot.com dividiti blog]<br />
*[http://c-mind.org/repo Collective Mind repository] (outdated)<br />
*[http://cknowledge.org/repo Collective Knowledge repository] (new)</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility&diff=838Reproducibility2015-10-11T13:01:45Z<p>Gfursin: /* Resources */</p>
<hr />
<div><center><br />
<span style="font-size:x-large"><span style="color: rgb(178, 34, 34)">Enabling collaborative, systematic and reproducible research and experimentation in computer engineering with an open publication model</span></span><br />
</center><p style="text-align: center">[[File:Ae-stamp-cgo.png|Ae-stamp-cgo.png|link=http://cTuning.org/event/ae-cgo2016]] [[File:Ae-stamp-ppopp2015.png|Ae-stamp-ppopp2015.png|link=http://cTuning.org/event/ae-ppopp2016]] [[File:Logo-validated-by-the-community.png|Logo-validated-by-the-community.png|link=http://cTuning.org/reproducibility]] &nbsp; &nbsp;&nbsp; [[File:CTuning foundation logo1.png|cTuning_foundation_logo1.png|link=http://cTuning.org]]&nbsp; &nbsp;&nbsp; [[File:dividiti.png|dividiti.png|link=http://dividiti.com]]</p><p style="text-align: center"></p><p style="text-align: center">This wiki is maintained by the [http://cTuning.org non-profit Tuning foundation].</p><br />
== News and upcoming events<br/> ==<br />
<br />
*''Brand new, open-source, BSD-licensed Collective Knowledge SDK (cTuning 4 aka CK) for collaborative and reproducible R&D is now publicly available at [http://github.com/ctuning/ck GitHub] with an [http://github.com/ctuning/ck/wiki online documentation] and [http://cknowledge.org/repo live demo repository].''<br />
*''[http://adapt-workshop.org ADAPT'16 @ HiPEAC'16] - will feature [http://dl.acm.org/citation.cfm?id=2618142 our open publication model with community-driven reviewing, reddit-based discussions and artifact evaluation] ''. Paper submission deadline: '''9 October 2015'''<br />
*''[http://cTuning.org/event/ae-ppopp2016 PPoPP'16 artifact evaluation]''<br />
*''[http://cTuning.org/event/ae-cgo2016 CGO'16 artifact evaluation]''<br />
*''[http://www.dagstuhl.de/de/programm/kalender/semhp/?semnr=15452 Dagstuhl perspective workshop on artifact evaluation for confernces and journals]''<br />
<br />
== Motivation<br/> ==<br />
<br />
Since 2006 we have been trying to solve problems with reproducibility of experimental results in computer engineering ''as a side effect'' of our [http://cTuning.org/project-milepost MILEPOST] , [http://cTuning.org cTuning.org], [http://c-mind.org Collective Mind] and [http://github.com/ctuning/ck Collective Knowledge] projects (speeding up optimization, benchmarking and co-design of computer systems using auto-tuning, big data, predictive analytics and crowdsourcing). <span style="font-size: small">We focus on the following technological and social aspects to enable collaborative, systematic and reproducible research and experimentation particularly related to benchmarking, optimization and co-design of faster, smaller, cheaper, more power efficient and reliable software and hardware:</span><br />
<br />
*developing public and open source Collective Mind repositories of knowledge (see our pilot live repository [[http://cknowledge.org/repo CK], [http://c-mind.org/repo cMind]] and our vision papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]]);<br />
*developing [http://github.com/ctuning/ck collaborative research and experimentation infrastructure] that can share artifacts as reusable components together with the whole experimental setups (see our papers [[http://arxiv.org/abs/1506.06256 1],[http://hal.inria.fr/hal-01054763 2]];<br />
*evangelizing and enabling new open publication model for online workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*setting up and improving procedure for sharing and evaluating experimental results and all related material for workshops, conferences and journals (see our proposal [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]);<br />
*improving sharing, description of dependencies, and statistical reproducibility of experimental results and related material.<br />
<br />
See our manifesto and history [http://cTuning.org/history here].<br />
<br />
=== Community-driven research and developments<br/> ===<br />
<br />
Together with the community and [http://cTuning.org not-for-profit cTuning foundation] we are working on the following topics:<br />
<br />
*developing tools and methodology to capture, preserve, formalize, systematize, exchange and improve knowledge and experimental results including negative ones<br />
*describing and cataloging whole experimental setups with all related material including algorithms, benchmarks, codelets, datasets, tools, models and any other artifact<br />
*developing specification to preserve experiments including all software and hardware dependencies<br />
*dealing with variability and rising amount of experimental data using statistical analysis, data mining, predictive modeling and other techniques<br />
*developing new predictive analytics techniques to explore large design and optimization spaces<br />
*validating and verifying experimental results by the community<br />
*developing common research interfaces for existing or new tools<br />
*developing common experimental frameworks and repositories (enable automation, re-execution and sharing of experiments)<br />
*sharing rare hardware and computational resources for experimental validation<br />
*implementing previously published experimental scenarios (auto-tuning, run-time adaptation) using common infrastructure<br />
*implementing open access to publications and data (particularly discussing intellectual property IP and legal issues)<br />
*speeding up analysis of "big" experimental data<br />
*developing new (interactive) visualization techniques for "big" experimental data<br />
*enabling interactive articles<br />
<br />
== Our interdisciplinary events<br/> ==<br />
<br />
==== Featuring new open publication model and validation of experimental results<br/> ====<br />
<br />
*[[Reproducibility:AE:PPoPP2015|PPoPP'15 artifact evaluation]]<br />
*[[Reproducibility:AE:CGO2015|CGO'15 artifact evaluation]]<br />
*[http://adapt-workshop.org ADAPT'15] @ HiPEAC'15 - workshop on adaptive self-tuning computer systems<br />
*[http://adapt-workshop.org/2014 ADAPT'14] @ HiPEAC'14 - workshop on adaptive self-tuning computer systems [ [http://adapt-workshop.org/2014/program.htm program and publications] ]<br />
<br />
==== Discussing technical aspects to enable reproducibility and open publication model<br/> ====<br />
<br />
*[http://www.occamportal.org/images/reproduce/TETC-SI-REPRODUCE.pdf Special journal issue] on Reproducible &nbsp;Research &nbsp;Methodologies at IEEE TETC<br />
*[http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14] @ PLDI'14<br />
*[http://www.occamportal.org/reproduce REPRODUCE'14] @ HPCA'14<br />
*[http://www.adapt-workshop.org/2014/program.htm ADAPT'14 panel] @ HiPEAC'14<br />
*[http://ctuning.org/making-computer-engineering-a-science-2013 HiPEAC'13 CSW thematic session] @ ACM ECRC "Making computer engineering a science"<br />
*[http://ctuning.org/hipeac3-thematic-session-2012-04 HiPEAC'12 CSW thematic session]<br />
*[http://exadapt.org/2012/program.html ASPLOS/EXADAPT'12 panel] @ ASPLOS'12<br />
*[http://ctuning.org/lab/education cTuning lectures (2008-2010)]<br />
*[http://hal.inria.fr/inria-00436029 GCC Summit'09 discussion]<br />
<br />
== Acknowledgments<br/> ==<br />
<br />
We would like to thank our colleagues from the [http://cTuning.org/lab/people cTuning foundation], [https://www.linkedin.com/company/dividiti dividiti], [http://www.artifact-eval.org artifact-eval.org], [http://www.occamportal.org OCCAM project] for their help, feedback, participation and support.<br />
<br />
== Paper and artifact evaluation committee<br/> ==<br />
<br />
Rather than pre-selecting a dedicated committee for conferences, we select reviewers for reseach material (artifacts) and publications from a [http://cTuning.org/lab/people pool of our supporters] based on '''''submitted and publicly available''''' publications, their keywords and '''''public discussions''''' as described in our proposal [[http://arxiv.org/abs/1406.4020 arXiv]], [[http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]. Validated papers receive a stamp "Validated by the community". Artifacts can be shared along with publication in the ACM Digital LIbrary, HAL, Collective Mind Repository or any other public archive.<br />
<br />
As for the workshops, conferences and journals with the traditional publication model (CGO, PPoPP, PLDI), we select artifact evaluation committee (AEC) as described [[Reproducibility:AE|here]].<br />
<br />
== Packing and sharing research and experimental material<br/> ==<br />
<br />
Rather than enforcing specific procedure for packing, sharing and validation of experimental results, we allow authors of the accepted papers to include an archive with all related research material (using any [[Reproducibility:Links|publicly available tool]]) and ''readme.txt'' file describing how to validate their experiments. The main reason is the lack of a universally acceptable solution to pack and share experimental setups. For example, it is not always possible to use Virtual Machines and similar approaches for our research on performance/energy tuning or when some new hardware is being co-designed as we discuss in our proposal [[http://arxiv.org/abs/1406.4020 arXiv], [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]. Therefore, our current intention is to gradually and collaboratively find best procedure for packing using practical experience from our events such as [http://adapt-workshop.org ADAPT workshop] and from common discussions during [http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14 workshops]. See also nice guidelines for packing code and data along with publications [http://www.artifact-eval.org/guidelines.html here].<br />
<br />
== Validation<br/> ==<br />
<br />
After many years of evangelizing collaborative and reproducible research in computer engineering based on our practical experience, we finally start seeing some change in the mentality in academia, industry and funding agencies. Authors of two papers of our [http://adapt-workshop.org/2014 ADAPT'14 workshop] (out of nine accepted) agreed to have experimental results of their papers validated by volunteers. Note that rather than enforcing specific validation rules, we decided to ask authors to pack all their research artifacts as they wish (for example, using a shared virtual machine or as a standard archive) and describe their own validation procedure. Thanks to [http://cTuning.org/lab/people our volunteers], experiments from these papers have been validated, archives shared in [http://c-mind.org/repo our public repository] , and papers marked with a "validated by the community" stamp:<br />
<p style="text-align: center">[[File:Logo-validated-by-the-community.png|center|Logo-validated-by-the-community.png]]</p><br />
== Archive<br/> ==<br />
<br />
*[http://ctuning.org/wiki/index.php?title=Discussions:New_Publication_Model Outdated cTuning wiki page related to reproducible research and open publication model]<br />
*Outdated cTuning repository for program/processor performance/power/size optimization (2008-2010): [ [http://ctuning.org/wiki/index.php/CDatabase database], [http://ctuning.org/wiki/index.php?title=Special:CPredict web-service for online prediction of optimizations] ]<br />
<br />
== Resources<br/> ==<br />
<br />
*[[Reproducibility:Jokes|Jokes]]<br />
*[[Reproducibility:Initiatives|Collection of related initiatives]]<br />
*[[Reproducibility:Links|Collection of related tools]]<br />
*[[Reproducibility:Datasets|Collection of related benchmarks and data sets]]<br />
*[[Reproducibility:Repositories|Collection of public repositories]]<br />
*[[Reproducibility:Lectures|Collection of related lectures]]<br />
*[[Reproducibility:Articles|Collection of related articles]]<br />
*[[Reproducibility:Blogs|Collection of related blogs]]<br />
*[[Reproducibility:Events|Collection of related events]]<br />
<br />
== Discussions<br/> ==<br />
<br />
*[http://www.linkedin.com/groups/Reproducible-research-experimentation-in-computer-7433414 LinkedIn group on reproducible research]<br />
*[http://groups.google.com/group/collective-knowledge Main mailing list] (general collaborative and reproducible R&D in computer engineering)<br />
*[http://groups.google.com/group/ctuning-discussions cTuning foundation mailing list] (collaborative and reproducible hardware and software benchmarking, auto-tuning and co-design)<br />
<br />
== Follow us<br/> ==<br />
<br />
*[https://twitter.com/c_tuning cTuning foundation twitter]<br />
*[https://www.facebook.com/pages/CrowdTuning-Foundation/668405119902805 cTuning foundation facebook page] (recent)<br />
*[http://dividiti.blogspot.com dividiti blog]<br />
*[http://c-mind.org/repo Collective Mind repository] (outdated)<br />
*[http://cknowledge.org/repo Collective Knowledge repository] (new)</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Articles&diff=837Reproducibility:Articles2015-10-01T11:23:33Z<p>Gfursin: /* Assorted articles and presentations */</p>
<hr />
<div>[ '''''[[Reproducibility|Back to main page]]''''' ]<br />
<br />
=== '''Assorted articles and presentations'''<br/> ===<br />
<br />
*[http://www.pl-enthusiast.net/2015/09/01/pl-conference-papers-to-get-a-journal PL conference papers to get a journal] - discussing ACM proposal to formally recognize conference publications as equal in quality as journal publications - controversial<br />
*Our experience report with encountered problems in computer systems' research and new proposal for ''community-driven reviewing and validation of publications'' [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]<br />
*[https://hal.inria.fr/inria-00436029 Collective Tuning Initiative: automating and accelerating development and optimization of computing systems]<br />
*[http://www.the-scientist.com/?articles.view/articleNo/33719/title/Science-s-Reproducibility-Problem Science's Reproducibility Problem], Bob Grant, The Scientist<br />
*[http://www.cs.huji.ac.il/~feit/papers/Repeat15SIGOPS.pdf From Repeatability to Reproducibility and Corroboration], Dror G. Feitelson, SIGOPS Operating Syst. Rev. 49(1), pp. 3-11, Jan 2015<br />
*[http://cacm.acm.org/blogs/blog-cacm/98560 Reviewing peer review], Jeannette M. Wing, ACM Blog<br />
*[http://cacm.acm.org/blogs/blog-cacm/100284 How Should Peer Review Evolve?], Ed H. Chi, ACM Blog<br />
*[http://crd.lbl.gov/~dhbailey/dhbpapers/twelve-ways.pdf Twelve Ways to Fool the Masses When Giving Performance Results on Parallel Computers], David H. Bailey, Supercomputing Review, June 11, 1991<br />
*[Ten Ways to Fool the Masses When Giving Performance Results on GPUs], Scott Pakin, HPCWire, December 13, 2011 <br />
*[http://www.cam.ac.uk/research/news/new-gold-standard-established-for-open-and-reproducible-research Cambridge University news about open and reproducible research]<br />
*[http://reproduciblescience.blogspot.fr/2015/06/interesting-replicable-badge-for.html Interesting Replicable "Badge" for journal articles], Daniel S. Katz<br />
*[http://thomasleeper.com/2015/05/open-science-language What's in a Name? The Concepts and Language of Replication and Reproducibility]<br />
*[http://web.eece.maine.edu/~vweaver/papers/wddd08/wddd08_workshop.pdf Are Cycle Accurate Simulations a Waste of Time?], Vincent M. Weaver and Sally A. McKee<br />
*[http://ehp.niehs.nih.gov/122-a188 Research Wranglers: Initiatives to Improve Reproducibility of Study Findings] (Environmental Health Perspectives)<br />
*[https://web.stanford.edu/~vcs/talks/VictoriaStoddenICML2010.pdf Reproducible Research in Computational Science: Problems and Solutions For Data and Code Sharing], Victoria Stodden, ICML workshop 2010<br />
*[http://www.the-scientist.com/?articles.view/articleNo/32426/title/Predatory-Publishing Predatory publishing]<br />
*[http://michaelnielsen.org/blog/three-myths-about-scientific-peer-review Three myths about scientific peer review]<br />
*[http://spyros.zoupanos.net/papers/p39.open-repeatability.pdf The repeatability experiment of SIGMOD 2008], SIGMOD Record, vol. 37, no. 1 (March 2008), 39-45. <br />
*[http://dl.acm.org/citation.cfm?id=1536616.1536619 "Begin with an Author's response to Reviews"] - proposal to submit past reviews along with articles<br />
*[https://www.dropbox.com/s/67n49i9k3cb28mo/GOBLE-CW2014.ppt Presentation by Professor Carole Goble]<br />
*Dennis McCafferty, [http://dl.acm.org/citation.cfm?id=1831415 "Should Code be Released?"]<br />
*[http://www.elsevier.com/connect/can-data-be-peer-reviewed Elsevier:Can data be peer-reviewed?]<br />
*[https://hal.inria.fr/hal-01054763 Collective mind: Towards practical and collaborative auto-tuning]<br />
*[http://arxiv.org/abs/1506.06256 Collective Mind, Part II: Towards Performance- and Cost-Aware Software Engineering as a Natural Science]<br />
*[http://www.executablepapers.com Elsevier:The Executable Paper Grand Challenge]<br />
*[http://www.sciencemag.org/content/334/6060/1225.full Special section on Data replication & reproducibility], Science magazine, 2 December 2011<br />
*Chris Drummond, [http://cogprints.org/7691/7/ICMLws09.pdf "Replicability is not Reproducibility: Nor is it Good Science"]<br />
*[http://theconversation.com/science-is-in-a-reproducibility-crisis-how-do-we-resolve-it-16998 Science is in a reproducibility crisis - how do we resolve it?]<br />
*[http://www.w3.org/2001/sw/hcls/notes/hcls-dataset W3C Dataset Descriptions: HCLS Community Profile]<br />
*My blog article on [http://www.software.ac.uk/blog/2014-07-22-automatic-performance-tuning-and-reproducibility-side-effect "Automatic performance tuning and reproducibility as a side effect"]&nbsp; for the Software Sustainability Institute<br />
*[http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble Trouble at the Lab], The Economist, 2013<br />
*[https://www.force11.org/group/joint-declaration-data-citation-principles-final Joint Declaration of Data Citation Principles]<br />
*[http://www.scientificamerican.com/article/puzzling-measurement-of-big-g-gravitational-constant-ignites-debate-slide-show/ Puzzling Measurement of "Big G" Gravitational Constant Ignites Debate]<br />
*[http://www.bipm.org/utils/common/documents/jcgm/JCGM_200_2012.pdf International vocabulary of metrology – Basic and general concepts and associated terms (VIM)]<br />
*[http://retractionwatch.com/2014/09/05/white-house-takes-notice-of-reproducibility-in-science-and-wants-your-opinion White House takes notice of reproducibility in science, and wants your opinion]<br />
*Problems during performance benchmarking:<br />
**[https://homes.cs.washington.edu/~bornholt/post/performance-evaluation.html [https://homes.cs.washington.edu/~bornholt/post/performance-evaluation.html]]<br/>''We also experienced many similar issues during our work on auto-tuning and machine learning:''<br />
**[http://hal.inria.fr/hal-01054763 [http://hal.inria.fr/hal-01054763]]<br />
**[http://arxiv.org/abs/1406.4020 [http://arxiv.org/abs/1406.4020]]<br />
*[http://dl.acm.org/citation.cfm?id=2723872 ACM SIGOPS Operating Systems Review - Special Issue on Repeatability and Sharing of Experimental Artifacts]<br />
*[http://ieeexplore.ieee.org/xpl/articleDetails.jsp?reload=true&arnumber=5768098 Vinton G. Cerf. "Bit Rot: Long-Term Preservation of Digital Information" [Point of View]]<br />
*[http://www.nature.com/news/the-top-100-papers-1.16224 About citations: less related though interesting]<br />
<br />
=== Journals with artifact sharing and evaluation ===<br />
<br />
*[http://www.ipol.im IPOL Journal: Image Processing On Line<br/>]<br />
*[http://netlib.org/toms ACM TRANSACTIONS ON MATHEMATICAL SOFTWARE (TOMS)]<br />
*[http://db-reproducibility.seas.harvard.edu ACM SIGMOD 2015]<br />
<br />
=== National requirements ===<br />
* [https://www.epsrc.ac.uk/files/aboutus/standards/clarificationsofexpectationsresearchdatamanagement UK EPSRC expectations on research data management]<br />
* University of Manchester, UK:<br />
** http://www.library.manchester.ac.uk/services-and-support/staff/research/services/research-data-management/policy<br />
** http://www.library.manchester.ac.uk/services-and-support/staff/research/services/research-data-management/data-management-planning<br />
* Cornell University, USA: [http://data.research.cornell.edu/content/best-practices Best Practices]</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Articles&diff=836Reproducibility:Articles2015-10-01T11:01:07Z<p>Gfursin: /* Assorted articles and presentations */</p>
<hr />
<div>[ '''''[[Reproducibility|Back to main page]]''''' ]<br />
<br />
=== '''Assorted articles and presentations'''<br/> ===<br />
<br />
*[http://www.pl-enthusiast.net/2015/09/01/pl-conference-papers-to-get-a-journal PL conference papers to get a journal] - discussing ACM proposal to formally recognize conference publications as equal in quality as journal publications - controversial<br />
*Our experience report with encountered problems in computer systems' research and new proposal for ''community-driven reviewing and validation of publications'' [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]<br />
*[https://hal.inria.fr/inria-00436029 Collective Tuning Initiative: automating and accelerating development and optimization of computing systems]<br />
*[http://www.the-scientist.com/?articles.view/articleNo/33719/title/Science-s-Reproducibility-Problem Science's Reproducibility Problem], Bob Grant, The Scientist<br />
*[http://www.cs.huji.ac.il/~feit/papers/Repeat15SIGOPS.pdf From Repeatability to Reproducibility and Corroboration], Dror G. Feitelson, SIGOPS Operating Syst. Rev. 49(1), pp. 3-11, Jan 2015<br />
*[http://cacm.acm.org/blogs/blog-cacm/98560 Reviewing peer review], Jeannette M. Wing, ACM Blog<br />
*[http://cacm.acm.org/blogs/blog-cacm/100284 How Should Peer Review Evolve?], Ed H. Chi, ACM Blog<br />
*[http://crd.lbl.gov/~dhbailey/dhbpapers/twelve-ways.pdf Twelve Ways to Fool the Masses When Giving Performance Results on Parallel Computers], David H. Bailey, Supercomputing Review, June 11, 1991<br />
*[Ten Ways to Fool the Masses When Giving Performance Results on GPUs], Scott Pakin, HPCWire, December 13, 2011 <br />
*[http://www.cam.ac.uk/research/news/new-gold-standard-established-for-open-and-reproducible-research Cambridge University news about open and reproducible research]<br />
*[http://reproduciblescience.blogspot.fr/2015/06/interesting-replicable-badge-for.html Interesting Replicable "Badge" for journal articles], Daniel S. Katz<br />
*[http://thomasleeper.com/2015/05/open-science-language What's in a Name? The Concepts and Language of Replication and Reproducibility]<br />
*[http://ehp.niehs.nih.gov/122-a188 Research Wranglers: Initiatives to Improve Reproducibility of Study Findings] (Environmental Health Perspectives)<br />
*[https://web.stanford.edu/~vcs/talks/VictoriaStoddenICML2010.pdf Reproducible Research in Computational Science: Problems and Solutions For Data and Code Sharing], Victoria Stodden, ICML workshop 2010<br />
*[http://www.the-scientist.com/?articles.view/articleNo/32426/title/Predatory-Publishing Predatory publishing]<br />
*[http://michaelnielsen.org/blog/three-myths-about-scientific-peer-review Three myths about scientific peer review]<br />
*[http://spyros.zoupanos.net/papers/p39.open-repeatability.pdf The repeatability experiment of SIGMOD 2008], SIGMOD Record, vol. 37, no. 1 (March 2008), 39-45. <br />
*[http://dl.acm.org/citation.cfm?id=1536616.1536619 "Begin with an Author's response to Reviews"] - proposal to submit past reviews along with articles<br />
*[https://www.dropbox.com/s/67n49i9k3cb28mo/GOBLE-CW2014.ppt Presentation by Professor Carole Goble]<br />
*Dennis McCafferty, [http://dl.acm.org/citation.cfm?id=1831415 "Should Code be Released?"]<br />
*[http://www.elsevier.com/connect/can-data-be-peer-reviewed Elsevier:Can data be peer-reviewed?]<br />
*[http://www.executablepapers.com Elsevier:The Executable Paper Grand Challenge]<br />
*[http://www.sciencemag.org/content/334/6060/1225.full Special section on Data replication & reproducibility], Science magazine, 2 December 2011<br />
*Chris Drummond, [http://cogprints.org/7691/7/ICMLws09.pdf "Replicability is not Reproducibility: Nor is it Good Science"]<br />
*[http://theconversation.com/science-is-in-a-reproducibility-crisis-how-do-we-resolve-it-16998 Science is in a reproducibility crisis - how do we resolve it?]<br />
*[http://www.w3.org/2001/sw/hcls/notes/hcls-dataset W3C Dataset Descriptions: HCLS Community Profile]<br />
*My blog article on [http://www.software.ac.uk/blog/2014-07-22-automatic-performance-tuning-and-reproducibility-side-effect "Automatic performance tuning and reproducibility as a side effect"]&nbsp; for the Software Sustainability Institute<br />
*[http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble Trouble at the Lab], The Economist, 2013<br />
*[https://www.force11.org/group/joint-declaration-data-citation-principles-final Joint Declaration of Data Citation Principles]<br />
*[http://www.scientificamerican.com/article/puzzling-measurement-of-big-g-gravitational-constant-ignites-debate-slide-show/ Puzzling Measurement of "Big G" Gravitational Constant Ignites Debate]<br />
*[http://www.bipm.org/utils/common/documents/jcgm/JCGM_200_2012.pdf International vocabulary of metrology – Basic and general concepts and associated terms (VIM)]<br />
*[http://retractionwatch.com/2014/09/05/white-house-takes-notice-of-reproducibility-in-science-and-wants-your-opinion White House takes notice of reproducibility in science, and wants your opinion]<br />
*Problems during performance benchmarking:<br />
**[https://homes.cs.washington.edu/~bornholt/post/performance-evaluation.html [https://homes.cs.washington.edu/~bornholt/post/performance-evaluation.html]]<br/>''We also experienced many similar issues during our work on auto-tuning and machine learning:''<br />
**[http://hal.inria.fr/hal-01054763 [http://hal.inria.fr/hal-01054763]]<br />
**[http://arxiv.org/abs/1406.4020 [http://arxiv.org/abs/1406.4020]]<br />
*[http://dl.acm.org/citation.cfm?id=2723872 ACM SIGOPS Operating Systems Review - Special Issue on Repeatability and Sharing of Experimental Artifacts]<br />
*[http://ieeexplore.ieee.org/xpl/articleDetails.jsp?reload=true&arnumber=5768098 Vinton G. Cerf. "Bit Rot: Long-Term Preservation of Digital Information" [Point of View]]<br />
*[http://www.nature.com/news/the-top-100-papers-1.16224 About citations: less related though interesting]<br />
<br />
=== Journals with artifact sharing and evaluation ===<br />
<br />
*[http://www.ipol.im IPOL Journal: Image Processing On Line<br/>]<br />
*[http://netlib.org/toms ACM TRANSACTIONS ON MATHEMATICAL SOFTWARE (TOMS)]<br />
*[http://db-reproducibility.seas.harvard.edu ACM SIGMOD 2015]<br />
<br />
=== National requirements ===<br />
* [https://www.epsrc.ac.uk/files/aboutus/standards/clarificationsofexpectationsresearchdatamanagement UK EPSRC expectations on research data management]<br />
* University of Manchester, UK:<br />
** http://www.library.manchester.ac.uk/services-and-support/staff/research/services/research-data-management/policy<br />
** http://www.library.manchester.ac.uk/services-and-support/staff/research/services/research-data-management/data-management-planning<br />
* Cornell University, USA: [http://data.research.cornell.edu/content/best-practices Best Practices]</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Events&diff=835Reproducibility:Events2015-09-25T12:08:29Z<p>Gfursin: </p>
<hr />
<div>[ '''''[[Reproducibility|Back to main page]]''''' ]<br />
<br />
=== '''Assorted events'''<br/> ===<br />
<br />
*2016.May: [http://mmsys2016.itec.aau.at/dataset-track MMSys 2016 Dataset Track]<br />
*2016.March: Dagstuhl seminar on "Rethinking Experimental Methods in Computing": [http://www.dagstuhl.de/programm/kalender/semhp/?semnr=16111 Home Page]<br />
*2016.Jan: [http://adapt-workshop.org/index2016.html ADAPT'16] - workshop on adaptive, self-tuning computer systems that will feature our open publication model with community-driven reviewing, reddit-based discussions and artifact evaluation<br />
*2015.Nov.20: Deadline for artifact submission for accepted papers for [http://cTuning.org/event/ae-ppopp2016 PPoPP'16] and [http://cTuning.org/event/ae-cgo2016 CGO'16]<br />
*2015.Nov.1-4: Dastuhl perspective workshop on artifact evaluation: [http://www.dagstuhl.de/de/programm/kalender/semhp/?semnr=15452 Home page], [http://www.dagstuhl.de/program/calendar/partlist/?semnr=15452 participants]<br />
*2015.Oct.19: [http://www.software.ac.uk/software-credit Software Credit Workshop] organized by Software Sustainability Institute in London. It will will explore what contribution software can and should make for academic reputational credit.<br />
*2015.Sep.23-25: [https://rd-alliance.org/plenary-meetings/rda-sixth-plenary-meeting.html Research Data Alliance 6th plenary meeting in Paris]<br />
*2015.Jul.27: [http://www.ntms-conf.org/ntms-2015/index.php/workshops/reproducibility-of-computation-based-research-workshop Workshop on Reproducibility of Computation Based Research: Languages, Standards, Methodologies and Platforms]<br />
*2015.Feb.9: our joint CGO/PPoPP'15 session on artifact evaluation experience [http://www.slideshare.net/GrigoriFursin/presentation-fursin-aecgoppopp2015 PDF]<br />
*2015.Jan.: [http://www.sigops.org/osr.html ACM SIGOPS Operating Systems Review . Special Issue on Repeatability and Sharing of Experimental Artifacts]<br />
*2015.Jan.: [http://adapt-workshop.org ADAPT'15] - workshop on adaptive self-tuning computer systems. <br />
*2014.Oct.27-30: [http://bigscientificdata.org/cask14 "1st International Workshop on Collaborative methodologies to Accelerate Scientific Knowledge discovery in big data (CASK) 2014"]<br />
*[http://www.occamportal.org/images/reproduce/TETC-SI-REPRODUCE.pdf Special journal issue] on Reproducible &nbsp;Research &nbsp;Methodologies at IEEE TETC<br />
*[http://wssspe.researchcomputing.org.uk/wssspe2 2nd Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE2)] to be held in conjunction with SC14, Sunday, 16 November 2014, New Orleans, LA, USA<br />
*[https://www.xsede.org/web/reproducibility Reproducibility @ XSEDE: An XSEDE14 workshop (July, 2014)]<br />
*[http://adapt-workshop.org/2014 ADAPT'14] - workshop on adaptive self-tuning computer systems [ [http://adapt-workshop.org/2014/program.htm program and publications] ]<br />
*[http://c-mind.org/events/trust2014 ACM SIGPLAN TRUST'14] @ PLDI'14<br />
*[http://www.software.ac.uk/cw14 Collaborative Workshop 2014 (CW14) - software in your reproducible research]<br />
*[http://www.occamportal.org/reproduce REPRODUCE'14] @ HPCA'14<br />
*[http://www.adapt-workshop.org/2014/program.htm ADAPT'14 panel] @ HiPEAC'14<br />
*[https://icerm.brown.edu/tw14-5-cemc Challenges in 21st Century Experimental Mathematical Computation] July 21-25, 2014<br />
*[http://www.eecg.toronto.edu/%7Eenright/wddd WDDD (Workshop on Duplicating, Deconstructing, and Debunking)]<br />
*[http://ctuning.org/making-computer-engineering-a-science-2013 HiPEAC'13 CSW thematic session] @ ACM ECRC "Making computer engineering a science"<br />
*[http://ctuning.org/hipeac3-thematic-session-2012-04 HiPEAC'12 CSW thematic session]<br />
*[http://www.executablepapers.com Elsevier:The Executable Paper Grand Challenge]<br />
*[http://exadapt.org/2012/program.html ASPLOS/EXADAPT'12 panel] @ ASPLOS'12<br />
*[http://www.stodden.net/AMP2011 AMP workshop 2011 (Reproducible Research: Tools and Strategies for Scientific Computing)]<br />
*[http://ctuning.org/lab/education cTuning lectures (2008-2010)]<br />
*[http://hal.inria.fr/inria-00436029 GCC Summit'09 discussion]</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Links&diff=834Reproducibility:Links2015-09-24T15:40:14Z<p>Gfursin: </p>
<hr />
<div>[ '''''[[Reproducibility|Back to main page]]''''' ]<br />
<br />
=== '''Assorted (offline and online) tools'''<br/> ===<br />
<br />
*[http://github.com/ctuning/ck Collective Knowledge Technology] - preserve (with distributed ID), organize, desribe, share and reuse your code and data via GitHub<br />
**[http://cknowledge.org/repo CK-based interactive article]<br />
**[http://github.com/ctuning/reproduce-clsmith PLDI'15 artifact converted to CK format]<br />
**[http://github.com/ctuning/reproduce-pamela-project Artifact repository for UK PAMELA project]<br />
**[http://github.com/ctuning/reproduce-carp-project Artifact repository for EU FP7 CARP project]<br />
**[http://github.com/ctuning/ctuning-programs Example of benchmarks shared in CK format]<br />
**[http://github.com/ctuning/ctuning-datasets-min Example of data sets shared in CK format]<br />
**[https://drive.google.com/folderview?id=0B-wXENVfIO82dzYwaUNIVElxaGc&usp=sharing Examples of CK repositories shared as zip files]<br />
*[https://www.docker.io Docker tool] (pack, ship and run applications as a lightweight container)<br />
*[http://vida-nyu.github.io/reprozip ReproZip] automatically packing experiments ([https://www.usenix.org/conference/tapp13/technical-sessions/presentation/chirigati related article])<br />
*[http://reproducible.io CARE tool from STMicroelectronics] (Comprehensive Archiver for Reproducible Execution)<br />
*[http://rr-project.org RR] (Mozilla project: records nondeterministic executions and debugs them deterministically)<br />
*[http://www.pgbovine.net/cde.html CDE tool] (automatically create portable Linux applications with all dependencies)<br />
*[http://ipython.org/notebook.html IPython Notebook] (a web-based interactive computational environment where you can combine code execution, text, mathematics, plots and rich media into a single document)<br />
*[http://www.rstudio.com R-studio] (Open source and enterprise-ready professional software for R)<br />
*[https://www.codalab.org Codelab] (an experimental platform for collaboration and competition)<br />
*[https://www.grid5000.fr Grid5000] large-scale and versatile testbed for experiment-driven research in all areas of computer science, with a focus on parallel and distributed computing including Cloud, HPC and Big Data<br />
*[http://www.mygrid.org.uk MyGrid] - develops a suite of tools designed to "help e-Scientists get on with science and get on with scientists"<br />
*[http://figshare.com FigShare] (managing research in a cloud)<br />
*[http://www.runmycode.org RunMyCode] (online workflows)<br />
*[https://www.aptlab.net AptLab] (online workflows)<br />
*[http://www.taverna.org.uk Taverna] (designing and executing workflows)<br />
*[http://boinc.berkeley.edu BOINC] (open-source software for volunteer computing and grid computing)<br />
*[https://mulcyber.toulouse.inra.fr/projects/ngspipelines NGS pipelines] (integrates pipelines and user interfaces to help biologists to analyse data outputed from biological applications such as RNAseq, sRNAseq, ChipSeq, BS-seq)<br />
*[http://www.seek4science.org SEEK for Science] - finding, sharing and exchanging Data, Models, Simulations and Processes in Science<br />
*[http://www.cs.umd.edu/projects/skoll/Skoll/Home.html Skoll] (A process & Infrastructure for Distributed, continuous Quality assurance)<br />
*[http://nepi.inria.fr NEPI] (Simplifying network experimentation)<br />
*[http://pgbovine.net/burrito.html Burrito] (Rethinking the Electronic Lab Notebook)<br/>[http://hal.inria.fr/inria-00436029 open source cTuning technology] (crowdsource auto-tuning and combine with machine learning using big data, predictive analytics and crowdsourcing) (2006-cur.)<br />
*[http://c-mind.org/repo Collective Mind technology] (towards collaborative, systematic and reproducible computer engineering using big data, predictive analytics and collective intelligence) (2011-2014)<br />
*[http://www.occamportal.org OCCAM] - open curation for computer architecture modeling<br />
*[http://orgmode.org Org mode] - keeping notes, maintaining TODO lists, planning projects, and authoring documents with a fast and effective plain-text system</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Links&diff=833Reproducibility:Links2015-09-24T15:37:01Z<p>Gfursin: </p>
<hr />
<div>[ '''''[[Reproducibility|Back to main page]]''''' ]<br />
<br />
=== '''Assorted (offline and online) tools'''<br/> ===<br />
<br />
*[http://github.com/ctuning/ck Collective Knowledge Technology] - preserve (with distributed ID), organize, desribe, share and reuse your code and data via GitHub<br />
**[http://cknowledge.org/repo CK-based interactive article]<br />
**[http://github.com/ctuning/reproduce-clsmith PLDI'15 artifact converted to CK format]<br />
**[http://github.com/ctuning/reproduce-pamela-project Artifact repository for UK PAMELA project]<br />
**[http://github.com/ctuning/reproduce-carp-project Artifact repository for EU FP7 CARP project]<br />
**[http://github.com/ctuning/ctuning-programs Example of benchmarks shared in CK format]<br />
**[http://github.com/ctuning/ctuning-datasets-min Example of data sets shared in CK format]<br />
**[https://drive.google.com/folderview?id=0B-wXENVfIO82dzYwaUNIVElxaGc&usp=sharing Examples of CK repositories shared as zip files]<br />
*[https://www.docker.io Docker tool] (pack, ship and run applications as a lightweight container)<br />
*[http://vida-nyu.github.io/reprozip ReproZip] automatically packing experiments ([https://www.usenix.org/conference/tapp13/technical-sessions/presentation/chirigati related article])<br />
*[http://reproducible.io CARE tool from STMicroelectronics] (Comprehensive Archiver for Reproducible Execution)<br />
*[http://rr-project.org RR] (Mozilla project: records nondeterministic executions and debugs them deterministically)<br />
*[http://www.pgbovine.net/cde.html CDE tool] (automatically create portable Linux applications with all dependencies)<br />
*[http://ipython.org/notebook.html IPython Notebook] (a web-based interactive computational environment where you can combine code execution, text, mathematics, plots and rich media into a single document)<br />
*[http://www.rstudio.com R-studio] (Open source and enterprise-ready professional software for R)<br />
*[https://www.codalab.org Codelab] (an experimental platform for collaboration and competition)<br />
*[https://www.grid5000.fr Grid5000] large-scale and versatile testbed for experiment-driven research in all areas of computer science, with a focus on parallel and distributed computing including Cloud, HPC and Big Data<br />
*[http://www.mygrid.org.uk MyGrid] - develops a suite of tools designed to "help e-Scientists get on with science and get on with scientists"<br />
*[http://figshare.com FigShare] (managing research in a cloud)<br />
*[http://www.runmycode.org RunMyCode] (online workflows)<br />
*[https://www.aptlab.net AptLab] (online workflows)<br />
*[http://www.taverna.org.uk Taverna] (designing and executing workflows)<br />
*[http://boinc.berkeley.edu BOINC] (open-source software for volunteer computing and grid computing)<br />
*[https://mulcyber.toulouse.inra.fr/projects/ngspipelines NGS pipelines] (integrates pipelines and user interfaces to help biologists to analyse data outputed from biological applications such as RNAseq, sRNAseq, ChipSeq, BS-seq)<br />
*[http://www.cs.umd.edu/projects/skoll/Skoll/Home.html Skoll] (A process & Infrastructure for Distributed, continuous Quality assurance)<br />
*[http://nepi.inria.fr NEPI] (Simplifying network experimentation)<br />
*[http://pgbovine.net/burrito.html Burrito] (Rethinking the Electronic Lab Notebook)<br/>[http://hal.inria.fr/inria-00436029 open source cTuning technology] (crowdsource auto-tuning and combine with machine learning using big data, predictive analytics and crowdsourcing) (2006-cur.)<br />
*[http://c-mind.org/repo Collective Mind technology] (towards collaborative, systematic and reproducible computer engineering using big data, predictive analytics and collective intelligence) (2011-2014)<br />
*[http://www.occamportal.org OCCAM] - open curation for computer architecture modeling<br />
*[http://orgmode.org Org mode] - keeping notes, maintaining TODO lists, planning projects, and authoring documents with a fast and effective plain-text system</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Repositories&diff=832Reproducibility:Repositories2015-09-24T15:36:34Z<p>Gfursin: </p>
<hr />
<div>[ '''''[[Reproducibility|Back to main page]]'''''<br />
<br />
=== '''Assorted misc repositories and initiatives'''<br/> ===<br />
<br />
*[http://github.com/ctuning/ck Collective Knowledge] - sharing and reusing artifacts with distributed ID via GitHub<br />
**[http://cknowledge.org/repo CK-based interactive article]<br />
**[http://github.com/ctuning/reproduce-clsmith PLDI'15 artifact converted to CK format]<br />
**[http://github.com/ctuning/reproduce-pamela-project Artifact repository for UK PAMELA project]<br />
**[http://github.com/ctuning/reproduce-carp-project Artifact repository for EU FP7 CARP project]<br />
**[http://github.com/ctuning/ctuning-programs Example of benchmarks shared in CK format]<br />
**[http://github.com/ctuning/ctuning-datasets-min Example of data sets shared in CK format]<br />
**[https://drive.google.com/folderview?id=0B-wXENVfIO82dzYwaUNIVElxaGc&usp=sharing Examples of CK repositories shared as zip files]<br />
*[http://gigadb.org (GIGA)n DB]<br />
*[http://galaxyproject.org Data Intensive Biology]<br />
*[http://www.galaxyzoo.org/?lang=en GalaxyZoo] (classification of galaxies)<br />
*[https://olivearchive.org Olive Archive] (preserving executable content)<br />
*[http://openscience.us/repo Tera-PROMISE<br/>]<br />
*[https://www.openaire.eu OpenAire (CERN)<br/>]<br />
*[https://zenodo.org Zenodo]<br />
*[http://fair-dom.org FAIRDOM - establish a data and model management service facility for Systems Biology]<br />
*[http://researchcompendia.org ResearchCompendia<br/>]<br />
*[http://www.researchobject.org ResearchObject]<br />
*[http://www.researchobject.org/initiative Other initiatives related to ResearchObject]<br />
*[https://archive.org Internet Archive<br/>]<br />
*[http://www.nationalarchives.gov.uk The national archives<br/>]<br />
*[http://www.dcc.ac.uk Digital Curation Center]<br />
*[http://riojournal.com Research Ideas and Outcome Journal] - controversial<br />
*[http://www.wikidata.org/wiki/Wikidata:Introduction WikiData]<br />
*[http://www.dpn.org The digital preservation network<br/>]<br />
*[http://www.ddialliance.org Data Documentation Initiative]<br />
*[http://recomputation.org recomputation.org]<br />
*[https://open-data.europa.eu Open datasets<br/>]<br />
*[http://rmap-project.info/rmap RMap PROJECT] - preserve the many-to-many complex relationships among scholarly publications and their underlying data<br />
*[http://datahub.io DataHub]<br />
*[http://www.execandshare.org/CompanionSite www.execandshare.org] - creates a companion website associated with a submitted paper to implement the methodology presented in the paper<br />
*[https://www.datacite.org/contact Datacite] (citing data as DOI, Germany, has connections with CERN)<br />
*[http://www.crossref.org CrossRef]<br />
*[http://www.knowledge-exchange.info Knowledge Exchange]<br />
*[http://www.doi.org International DOI Foundation]<br />
*[http://c-mind.org/repo Collective Mind pilot live repository] (deprecated in 2014 for [http://cknowledge.org/repo Collective Knowledge])<br />
*[http://ctuning.org/repo cTuning.org (performance tuning database, outdated)]<br />
*[http://www.occamportal.org OCCAM project] - open curation for computer architecture modeling</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Links&diff=831Reproducibility:Links2015-09-24T15:35:09Z<p>Gfursin: </p>
<hr />
<div>[ '''''[[Reproducibility|Back to main page]]''''' ]<br />
<br />
=== '''Assorted (offline and online) tools'''<br/> ===<br />
<br />
*[http://github.com/ctuning/ck Collective Knowledge] - preserve (with distributed ID), organize, desribe, share and reuse your code and data via GitHub<br />
**[http://cknowledge.org/repo CK-based interactive article]<br />
**[http://github.com/ctuning/reproduce-clsmith PLDI'15 artifact converted to CK format]<br />
**[http://github.com/ctuning/reproduce-pamela-project Artifact repository for UK PAMELA project]<br />
**[http://github.com/ctuning/reproduce-carp-project Artifact repository for EU FP7 CARP project]<br />
**[http://github.com/ctuning/ctuning-programs Example of benchmarks shared in CK format]<br />
**[http://github.com/ctuning/ctuning-datasets-min Example of data sets shared in CK format]<br />
**[https://drive.google.com/folderview?id=0B-wXENVfIO82dzYwaUNIVElxaGc&usp=sharing Examples of CK repositories shared as zip files]<br />
*[https://www.docker.io Docker tool] (pack, ship and run applications as a lightweight container)<br />
*[http://vida-nyu.github.io/reprozip ReproZip] automatically packing experiments ([https://www.usenix.org/conference/tapp13/technical-sessions/presentation/chirigati related article])<br />
*[http://reproducible.io CARE tool from STMicroelectronics] (Comprehensive Archiver for Reproducible Execution)<br />
*[http://rr-project.org RR] (Mozilla project: records nondeterministic executions and debugs them deterministically)<br />
*[http://www.pgbovine.net/cde.html CDE tool] (automatically create portable Linux applications with all dependencies)<br />
*[http://ipython.org/notebook.html IPython Notebook] (a web-based interactive computational environment where you can combine code execution, text, mathematics, plots and rich media into a single document)<br />
*[http://www.rstudio.com R-studio] (Open source and enterprise-ready professional software for R)<br />
*[https://www.codalab.org Codelab] (an experimental platform for collaboration and competition)<br />
*[https://www.grid5000.fr Grid5000] large-scale and versatile testbed for experiment-driven research in all areas of computer science, with a focus on parallel and distributed computing including Cloud, HPC and Big Data<br />
*[http://figshare.com FigShare] (managing research in a cloud)<br />
*[http://www.runmycode.org RunMyCode] (online workflows)<br />
*[https://www.aptlab.net AptLab] (online workflows)<br />
*[http://www.taverna.org.uk Taverna] (designing and executing workflows)<br />
*[http://boinc.berkeley.edu BOINC] (open-source software for volunteer computing and grid computing)<br />
*[https://mulcyber.toulouse.inra.fr/projects/ngspipelines NGS pipelines] (integrates pipelines and user interfaces to help biologists to analyse data outputed from biological applications such as RNAseq, sRNAseq, ChipSeq, BS-seq)<br />
*[http://www.cs.umd.edu/projects/skoll/Skoll/Home.html Skoll] (A process & Infrastructure for Distributed, continuous Quality assurance)<br />
*[http://nepi.inria.fr NEPI] (Simplifying network experimentation)<br />
*[http://pgbovine.net/burrito.html Burrito] (Rethinking the Electronic Lab Notebook)<br/>[http://hal.inria.fr/inria-00436029 open source cTuning technology] (crowdsource auto-tuning and combine with machine learning using big data, predictive analytics and crowdsourcing) (2006-cur.)<br />
*[http://c-mind.org/repo Collective Mind technology] (towards collaborative, systematic and reproducible computer engineering using big data, predictive analytics and collective intelligence) (2011-2014)<br />
*[http://www.occamportal.org OCCAM project] - open curation for computer architecture modeling<br />
*[http://orgmode.org Org mode] - keeping notes, maintaining TODO lists, planning projects, and authoring documents with a fast and effective plain-text system</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Repositories&diff=830Reproducibility:Repositories2015-09-24T15:32:36Z<p>Gfursin: </p>
<hr />
<div>[ '''''[[Reproducibility|Back to main page]]'''''<br />
<br />
=== '''Assorted misc repositories and initiatives'''<br/> ===<br />
<br />
*[http://github.com/ctuning/ck Collective Knowledge] - sharing and reusing artifacts with distributed ID via GitHub<br />
**[http://cknowledge.org/repo CK-based interactive article]<br />
**[http://github.com/ctuning/reproduce-clsmith PLDI'15 artifact converted to CK format]<br />
**[http://github.com/ctuning/reproduce-pamela-project Artifact repository for UK PAMELA project]<br />
**[http://github.com/ctuning/reproduce-carp-project Artifact repository for EU FP7 CARP project]<br />
**[http://github.com/ctuning/ctuning-programs Example of benchmarks shared in CK format]<br />
**[http://github.com/ctuning/ctuning-datasets-min Example of data sets shared in CK format]<br />
**[https://drive.google.com/folderview?id=0B-wXENVfIO82dzYwaUNIVElxaGc&usp=sharing Examples of CK repositories shared as zip files]<br />
*[http://gigadb.org (GIGA)n DB]<br />
*[http://galaxyproject.org Data Intensive Biology]<br />
*[http://www.galaxyzoo.org/?lang=en GalaxyZoo] (classification of galaxies)<br />
*[https://olivearchive.org Olive Archive] (preserving executable content)<br />
*[http://openscience.us/repo Tera-PROMISE<br/>]<br />
*[https://www.openaire.eu OpenAire (CERN)<br/>]<br />
*[https://zenodo.org Zenodo]<br />
*[http://researchcompendia.org ResearchCompendia<br/>]<br />
*[http://www.researchobject.org ResearchObject]<br />
*[http://www.researchobject.org/initiative Other initiatives related to ResearchObject]<br />
*[https://archive.org Internet Archive<br/>]<br />
*[http://www.nationalarchives.gov.uk The national archives<br/>]<br />
*[http://www.dcc.ac.uk Digital Curation Center]<br />
*[http://riojournal.com Research Ideas and Outcome Journal] - controversial<br />
*[http://www.wikidata.org/wiki/Wikidata:Introduction WikiData]<br />
*[http://www.dpn.org The digital preservation network<br/>]<br />
*[http://www.ddialliance.org Data Documentation Initiative]<br />
*[http://recomputation.org recomputation.org]<br />
*[https://open-data.europa.eu Open datasets<br/>]<br />
*[http://rmap-project.info/rmap RMap PROJECT] - preserve the many-to-many complex relationships among scholarly publications and their underlying data<br />
*[http://datahub.io DataHub]<br />
*[http://www.execandshare.org/CompanionSite www.execandshare.org] - creates a companion website associated with a submitted paper to implement the methodology presented in the paper<br />
*[https://www.datacite.org/contact Datacite] (citing data as DOI, Germany, has connections with CERN)<br />
*[http://www.crossref.org CrossRef]<br />
*[http://www.knowledge-exchange.info Knowledge Exchange]<br />
*[http://www.doi.org International DOI Foundation]<br />
*[http://c-mind.org/repo Collective Mind pilot live repository] (deprecated in 2014 for [http://cknowledge.org/repo Collective Knowledge])<br />
*[http://ctuning.org/repo cTuning.org (performance tuning database, outdated)]<br />
*[http://www.occamportal.org OCCAM project] - open curation for computer architecture modeling</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Articles&diff=829Reproducibility:Articles2015-09-24T15:31:09Z<p>Gfursin: </p>
<hr />
<div>[ '''''[[Reproducibility|Back to main page]]''''' ]<br />
<br />
=== '''Assorted articles and presentations'''<br/> ===<br />
<br />
*[http://www.pl-enthusiast.net/2015/09/01/pl-conference-papers-to-get-a-journal PL conference papers to get a journal] - discussing ACM proposal to formally recognize conference publications as equal in quality as journal publications - controversial<br />
*Our experience report with encountered problems in computer systems' research and new proposal for ''community-driven reviewing and validation of publications'' [[http://arxiv.org/abs/1406.4020 arXiv] , [http://dl.acm.org/citation.cfm?id=2618142 ACM DL]]<br />
*[http://www.the-scientist.com/?articles.view/articleNo/33719/title/Science-s-Reproducibility-Problem Science's Reproducibility Problem], Bob Grant, The Scientist<br />
*[http://www.cs.huji.ac.il/~feit/papers/Repeat15SIGOPS.pdf From Repeatability to Reproducibility and Corroboration], Dror G. Feitelson, SIGOPS Operating Syst. Rev. 49(1), pp. 3-11, Jan 2015<br />
*[http://cacm.acm.org/blogs/blog-cacm/98560 Reviewing peer review], Jeannette M. Wing, ACM Blog<br />
*[http://cacm.acm.org/blogs/blog-cacm/100284 How Should Peer Review Evolve?], Ed H. Chi, ACM Blog<br />
*[http://crd.lbl.gov/~dhbailey/dhbpapers/twelve-ways.pdf Twelve Ways to Fool the Masses When Giving Performance Results on Parallel Computers], David H. Bailey, Supercomputing Review, June 11, 1991<br />
*[Ten Ways to Fool the Masses When Giving Performance Results on GPUs], Scott Pakin, HPCWire, December 13, 2011 <br />
*[http://www.cam.ac.uk/research/news/new-gold-standard-established-for-open-and-reproducible-research Cambridge University news about open and reproducible research]<br />
*[http://reproduciblescience.blogspot.fr/2015/06/interesting-replicable-badge-for.html Interesting Replicable "Badge" for journal articles], Daniel S. Katz<br />
*[http://thomasleeper.com/2015/05/open-science-language What's in a Name? The Concepts and Language of Replication and Reproducibility]<br />
*[http://ehp.niehs.nih.gov/122-a188 Research Wranglers: Initiatives to Improve Reproducibility of Study Findings] (Environmental Health Perspectives)<br />
*[https://web.stanford.edu/~vcs/talks/VictoriaStoddenICML2010.pdf Reproducible Research in Computational Science: Problems and Solutions For Data and Code Sharing], Victoria Stodden, ICML workshop 2010<br />
*[http://www.the-scientist.com/?articles.view/articleNo/32426/title/Predatory-Publishing Predatory publishing]<br />
*[http://michaelnielsen.org/blog/three-myths-about-scientific-peer-review Three myths about scientific peer review]<br />
*[http://spyros.zoupanos.net/papers/p39.open-repeatability.pdf The repeatability experiment of SIGMOD 2008], SIGMOD Record, vol. 37, no. 1 (March 2008), 39-45. <br />
*[http://dl.acm.org/citation.cfm?id=1536616.1536619 "Begin with an Author's response to Reviews"] - proposal to submit past reviews along with articles<br />
*[https://www.dropbox.com/s/67n49i9k3cb28mo/GOBLE-CW2014.ppt Presentation by Professor Carole Goble]<br />
*Dennis McCafferty, [http://dl.acm.org/citation.cfm?id=1831415 "Should Code be Released?"]<br />
*[http://www.elsevier.com/connect/can-data-be-peer-reviewed Elsevier:Can data be peer-reviewed?]<br />
*[http://www.executablepapers.com Elsevier:The Executable Paper Grand Challenge]<br />
*[http://www.sciencemag.org/content/334/6060/1225.full Special section on Data replication & reproducibility], Science magazine, 2 December 2011<br />
*Chris Drummond, [http://cogprints.org/7691/7/ICMLws09.pdf "Replicability is not Reproducibility: Nor is it Good Science"]<br />
*[http://theconversation.com/science-is-in-a-reproducibility-crisis-how-do-we-resolve-it-16998 Science is in a reproducibility crisis - how do we resolve it?]<br />
*[http://www.w3.org/2001/sw/hcls/notes/hcls-dataset W3C Dataset Descriptions: HCLS Community Profile]<br />
*My blog article on [http://www.software.ac.uk/blog/2014-07-22-automatic-performance-tuning-and-reproducibility-side-effect "Automatic performance tuning and reproducibility as a side effect"]&nbsp; for the Software Sustainability Institute<br />
*[http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble Trouble at the Lab], The Economist, 2013<br />
*[https://www.force11.org/group/joint-declaration-data-citation-principles-final Joint Declaration of Data Citation Principles]<br />
*[http://www.scientificamerican.com/article/puzzling-measurement-of-big-g-gravitational-constant-ignites-debate-slide-show/ Puzzling Measurement of "Big G" Gravitational Constant Ignites Debate]<br />
*[http://www.bipm.org/utils/common/documents/jcgm/JCGM_200_2012.pdf International vocabulary of metrology – Basic and general concepts and associated terms (VIM)]<br />
*[http://retractionwatch.com/2014/09/05/white-house-takes-notice-of-reproducibility-in-science-and-wants-your-opinion White House takes notice of reproducibility in science, and wants your opinion]<br />
*Problems during performance benchmarking:<br />
**[https://homes.cs.washington.edu/~bornholt/post/performance-evaluation.html [https://homes.cs.washington.edu/~bornholt/post/performance-evaluation.html]]<br/>''We also experienced many similar issues during our work on auto-tuning and machine learning:''<br />
**[http://hal.inria.fr/hal-01054763 [http://hal.inria.fr/hal-01054763]]<br />
**[http://arxiv.org/abs/1406.4020 [http://arxiv.org/abs/1406.4020]]<br />
*[http://dl.acm.org/citation.cfm?id=2723872 ACM SIGOPS Operating Systems Review - Special Issue on Repeatability and Sharing of Experimental Artifacts]<br />
*[http://ieeexplore.ieee.org/xpl/articleDetails.jsp?reload=true&arnumber=5768098 Vinton G. Cerf. "Bit Rot: Long-Term Preservation of Digital Information" [Point of View]]<br />
*[http://www.nature.com/news/the-top-100-papers-1.16224 About citations: less related though interesting]<br />
<br />
=== Journals with artifact sharing and evaluation ===<br />
<br />
*[http://www.ipol.im IPOL Journal: Image Processing On Line<br/>]<br />
*[http://netlib.org/toms ACM TRANSACTIONS ON MATHEMATICAL SOFTWARE (TOMS)]<br />
*[http://db-reproducibility.seas.harvard.edu ACM SIGMOD 2015]<br />
<br />
=== National requirements ===<br />
* [https://www.epsrc.ac.uk/files/aboutus/standards/clarificationsofexpectationsresearchdatamanagement UK EPSRC expectations on research data management]<br />
* University of Manchester, UK:<br />
** http://www.library.manchester.ac.uk/services-and-support/staff/research/services/research-data-management/policy<br />
** http://www.library.manchester.ac.uk/services-and-support/staff/research/services/research-data-management/data-management-planning<br />
* Cornell University, USA: [http://data.research.cornell.edu/content/best-practices Best Practices]</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Repositories&diff=828Reproducibility:Repositories2015-09-24T15:27:53Z<p>Gfursin: </p>
<hr />
<div>[ '''''[[Reproducibility|Back to main page]]'''''<br />
<br />
=== '''Assorted misc repositories'''<br/> ===<br />
<br />
*[http://github.com/ctuning/ck Collective Knowledge] - sharing and reusing artifacts with distributed ID via GitHub<br />
**[http://cknowledge.org/repo CK-based interactive article]<br />
**[http://github.com/ctuning/reproduce-clsmith PLDI'15 artifact converted to CK format]<br />
**[http://github.com/ctuning/reproduce-pamela-project Artifact repository for UK PAMELA project]<br />
**[http://github.com/ctuning/reproduce-carp-project Artifact repository for EU FP7 CARP project]<br />
**[http://github.com/ctuning/ctuning-programs Example of benchmarks shared in CK format]<br />
**[http://github.com/ctuning/ctuning-datasets-min Example of data sets shared in CK format]<br />
**[https://drive.google.com/folderview?id=0B-wXENVfIO82dzYwaUNIVElxaGc&usp=sharing Examples of CK repositories shared as zip files]<br />
*[http://gigadb.org (GIGA)n DB]<br />
*[http://galaxyproject.org Data Intensive Biology]<br />
*[http://www.galaxyzoo.org/?lang=en GalaxyZoo] (classification of galaxies)<br />
*[https://olivearchive.org Olive Archive] (preserving executable content)<br />
*[http://openscience.us/repo Tera-PROMISE<br/>]<br />
*[https://www.openaire.eu OpenAire (CERN)<br/>]<br />
*[https://zenodo.org Zenodo]<br />
*[http://researchcompendia.org ResearchCompendia<br/>]<br />
*[http://www.researchobject.org ResearchObject]<br />
*[http://www.researchobject.org/initiative Other initiatives related to ResearchObject]<br />
*[https://archive.org Internet Archive<br/>]<br />
*[http://www.nationalarchives.gov.uk The national archives<br/>]<br />
*[http://www.dcc.ac.uk Digital Curation Center]<br />
*[http://riojournal.com Research Ideas and Outcome Journal] - controversial<br />
*[http://www.wikidata.org/wiki/Wikidata:Introduction WikiData]<br />
*[http://www.dpn.org The digital preservation network<br/>]<br />
*[http://www.ddialliance.org Data Documentation Initiative]<br />
*[http://recomputation.org recomputation.org]<br />
*[https://open-data.europa.eu Open datasets<br/>]<br />
*[http://rmap-project.info/rmap RMap PROJECT] - preserve the many-to-many complex relationships among scholarly publications and their underlying data<br />
*[http://datahub.io DataHub]<br />
*[http://www.execandshare.org/CompanionSite www.execandshare.org] - creates a companion website associated with a submitted paper to implement the methodology presented in the paper<br />
*[https://www.datacite.org/contact Datacite] (citing data as DOI, Germany, has connections with CERN)<br />
*[http://www.crossref.org CrossRef]<br />
*[http://www.doi.org International DOI Foundation]<br />
*[http://c-mind.org/repo Collective Mind pilot live repository] (deprecated in 2014 for [http://cknowledge.org/repo Collective Knowledge])<br />
*[http://ctuning.org/repo cTuning.org (performance tuning database, outdated)]<br />
*[http://www.occamportal.org OCCAM project] - open curation for computer architecture modeling</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Links&diff=827Reproducibility:Links2015-09-24T15:27:24Z<p>Gfursin: </p>
<hr />
<div>[ '''''[[Reproducibility|Back to main page]]''''' ]<br />
<br />
=== '''Assorted (offline and online) tools'''<br/> ===<br />
<br />
*[http://github.com/ctuning/ck Collective Knowledge] - preserve (with distributed ID), organize, desribe, share and reuse your code and data via GitHub<br />
**[http://cknowledge.org/repo CK-based interactive article]<br />
**[http://github.com/ctuning/reproduce-clsmith PLDI'15 artifact converted to CK format]<br />
**[http://github.com/ctuning/reproduce-pamela-project Artifact repository for UK PAMELA project]<br />
**[http://github.com/ctuning/reproduce-carp-project Artifact repository for EU FP7 CARP project]<br />
**[http://github.com/ctuning/ctuning-programs Example of benchmarks shared in CK format]<br />
**[http://github.com/ctuning/ctuning-datasets-min Example of data sets shared in CK format]<br />
**[https://drive.google.com/folderview?id=0B-wXENVfIO82dzYwaUNIVElxaGc&usp=sharing Examples of CK repositories shared as zip files]<br />
*[https://www.docker.io Docker tool] (pack, ship and run applications as a lightweight container)<br />
*[http://vida-nyu.github.io/reprozip ReproZip] automatically packing experiments ([https://www.usenix.org/conference/tapp13/technical-sessions/presentation/chirigati related article])<br />
*[http://reproducible.io CARE tool from STMicroelectronics] (Comprehensive Archiver for Reproducible Execution)<br />
*[http://rr-project.org RR] (Mozilla project: records nondeterministic executions and debugs them deterministically)<br />
*[http://www.pgbovine.net/cde.html CDE tool] (automatically create portable Linux applications with all dependencies)<br />
*[http://ipython.org/notebook.html IPython Notebook] (a web-based interactive computational environment where you can combine code execution, text, mathematics, plots and rich media into a single document)<br />
*[http://www.rstudio.com R-studio] (Open source and enterprise-ready professional software for R)<br />
*[https://www.codalab.org Codelab] (an experimental platform for collaboration and competition)<br />
*[http://figshare.com FigShare] (managing research in a cloud)<br />
*[http://www.runmycode.org RunMyCode] (online workflows)<br />
*[https://www.aptlab.net AptLab] (online workflows)<br />
*[http://www.taverna.org.uk Taverna] (designing and executing workflows)<br />
*[http://boinc.berkeley.edu BOINC] (open-source software for volunteer computing and grid computing)<br />
*[https://mulcyber.toulouse.inra.fr/projects/ngspipelines NGS pipelines] (integrates pipelines and user interfaces to help biologists to analyse data outputed from biological applications such as RNAseq, sRNAseq, ChipSeq, BS-seq)<br />
*[http://www.cs.umd.edu/projects/skoll/Skoll/Home.html Skoll] (A process & Infrastructure for Distributed, continuous Quality assurance)<br />
*[http://nepi.inria.fr NEPI] (Simplifying network experimentation)<br />
*[http://pgbovine.net/burrito.html Burrito] (Rethinking the Electronic Lab Notebook)<br/>[http://hal.inria.fr/inria-00436029 open source cTuning technology] (crowdsource auto-tuning and combine with machine learning using big data, predictive analytics and crowdsourcing) (2006-cur.)<br />
*[http://c-mind.org/repo Collective Mind technology] (towards collaborative, systematic and reproducible computer engineering using big data, predictive analytics and collective intelligence) (2011-2014)<br />
*[http://www.occamportal.org OCCAM project] - open curation for computer architecture modeling<br />
*[http://orgmode.org Org mode] - keeping notes, maintaining TODO lists, planning projects, and authoring documents with a fast and effective plain-text system</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Links&diff=826Reproducibility:Links2015-09-24T15:24:35Z<p>Gfursin: </p>
<hr />
<div>[ '''''[[Reproducibility|Back to main page]]''''' ]<br />
<br />
=== '''Assorted (offline and online) tools'''<br/> ===<br />
<br />
*[http://github.com/ctuning/ck Collective Knowledge] - preserve (with distributed ID), organize, desribe, share and reuse your code and data via GitHub<br />
**[http://cknowledge.org/repo CK-based interactive article]<br />
**[http://github.com/ctuning/reproduce-clsmith PLDI'15 artifact converted to CK format]<br />
**[http://github.com/ctuning/reproduce-pamela-project Artifact repository for UK PAMELA project]<br />
**[http://github.com/ctuning/reproduce-carp-project Artifact repository for EU FP7 CARP project]<br />
**[http://github.com/ctuning/ctuning-programs Example of benchmarks shared in CK format]<br />
**[http://github.com/ctuning/ctuning-datasets-min Example of data sets shared in CK format]<br />
**[https://drive.google.com/folderview?id=0B-wXENVfIO82dzYwaUNIVElxaGc&usp=sharing Examples of CK repositories shared as zip files]<br />
*[https://www.docker.io Docker tool] (pack, ship and run applications as a lightweight container)<br />
*[http://vida-nyu.github.io/reprozip ReproZip] automatically packing experiments ([https://www.usenix.org/conference/tapp13/technical-sessions/presentation/chirigati related article])<br />
*[http://reproducible.io CARE tool from STMicroelectronics] (Comprehensive Archiver for Reproducible Execution)<br />
*[http://rr-project.org RR] (Mozilla project: records nondeterministic executions and debugs them deterministically)<br />
*[http://www.pgbovine.net/cde.html CDE tool] (automatically create portable Linux applications with all dependencies)<br />
*[http://ipython.org/notebook.html IPython Notebook] (a web-based interactive computational environment where you can combine code execution, text, mathematics, plots and rich media into a single document)<br />
*[http://www.rstudio.com R-studio] (Open source and enterprise-ready professional software for R)<br />
*[https://www.codalab.org Codelab] (an experimental platform for collaboration and competition)<br />
*[http://figshare.com FigShare] (managing research in a cloud)<br />
*[http://www.runmycode.org RunMyCode] (online workflows)<br />
*[https://www.aptlab.net AptLab] (online workflows)<br />
*[http://www.taverna.org.uk Taverna] (designing and executing workflows)<br />
*[http://boinc.berkeley.edu BOINC] (open-source software for volunteer computing and grid computing)<br />
*[https://mulcyber.toulouse.inra.fr/projects/ngspipelines NGS pipelines] (integrates pipelines and user interfaces to help biologists to analyse data outputed from biological applications such as RNAseq, sRNAseq, ChipSeq, BS-seq)<br />
*[http://www.cs.umd.edu/projects/skoll/Skoll/Home.html Skoll] (A process & Infrastructure for Distributed, continuous Quality assurance)<br />
*[http://nepi.inria.fr NEPI] (Simplifying network experimentation)<br />
*[http://pgbovine.net/burrito.html Burrito] (Rethinking the Electronic Lab Notebook)<br/>[http://hal.inria.fr/inria-00436029 open source cTuning technology] (crowdsource auto-tuning and combine with machine learning using big data, predictive analytics and crowdsourcing) (2006-cur.)<br />
*[http://c-mind.org/repo Collective Mind technology] (towards collaborative, systematic and reproducible computer engineering using big data, predictive analytics and collective intelligence) (2011-2014)<br />
*[http://http://www.occamportal.org OCCAM project] - open curation for computer architecture modeling</div>Gfursinhttp://ctuning.org/cm/wiki/index.php?title=Reproducibility:Repositories&diff=825Reproducibility:Repositories2015-09-24T15:20:28Z<p>Gfursin: </p>
<hr />
<div>[ '''''[[Reproducibility|Back to main page]]'''''<br />
<br />
=== '''Assorted misc repositories'''<br/> ===<br />
<br />
*[http://github.com/ctuning/ck Collective Knowledge] - sharing and reusing artifacts with distributed ID via GitHub<br />
**[http://cknowledge.org/repo CK-based interactive article]<br />
**[http://github.com/ctuning/reproduce-clsmith PLDI'15 artifact converted to CK format]<br />
**[http://github.com/ctuning/reproduce-pamela-project Artifact repository for UK PAMELA project]<br />
**[http://github.com/ctuning/reproduce-carp-project Artifact repository for EU FP7 CARP project]<br />
**[http://github.com/ctuning/ctuning-programs Example of benchmarks shared in CK format]<br />
**[http://github.com/ctuning/ctuning-datasets-min Example of data sets shared in CK format]<br />
**[https://drive.google.com/folderview?id=0B-wXENVfIO82dzYwaUNIVElxaGc&usp=sharing Examples of CK repositories shared as zip files]<br />
*[http://gigadb.org (GIGA)n DB]<br />
*[http://galaxyproject.org Data Intensive Biology]<br />
*[http://www.galaxyzoo.org/?lang=en GalaxyZoo] (classification of galaxies)<br />
*[https://olivearchive.org Olive Archive] (preserving executable content)<br />
*[http://openscience.us/repo Tera-PROMISE<br/>]<br />
*[https://www.openaire.eu OpenAire (CERN)<br/>]<br />
*[https://zenodo.org Zenodo]<br />
*[http://researchcompendia.org ResearchCompendia<br/>]<br />
*[http://www.researchobject.org ResearchObject]<br />
*[http://www.researchobject.org/initiative Other initiatives related to ResearchObject]<br />
*[https://archive.org Internet Archive<br/>]<br />
*[http://www.nationalarchives.gov.uk The national archives<br/>]<br />
*[http://www.dcc.ac.uk Digital Curation Center]<br />
*[http://riojournal.com Research Ideas and Outcome Journal] - controversial<br />
*[http://www.wikidata.org/wiki/Wikidata:Introduction WikiData]<br />
*[http://www.dpn.org The digital preservation network<br/>]<br />
*[http://www.ddialliance.org Data Documentation Initiative]<br />
*[http://recomputation.org recomputation.org]<br />
*[https://open-data.europa.eu Open datasets<br/>]<br />
*[http://rmap-project.info/rmap RMap PROJECT] - preserve the many-to-many complex relationships among scholarly publications and their underlying data<br />
*[http://datahub.io DataHub]<br />
*[http://www.execandshare.org/CompanionSite www.execandshare.org] - creates a companion website associated with a submitted paper to implement the methodology presented in the paper<br />
*[https://www.datacite.org/contact Datacite] (citing data as DOI, Germany, has connections with CERN)<br />
*[http://www.crossref.org CrossRef]<br />
*[http://www.doi.org International DOI Foundation]<br />
*[http://c-mind.org/repo Collective Mind pilot live repository] (not supported, development moved to Collective Knowledge)<br />
*[http://ctuning.org/repo cTuning.org (performance tuning database, outdated)]<br />
*[http://http://www.occamportal.org OCCAM project] - open curation for computer architecture modeling</div>Gfursin