Difference between revisions of "Events:TRUST2014"

Latest revision as of 15:41, 25 June 2014

Important dates:

~~Submission: March 19, 2014~~
~~Notification: April 14, 2014~~
~~Final version: May 6, 2014~~
Early registration: May 7, 2014
~~Workshop: June 12, 2014~~

New publication model

[ ACM DL ] [ ArXiv ]

LinkedIn group

[ Link ]

Collective Mind project:

[ Live repository ], [ Framework ], [Discussions]

OCCAM project

[ Link ]

Special journal issue

IEEE TETC

[ Link ]

Artifact evaluation for software conferences

[ Link ]

Sponsors

ACM SIGPLAN

If your company or institution is interested to become our sponsor, please don't hesitate to contact us !

ACM SIGPLAN TRUST 2014 @ PLDI 2014

1st Workshop on Reproducible Research Methodologies
and New Publication Models in Computer Engineering

June 12, 2014 (afternoon), Edinburgh, UK

We particularly focus on technological aspects to enable reproducible research and experimentation on program and architecture empirical analysis, optimization, simulation, co-design and run-time adaptation.

Panel on conference artifact evaluation experience

Jan Vitek (Purdue University, USA), Shriram Krishnamurthi (Brown University, USA) - presenting and discussing the outcome of the Artifact Evaluation at PLDI'14 and other software conferences
Grigori Fursin (INRIA, France), Christophe Dubach (University of Edinburgh, UK) - discussing possibility to crowdsource validation of publications and research results

Final program

Proceedings in ACM Digital Library

1:30 - 1:35	Workshop introduction Grigori Fursin, INRIA, France
1:35 - 1:55	Introducing OCCAM project Bruce Chillders, University of Pittsburgh, USA
1:55 - 2:20	Software in reproducible research: advice and best practice collected from experiences at the Collaborations Workshop. Mario Antonioletti¹, Neil Chue Hong¹, Stephen Crouch², Alexander Hay², Simon Hettrick², Devasena Inupakutika², Mike Jackson¹, Aleksandra Pawlik³, Giacomo Peru¹, John Robinson², Shoaib Sufi³, Les Carr², David De Roure⁴, Carole Goble³, and Mark Parsons¹. ¹ EPCC, University of Edinburgh, UK ² WAIS, University of Southampton, UK ³ University of Manchester, UK ⁴ Oxford eResearch Centre, Oxford University, UK [ Slides ] [ Paper in ACM DL ]
2:20 - 2:45	Academia 2.0: removing the publisher middle-man while retaining impact Raphael Poss¹, Sebastian Altmeyer¹,Mark Thompson², Rob Jelier³ ¹ University of Amsterdam, Netherlands ² L.U.M.C., Netherlands ³ KU Leuven, Belgium [ Slides ] [ Paper in ACM DL ]
2:45 - 3:15	Coffee break (synchronized with all PLDI workshops)
3:15 - 3:50	Invited presentation: Sharing Specifications Christian Collberg & Todd Proebsting, University of Arizona, USA [ Slides ]
3:50 - 4:15	Falsifiability of network security research: the Good, the Bad, and the Ugly Dennis Gamayunov, Lomonosov Moscow State University, Russia [ Paper in ACM DL ]
4:15 - 4:40	CARE, the Comprehensive Archiver for Reproducible Execution Yves Janin, Cedric Vincent and Remi Duraffort STMicroelectronics, France [ Slides ] [ Paper in ACM DL ]
4:40 - 6:00	Panel on conference artifact evaluation experience Jan Vitek¹ , Shriram Krishnamurthi² , Christophe Dubach³ , Grigori Fursin⁴ ¹ Purdue University, USA ² Brown University, USA ³ University of Edinburgh, UK ⁴ INRIA, France [ Slides ] [ Paper in ACM DL ]

Workshop organizers

Grigori Fursin (INRIA, France)
Bruce Childers, Alex K.Jones and Daniel Mosse (University of Pittsburgh, USA)

Program committee

Jose Nelson Amaral (University of Alberta, Canada)
Calin Cascaval (Qualcomm, USA)
Jack Davidson (University of Virginia, USA)
Evelyn Duesterwald (IBM, USA)
Lieven Eeckhout (Ghent University, Belgium)
Eric Eide (University of Utah, USA)
Sebastian Fischmeister (University of Waterloo, Canada)
Michael Gerndt (TU Munich, Germany)
Christophe Guillon (STMicroelectronics, France)
Shriram Krishnamurthi (Brown University, USA)
Hugh Leather (University of Edinburgh, UK)
Anton Lokhmotov (ARM, UK)
Mikel Lujan (University of Manchester, UK)
David Padua (University of Illinois at Urbana-Champaign, USA)
Christoph Reichenbach (Johann-Wolfgang Goethe Universitat Frankfurt, Germany)
Arun Rodrigues (Sandia National Laboratories, USA)
Reiji Suda (University of Tokyo, Japan)
Sid Touati (INRIA, France)
Jesper Larsson Traff (Vienna University of Technology, Austria)
Petr Tuma (Charles University, Czech Republic)
Jan Vitek (Purdue University, USA)
Vladimir Voevodin (Moscow State University, Russia)
Vittorio Zaccaria (Politecnico di Milano, Italy)
Xiaoyun Zhu (VMware, USA)

Call for papers

It becomes excessively challenging or even impossible to capture, share and accurately reproduce experimental results in computer engineering for fair and trustable evaluation and future improvement. This is often due to ever rising complexity of the design, analysis and optimization of computer systems, increasing number of ad-hoc tools, interfaces and techniques, lack of a common experimental methodology, and lack of simple and unified mechanisms, tools and repositories to preserve and exchange knowledge apart from numerous publications where reproducibility is often not even considered. This ACM SIGPLAN workshop is intended to become an interdisciplinary forum for academic and industrial researchers, practitioners and developers in computer engineering to discuss challenges, ideas, experience, trustable and reproducible research methodologies, practical techniques, tools and repositories to:

capture, preserve, formalize, systematize, exchange and improve knowledge and experimental results including negative ones
describe and catalog whole experimental setups with all related material including algorithms, benchmarks, codelets, datasets, tools, models and any other artifact
validate and verify experimental results by the community
develop common research interfaces for existing or new tools
develop common experimental frameworks and repositories
share rare hardware and computational resources for experimental validation
deal with variability and rising amount of experimental data using statistical analysis, data mining, predictive modeling and other techniques
implement previously published experimental scenarios (auto-tuning, run-time adaptation) using common infrastructure
discuss open access to publications and data including intellectual property IP and legal issues
improve reviewing and evaluation process for publications and shared artifacts
enable interactive articles

Submission guidelines

Easychair submission website is now closed.

We invite papers in three categories (please use these prefixes for your submission title):

T1: Position papers should be at most 3 pages long (excluding bibliography). We welcome preliminary and exploratory work, presentation of related tools and repositories in development, experience reports particularly related to recent research validation initiatives at OOPSLA, PDLI and ADAPT, and any related ideas.
T2: Full papers should be at most 6 pages long (excluding bibliography). Papers in this category are expected to have relatively mature content.
T3: Papers validating and sharing past research on design and optimization of computer systems published in relevant conferences. These papers should be at most 6 pages long (excluding bibliography).

Submissions should be in PDF formatted with double column/single-spacing using 10pt fonts and printable on US letter or A4 sized paper using standard ACM LaTeX2e template. All papers will be peer-reviewed. The authors will have a choice to publish accepted papers at the conference website (that will not prevent later publication of extended papers) or in the ACM Digital Library (International Conference Proceedings Series; ISBN 978-1-4503-2918-7).

Important dates

~~Paper submission: March 19, 2014 (Anywhere on Earth)~~
~~Notification: April 14, 2014~~
~~Final version: May 6, 2014~~
Early registration: May 7, 2014 (link)
~~Workshop: June 12, 2014~~

Assorted related projects, initiatives and tools

Assorted tools

CARE tool from STMicroelectronics (Comprehensive Archiver for Reproducible Execution)
CDE tool (automatically create portable Linux applications with all dependencies)
Docker tool (pack, ship and run applications as a lightweight container)
IPython Notebook (a web-based interactive computational environment where you can combine code execution, text, mathematics, plots and rich media into a single document)
cTuning technology (crowdsource auto-tuning and combine with machine learning) (2006-cur.)
Collective Mind technology (towards collaborative, systematic and reproducible computer engineering) (2011-cur.)

@@ Line 2: / Line 2: @@
 |- valign="top"
 | width="170" |
-[[File:Logo-validated-by-the-community.png|none|Logo-validated-by-the-community.png]]
+[[File:ACM-sigplan.png|none|ACM-sigplan.png|link=http://sigplan.org]] [[File:Logo-validated-by-the-community.png|none|Logo-validated-by-the-community.png|link=http://cTuning.org/reproducibility]] [[File:Edinburgh.jpg|none|Edinburgh.jpg|link=http://en.wikipedia.org/wiki/Edinburgh]]
-[[File:ACM-sigplan.png|none|ACM-sigplan.png]]
-[[File:Edinburgh.jpg|none|Edinburgh.jpg|link=http://en.wikipedia.org/wiki/Edinburgh]]
 <p style="text-align: center"></p><p style="text-align: center"><span style="font-size:medium">'''Important dates:'''</span></p>
-<strike><span style="font-size:small"><span style="color:#ff0000">Submission: March 19, 2014</span></span></strike><br/><span style="font-size:small">Notification: April 14, 2014<br/>Final version: May 2, 2014</span><br/>[https://regmaster4.com/2014conf/PLDI14/register.php Early registration]: May 7, 2014<br/><span style="font-size:small">Workshop: June 12, 2014</span>
+<strike><span style="font-size:small">Submission: March 19, 2014</span></strike><br/><span style="font-size:small"><strike>Notification: April 14, 2014</strike><br/><strike>Final version: May 6, 2014</strike></span><br/><strike>[https://regmaster4.com/2014conf/PLDI14/register.php Early registration]: '''<span style="color:#ff0000">May 7, 2014</span>'''</strike><br/><strike><span style="font-size:small">Workshop: June 12, 2014</span></strike>
 ----
-<p style="text-align: center"><span style="font-size:small"><span style="font-size:medium">'''LinkedIn group'''</span></span></p><p style="text-align: center"><span style="font-size:small">[ [http://www.linkedin.com/groups/Reproducible-research-experimentation-in-computer-7433414 Link] ]</span></p><p style="text-align: center"><span style="font-size:small"><span style="font-size:medium">'''Collective Mind project:'''</span></span></p><p style="text-align: center"><span style="font-size:small">[[File:C-mind.org.png|Collective Mind|link=http://c-mind.org]]<br/>[&nbsp;[http://c-mind.org/repo Live repository]&nbsp;], [&nbsp;[http://cTuning.org/tools/cm Framework]&nbsp;]</span>, [[http://cTuning.org/reproducibility Discussions]]</p><p style="text-align: center"><span style="font-size:small"><span style="font-size:medium">'''OCCAM project'''</span></span></p><p style="text-align: center"><span style="font-size:small">[ [http://www.occamportal.org Link] ]</span></p><p style="text-align: center"><span style="font-size:small"><span style="font-size:medium">'''Special journal issue&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; IEEE TETC'''</span></span></p><p style="text-align: center"><span style="font-size:small">[ [http://mc.manuscriptcentral.com/tetc-cs Link] ]</span></p><p style="text-align: center"><span style="font-size:small"><span style="font-size:medium">'''Artifact evaluation for software conferences'''</span></span></p><p style="text-align: center"><span style="font-size:small">[ [http://www.artifact-eval.org Link] ]</span></p>
+<p style="text-align: center"><span style="font-size:small"><span style="font-size:medium">'''New publication model'''</span></span></p><p style="text-align: center"><span style="font-size:small">[ [http://dl.acm.org/citation.cfm?doid=2618137.2618142 ACM DL] ]</span> [ [http://fr.arxiv.org/abs/1406.4020 ArXiv] ]</p><p style="text-align: center"><span style="font-size:small"><span style="font-size:medium">'''LinkedIn group'''</span></span></p><p style="text-align: center"><span style="font-size:small">[ [http://www.linkedin.com/groups/Reproducible-research-experimentation-in-computer-7433414 Link] ]</span></p><p style="text-align: center"><span style="font-size:small"><span style="font-size:medium">'''Collective Mind project:'''</span></span></p><p style="text-align: center"><span style="font-size:small">[[File:C-mind.org.png|Collective Mind|link=http://c-mind.org]]<br/>[&nbsp;[http://c-mind.org/repo Live repository]&nbsp;], [&nbsp;[http://cTuning.org/tools/cm Framework]&nbsp;]</span>, [[http://cTuning.org/reproducibility Discussions]]</p><p style="text-align: center"><span style="font-size:small"><span style="font-size:medium">'''OCCAM project'''</span></span></p><p style="text-align: center"><span style="font-size:small">[ [http://www.occamportal.org Link] ]</span></p><p style="text-align: center"><span style="font-size:small"><span style="font-size:medium">'''Special journal issue'''</span></span></p><p style="text-align: center"><span style="font-size:small"><span style="font-size:medium">'''IEEE TETC'''</span></span></p><p style="text-align: center"><span style="font-size:small">[ [http://mc.manuscriptcentral.com/tetc-cs Link] ]</span></p><p style="text-align: center"><span style="font-size:small"><span style="font-size:medium">'''Artifact evaluation for software conferences'''</span></span></p><p style="text-align: center"><span style="font-size:small">[ [http://www.artifact-eval.org Link] ]</span></p>
 ----
 <p style="text-align: center"><span style="font-size:small"><span style="font-size:medium">'''Sponsors'''</span></span></p><p style="text-align: center"><span style="font-size:small">[http://www.sigplan.org ACM SIGPLAN]</span></p>
@@ Line 16: / Line 14: @@
-| <p style="text-align: center"><span style="font-size:small">'''<span style="font-size:x-large">TRUST 2014</span>'''</span></p><p style="text-align: center">'''<span style="color:#b22222">You can now register for TRUST and other PLDI events [https://regmaster4.com/2014conf/PLDI14/register.php here]</span><span style="color:#b22222">! Looking forward to see you in Edinburgh!</span>'''</p><p style="text-align: center"><span style="font-size:small"><span style="font-size:medium">1st ACM SIGPLAN Workshop on Reproducible Research Methodologies<br/>and New Publication Models in Computer Engineering</span>&nbsp;</span></p><p style="text-align: center"><span style="font-size:small">co-located with [http://conferences.inf.ed.ac.uk/pldi2014 ACM PLDI 2014]</span></p><p style="text-align: center">'''<span style="font-size:small">''June 12, 2014 (afternoon), Edinburgh, UK''</span>'''</p><p style="text-align: center"></p><p style="text-align: center"><span style="font-size: small">We particularly focus on technological aspects to enable reproducible research and experimentation on program and architecture empirical analysis, optimization, simulation, co-design and run-time adaptation.</span></p>
+| <p style="text-align: center"><span style="font-size:small">'''<span style="font-size:x-large">ACM SIGPLAN TRUST 2014 @ PLDI 2014</span>'''</span></p><p style="text-align: center"><span style="font-size:small"><span style="font-size:medium">1st Workshop on Reproducible Research Methodologies<br/>and New Publication Models in Computer Engineering</span>&nbsp;</span></p><p style="text-align: center">'''<span style="font-size:small">''June 12, 2014 (afternoon), Edinburgh, UK''</span>'''</p><p style="text-align: center"></p><p style="text-align: center"><span style="font-size: small">We particularly focus on technological aspects to enable reproducible research and experimentation on program and architecture empirical analysis, optimization, simulation, co-design and run-time adaptation.</span></p>
 <span style="font-size:small">'''<span style="font-size:large">Panel on conference artifact evaluation experience</span>'''</span>
 *[https://www.cs.purdue.edu/homes/jv Jan Vitek] (Purdue University, USA), [http://cs.brown.edu/~sk/ Shriram Krishnamurthi ](Brown University, USA) - presenting and discussing the outcome of the&nbsp;<span style="font-size:small">[http://www.artifact-eval.org Artifact Evaluation at PLDI]</span><span style="font-size:small">[http://www.artifact-eval.org '14 and other software conferences]</span>
-*<span style="font-size:small">[http://cTuning.org/lab/people/gfursin Grigori Fursin ](INRIA, France), [http://homepages.inf.ed.ac.uk/cdubach Christophe Dubach]</span> (University of Edinburgh, UK) - presenting their experience with the [http://adapt-workshop.org/2014/program.htm ADAPT'14 experiment validation] and discussing possible technical solutions
+*<span style="font-size:small">[http://cTuning.org/lab/people/gfursin Grigori Fursin ](INRIA, France), [http://homepages.inf.ed.ac.uk/cdubach Christophe Dubach]</span> (University of Edinburgh, UK) - discussing possibility to crowdsource validation of publications and research results
-<span style="color:#b22222">''Please, do not forget to register at the TRUST workshop ''</span>''[https://regmaster4.com/2014conf/PLDI14/register.php here]''<span style="color:#b22222">'' (early registration deadline is 7th of May)!''</span>
+<span style="font-size:small">'''<span style="font-size:large">Final program</span>'''</span>
+<p style="text-align: center">[http://dl.acm.org/citation.cfm?id=2618137 Proceedings in ACM Digital Library]</p>
+{| style="background:#FFFFFF;  color:black" cellpadding="10" cellspacing="0" border="1" width="100%"
+|- style="background:#CFCFFF;  color:black"
+| style="width:70px" valign="top" | 1:30 - 1:35<br/>
+| valign="top" | '''Workshop introduction'''
+''<span style="font-size:small">[http://cTuning.org/lab/people/gfursin Grigori Fursin]</span>, ''INRIA, France
-<span style="font-size:small">'''<span style="font-size:large">Program</span>'''</span>
+|-
+| style="width:70px" valign="top" | 1:35 - 1:55<br/>
+| valign="top" | '''[http://www.occamportal.org Introducing OCCAM project]'''
+''[http://people.cs.pitt.edu/~childers Bruce Chillders], ''University of Pittsburgh, USA
-''Under preparation''
+|-
+| style="width:70px" valign="top" | 1:55 - 2:20<br/>
+| valign="top" | '''Software in reproducible research: advice and best practice collected from experiences at the [http://www.software.ac.uk/collaborations-workshop-2014-cw14-software-your-reproducible-research Collaborations Workshop].'''
+''Mario Antonioletti<sup>1</sup>, Neil Chue Hong''<sup>1</sup>'', Stephen Crouch''<sup>2</sup>'', Alexander Hay<sup>2</sup>, Simon Hettrick''<sup>2</sup>'', Devasena Inupakutika''<sup>2</sup>'', Mike Jackson''<sup>1</sup>'', Aleksandra Pawlik''<sup>3</sup>'', Giacomo Peru''<sup>1</sup>'', John Robinson''<sup>2</sup>'', Shoaib Sufi''<sup>3</sup>'', Les Carr''<sup>2</sup>'', David De Roure<sup>4</sup>, Carole Goble<sup>3</sup>, and Mark Parsons''<sup>1</sup>''.''
-<span style="font-size:small">'''<span style="font-size:large">Call for papers</span>'''</span>
+<sup>1</sup> EPCC, University of Edinburgh, UK<br/><sup>2</sup> WAIS, University of Southampton, UK<br/><sup>3</sup> University of Manchester, UK<br/><sup>4</sup> Oxford eResearch Centre, Oxford University, UK
-<span style="font-size:small">It becomes excessively challenging or even impossible to capture, share and accurately reproduce experimental results in computer engineering for fair and trustable evaluation and future improvement. This is often due to ever rising complexity of the design, analysis and optimization of computer systems, increasing number of ad-hoc tools, interfaces and techniques, lack of a common experimental methodology, and lack of simple and unified mechanisms, tools and repositories to preserve and exchange knowledge apart from numerous publications where reproducibility is often not even considered. This ACM SIGPLAN workshop is intended to become an interdisciplinary forum for academic and industrial researchers, practitioners and developers in computer engineering to discuss challenges, ideas, experience, trustable and reproducible research methodologies, practical techniques, tools and repositories to:</span>
+[ [http://c-mind.org/events/trust2014/presentations/trust14_ssi_4.pdf Slides] ] [ [http://dl.acm.org/citation.cfm?doid=2618137.2618140 Paper in ACM DL] ]
-*<span style="font-size:small">capture, preserve, formalize, systematize, exchange and improve knowledge and experimental results including negative ones</span>
+|-
-*<span style="font-size:small">describe and catalog whole experimental setups with all related material including algorithms, benchmarks, codelets, datasets, tools, models and any other artifact</span>
+| style="width:70px" valign="top" | 2:20 - 2:45<br/>
-*<span style="font-size:small">validate and verify experimental results by the community</span>
+| valign="top" | '''Academia 2.0: removing the publisher middle-man while retaining impact'''
-*<span style="font-size:small">develop common research interfaces for existing or new tools</span>
+''Raphael Poss<sup>1</sup>, Sebastian Altmeyer<sup>1</sup>,Mark Thompson<sup>2</sup>, Rob Jelier<sup>3</sup>''<br/><sup>1</sup> University of Amsterdam, Netherlands<br/><sup>2</sup> L.U.M.C., Netherlands<br/><sup>3</sup> KU Leuven, Belgium
-*<span style="font-size:small">develop common experimental frameworks and repositories</span>
-*<span style="font-size:small">share rare hardware and computational resources for experimental validation</span>
-*<span style="font-size:small">deal with variability and rising amount of experimental data using statistical analysis, data mining, predictive modeling and other techniques</span>
-*<span style="font-size:small">implement previously published experimental scenarios (auto-tuning, run-time adaptation) using common infrastructure</span>
-*<span style="font-size:small">implement open access to publications and data (particularly discussing intellectual property IP and legal issues)</span>
-*<span style="font-size:small">improve reviewing and evaluation process for publications and shared artifacts</span>
-*<span style="font-size:small">enable interactive articles</span>
-<span style="font-size:small">'''<span style="font-size:large">Submission guidelines</span>'''</span>
+[ [http://c-mind.org/events/trust2014/presentations/trust14_academia.pdf Slides] ] [ [http://dl.acm.org/citation.cfm?doid=2618137.2618141 Paper in ACM DL] ]
-<span style="font-size:small">'''''Easychair submission website is now [https://www.easychair.org/conferences/?conf=trust20140 closed]''.'''</span>
+|- style="background:#CFCFFF;  color:black"
+| style="width:70px" valign="top" | 2:45 - 3:15<br/>
+| valign="top" | Coffee break (synchronized with all PLDI workshops)<br/>
+|-
+| style="width:70px" valign="top" | 3:15 - 3:50<br/>
+| valign="top" | '''Invited presentation: Sharing Specifications'''<br/>
+''Christian Collberg & Todd Proebsting, ''University of Arizona, USA
-<span style="font-size:small">We invite papers in three categories (''please use these prefixes for your submission title''):</span>
+[ [http://reproducibility.cs.arizona.edu/TRUST Slides] ]
-*<span style="font-size:small">'''T1:''' Position papers should be at most 3 pages long (excluding bibliography). We welcome preliminary and exploratory work, presentation of related tools and repositories in development, experience reports particularly related to recent research validation initiatives at OOPSLA, PDLI and ADAPT, and any related ideas.</span>
+|-
-*<span style="font-size:small">'''T2:''' Full papers should be at most 6 pages long (excluding bibliography). Papers in this category are expected to have relatively mature content.</span>
+| style="width:70px" valign="top" | 3:50 - 4:15<br/>
-*<span style="font-size:small">'''T3:''' Papers validating and sharing past research on design and optimization of computer systems published in relevant conferences. These papers should be at most 6 pages long (excluding bibliography).</span>
+| valign="top" | '''Falsifiability of network security research: the Good, the Bad, and the Ugly'''
+''Dennis Gamayunov, ''Lomonosov Moscow State University, Russia
-<span style="font-size:small">Submissions should be in PDF formatted with double column/single-spacing using 10pt fonts and printable on US letter or A4 sized paper. All papers will be peer-reviewed. Accepted papers can be published online on the conference website that will not prevent later publication of extended papers. We currently arrange proceedings to be published in the ACM Digital Library.</span>
+[ [http://dl.acm.org/citation.cfm?doid=2618137.2618141 Paper in ACM DL] ]
-<span style="font-size:small">'''<span style="font-size:large">Important dates</span>'''</span>
+|-
+| style="width:70px" valign="top" | 4:15 - 4:40<br/>
+| valign="top" | '''[http://reproducible.io CARE], the Comprehensive Archiver for Reproducible Execution'''
+''Yves Janin, Cedric Vincent and Remi Duraffort''<br/>STMicroelectronics, France
+[ [http://c-mind.org/events/trust2014/presentations/trust14_care.pdf Slides] ] [ [http://dl.acm.org/citation.cfm?doid=2618137.2618138 Paper in ACM DL] ]
+|- style="background:#CFCFFF;  color:black"
+| style="width:70px" valign="top" | 4:40 - 6:00<br/>
+| valign="top" | '''Panel on [http://www.artifact-eval.org conference artifact evaluation] experience'''
+[https://www.cs.purdue.edu/homes/jv Jan Vitek]<sup>1</sup> , [http://cs.brown.edu/%7Esk/ Shriram Krishnamurthi]<sup>2</sup> ,&nbsp; [http://homepages.inf.ed.ac.uk/cdubach Christophe Dubach]<sup>3</sup> , [http://cTuning.org/lab/people/gfursin Grigori Fursin]<sup>4</sup>
+<sup>1</sup> Purdue University, USA <sup>2</sup> Brown University, USA <sup>3</sup> University of Edinburgh, UK <sup>4</sup> INRIA, France
+[ [http://www.slideshare.net/GrigoriFursin/panel-at-acmsigplantrust2014 Slides] ] [ [http://dl.acm.org/citation.cfm?doid=2618137.2618142 Paper in ACM DL] ]
-*<strike><span style="font-size:small"><span style="color:#ff0000">'''''Paper submission: March 19, 2014 (Anywhere on Earth)'''''</span></span></strike>
+|}
-*<span style="font-size:small">Notification: April 14, 2014</span>
-*<span style="font-size:small">Final version: May 2, 2014</span>
-*<span style="font-size:small">Early registration: May 7, 2014 ([https://regmaster4.com/2014conf/PLDI14/register.php link])</span>
-*<span style="font-size:small">Workshop: June 12, 2014</span>
 <span style="font-size:small">'''<span style="font-size:large">Workshop organizers</span>'''</span>
 *<span style="font-size:small">[http://cTuning.org/lab/people/gfursin Grigori Fursin] (INRIA, France)</span>
-*<span style="font-size:small">[http://people.cs.pitt.edu/~childers Bruce Childers], [http://www.pitt.edu/~akjones Alex K.Jones] and [http://people.cs.pitt.edu/~mosse Daniel Mosse] (University of Pittsburgh, USA)</span>
+*<span style="font-size:small">[http://people.cs.pitt.edu/%7Echilders Bruce Childers], [http://www.pitt.edu/%7Eakjones Alex K.Jones] and [http://people.cs.pitt.edu/%7Emosse Daniel Mosse] (University of Pittsburgh, USA)</span>
 <span style="font-size:small">'''<span style="font-size:large">Program committee</span>'''</span>
-*<span style="font-size:small">[http://webdocs.cs.ualberta.ca/~amaral Jose Nelson Amaral] (University of Alberta, Canada)</span>
+*<span style="font-size:small">[http://webdocs.cs.ualberta.ca/%7Eamaral Jose Nelson Amaral] (University of Alberta, Canada)</span>
 *<span style="font-size:small">[http://www.cascaval.org/calin Calin Cascaval] (Qualcomm, USA)</span>
-*<span style="font-size:small">[http://www.cs.virginia.edu/~jwd Jack Davidson] (University of Virginia, USA)</span>
+*<span style="font-size:small">[http://www.cs.virginia.edu/%7Ejwd Jack Davidson] (University of Virginia, USA)</span>
 *<span style="font-size:small">[http://researcher.watson.ibm.com/researcher/view.php?person=us-duester Evelyn Duesterwald] (IBM, USA)</span>
-*<span style="font-size:small">[http://users.elis.ugent.be/~leeckhou Lieven Eeckhout] (Ghent University, Belgium)</span>
+*<span style="font-size:small">[http://users.elis.ugent.be/%7Eleeckhou Lieven Eeckhout] (Ghent University, Belgium)</span>
-*<span style="font-size:small">[https://www.cs.utah.edu/~eeide Eric Eide] (University of Utah, USA)</span>
+*<span style="font-size:small">[https://www.cs.utah.edu/%7Eeeide Eric Eide] (University of Utah, USA)</span>
 *<span style="font-size:small">[https://uwaterloo.ca/embedded-software-group/people-profiles/sebastian-fischmeister Sebastian Fischmeister] (University of Waterloo, Canada)</span>
-*<span style="font-size:small">[http://www.lrr.in.tum.de/~gerndt Michael Gerndt] (TU Munich, Germany)</span>
+*<span style="font-size:small">[http://www.lrr.in.tum.de/%7Egerndt Michael Gerndt] (TU Munich, Germany)</span>
 *<span style="font-size:small">[http://fr.linkedin.com/pub/christophe-guillon/4/387/75a Christophe Guillon] (STMicroelectronics, France)</span>
-*<span style="font-size:small">[http://cs.brown.edu/~sk Shriram Krishnamurthi] (Brown University, USA)</span>
+*<span style="font-size:small">[http://cs.brown.edu/%7Esk Shriram Krishnamurthi] (Brown University, USA)</span>
 *<span style="font-size:small">[http://homepages.inf.ed.ac.uk/hleather Hugh Leather] (University of Edinburgh, UK)</span>
 *<span style="font-size:small">[http://uk.linkedin.com/in/lokhmotov Anton Lokhmotov] (ARM, UK)</span>
 *<span style="font-size:small">[http://apt.cs.man.ac.uk/people/mlujan Mikel Lujan] (University of Manchester, UK)</span>
-*<span style="font-size:small">[http://polaris.cs.uiuc.edu/~padua David Padua] (University of Illinois at Urbana-Champaign, USA)</span>
+*<span style="font-size:small">[http://polaris.cs.uiuc.edu/%7Epadua David Padua] (University of Illinois at Urbana-Champaign, USA)</span>
 *<span style="font-size:small">[http://creichen.net Christoph Reichenbach] (Johann-Wolfgang Goethe Universitat Frankfurt, Germany)</span>
 *<span style="font-size:small">[http://www.researchgate.net/profile/Arun_Rodrigues Arun Rodrigues] (Sandia National Laboratories, USA)</span>
-*<span style="font-size:small">[http://olab.is.s.u-tokyo.ac.jp/~reiji Reiji Suda] (University of Tokyo, Japan)</span>
+*<span style="font-size:small">[http://olab.is.s.u-tokyo.ac.jp/%7Ereiji Reiji Suda] (University of Tokyo, Japan)</span>
-*<span style="font-size:small">[http://www-sop.inria.fr/members/Sid.Touati/ Sid Touat][http://www-sop.inria.fr/members/Sid.Touati i] (INRIA, France)</span>
+*<span style="font-size:small">[http://www-sop.inria.fr/members/Sid.Touati/ Sid Touati] (INRIA, France)</span>
 *<span style="font-size:small">[http://www.par.tuwien.ac.at/traff.html Jesper Larsson Traff] (Vienna University of Technology, Austria)</span>
-*<span style="font-size:small">[http://d3s.mff.cuni.cz/~ceres Petr Tuma] (Charles University, Czech Republic)</span>
+*<span style="font-size:small">[http://d3s.mff.cuni.cz/%7Eceres Petr Tuma] (Charles University, Czech Republic)</span>
 *<span style="font-size:small">[https://www.cs.purdue.edu/homes/jv Jan Vitek] (Purdue University, USA)</span>
 *<span style="font-size:small">[http://parallel.ru/en/person/vvv Vladimir Voevodin] (Moscow State University, Russia)</span>
@@ Line 96: / Line 116: @@
 *<span style="font-size:small">[http://www.linkedin.com/in/xiaoyunzhu Xiaoyun Zhu] (VMware, USA)</span>
-<span style="font-size:small">'''<span style="font-size:large">Assorted related projects and initiatives</span>'''&nbsp;</span>
+<span style="font-size:small">'''<span style="font-size:large">Call for papers</span>'''</span>
+<span style="font-size:small">It becomes excessively challenging or even impossible to capture, share and accurately reproduce experimental results in computer engineering for fair and trustable evaluation and future improvement. This is often due to ever rising complexity of the design, analysis and optimization of computer systems, increasing number of ad-hoc tools, interfaces and techniques, lack of a common experimental methodology, and lack of simple and unified mechanisms, tools and repositories to preserve and exchange knowledge apart from numerous publications where reproducibility is often not even considered. This ACM SIGPLAN workshop is intended to become an interdisciplinary forum for academic and industrial researchers, practitioners and developers in computer engineering to discuss challenges, ideas, experience, trustable and reproducible research methodologies, practical techniques, tools and repositories to:</span>
+*<span style="font-size:small">capture, preserve, formalize, systematize, exchange and improve knowledge and experimental results including negative ones</span>
+*<span style="font-size:small">describe and catalog whole experimental setups with all related material including algorithms, benchmarks, codelets, datasets, tools, models and any other artifact</span>
+*<span style="font-size:small">validate and verify experimental results by the community</span>
+*<span style="font-size:small">develop common research interfaces for existing or new tools</span>
+*<span style="font-size:small">develop common experimental frameworks and repositories</span>
+*<span style="font-size:small">share rare hardware and computational resources for experimental validation</span>
+*<span style="font-size:small">deal with variability and rising amount of experimental data using statistical analysis, data mining, predictive modeling and other techniques</span>
+*<span style="font-size:small">implement previously published experimental scenarios (auto-tuning, run-time adaptation) using common infrastructure</span>
+*discuss open access to publications and data including intellectual property IP and legal issues
+*<span style="font-size:small">improve reviewing and evaluation process for publications and shared artifacts</span>
+*<span style="font-size:small">enable interactive articles</span>
+<span style="font-size:small">'''<span style="font-size:large">Submission guidelines</span>'''</span>
+<span style="font-size:small">'''''Easychair submission website is now [https://www.easychair.org/conferences/?conf=trust20140 closed]''.'''</span>
+<span style="font-size:small">We invite papers in three categories (''please use these prefixes for your submission title''):</span>
+*<span style="font-size:small">'''T1:''' Position papers should be at most 3 pages long (excluding bibliography). We welcome preliminary and exploratory work, presentation of related tools and repositories in development, experience reports particularly related to recent research validation initiatives at OOPSLA, PDLI and ADAPT, and any related ideas.</span>
+*<span style="font-size:small">'''T2:''' Full papers should be at most 6 pages long (excluding bibliography). Papers in this category are expected to have relatively mature content.</span>
+*<span style="font-size:small">'''T3:''' Papers validating and sharing past research on design and optimization of computer systems published in relevant conferences. These papers should be at most 6 pages long (excluding bibliography).</span>
+<span style="font-size:small">Submissions should be in PDF formatted with double column/single-spacing using 10pt fonts and printable on US letter or A4 sized paper using [http://www.acm.org/sigs/publications/proceedings-templates standard ACM LaTeX2e template]. All papers will be peer-reviewed. The authors will have a choice to publish accepted papers at the conference website (that will not prevent later publication of extended papers) or in the ACM Digital Library (International Conference Proceedings Series; ISBN 978-1-4503-2918-7).</span>
+<span style="font-size:small">'''<span style="font-size:large">Important dates</span>'''</span>
+*<strike><span style="font-size:small"><span style="color:#ff0000">Paper submission: March 19, 2014 (Anywhere on Earth)</span></span></strike>
+*<strike><span style="font-size:small">Notification: April 14, 2014</span></strike>
+*<strike><span style="font-size:small">Final version: May 6, 2014</span></strike>
+*<span style="font-size:small"><strike>Early registration: <span style="color:#ff0000">'''May 7, 2014'''</span></strike> ([https://regmaster4.com/2014conf/PLDI14/register.php link])</span>
+*<strike><span style="font-size:small">Workshop: June 12, 2014</span></strike>
+<span style="font-size:small">'''<span style="font-size:large">Assorted related projects, initiatives and tools</span>'''&nbsp;</span>
-*<span style="font-size:small">[http://www.artifact-eval.org Artifact Evaluation for software conferences]</span>
+*[http://dl.acm.org/citation.cfm?id=2618142 Our new proposal to crowdsource reviewing of publications and artifacts]
+*[http://www.artifact-eval.org Artifact Evaluation for software conferences]&nbsp;; [http://www.artifact-eval.org/guidelines.html Guidelines for packaging artifacts for submissions]
 *<span style="font-size:small">[http://adapt-workshop.org/program.htm ADAPT panel on reproducible research methodologies and new publication models] (January 2014)</span>
 *<span style="font-size:small">[http://hal.inria.fr/inria-00436029/ GCC Summit'09 discussion]</span>
 *<span style="font-size:small">[http://www.hipeac.net/thematic-session/making-computer-engineering-science HiPEAC thematic session on making computer engineering a science] (2013)</span>
 *<span style="font-size:small">[http://www.occamportal.org/reproduce/11-community REPRODUCE'14 workshop at HPCA'14] (February 2014)</span>
-*<span style="font-size:small">[http://c-mind.org/repo Collective Mind technology for collaborative, systematic and reproducible computer engineering]</span>
-*<span style="font-size:small">[http://hal.inria.fr/inria-00436029 cTuning technology to crowdsource auto-tuning and combine with machine learning] (2006-2011)</span>
 *<span style="font-size:small">[http://www.occamportal.org OCCAM project for reproducible computer architecture simulation]</span>
 *<span style="font-size:small">[http://evaluate.inf.usi.ch Evaluate project]</span>
-*<span style="font-size:small">[http://reproducible.io CARE tool from STMicroelectronics (Comprehensive Archiver for Reproducible Execution)]</span>
 *[http://www.eecg.toronto.edu/~enright/wddd WDDD (Workshop on Duplicating, Deconstructing, and Debunking)]
 *[http://www.stodden.net/AMP2011 AMP workshop 2011 (Reproducible Research: Tools and Strategies for Scientific Computing)]
@@ Line 115: / Line 169: @@
 *[http://reproducibleresearch.net/blog Reproducible Research blog]
 *[http://reproduciblescience.org Reproducible Science project]
+*[http://www.jstatsoft.org Journal of Statistical Software]
+*[http://matt.might.net/articles/crapl CRAPL license]
+*[http://dl.acm.org/citation.cfm?id=1536616.1536619 "Begin with an Author's response to Reviews" - Proposal to improve paper reviewing (submitting past reviews)]
 *[https://www.dropbox.com/s/67n49i9k3cb28mo/GOBLE-CW2014.ppt Presentation by Professor Carole Globe]
+*[http://www.structured-commons.org Structured Commons initiative]
 *[http://reproducibility.cs.arizona.edu Reproducibility in Computer Science]
+<span style="font-size:small">'''<span style="font-size:large">Assorted tools</span>'''</span>
+*<span style="font-size:small">[http://reproducible.io CARE tool from STMicroelectronics] (Comprehensive Archiver for Reproducible Execution)</span>
+*[http://www.pgbovine.net/cde.html CDE tool] (automatically create portable Linux applications with all dependencies)
+*[https://www.docker.io Docker tool] (pack, ship and run applications as a lightweight container)
+*[http://ipython.org/notebook.html IPython Notebook] (a web-based interactive computational environment where you can combine code execution, text, mathematics, plots and rich media into a single document)
+*<span style="font-size:small">[http://hal.inria.fr/inria-00436029 cTuning technology] (crowdsource auto-tuning and combine with machine learning) (2006-cur.)</span>
+*<span style="font-size:small">[http://c-mind.org/repo Collective Mind technology] (towards collaborative, systematic and reproducible computer engineering</span>) (2011-cur.)
 |}