Discussions:New Publication Model

From cTuning.org

(Difference between revisions)

Current revision

This is archive. We moved to c-mind.org/reproducibility ...

This wiki is dedicated to:

New publication model for computer engineering that favors reproducible experimental results validated by the community
New plugin-based open source collaborative R&D infrastructure and repository to systematize design and optimization of computer systems

If you have questions, comments or suggestions, interested to participate in the future events or would like to join this effort, please get in touch with Grigori Fursin.

Designing novel many-core computer systems becomes intolerably complex, ad-hoc, costly and error prone due to limitations of available technology, enormous number of available design and optimization choices, and complex interactions between all software and hardware components. At the same time, research and development methodology for computer systems has hardly changed in the past decades. Users and developers often have to resort to non-scientific, non-systematic, non-rigorous, intuitive and error-prone methods combined with multiple ad-hoc tools, often limited, non-representative and simply outdated benchmarks and datasets, complex interfaces and data formats to select the most appropriate solution that satisfies all their needs. Such an outdated technology results in an enormous waste of expensive computing resources and energy, and considerably increases development costs and time-to-market for the new systems.

Empirical auto-tuning combined with run-time adaptation and machine learning has been demonstrating good potential to address above challenges for more than a decade but still far from the widespread production use due to unbearably long exploration and training times, ever changing tools and their interfaces, lack of a common experimental methodology, and lack of unified mechanisms for knowledge building and exchange apart from publications where reproducibility of results is often not even considered.

We strongly believe that the time has come to start collaborative systematization and unification of design and optimization of computer systems combined with a new publication model where experimental results are validated by the community. One of the possible promising solutions is to combine public repository of knowledge with online auto-tuning, machine learning and crowdsourcing techniques where HiPEAC and cTuning communities already have a good practical experience. Such collaborative approach should allow community to continuously validate, systematize and improve collective knowledge about computer systems, and extrapolate it to build faster, more power efficient and reliable computer systems. Furthermore, it should be able to suggest researchers and engineers where to focus their effort and creativity when designing or optimizing computer systems, thus boosting innovation and dramatically reducing development and optimization costs, and time to market for new systems. It can also help to restore the attractiveness of computer engineering making it a more systematic and rigorous discipline rather than "hacking".

We are building an academic and industrial workgroup interested in:

building common extensible HiPEAC repository and infrastructure (possibly based on already existing modular cTuning/Collective Mind technology) to collect statistics, benchmarks, codelets, data sets, tools, auto-tuning and run-time adaptation plugins, and predictive models from the community
systematizing and unifying optimization, design space exploration and run-time adaptation techniques (co-design and auto-tuning)
collaboratively evaluating and improving various data mining, classification and predictive modeling techniques for off-line and on-line auto-tuning
substituting ad-hoc and possibly outdated benchmarks and data sets with realistic and representative ones from the community
preparing new publication model (workshops, conferences, journals) with validation of experimental results by the community

Events

22/01/2014 - Panel at ADAPT2014
2-3/05/2013 - Thematic session on "making computer engineering a science" at HiPEAC Computing Week/ACM ECRC 2013 in Paris, France
- Meeting notes
24-25/04/2012 - Thematic session on "Collective characterization, optimization and design of computer systems" at HiPEAC Computing Week 2012 in Göteborg, Sweden
3/04/2012 - ASPLOS/EXADAPT 2012 panel on "Collaborative research methodology for large-scale computer systems", London, UK

Comments

Grigori: from our experience with sharing our experimental results and tools or validating existing results (such as at ADAPT'14), just packing and sharing your artifacts is not enough - we would like to have them as reusable components with defined API and meta-description.

Grigori: During conference on auto-tuning in NTU (Taiwan), colleagues mentioned a on-line journal in statistics where algorithm and datasets are submitted along publication to validate results by the PC/community. Later we also had more discussions at HiPEAC computing week in Paris and we all agreed that we can't simply reproduce the same scheme since experimental setups are usually much more complex in computer engineering (design, optimization and run-time adaptation) while there is still no clear methodology for reproducibility (for example, how to deal with general variations in computer system behavior across different users, setups or runs). That's why we believe that we need a common collaborative R&D repository and infrastructure to reproduce and validate experiments across many users, architectures, tools, data sets. That is exact reason why I have been designing open source plugin-based cTuning and Collective Mind repository and infrastructure which allows to build customized experimental setups while supporting most of the architectures, operating systems (including off-the-shelf Android mobiles), compilers, run-time systems, benchmarks, data sets, etc. It is now available for preview and we are now testing it in a few industrial and academic projects before pre-release to continue collaborative and public discussions.

Thanks

We would like to thank HiPEAC for funding our meetings and all colleagues who provided interesting feedback on cTuning/Collective Mind technology, and evaluated, validated or extended various collaborative tools, benchmarks, data sets and predictive models for program and architecture optimization!

@@ Line 1: / Line 1: @@
-''New publication model for computer engineering that favors reproducible experimental results validated by the community''
+'''''This is archive. We moved to [http://c-mind.org/reproducibility c-mind.org/reproducibility] ...'''''
-With our background in physics, we found it extremely disappointing that reproducibility, sharing and statistical mindfulness of results in computer engineering is rarely considered. In fact, it is often simply impossible due to lack of common tools and data repositories. Therefore, we decided to develop cTuning technology for collaborative and reproducible research that helps to systematize computer engineering and enables a new publication model which favors sharing of data, models, tools and interfaces for validation and reproducibility by the community.
+----
-Since 2008 we released all our benchmarks (cBench), datasets (cDatasets/KDataSets), tools, and experimental data used in our publications in the cTuning repository in most of our publications since 2008. Since 2005, we also made our cTuning-related lectures available on-line. This model resulted in multiple collaborative projects to improve predictive models and tools to design and optimize computer systems together with IBM (Israel), Google (USA), ICT (China), University of Edinburgh (UK), UPC (Spain), CAPS Entreprise (France), ISP RAS (Russia), Intel (Illinois), Ghent University (Belgium), UVSQ (France), NCAR (USA), ARC/Synopsys (UK) and others.This topic has been accepted for the HiPEAC3 network of excellence (2012-2015) and we are currently building a community around this model.
+This wiki is dedicated to:
+* ''New publication model for computer engineering that favors reproducible experimental results validated by the community''
+* ''New plugin-based open source collaborative R&D infrastructure and repository to systematize design and optimization of computer systems''
-=== Next event ===
+''If you have questions, comments or suggestions, interested to participate in the future events or would like to join this effort, please get in touch with [http://cTuning.org/lab/people/gfursin Grigori Fursin].''
-* '''2-3/May/2013''' - [cTuning.org/making-computer-engineering-a-science-2013 Thematic session] at HiPEAC Computing Week/ACM ECRC 2013 in Paris, France
-If you are interested to join this effort, please get in touch with Grigori Fursin.
-=== Current work in progress ===
+Designing novel many-core computer systems becomes intolerably complex, ad-hoc, costly and error prone due to limitations of available technology, enormous number of available design and optimization choices, and complex interactions between all software and hardware components. At the same time, research and development methodology for computer systems has hardly changed in the past decades. Users and developers often have to resort to non-scientific, non-systematic, non-rigorous, intuitive and error-prone methods combined with multiple ad-hoc tools, often limited, non-representative and simply outdated benchmarks and datasets, complex interfaces and data formats to select the most appropriate solution that satisfies all their needs. Such an outdated technology results in an enormous waste of expensive computing resources and energy, and considerably increases development costs and time-to-market for the new systems.
-We are starting building an academic and industrial workgroup interested in:
-# setting up practical public repository and infrastructure to share and reproduce experimental results from the community
+Empirical auto-tuning combined with run-time adaptation and machine learning has been demonstrating good potential to address above challenges for more than a decade but still far from the widespread production use due to unbearably long exploration and training times, ever changing tools and their interfaces, lack of a common experimental methodology, and lack of unified mechanisms for knowledge building and exchange apart from publications where reproducibility of results is often not even considered.
-# sharing benchmarks, data sets, tools, interfaces, predictive models, etc. to prepare common experimental methodology
-# pushing forward new publication model for HiPEAC where experimental results are validated by the community before being published (similar to conferences and journals in statistics, machine learning, biology, etc) - the main challenge is to enable very simple validation across multiple ever-changing architectures, tools, benchmarks and data sets
+We strongly believe that the time has come to start collaborative systematization and unification of design and optimization of computer systems combined with a new publication model where experimental results are validated by the community. One of the possible promising solutions is to combine public repository of knowledge with online auto-tuning, machine learning and crowdsourcing techniques where HiPEAC and cTuning communities already have a good practical experience. Such collaborative approach should allow community to continuously validate, systematize and improve collective knowledge about computer systems, and extrapolate it to build faster, more power efficient and reliable computer systems. Furthermore, it should be able to suggest researchers and engineers where to focus their effort and creativity when designing or optimizing computer systems, thus boosting innovation and dramatically reducing development and optimization costs, and time to market for new systems. It can also help to restore the attractiveness of computer engineering making it a more systematic and rigorous discipline rather than "hacking".
+We are building an academic and industrial workgroup interested in:
+* building common extensible HiPEAC repository and infrastructure (possibly based on already existing modular cTuning/Collective Mind technology) to collect statistics, benchmarks, codelets, data sets, tools, auto-tuning and run-time adaptation plugins, and predictive models from the community
+* systematizing and unifying optimization, design space exploration and run-time adaptation techniques (co-design and auto-tuning)
+* collaboratively evaluating and improving various data mining, classification and predictive modeling techniques for off-line and on-line auto-tuning
+* substituting ad-hoc and possibly outdated benchmarks and data sets with realistic and representative ones from the community
+* preparing new publication model (workshops, conferences, journals) with validation of experimental results by the community
+=== Events ===
+* '''22/01/2014''' - [[Discussions:New_Publication_Model:ADAPT2014_Panel_Discussion|Panel at ADAPT2014]]
+* '''2-3/05/2013''' - [http://cTuning.org/making-computer-engineering-a-science-2013 Thematic session on "making computer engineering a science"] at HiPEAC Computing Week/ACM ECRC 2013 in Paris, France
+** [[Discussions:New_Publication_Model:Notes_from_session_paris_2013|Meeting notes]]
+* '''24-25/04/2012''' - [http://www.hipeac.net/content/goeteborg-hipeac-computing-systems-week-april-2012 Thematic session on "Collective characterization, optimization and design of computer systems"] at HiPEAC Computing Week 2012 in Göteborg, Sweden
+* '''3/04/2012''' - [http://exadapt.org/2012/presentation-exadapt2012-grigori-fursin.pdf ASPLOS/EXADAPT 2012 panel on "Collaborative research methodology for large-scale computer systems"], London, UK
 === Comments ===
-* Grigori: after the keynote, I had interesting discussions with mathematicians from NTU during conference on [http://goo.gl/iutx auto-tuning] in Taiwan. They mentioned a [http://www.jstatsoft.org journal in statistics] where results have to be validated by the community. However, we agreed that the biggest challenge in computer engineering is not in reproducing similar scheme, but in enabling validation of results across many architectures, tools, data sets which is impossible right now without special framework and repository. That's why there was a considerable interest to the new plugin-based Collective Mind framework that is intended to enable collaborative and reproducible research while supporting most of the architectures, operating systems (including off-the-shelf Android mobiles), compilers, run-time systems, benchmarks, data sets, etc. I plan to pre-release this framework for a broader community during next HiPEAC computing week to continue this collaborative effort.
+* Grigori: from our experience with sharing our experimental results and tools or validating existing results (such as at ADAPT'14), just packing and sharing your artifacts is not enough - we would like to have them as reusable components with defined API and meta-description.
+* Grigori: During conference on [http://goo.gl/iutx auto-tuning] in NTU (Taiwan), colleagues mentioned a [http://www.jstatsoft.org on-line journal in statistics] where algorithm and datasets are submitted along publication to validate results by the PC/community. Later we also had more [[Discussions:New_Publication_Model:Notes_from_session_paris_2013|discussions]] at HiPEAC computing week in Paris and we all agreed that we can't simply reproduce the same scheme since experimental setups are usually much more complex in computer engineering (design, optimization and run-time adaptation) while there is still no clear methodology for reproducibility (for example, how to deal with general variations in computer system behavior across different users, setups or runs). That's why we believe that we need a common collaborative R&D repository and infrastructure to reproduce and validate experiments across many users, architectures, tools, data sets. That is exact reason why I have been designing open source plugin-based cTuning and Collective Mind repository and infrastructure which allows to build customized experimental setups while supporting most of the architectures, operating systems (including off-the-shelf Android mobiles), compilers, run-time systems, benchmarks, data sets, etc. It is now available for preview and we are now testing it in a few industrial and academic projects before pre-release to continue collaborative and public discussions.
-=== People ===
+=== Thanks ===
-* We would like to thank [http://ctuning.org/lab/people all our colleagues] who provided interesting feedback on cTuning technology, and evaluated, validated or extended various collaborative tools, benchmarks, data sets and predictive models for program and architecture optimization!
+* We would like to thank HiPEAC for funding our meetings and [http://ctuning.org/lab/people all colleagues] who provided interesting feedback on cTuning/Collective Mind technology, and evaluated, validated or extended various collaborative tools, benchmarks, data sets and predictive models for program and architecture optimization!