Since 2006 we are actively developing an open common methodology, tools and public repository of knowledge to bring together academia, industry and end-users for collaborative and reproducible research and experimentation in computer engineering similar to physics, biology and other sciences.
We are focusing on the development of a common research SDK (Collective Knowledge aka CK) to let the community share their artifacts and experimental workflows as reusable components with a unified API, crowdsource experiments, collaboratively validate them and build upon them as described in our manifesto [ ACM DL, PDF, DATE'16, ADAPT workshops ].
It originally started as a side effect of the MHAOTEU project (1999-2001) where Grigori Fursin led development of a common framework for performance tuning of real HPC applications from supercomputer centers using polyhedral transformations [ white paper ], and later during our MILEPOST project (2006-2009) to enable machine learning based self-tuning compiler.
In the MILEPOST project we developed and tried to combine the first cTuning repository of optimization knowledge, common machine learning-based autotuning infrastructure based on customizable plugins, performance statistics continuously collected from multiple users (big data), and predictive analytics (statistical analysis, data mining and machine learning and feature detection) to adaptively explore and prune large optimization spaces, and predict better optimizations and hardware designs.
The main challenge we faced was lack of common experimental framework, lack of large, diverse and realistic programs and data sets, lack of reusability of shared research tools, lack of computational resources to train ML-based compiler, and difficulties to the reproduce experimental results collected from multiple users due to constantly changing software and hardware stack.
Based on our practical knowledge and experience, the non-profit cTuning foundation helps the community with:
- various technical aspects of reproducibility including methodology, tools and repositories to share experimental results and all related material in a reproducible way with full specification including all software and hardware dependencies [ details ];
- setting up procedures to share and validate experimental results from publications for workshops, conferences and journals related to computer engineering [ details ];
- setting up artifact evaluation committee from a large pool of supporters [ details ];
- promoting and improving our open publication model in computer engineering where publications and experimental results are reviewed and validated by the community [ ACM DL, arXiv, ADAPT workshops];
- enabling systematic and collaborative optimization and co-design of the emerging workloads such as deep learning across the whole software and hardware stack to develop faster, smaller, cheaper and more power efficient computer systems [ details ];
- organizing events to discuss social and technological aspects of open science [ list of events ].