Since 2006 we are actively developing an open common methodology, tools and public repository of knowledge to bring together academia, industry and end-users for collaborative and reproducible research and experimentation in computer engineering similar to physics, biology and other sciences.
We are focusing on the development of a common research SDK (Collective Knowledge aka CK) to let the community share their artifacts and experimental workflows as reusable components with a unified API, crowdsource experiments, collaboratively validate them and build upon them as described in our manifesto [ ACM DL, PDF, DATE'16, ADAPT workshops ].
It originally started as a side effect of the MHAOTEU project (1999-2001) where Grigori Fursin led development of a common framework for performance tuning of real HPC applications from supercomputer centers using polyhedral transformations [ white paper ], and later during our MILEPOST project (2006-2009) to enable machine learning based self-tuning compiler.
In the MILEPOST project we developed and tried to combine the first cTuning repository of optimization knowledge, common machine learning-based autotuning infrastructure based on customizable plugins, performance statistics continuously collected from multiple users (big data), and predictive analytics (statistical analysis, data mining and machine learning and feature detection) to adaptively explore and prune large optimization spaces, and predict better optimizations and hardware designs.
The main challenge we faced was lack of common experimental framework, lack of large, diverse and realistic programs and data sets, lack of reusability of shared research tools, lack of computational resources to train ML-based compiler, and difficulties to the reproduce experimental results collected from multiple users due to constantly changing software and hardware stack.
Based on our practical knowledge and experience, the non-profit cTuning foundation helps the community with: