Enabling collaborative, systematic and reproducible research, experimentation and development with an open publication model in computer engineering

Manifesto / motivation

Rather than writing yet another manifesto on reproducible research and experimentation in computer engineering, we have been working on enabling sharing and reproducing experimental results and artifacts in computer engineering since 2006 as a side effect of our MILEPOST and cTuning.org projects. We attempted to build a practical machine learning based self-tuning compiler combining plugin-based auto-tuning framework with a public cTuning repository of knowledge, crowdsourcing predictive analytics, but faced numerous problems including:

• Lack of common, large and diverse benchmarks and data sets needed to build statistically meaningful predictive models;
• Lack of common experimental methodology and unified ways to preserve, systematize and share our growing optimization knowledge and research material including benchmarks, data sets, tools, tuning plugins, predictive models and optimization results;
• Problem with continuously changing, "black box" and complex software and hardware stack with many hardwired and hidden optimization choices and heuristics not well suited for auto-tuning and machine learning;
• Difficulty to reproduce performance results from the cTuning.org database submitted by users due to a lack of full software and hardware dependencies;
• Difficulty to validate related auto-tuning and machine learning techniques from existing publications due to a lack of culture of sharing research artifacts with full experiment specifications along with publications in computer engineering.