Revision as of 07:02, 13 December 2013

NEWS: Pleas, join our panel on reproducible research at ADAPT'14 @ HiPEAC 2014 in January 2014, or submit papers to TRUST'14 @ PLDI 2014

Since 2006, we are working on a common methodology, infrastructure and repository to enable collaborative and reproducible research and experimentation in computer engineering while focusing on auto-tuning, co-design machine learning and run-time adaptation of computer systems! Such approach enables new publication model where all research materials (artifacts) are continuously shared, validated and improved by the community! To set up an example, we started collecting, unifying and releasing all benchmarks, data sets, models and tools with unified interfaces at cTuning.org and later at c-mind.org since 2008. In spite of original hostility to this project from the academic community, we glad to eventually see recent similar initiatives at major conferences! However, our project is complementary and focuses more on technological aspects of collaborative and reproducible research in computer engineering rather than just sharing and validating artifacts. If you are interested in this community project, join our events and effort, collaborate, invest or contact Grigori Fursin (project founder) for more details!

Collaborative, systematic and reproducible computer engineering

Motivation

With the rapid advances in information technology and all other fields of science comes dramatic growth in the amount of processing data ("big data"). Scientists, engineers and students are drowning in experimental data and often have to divert their research path towards data management, mining, and visualization. Such approaches often require additional interdisciplinary skills including statistical analysis, machine learning, programming and parallelization, database management, and Internet technologies, which still few researchers have or can afford to learn in parallel with their main research work. Multiple frameworks, languages and public data repositories started appearing recently to enable collaborative data analysis and processing but they are often either covering very narrow research topics and too simplistic (just data and code sharing) or very formal and still require special programming skills often including Object Oriented Programming.

Collective Mind technology (cM) attempts to fill in this gap by providing researchers and companies a simple, portable, technology-neutral and practically transparent way to gradually systematize and classify all their data, code and tools. Open source cM framework and repository fully relies on customizable public or private plugins (mostly written in python with support of any other language through OpenME interface) to gradually describe and classify similar data and code objects, or abstract interfaces of ever changing tools thus effectively protecting researchers' experimental setups. cM helps to easily preserve any complex research artifact (collection of files, benchmarks, codelets, datasets, tools, traces, models) with gradually and easily extensible JSON based meta description including classification, properties and either direct or semantic data connections. Furthermore, meta descriptions of all data can be transparently and easily indexed using third-party ElasticSearch enabling very fast and complex queries. At the same time, all research artifacts can be exposed to any public or workgroup user through unified web services to crowdsource experimentation, ranking, online learning and knowledge management.

cM uses agile top-down methodology originating from physics to represent any experimental scenario and gradually decompose it into connected plugins with associated data or compose it from already shared plugins similar to "research LEGO". Universal structure immediately enables replay mode for any experiment, thus making this framework suitable for recent projects on reproducibility of experimental results and new publication model where experiments and techniques are validated, ranked and improved by the community. For example, we easily moved all our past R&D on program and architecture multi-objective auto-tuning, co-design and dynamic adaptation to cM plugins and gradually make them available together with all research artifacts at http://c-mind.org/repo. We hope that cM will be useful to a broad range of researchers and companies either as an open-source, community driven solution to systematize their research and experimentation, or possibly as an intermediate step before investing into more complex or commercial knowledge management systems.

Public repository of knowledge

Do not waste your research material - use Collective Mind Framework and Repository to describe, run and share your experiments with the community!

Beta live Collective Mind repository (3rd generation opened in 2013 substituting previous cTuning repository and infrastructure available since 2008) - we described and shared all our past research developments, codelets, benchmarks, data sets, models, statistical analysis, modeling and online learning plugins and tools to start top-down analysis and optimization of existing computer systems. We used it as the first practical example to motivate new publication model where all research artifacts are continuously shared, validated and improved by the community. After many years, it seems that community finally started moving in this direction and we even see some related initiatives in major conferences including OOPSLA and PLDI. However, our project is complementary and focuses more on technological aspects of collaborative and reproducible research in computer engineering rather than just sharing and validating artifacts.

Common infrastructure and support tools

Collective Mind Infrastructure - plugin-based framework and repository for collaborative and reproducible research and experimentation
- OpenME - interface to "open up" third-party tools and applications to make them prepared for auto-tuning using cM
- Alchemist - OpenME plugin to convert compilers into interactive analysis and optimization toolsets

Events

Upcoming

Workshop TRUST 2014 on reproducible research methodologies and new publication models @ PLDI 2014 (Edinburgh, UK)
Workshop REPRODUCE 2014 on reproducible research methodologies and new publication models @ HPCA 2014 (Orlando, Florida, USA)
Panel on reproducible research methodologies and new publication models at ADAPT 2014 @ HiPEAC 2014 (Vienna, Austria)

Past

Thematic session on making computer engineering a science @ ACM ECRC 2013 / HiPEAC computing week 2013 (Paris, France)
Thematic session on collective characterization, optimization and design of computer systems @ HiPEAC spring computing week 2012 (Goteborg, Sweden)
Tutorial on Speedup-Test: Statistical Methodology to Evaluate Program Speedups and their Optimisation Techniques @ HiPEAC 2010 (Pisa, Italy)
Tutorial on cTuning tools for collaborative and reproducible program and architecture characterization and auto-tuning @ HiPEAC computing systems week 2009 (Infineon, Munich, Germany)

Collective Mind is a community-based and continuously evolving project that uses agile development methodology. Hence, interfaces and modules may be changing from time to time to provide needed functionality. We are very thankful for your understanding, patience and any help to extend and improve this framework while making it clean, simple and easy to use.

@@ Line 57: / Line 57: @@
 <li>[http://c-mind.org/repo/?view_cid=77154d189d2e226c:0053bdf524fb9a58 Tutorial on cTuning tools for collaborative and reproducible program and architecture characterization and auto-tuning] @ HiPEAC computing systems week 2009 (Infineon, Munich, Germany)</li>
 </ul>
-= Motivation<br/> =
-Designing, analyzing and optimizing computer systems is nowadays a tremendously tedious, messy, ad-hoc, costly and error prone process due to an enormous number of available design and optimization choices combined with complex interactions between all components, and rapidly changing technology, tools and interfaces.
-Auto-tuning, run-time adaptation and machine learning based techniques have been investigated for more than a decade to address some of these challenges but are still far from the widespread production use. This is not only due to unbearably long tuning times and ever changing interfaces of analysis and optimization tools, but also due to a lack of a common methodology to discover, preserve and share knowledge about behavior of existing computer systems. Current solutions are mainly proprietary and include redesign of the whole SW/HW stack by very large groups (Liquid Metal project from IBM, SW/HW co-design initiatives from Intel, SciDAC SUPER project).
-When developing cTuning and Collective Mind technology our main goal was to develop such a holistic methodology, framework and repository for collaborative and systematic research and experimentation on software/hardware co-design which will not require redesign of the whole SW/HW stack but could easily and transparently evolve with the evolution of existing systems and tools. At the same time we wanted to implement the core of cTuning/Collective Mind by just a few engineers using Agile methodology and enable easy extensibility by the community to share, reproduce and validate various benchmarks, data sets, tools, models, experimental results, etc.
-We believe that in the past 14 years, we found all the missing pieces of the puzzle to address the above challenges and enable systematic and reproducible characterization and optimization of computer systems through unified and scalable repositories of knowledge and crowdsourcing. In this approach, multi-objective program and architecture tuning to balance performance, power consumption, compilation time, code size and any other important metric is transparently distributed among multiple users while utilizing any available mobile, cluster or cloud computer services. Collected information about program and architecture properties and behavior is continuously processed using statistical and predictive modeling techniques to build, keep and share only useful knowledge at multiple levels of granularity. Gradually increasing and systematized knowledge can be used to predict most profitable program optimizations, run-time adaptation scenarios and architecture configurations depending on user requirements.
-Collective Mind Framework (cM) is a public, open-source, plugin-based infrastructure and repository that attempts to implement the above methodology. Motivated by physics, biology and AI sciences, this framework helps researchers to gradually expose tuning choices, properties and characteristics of any tool or application at multiple granularity levels in existing systems through multiple plugins ("wrappers"). These wrappers use simple and extensible no-type interfaces and no-SQL JSON-based extensible file-based repositories. Such wrappers can be easily combined like research "LEGO" to prepare various exploration, analysis and optimization scenarios and connected to customizable public or private in-house repositories of shared data (applications, data sets, codelets, micro-benchmarks and architecture descriptions), modules (classification, predictive modeling, run-time adaptation) and statistics about behavior of computer systems. Collected data can be continuously analyzed and extrapolated using online learning to predict better optimizations or hardware configurations to effectively balance performance, power consumption and other characteristics. We start from the top-down analysis and optimization of existing computer systems and together with the community start gradually increasing the complexity until we understand and systematize the behavior of existing computer systems to be able to either quickly predict how to optimize them or to build new better systems by extrapolating existing knowledge.
-[[File:Universal learning node small.png|center|Universal learning node small.png]]
 ----
 ''Collective Mind is a community-based and continuously evolving project that uses agile development methodology. Hence, interfaces and modules may be changing from time to time to provide needed functionality. We are very thankful for your understanding, patience and any help to extend and improve this framework while making it clean, simple and easy to use.''