Tools:CM:Demos - Revision history

Gfursin at 14:32, 13 August 2014

2014-08-13T14:32:45Z

Gfursin at 14:35, 23 December 2013

2013-12-23T14:35:59Z

Gfursin at 13:39, 23 December 2013

2013-12-23T13:39:19Z

Gfursin at 13:38, 23 December 2013

2013-12-23T13:38:10Z

Gfursin at 13:28, 23 December 2013

2013-12-23T13:28:40Z

Gfursin at 13:26, 23 December 2013

2013-12-23T13:26:54Z

Gfursin at 13:25, 23 December 2013

2013-12-23T13:25:37Z

Gfursin at 13:22, 23 December 2013

2013-12-23T13:22:05Z

Gfursin at 13:15, 23 December 2013

2013-12-23T13:15:48Z

Gfursin at 13:15, 23 December 2013

2013-12-23T13:15:24Z

@@ Line 233: / Line 233: @@
 |-
 | '''P1'''=Samsung Galaxy Y<br/>
-| &nbsp;?<br/>
+| ~4 Watts<br/>
 | ~100 euros<br/>
 | Broadcom BCM21553 ARM11 processor (ARMv6), 0.832 GHz, Memory 290MB<br/>

@@ Line 138: / Line 138: @@
 *''prepare_experiment_repo.bat'' - above demos run experiments in the ''ctuning-shared'' repository. However, it is often desirable to keep original repository and benchmarks "clean". Therefore, it is possible to copy all benchmarks to another ''ctuning-experiments'' repository using this script and changing 'target_repo_uoa="''ctuning-experiments''"' in the above input json files.
-== Crowdsourcing compiler flags tuning (on-going)<br/> ==
+= On-going projects<br/> =
 Tuning of compiler flags was not solved in decades even with machine learning compiler (MILEPOST GCC) mainly because it requires huge amount of training runs. Having common infrastructure, we are trying to solve this problem using crowdsourcing. We provided a scenario where users can start experiments with just a few clicks on their machine, cloud service, supercomputer or even mobile phone or tablet (download and try Android application [https://play.google.com/store/apps/details?id=com.collective_mind.node "Collective Mind Node" from Google Play Store]) and submit experimental data to the common repository (c-mind.org) to find most profitable optimizations for different platforms, OS, compilers, programs and datasets. It is an on-going work and collaborations are appreciated!

@@ Line 137: / Line 137: @@
 *''del_ctuning_space.bat'' - delete (clean) cTuning space entries from the above demos
 *''prepare_experiment_repo.bat'' - above demos run experiments in the ''ctuning-shared'' repository. However, it is often desirable to keep original repository and benchmarks "clean". Therefore, it is possible to copy all benchmarks to another ''ctuning-experiments'' repository using this script and changing 'target_repo_uoa="''ctuning-experiments''"' in the above input json files.
 == Predicting optimizations and run-time adaptation using collective knowledge (on-going, a bit complex)<br/> ==
@@ Line 207: / Line 211: @@
 Naturally, there is much more to be done in the future and we hope to improve/extend this technology collaboratively, including run-time adaptation based on our past techniques (static multi-versioning combined with machine learning and run-time adaptation dynamic plugins using OpenME interface) - see our [http://ctuning.org/cm/wiki/Tools%3ACM%3Aprojects possible collaborative projects] and the following publications {{CMIND_PUB|fd899287e64b4999|FCOP2005}} {{CMIND_PUB|199e2c0b9d4abcb9|LCWP2009}} {{CMIND_PUB|3b9c0386749be8ad|JVGP2009}} {{CMIND_PUB|9404f8911595b11b|FT2010}}:
-http://ctuning.org/cm/wiki/images/1/19/Cm_run_time_adaptation.png
+[http://ctuning.org/cm/wiki/images/1/19/Cm_run_time_adaptation.png http://ctuning.org/cm/wiki/images/1/19/Cm_run_time_adaptation.png]
 = Collective Mind web-based demos<br/> =

@@ Line 133: / Line 133: @@
 == Miscellaneous scripts<br/> ==
-We prepared a few more demo support scripts available in the directory ''''$CM_ROOT/scripts/2-examples-of-auto-tuning/''6-misc'':
+We prepared a few more demo support scripts available in the directory ''''$CM_ROOT/scripts/2-examples-of-auto-tuning/''6-misc'':'''
 *''del_ctuning_space.bat'' - delete (clean) cTuning space entries from the above demos
 *''prepare_experiment_repo.bat'' - above demos run experiments in the ''ctuning-shared'' repository. However, it is often desirable to keep original repository and benchmarks "clean". Therefore, it is possible to copy all benchmarks to another ''ctuning-experiments'' repository using this script and changing 'target_repo_uoa="''ctuning-experiments''"' in the above input json files.
 = Collective Mind web-based demos<br/> =

@@ Line 78: / Line 78: @@
 == Multi-objective compiler auto-tuning<br/> ==
-Research and experimental scenarios such as auto-tuning are implemented in Collective Mind as standard modules (plugins) around above universal "build and run" pipeline. However, before sharing stable scenario as a cM module, it can be first prototyped and implemented outside as a python module. The names of such prototype modules always start with cme (Collective Mind External module) and have a direct access to Collective Mind API. Directory ''''$CM_ROOT/scripts/2-examples-of-auto-tuning/''2-tune-compiler-flags'' presents such compiler flag auto-tuning scenario with a module ''cme.scenario.explore_compiler_flags.py .'' As a demo, user can run script ''run_scenario_explore_compiler_flags_linux_gcc_1.bat'' to explore based flags (-Ox) for susan corners benchmark and generic GCC. This scenario can also be customized through json input file (''run_scenario_explore_compiler_flags_linux_gcc_1.json'') as following:
+Research and experimental scenarios such as auto-tuning are implemented in Collective Mind as standard modules (plugins) around above universal "build and run" pipeline. However, before sharing stable scenario as a cM module, it can be first prototyped and implemented outside as a python module. The names of such prototype modules always start with cme (Collective Mind External module) and have a direct access to Collective Mind API. Directory ''$CM_ROOT/scripts/2-examples-of-auto-tuning/2-tune-compiler-flags''presents such compiler flag auto-tuning scenario with a module ''cme.scenario.explore_compiler_flags.py .'' As a demo, user can run script ''run_scenario_explore_compiler_flags_linux_gcc_1.bat'' to explore based flags (-Ox) for susan corners benchmark and generic GCC. This scenario can also be customized through json input file (''run_scenario_explore_compiler_flags_linux_gcc_1.json''
 {<br/>"prepare_state":<br/>{
@@ Line 100: / Line 100: @@
 Note, that there are two prepared similar demo scripts for LLVM, i.e. ''run_scenario_explore_compiler_flags_linux_llvm_1.bat'' and ''run_scenario_explore_compiler_flags_linux_llvm_2.bat .''
 == Dataset exploration<br/> ==

@@ Line 7: / Line 7: @@
 == "Build and run" experimental pipeline<br/> ==
-Here we describe how to use our current common "build and run" experimental pipeline (''sub-directory 1-basic-build-and-run). Associated cM code for this pipeline can be found in ''$CM_ROOT/default/.cmr/module/ctuning.pipeline.build_and_run_program/module.py''. ''It connects all the related artifacts from repository such as source code, data set, compiler, profiling tools (gprog, Intel vTune, likwid), OS, processor, etc. or chain other modules (such as statistical analysis of characteristics, Pareto frontier filter, etc) to build, run program and measure/collect related characteristics such as execution time, code size, compilation time, correctness of the output, hardware counters, etc. Input is customized through cM json file or command line parameters (which are internally converted to json). Output is also recorded as json format. Internally json is directly converted to python dictionary.
+Here we describe how to use our current common "build and run" experimental pipeline (''directory $CM_ROOT/scripts/2-examples-of-auto-tuning/1-basic-build-and-run). Associated cM code for this pipeline can be found in ''$CM_ROOT/default/.cmr/module/ctuning.pipeline.build_and_run_program/module.py''. ''It connects all the related artifacts from repository such as source code, data set, compiler, profiling tools (gprog, Intel vTune, likwid), OS, processor, etc. or chain other modules (such as statistical analysis of characteristics, Pareto frontier filter, etc) to build, run program and measure/collect related characteristics such as execution time, code size, compilation time, correctness of the output, hardware counters, etc. Input is customized through cM json file or command line parameters (which are internally converted to json). Output is also recorded as json format. Internally json is directly converted to python dictionary.
 Note that these demos are available in the standalone Collective Mind package and do not require downloading of extended cTuning data from the [http://c-mind.org/repo live Collective Mind repository] (though it will enable much more functionality, packages, benchmarks and data sets).&nbsp;
@@ Line 78: / Line 78: @@
 == Multi-objective compiler auto-tuning<br/> ==
-Research and experimental scenarios such as auto-tuning are implemented in Collective Mind as standard modules (plugins) around above universal "build and run" pipeline. However, before sharing stable scenario as a cM module, it can be first prototyped and implemented outside as a python module. The names of such prototype modules always start with cme (Collective Mind External module) and have a direct access to Collective Mind API. Subdirectory ''2-tune-compiler-flags'' presents such compiler flag auto-tuning scenario with a module ''cme.scenario.explore_compiler_flags.py .''
+Research and experimental scenarios such as auto-tuning are implemented in Collective Mind as standard modules (plugins) around above universal "build and run" pipeline. However, before sharing stable scenario as a cM module, it can be first prototyped and implemented outside as a python module. The names of such prototype modules always start with cme (Collective Mind External module) and have a direct access to Collective Mind API. Directory ''''$CM_ROOT/scripts/2-examples-of-auto-tuning/''2-tune-compiler-flags'' presents such compiler flag auto-tuning scenario with a module ''cme.scenario.explore_compiler_flags.py .'' As a demo, user can run script ''run_scenario_explore_compiler_flags_linux_gcc_1.bat'' to explore based flags (-Ox) for susan corners benchmark and generic GCC. This scenario can also be customized through json input file (''run_scenario_explore_compiler_flags_linux_gcc_1.json'') as following:
 As a demo, user can run script ''run_scenario_explore_compiler_flags_linux_gcc_1.bat'' to explore based flags (-Ox) for susan corners benchmark and generic GCC. This scenario can also be customized through json input file (''run_scenario_explore_compiler_flags_linux_gcc_1.json'') as following:
 {<br/>"prepare_state":<br/>{
@@ Line 104: / Line 103: @@
 == Dataset exploration<br/> ==
-An important part of program (or compiler) auto-tuning and run-time adaptation, is understanding and optimizing program behavior across multiple data sets (see&nbsp;[http://ctuning.org/lab/people/gfursin/cv.html#cm_29db2248aba45e59_199e2c0b9d4abcb9 Finding representative sets of optimizations for adaptive multiversioning applications], [http://ctuning.org/lab/people/gfursin/cv.html#cm_29db2248aba45e59_404c91542f38bd58 MiDataSets: creating the conditions for a more realistic evaluation of Iterative optimizations], [http://ctuning.org/lab/people/gfursin/cv.html#cm_29db2248aba45e59_dabab2e9ff26c745 Evaluating iterative optimization across 1000 datasets], [http://ctuning.org/lab/people/gfursin/cv.html#cm_29db2248aba45e59_530e5f456ea259de Collective optimization: A practical collaborative approach]). Therefore, we provided demo scripts available in subdirectory ''3-explore-datasets'' to monitor behavior of programs (benchmarks, codelets) vs datasets and their parameters.
+An important part of program (or compiler) auto-tuning and run-time adaptation, is understanding and optimizing program behavior across multiple data sets (see&nbsp;[http://ctuning.org/lab/people/gfursin/cv.html#cm_29db2248aba45e59_199e2c0b9d4abcb9 Finding representative sets of optimizations for adaptive multiversioning applications], [http://ctuning.org/lab/people/gfursin/cv.html#cm_29db2248aba45e59_404c91542f38bd58 MiDataSets: creating the conditions for a more realistic evaluation of Iterative optimizations], [http://ctuning.org/lab/people/gfursin/cv.html#cm_29db2248aba45e59_dabab2e9ff26c745 Evaluating iterative optimization across 1000 datasets], [http://ctuning.org/lab/people/gfursin/cv.html#cm_29db2248aba45e59_530e5f456ea259de Collective optimization: A practical collaborative approach]). Therefore, we provided demo scripts available in the directory ''''$CM_ROOT/scripts/2-examples-of-auto-tuning/''3-explore-datasets'' to monitor behavior of programs (benchmarks, codelets) vs datasets and their parameters.
 Example ''run_scenario_explore_datasets_linux_gcc.bat'' explores behavior of a numerical benchmark LU (''benchmark-polybench-cpu-lu-demo'') from polybench collection versus matrix size (specified as a build parameter N and selected randomly during iterations) using GCC with -O3 flag while monitoring CPI and instruction count using likwid tool to collect hardware counters. Similar to auto-tuning scenario, this scenario is implemented as an external cM plugin ''cme.scenario.explore_datasets.py'' and can be customized through input json file ''run_scenario_explore_datasets_linux_gcc.json''. Experimental results are continuously aggregated in cTuning space entry ''demo-cmd-dataset-linux-gcc''.
@@ Line 112: / Line 111: @@
 == Universal visualization of multi-dimensional spaces<br/> ==
-Collective Mind enables universal visualization of multi-dimensional spaces generated by cM scenario plugins which are usually recorded as cTuning space entries. We prepared several demos available in sub-directory ''4-visualize'' to visualize:
+Collective Mind enables universal visualization of multi-dimensional spaces generated by cM scenario plugins which are usually recorded as cTuning space entries. We prepared several demos available in directory ''''$CM_ROOT/scripts/2-examples-of-auto-tuning/''4-visualize'' to visualize:
 *2D optimization space from compiler flag auto-tuning (such as execution time versus compilation time critical for companies developing mobile and embedded systems). Script ''view_ctuning_space_for_compiler_tuning_linux_gcc.bat'' plots such a graph for cTuning space entry ''demo-cmd-linux-gcc-1'' after iterative compilation from one of the above demos.
@@ Line 123: / Line 122: @@
 One of the major purposes of the first cTuning framework and its third generation presented here (Collective Mind) was to enable reproducibility of experimental results to be shared and validated by the community. Since Collective Mind can preserve the whole experimental setup and all related material (artifacts) while unifying input and output of all modules (plugins) through json files, it is possible to easily replay recorded experiments from cTuning spaces while monitoring varying output parameters.
-We preared a simple demo in subdirectory ''5-reproduce'' to replay recorded experiment 0 for:
+We preared a simple demo in directory ''''$CM_ROOT/scripts/2-examples-of-auto-tuning/''5-reproduce'' to replay recorded experiment 0 for:
 *compiler flag exploration from cTuning space entry ''demo-cmd-linux-llvm-1'' (script ''reproduce_experiment_flags_linux_llvm.bat'')
@@ Line 132: / Line 131: @@
 == Miscellaneous scripts<br/> ==
-We prepared a few more demo support scripts available in subdirectory ''6-misc'':
+We prepared a few more demo support scripts available in the directory ''''$CM_ROOT/scripts/2-examples-of-auto-tuning/''6-misc'':
-−
-−
 *''del_ctuning_space.bat'' - delete (clean) cTuning space entries from the above demos

@@ Line 130: / Line 130: @@
 Note, that this mode is useful for compiler developers when using cM to test compiler performance, code size and compilation time regressions or bugs, sharing setups across specialists and reproducing problematic cases.
-== Misc<br/> ==
+== Miscellaneous scripts<br/> ==
 = Collective Mind web-based demos<br/> =

← Older revision		Revision as of 13:22, 23 December 2013
Line 120:		Line 120:

	== Reproducibility<br/> ==		== Reproducibility<br/> ==
		+
		+	One of the major purposes of the first cTuning framework and its third generation presented here (Collective Mind) was to enable reproducibility of experimental results to be shared and validated by the community. Since Collective Mind can preserve the whole experimental setup and all related material (artifacts) while unifying input and output of all modules (plugins) through json files, it is possible to easily replay recorded experiments from cTuning spaces while monitoring varying output parameters.
		+
		+	We preared a simple demo in subdirectory ''5-reproduce'' to replay recorded experiment 0 for:
		+
		+	*compiler flag exploration from cTuning space entry ''demo-cmd-linux-llvm-1'' (script ''reproduce_experiment_flags_linux_llvm.bat'')
		+	*dataset exploration from cTuning space entry ''demo-cmd-dataset-linux-gcc ''(script ''reproduce_experiment_datasets_linux_gcc.bat'').
		+
		+	Note, that this mode is useful for compiler developers when using cM to test compiler performance, code size and compilation time regressions or bugs, sharing setups across specialists and reproducing problematic cases.

	== Misc<br/> ==		== Misc<br/> ==

@@ Line 110: / Line 110: @@
 We are gradually and collaboratively improving this plugin to support adaptive probabilistic sampling of large and multi-dimensional data set and optimization spaces to focus search on areas with an unusual behavior or highest speedup (concept described in [http://ctuning.org/lab/people/gfursin/cv.html#cm_29db2248aba45e59_c83fc3a0393b4339 Probabilistic source-level optimisation of embedded programs], [http://ctuning.org/lab/people/gfursin/cv.html#cm_29db2248aba45e59_3854f3510e3e6c91 Using Machine Learning to Focus Iterative Optimization] and [http://ctuning.org/lab/people/gfursin/cv.html#cm_29db2248aba45e59_530e5f456ea259de Collective optimization: A practical collaborative approach]), and to find representative data set/optimization pairs to build adaptive applications through static multi-versioning and optimized decision trees that select appropriate versions at run-time using [[Tools:OpenME|OpenME interface]] (as described in [http://ctuning.org/lab/people/gfursin/cv.html#cm_29db2248aba45e59_199e2c0b9d4abcb9 our paper]).
-== Visualization<br/> ==
+== Universal visualization of multi-dimensional spaces<br/> ==
 Collective Mind enables universal visualization of multi-dimensional spaces generated by cM scenario plugins which are usually recorded as cTuning space entries. We prepared several demos available in sub-directory ''4-visualize'' to visualize:

← Older revision		Revision as of 13:15, 23 December 2013
Line 111:		Line 111:

	== Visualization<br/> ==		== Visualization<br/> ==
		+
		+	Collective Mind enables universal visualization of multi-dimensional spaces generated by cM scenario plugins which are usually recorded as cTuning space entries. We prepared several demos available in sub-directory ''4-visualize'' to visualize:
		+
		+	*2D optimization space from compiler flag auto-tuning (such as execution time versus compilation time critical for companies developing mobile and embedded systems). Script ''view_ctuning_space_for_compiler_tuning_linux_gcc.bat'' plots such a graph for cTuning space entry ''demo-cmd-linux-gcc-1'' after iterative compilation from one of the above demos.
		+	*2D graph of a program behavior (CPI) vs matrix size (N). Script ''plot_2d_graph_datasets_linux_gcc.bat'' plots such a graph for cTuning space entry ''demo-cmd-dataset-linux-gcc ''from one of the above demos''.''This script can be customized through ''plot_2d_graph_datasets_linux_gcc.json'' to select different graph engines with cM entries and dimensions to visualize.
		+
		+	Note, that the same graphs can be plotted from cM web interface (cM web interface -> Usage -> Build graph of multi-dimensional optimization space).

	== Reproducibility<br/> ==		== Reproducibility<br/> ==