Tools:CM:User Guide:V1.0 - Revision history

Gfursin at 13:39, 23 December 2013

2013-12-23T13:39:25Z

Gfursin at 13:38, 23 December 2013

2013-12-23T13:38:16Z

Gfursin at 13:35, 23 December 2013

2013-12-23T13:35:24Z

Show changes

Gfursin at 13:32, 23 December 2013

2013-12-23T13:32:31Z

Gfursin at 13:31, 23 December 2013

2013-12-23T13:31:50Z

Gfursin at 13:31, 23 December 2013

2013-12-23T13:31:34Z

Gfursin at 13:30, 23 December 2013

2013-12-23T13:30:50Z

Gfursin: Created page with "= cM user guide = This user guide is continuously evolving and describes various cM usage scenarios. In the mean time, user may also be interested to look at at available T..."

2013-10-23T08:41:58Z

Created page with "= cM user guide = This user guide is continuously evolving and describes various cM usage scenarios. In the mean time, user may also be interested to look at at available T..."

Show changes

← Older revision		Revision as of 13:39, 23 December 2013
Line 416:		Line 416:

	Experimental results can be viewed, reproduced, pruned and plotted as was described above.		Experimental results can be viewed, reproduced, pruned and plotted as was described above.
−
−	~~==== Example: crowdsourcing compiler flags tuning ====~~
−
−	Tuning of compiler flags was not solved in decades even with machine learning compiler (MILEPOST GCC) mainly because it requires huge amount of training runs. Having common infrastructure, we are trying to solve this problem using crowdsourcing. We provided a scenario where users can start experiments with just a few clicks on their machine, cloud service, supercomputer or even mobile phone or tablet (download and try Android application [https://play.google.com/store/apps/details?id=com.collective_mind.node "Collective Mind Node" from Google Play Store]) and submit experimental data to the common repository (c-mind.org) to find most profitable optimizations for different platforms, OS, compilers, programs and datasets. It is an on-going work and collaborations are appreciated!

← Older revision		Revision as of 13:38, 23 December 2013
Line 420:		Line 420:

	Tuning of compiler flags was not solved in decades even with machine learning compiler (MILEPOST GCC) mainly because it requires huge amount of training runs. Having common infrastructure, we are trying to solve this problem using crowdsourcing. We provided a scenario where users can start experiments with just a few clicks on their machine, cloud service, supercomputer or even mobile phone or tablet (download and try Android application [https://play.google.com/store/apps/details?id=com.collective_mind.node "Collective Mind Node" from Google Play Store]) and submit experimental data to the common repository (c-mind.org) to find most profitable optimizations for different platforms, OS, compilers, programs and datasets. It is an on-going work and collaborations are appreciated!		Tuning of compiler flags was not solved in decades even with machine learning compiler (MILEPOST GCC) mainly because it requires huge amount of training runs. Having common infrastructure, we are trying to solve this problem using crowdsourcing. We provided a scenario where users can start experiments with just a few clicks on their machine, cloud service, supercomputer or even mobile phone or tablet (download and try Android application [https://play.google.com/store/apps/details?id=com.collective_mind.node "Collective Mind Node" from Google Play Store]) and submit experimental data to the common repository (c-mind.org) to find most profitable optimizations for different platforms, OS, compilers, programs and datasets. It is an on-going work and collaborations are appreciated!
−
−	~~=== Predicting optimizations and run-time adaptation using collective knowledge ===~~
−
−	When enough knowledge is collected about program/codelet and architecture behavior and optimization, it is possible to predict optimizations or run-time adaptation scenarios using relatively standard machine learning techniques (predictive models) that correlate program static features (semantic MILEPOST features, code patterns, control flow graphs), dynamic program and architecture features (hardware counters, micro-architecture independent features, etc), static architecture features (cache size, pipeline length, etc), OS features, etc.
−
−	We hope that community will gradually add many various existing ML models. As the first step, we added a concept of cM self-tuning compiler module (''ctuning.compiler'') and ''ctuning-cc/ctuning-cpp/ctuning-fortran'' scripts that can wrap up any existing compiler and predict compiler flags (or finer-grain optimizations using pragmas or OpenME interface) using semantic program features from MILEPOST GCC and nearest neighbour classifier to find best found optimizations by the community for the most similar program/codelet in the repository.
−
−	The live interactive demo is available at [http://c-mind.org/repo/?cm_menu=scenarios&cm_submenu=ctuning_pipeline_build_and_run_program_predict_opt c-mind.org]. This online web-service is used in ctuning-cc to predict optimizations on the fly during compilation. Simply prefix your compiler with ctuning-cc and you will be able to control various options, record compilation flow for reproducibility, extract features, predict optimizations simply by controlling related environment variable and json file.
−
−	~~In order to test use ctuning-cc locally, you need to:~~
−
−	~~#Go to cM web front end -> Usage -> Install package~~
−	~~#Install MILEPOST GCC~~
−	~~#Install cTuning CC~~
−	~~#Install XBS~~
−	~~#We suggest to install GCC 4.7.2 as reference compiler since there is already collected statistics on program/codelet optimization in c-mind.org~~
−
−	~~#Go to cM web front end -> Usage -> Predict optimizations~~
−	~~#Click on cTuning local setup -> Add~~
−	~~#Specify 'default' as internal alias (if configured for the fist time) and configure your system for optimization prediction~~
−
−	~~Now you should normally be able to use ctuning-cc. Find the entry with the demo in the repository using~~
−
−	~~cmx find demo:ctuning-cc-and-milepost-gcc-with-cm~~
−
−	~~This directory contains 3 sub-directories:~~
−
−	*'''susan''' - susan corners benchmark from cBench (uses batch script for compilation)
−	*'''bzip2-1.0.6''' - bzip2 real application (uses Makefile for compilation)
−	*'''lapack-3.4.2''' - lapack library (uses configure and Makefile for compilation)
−
−	~~Each directory has 2 subdirectories:~~
−
−	*cm_build_linux_gcc - demo for Linux
−	*cm_build_win32_gcc - demo for Windows
−
−	~~Each of the above directories has the following scripts:~~
−
−	*''cm_build_ctuning_test_repository.sh'' - test c-mind.org repository access (you must be registered)
−
−	*''cm_build_native.sh'' - compile program using native compiler
−	*''cm_build_ctuning_transparent.sh'' - compile program using ctuning-cc
−
−	*''cm_build_ctuning_record_compilation.s''h - record compilation flow for reproducibility (input in ''cm_input_rc.json'', output in ''cm_output_rc.json'')
−
−	*''cm_build_ctuning_select_collective_opt.sh'' - select some pre-defined optimization from the local repository (input in ''cm_input_sco.json'', output in ''cm_output_sco.json'')
−	*''cm_build_ctuning_select_collective_opt_cmind.sh'' - select some pre-defined optimization from the c-mind.org repository (useful when sharing optimizations from the publications such as from our MILEPOST GCC paper and then checking/reproducing them) (input in ''cm_input_sco_cmind.json'', output in ''cm_output_sco_cmind.json'')
−
−	*''cm_build_ctuning_extract_features.sh'' - extract program semantic features using MILEPOST GCC (input in ''cm_input_ef.json'', output in ''cm_output_ef.json'')
−
−	*''cm_build_ctuning_predict_opt.sh'' - predict optimizations on the fly using local repository (input in ''cm_input_po.json'', output in ''cm_output_po.json'')
−	*''cm_build_ctuning_predict_opt_cmind.sh'' - predict optimizations on the fly using c-mind.org repository (input in ''cm_input_po.json'', output in ''cm_output_po_cmind.json'')
−
−	'''Note: if GCC is not in your path (say on Windows) you need to add a call to cM package variable in scripts with installation UID (can be found on the right side of each package in cM web front-end -> Usage -> install package''': . cm_code_env_<cM UID of package installation>.sh
−
−	~~Ctuning-cc/ctuning-fortran/ctuning-cpp and cM module ''ctuning.compiler'' is an example of cM universal tuning and learning concept. Basically instead of using~~
−
−	~~gcc -O3 *.c~~
−
−	~~user now writes:~~
−
−	~~cur_path=$PWD~~
−	~~export CM_CTUNING_COMPILER_CMD="ct_output_file=\"$cur_path/cm_output_po_cmind.json\" @\"$cur_path/cm_input_po_cmind.json\""~~
−	~~export CM_CTUNING_LOCAL_SETUP_UOA=default~~
−	~~ctuning-cc gcc -O3 ../*.c -o a.out -lm~~
−
−	~~where compiler can now continuously learn and improve optimization heuristics based on collective knowledge. Note, that ''default'' is the UOA of the cTuning local setup that user prepared above.~~
−
−	Naturally, there is much more to be done in the future and we hope to improve/extend this technology collaboratively, including run-time adaptation based on our past techniques (static multi-versioning combined with machine learning and run-time adaptation dynamic plugins using OpenME interface) - see our [[Tools:CM:projects\|possible collaborative projects]] and the following publications {{CMIND_PUB\|fd899287e64b4999\|FCOP2005}} {{CMIND_PUB\|199e2c0b9d4abcb9\|LCWP2009}} {{CMIND_PUB\|3b9c0386749be8ad\|JVGP2009}} {{CMIND_PUB\|9404f8911595b11b\|FT2010}}:
−
−	~~[[File:Cm run time adaptation.png]]~~

@@ Line 55: / Line 55: @@
 If you have problems installing cM, please check [[Tools:CM:Knowledge Base|cM knowledge base]] and/or contact [[#Contact|us]]
-=== Checking cM<br/> ===
+== Checking cM<br/> ==
 Users can access all cM functionality from CMD using ''cm'' or ''cmx'' front-end (accessing repository, executing experimental pipelines, reproducing experiments, compose new experiments, etc). To test that basic cM CMD version is working, user should execute ''cm'' from the command line and should receive the following output:

@@ Line 55: / Line 55: @@
 If you have problems installing cM, please check [[Tools:CM:Knowledge Base|cM knowledge base]] and/or contact [[#Contact|us]]
-=== Check cM<br/> ===
+=== Checking cM<br/> ===
 Users can access all cM functionality from CMD using ''cm'' or ''cmx'' front-end (accessing repository, executing experimental pipelines, reproducing experiments, compose new experiments, etc). To test that basic cM CMD version is working, user should execute ''cm'' from the command line and should receive the following output:

@@ Line 55: / Line 55: @@
 If you have problems installing cM, please check [[Tools:CM:Knowledge Base|cM knowledge base]] and/or contact [[#Contact|us]]
-== Using cM ==
+=== Check cM<br/> ===
 Users can access all cM functionality from CMD using ''cm'' or ''cmx'' front-end (accessing repository, executing experimental pipelines, reproducing experiments, compose new experiments, etc). To test that basic cM CMD version is working, user should execute ''cm'' from the command line and should receive the following output:

@@ Line 275: / Line 275: @@
 === Installing packages for collaborative experimentation<br/> ===
-Go to ''cM web front end –> Usage –> Install / monitor packages''. Then select Collective Tuning Setup according to your system and install necessary packages (tools, libraries, etc). If package require compiler, please select available compiler in the top right menu before installation/buiding.
+Collective Mind enabled easy co-existance of multiple versions of tools and libraries. Go to ''cM web front end –> Usage –> Install / monitor packages''. Then select Collective Tuning Setup according to your system and install necessary packages (tools, libraries, etc). If package require compiler, please select available compiler in the top right menu before installation/buiding.
 === Compiling and running program using universal pipeline<br/> ===