From cTuning.org
Navigation: cTuning.org > CTools > CTuningCC
Contents |
cTuning CC V2.5 with MILEPOST GCC 4.4.x
General configuration
Check ./_set_environment_for_analysis_compiler__milepost_gcc.sh - normally all environment variables should be already properly set (check variable CCC_UUID - the uuid tool). You have to source this file before using cTuning CC - it tells cTuning CC to use MILEPOST GCC for program analysis (extraction of features and access to fine-grain optimizations through ICI).
Importantly, cTuning CC can now use any compiler that supports ICI and cTuning/MILEPOST technology for code analysis and characterization, that is configured through the following environment variables (using GCC as an example):
CTUNING_ANALYSIS_CC=gcc CTUNING_ANALYSIS_CPP=g++ CTUNING_ANALYSIS_FORTRAN=gfortran
File ./_set_environment_for_plugin_tests.sh sets up environment variables for low-level ICI tests and should also be already properly set. If you plan to use only high-level cTuning CC, you can skip it.
Compiler configuration
* You can find how to use cTuning Compiler Collection either transparently without Makefile modifications or explicitly using multiple benchmarks in the demo directory (bitcount, bzip2, libvorbis, matmul).
You need to first configure environment variables in the ___common_environment.sh which are user-dependent:
cTuning CC can use 2 separate compilers - one for analysis (should support ICI and cTuning/MILEPOST technology for program and architecture characterization, self-tuning and adaptation such as MILEPOST GCC) and another can be any user compiler (GCC, LLVM, ICC, ROSE, Open64, XL, etc) driven by the analysis compiler.
User compiler is defined using the following environment variables (using GCC as an example):
CTUNING_COMPILER_CC=gcc
CTUNING_COMPILER_CPP=g++
CTUNING_COMPILER_FORTRAN=gfortran
CCC_CTS_USER and CCC_CTS_PASS should be set to your username and password when self-registering at http://cTuning.org/wiki/index.php/Special:UserLogin
NOW YOU CAN TEST cTuning CC wrapper and communication with the cTuning database by invoking __test_milepost_gcc.sh. If everything is installed correctly, you should get a response from the cTuning web-service: "Test passed successfully".
In order to continue using cTuning CC, you can check the following variables: Note that they already have default parameters so you do not have to change that unless you want to tune cTuning CC:
CCC_CTS_URL=cTuning.org/wiki/index.php/Special:CDatabase?request=
- points to the cTuning web-service.
CCC_CTS_DB=cod_opt_cases - points to the database with optimization cases
from the community.
ICI_PLUGIN_VERBOSE=1 - if set to 1, additional diagnostic information from ICI plugins. ICI_VERBOSE=1 - if set to 1, additional diagnostic information from ICI.
ICI_PROG_FEAT_PASS=fre - sets pass after which to extract static program features.
CCC_COMPILER_FEATURES_ID=129504539516446542 - sets compiler ID which was used
to extract static program features for all programs
at cTuning.org. Do not changed it unless you really
understand what you are doing ;) !..
CCC_OPTS="-O3" - sets combination of flags to be used if cTuning prediction web-service
did not return optimization flags.
CCC_OPT_ARCH_USE=1 - if set to 1, cTuning CC will also use architecture-dependent flags
(such as -march=athlon64) from cTuning.org. If set to 0, architecture
dependent flags will be ignored.
TIME_THRESHOLD=0.3 - when calculating speedups at cTuning.org, only optimization cases
with EXECUTION TIME more than this threshold are considered.
NOTES= - when <>"", only those optimization cases are returned that have this NOTES.
PG_USE=0 - if set to 1, only those optimization cases are returned that have function and other
level profiling. If unset or set to 0, use only those cases that do not have profiling
to avoid speedup skewing due to profiling.
OUTPUT_CORRECT=1 - if set to 1, only those optimization cases are returned that have been
checked for correctness by comparing benchmark outputs for the original
and transformed program (note that it still does not guarantee that
the combination of optimizations is correct, but it helps to reduce
obvious wrong cases).
RUN_TIME=RUN_TIME - sets which execution time to use when calculating speedups
(RUN_TIME - overall program execution time,
while RUN_TIME USER - only user execution time)
SORT=012 - when predicting optimizations, the best combinations of optimizations
are selected from the most similar program. Naturally, that program
can have flags that improve not only execution time, but also code
size and compilation time among other parameters. Hence a user can
suggest an order of sorting speedups by:
0 - execution time
1 - code size,
2 - compilation time
before returning the top optimization. For example, when setting this variable to
012 - cTuning returns the optimization case with the highest execution time
and only then sorts them by code size improvement and compilation time speedup;
102 - cTuning returns the optimization case with the highest code size improvement,
then execution time speedup and then compilation time;
201 - cTuning returns the optimization case with the highest compilation time speedup,
then execution time speedup and only then code size.
CT_OPT_REPORT=1 - when set to 1, cTuning returns all optimization cases sorted according to SORT
environment variable together with the associated optimization ID so that user
could later force different optimization case, particularly when having multi-objective
optimization scenarios.
Here is an example of such output:
****************************************************************************
Checking program features (and aggregating them if generated) ...
Static program features:
ft1=9, ft2=2, ft3=1, ft4=0, ft5=4, ft6=1, ft7=0, ft8=2, ft9=1, ft10=0, ft11=0,
ft12=0, ft13=5, ft14=0, ft15=0, ft16=8, ft17=0, ft18=0, ft24=27, ft25=13.50, ft19=0,
ft39=0, ft20=1, ft21=0, ft33=0, ft21=24, ft35=2, ft22=11, ft23=0, ft34=6, ft36=3,
ft37=0, ft38=0, ft40=0, ft41=8, ft42=0, ft43=0, ft44=0, ft45=0, ft46=1, ft48=3, ft47=9,
ft49=0, ft51=0, ft50=55, ft52=21, ft53=0, ft54=2, ft55=0, ft26=0, ft27=0, ft28=0, ft29=0,
ft30=5, ft31=0, ft32=0
Submitting features to the cTuning web-service to predict good optimizations ...
cTuning Optimization Report (optimal optimization cases):
Distance from most close program (462.libquantum) = 0.639
Selected opt. case = 23011215880571251
Optimal cases on frontier (averaged speedups):
Ex.time: Code size: Comp. time: cTuning opt. case:
1.18 0.80 1.00 15423655473087225
1.21 0.80 0.80 29686176401405
1.25 0.70 0.80 4614589283098526
1.29 0.67 0.80 23011215880571251
1.25 0.70 0.80 15721270875126789
1.26 0.69 0.80 15128754576807000
1.29 0.67 1.00 19230939973657069
1.07 1.02 1.00 3258730975700728
1.21 0.80 1.00 23810155474721838
1.24 0.71 1.00 4699569679776380
1.26 0.68 0.83 15492934568598271
Predicted flags:
-O2 -fdelete-null-pointer-checks -fno-tree-pre -funroll-all-loops
Invoking command:
gcc -O2 -fdelete-null-pointer-checks -fno-tree-pre -funroll-all-loops bitarray.c
bitcnt_1.c bitcnt_2.c bitcnt_3.c bitcnt_4.c bitcnts.c bitfiles.c bitstrng.c
bstr_i.c loop-wrap.c
****************************************************************************
Multi-objective optimizations:
When there are many optimization cases that improve at the same time execution time, code size
and compilation time, the selection of an optimal optimization case depends on depends on end-user
usage scenarios: improving both execution time and code size is often required for embedded applications,
improving both compilation and execution time is important for data centers and real-time systems,
while improving only execution time is common for desktops and supercomputers. Hence, we provided several
other environment variables to select optimization cases on the frontier of the optimization space:
DIM=012 - returns optimization cases only on the frontier of all optimization cases.
For example DIM=01 produces 2D frontier for execution time speedup and code size improvement,
DIM=02 produces 2D frontier for execution time and compilation time speedups,
DIM=12 produces 2D frontier for code size improvement and compilation time speedup,
DIM=012 produces 3D frontier for all constraints.
CUT=0,0,0 - cuts optimization cases frontier on each dimension, i.e. if CUT=0,0,1.2
the frontier optimization cases should have compilation time speedup > 1.2,
if CUT=1,1,1, all optimization cases on frontier should have execution time
speedup > 1, code size improvement > 1 and compilation time > 1.
When using this mode with DIM=012 and CUT=1,1,1, only one optimization case will be returned (when using CT_OPT_REPORT=1):
1.07 1.02 1.00 3258730975700728
Note, that you have to select such cases manually, because cTuning CC will still use the top optimization case before building frontier since the last one really depend on user scenario.
The following info is very important to find optimization cases from similar program for the following architecture (you can most similar architecture to yours at with optimization case at http://cTuning.org/cdatabase)
CCC_PLATFORM_ID=2111574609159278179 (example for AMD Athlon 64 3700+) CCC_ENVIRONMENT_ID=2781195477254972989 (example for Linux Mandriva 2.6.17-10alchemy) CCC_COMPILER_ID=331350613878705696 (example for GCC 4.4.0)
When compiling large applications, feature extraction can take a very long time (and this is part of the future work to speed it up), so a user may want to extract features only of a few functions. In this case, a user should add the file _ctuning_select_functions.txt to the compilation directory where only those functions should be listed that need to be processed (one function per line).
* If you want to test low-level plugins, you can find self-explanatory tests in plugins-ici-2.05/tests directory.
Usage
* cTuning web-services test:
ctuning-cc --ct-test *.c
You can also use test script ./__test_ctuning_web_service_for_ctuning_cc.sh
* Using optimization cases directly from the Collective Optimization Database (referenced by unique ID) - it is useful for multi-objective optimization, to share optimization cases within the community or when publishing papers and results on program optimization:
ctuning-cc --ct-opt=11475790782770590 *.c
You can also use demo script ./__compile_using_ctuning_cc_with_fixed_optimization.sh to understand how to configure your own system.
* Predict good optimizations (execution time, code size, compilation time) based on correlation of program features and optimizations using collective optimization knowledge (empirical iterative feedback-directed compilation performed by multiple users and shared in the Collective Optimization Database):
ctuning-cc -Oml *.c
You can also use demo script ./__compile_using_ctuning_cc_with_predicted_optimization.sh (or ./__compile_using_ctuning_cc_with_predicted_optimization_tr.sh for transparent invocation of this mode without flags through environment variables) to understand how to configure your own system.
* Extract program structure:
ctuning-cc -O3 --ct-extract-structure *.c
You can also use demo script ./__extract_program_structure_using_ctuning_cc.sh (or __extract_program_structure_using_ctuning_cc_tr.sh. for transparent invocation of this mode without flags through environment variables) to understand how to configure your own system.
* Extract program features:
ctuning-cc -O3 --ct-extract-features *.c
You can also use demo script ./__extract_program_features_using_ctuning_cc.sh (or __extract_program_features_using_ctuning_cc_tr.sh. for transparent invocation of this mode without flags through environment variables) to understand how to configure your own system.
* Some of the above methods can be invoked transparently without any Makefile modifications, using CTUNING_* environment variables. Look at the scripts in demo directory.
Real usage cases
- Nikhil Kapur is trying to use cTuning CC/MILEPOST GCC to optimize Mozilla code. More info is available at his blog.
- Yuriy Kashikov has been using cTuning CC/MILEPOST GCC to optimize BerkeleyDB and reported speedups of 1.4 times in comparison with native GCC 4.4.0 (-O3) on several Intel Xeon machines without loosing much compilation time or code size.
- Grigori Fursin and Abdul Memon have been using cTuning CC/MILEPOST GCC to optimize several audio/video libraries for ARC/AMD/Intel platforms for execution time, code size and compilation time constraints.