From cTuning.org
Navigation: cTuning.org > CTools > CTuningCC
cTuning Compiler Collection V2.5 combined with MILEPOST GCC 4.4.x, ICI V2.05 (Interactive Compilation Interface), MILEPOST feature extractor V2.1 and any user compiler
Description
cTuning CC is a free, open source compiler collection that combines multiple tools and techniques developed during more than 10 years as the first practical step toward self-tuning, adaptive computing systems based on industrial tools, empirical techniques, collective optimization, statistical analysis and machine learning.
It may not always be visible to the IT users, but developing and optimizing computing systems using available over-complicated technology is too time consuming and costly often resulting in underperforming, power-hungry and inefficient computers and programs. Novel cTuning technology attempts to overcome the complexity of computing system by automating architecture, code and dataset analysis, characterization and multi- objective optimization (currently execution time, code size and compilation time) and enabling portable optimization using
- continuous parameterization of all components of a computing system (from architecture to operating system, compiler and code),
- continuous empirical collective optimization space exploration distributed among multiple users,
- continuous profiling and characterization of applications (extraction of program and architecture features), run-time behavior and resources,
- continuous sharing of analysis and optimization information in the Collective Optimization Database
- continuous refining and adaptation of performance models and optimization prediction based on standard statistical and machine learning techniques.
cTuning CC includes:
- New cTuning compiler wrapper to transparently extract program structure and features (using MILEPOST GCC), communicate with cTuning web services to share optimization data and predict optimizations, and invoke any other user compiler (GCC, LLVM, Open64, ICC, XL, ROSE, etc)
- MILEPOST GCC 4.4.x (self-tuning, adaptive, machine-learning based compiler) with ICI (plugin framework) v2.05 and MILEPOST feature extractor V2.1
- New Continuous Collective Compilation framework
- Collective Benchmark
- New plugins and web-services for multi-objective optimizations(balancing execution time, code size, compilation time)
We are developing cTuning infrastructure as a very simple, modular and portable tool so that users could easily download, install and use it to compile, execute, characterize and optimize their programs or share optimization knowledge. Our users managed to optimize some large industrial applications such as BerkeleyDB (1.4 speedup over GCC 4.4.0 -O3 on several Intel Xeon machines), some audio and video codecs, multiple standard benchmarks, Linux kernel, etc.
Please, note that this is an on-going, evolving project driven by the cTuning community, so please be patient or join the project and help to improve cTuning infrastructure.
Category:
self-tuning, adaptive computing systems
Keywords:
adaptive compiler, self-tuning compiler, intelligent compiler, iterative compilation, feedback-directed compilation, collective optimization, program features, optimization prediction, predictive modeling, machine learning, statistical analysis, multi-objective optimization (balance execution time, code size, compilation time), optimization space frontier, collaborative experimental data sharing, cTuning web-services
Coordination
- Grigori Fursin, UNIDAPT Group, UVSQ, France (original R&D for MILEPOST framework/ICI prototypes/CCC framework/Collective Optimization Database, cTuning.org and self-tuning computing systems)
I am on sabbatical from March, 2010 to help create a new Exascale Research Center in France so I have very little spare time to coordinate these developments. I hope that my students and the community will continue extending this framework ...
License
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License version 3 as published by the Free Software Foundation.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
If you found this software useful, you are welcome to reference http://cTuning.org website and these publications FMTP2008,Fur2009,FT2009 in your derivative works.
Development/testing/evaluation
- cTuning Compiler Collection V2.5 (and unreleased MILEPOST GCC V2.1) (2009-2010)
- Grigori Fursin (UVSQ, France) - new design, new cTuning compiler wrapper around MILEPOST GCC and other compilers such as LLVM, Open64, ROSE, ICC, XL, etc, CCC framework, new statistical analysis plugins and new optimization/prediction services.
- Yuriy Kashnikov (UVSQ, France) - testing/evaluation on Berkeley DB
- Abdul Wahid Memon (UVSQ, France) - testing/evaluation on cBench and bug fixes/extensions to support whole Linux recompilation (such as GENTOO)
- Jose Noudohouenou (UVSQ, France) - testing on large exascale applications
- Joern Rennecke (UK) - testing/providing support for g++ in MILEPOST GCC; moving some parts of ICI to mainline GCC 4.5 (plugin framework)
- Jeremy Singer (University of Manchester, UK) - adding new static program features
- Nikhil Kapur - testing on Mozilla/libvorbis
Includes ICI and CCC frameworks.
Framework high-level overview
History
cTuning CC V2.5 - 20100520 - * added cTuning CC wrapper that can be used with any
compiler to enable transparent architecture, code and
dataset analysis, characterization and multi-objective
optimization (currently execution time, code size and
compilation time) based on empirical iterative feedback
directed compilation, statistical analysis, collective
optimization and machine learning (predictive modeling).
cTuning-cc performs the following:
* detects special flags or environment variables to invoke
analysis compilers that support Interactive Compilation
Interface (currently MILEPOST GCC -
http://cTuning.org/milepost-gcc) to analyze code
structure, extract features, select and reorder
optimizations, etc.
* communicates with cTuning.org web-services and Collective
Optimization Database (http://cTuning.org/cdatabase) to
suggest better optimizations based on program, dataset
and architecture features and machine learning or to
return an optimization referenced by the unique cTuning
ID (useful for manual sharing of optimization data,
academic experiments or bug reports, etc).
* invokes any user compiler (for example, GCC, LLVM, ICC,
Open64, Rose, XL, etc) with the returned optimizations
from cTuning.org.
* created new cTuning wiki page for cTuning Compiler Collection
http://cTuning.org/ctuning-cc
* added full ccc-framework to enable characterization of
programs and architectures, iterative compilation for
multi-objective optimization (execution time, code size,
compilation time), training of machine learning models to
predict optimizations, collective optimization
* fixed compilation of GCC 4.4.x and 4.5.0 with new GMP and PPL
(also added a few flags during GCC compilation from
http://openwall.info/wiki/internal/gcc-local-build)
* added support for GCC 4.4.4
* added new cTuning CC flags and environment variables
to transparently/explicitly extract program structure
and program features
* added more demo scripts
MILEPOST GCC V2.1 (4.4.x) - 20100315 - Pre-release of the fully updated compiler that includes
parts of CCC framework and can transparently communicate
with cTuning web-services to suggest good optimization
cases to improve/balance execution time, code size and compilation time
using correlation between program features and optimizations.
The MILEPOST GCC wrapper from CCC framework can be easily converted
to work with any other compiler such as LLVM, Open64, Intel compilers, etc
to predict good optimizations based on correlation between program features,
optimization and run-time behavior.
It also allows to directly and transparently use optimizations
Collective Optimization Database (http://cTuning.org/cdatabase)
referenced by unique optimiation ID that is useful for sharing
of profitable optimization cases with the community.
Preliminary experiments show that it is now possible to transparently
recompile standard programs/libraries/Linux kernel and the whole
Linux with new MILEPOST GCC. We are looking for voolontiers to evaluate
performance for Linux individual programs/libraries/kernel.
MILEPOST GCC V2.1 now officially supports C,C++ and Fortran.
MILEPOST GCC V1.5 and V2.0 - Internal development versions of compiler that were not officially released.
MILEPOST GCC 4.4.0 - 20090629 - New official version of MILEPOST GCC with new ICI v2.0
and updated static feature extractor.
MILEPOST GCC 4.2.2 - 20080613 - Stable MILEPOST GCC version used in most MILEPOST Year 3 experiments.
Requirements
In order to install cTuning CC and MILEPOST GCC, you will need:
- C compiler that can compile GCC 4.x.
- uuid or uudigen tool to generate unique identifiers.
- PHP (needed to communicate with cTuning web-services).
Directory structure
milepost-gcc-4.4.x - MILEPOST GCC 4.4.x source directory (core + g++ + gfortran +
GRAPHITE support) with ICI v2.05 and MILEPOST feature extractor V2.1
ccc-framework - Continuous Collective Compilation Framework to distribute optimization space exploration among multiple users, share optimization data in Collective Optimization Database to enable machine learning for optimization predictions. This framework includes cTuning compiler wrapper to transparently extract program structure and features (using any compiler that supports ICI such as MILEPOST GCC), communicate with cTuning web services to share optimization data and predict optimizations, and invoke any other user compiler (GCC, LLVM, Open64, ROSE, etc).
src-third-party - Third party support tools | +-- gmp-5.0.1 - GMP library (user can update if needed) +-- gmp-4.3.0 - Different GMP version (*) +-- mpfr-2.4.2 - MPFR library (a user can update if needed) +-- mpfr-2.4.1 - Different MPFR version (*) +-- mxml-2.6 - XML library for plugins +-- ppl-0.10.2-modified - PPL library (for GRAPHITE), slightly modified by Grigori | to support GMP-5.0.1 +-- ppl-0.10.2 - Different PPL library (*) +-- cloog - CLOOG library (for GRAPHITE, latest version from GIT) +-- cloog.old-milepost-gcc-v1 - Different CLOOG versions (*) +-- mpc-0.8.1 - +-- XSB - Prolog to calculate program features
plugins-ici-2.0x - Plugins for GCC 4.4.x with ICI (see README inside this directory)
demo - Demo files for MILEPOST GCC | +-- bitcount - bitcount example written in C from cBench. +-- bzip2-1.0.5 - bzip2 written in C with a few scripts to show how to use MILEPOST GCC | with standard programs without any project changes. +-- libvorbis-1.2.3 - standard vorbis library to show how to use MILEPOST GCC | with standard libraries/kernel without any project changes. +-- matmul.c - simple matmul example written in C. +-- matmul.cpp - simple matmul example written in C++. +-- matmul.fortran - simple matmul example written in Fortran.
install - Installation directory
(*) included for compatibility with older versions of cTuning CC/MILEPOST GCC
to be able to reproduce experimental/optimization results
Installation
First, check in all scripts that you have the same BUILD_EXT variable that points to the install directory! You may have different names if you install MILEPOST GCC for several architectures on the shared file system ...
Invoke:
./_build_all.sh to build the whole cTuning Compiler Collection with all necessary tools.
This script invokes the following scripts: ./_build_gcc.sh to build GCC with all the third-party tools. ./_build_ccc.sh to build CCC framework with cTuning compiler wrapper. ./_build_plugins.sh will build all non-machine learning plugins. ./_build_plugins_ml.sh will build all machine learning plugins.
General configuration
Check ./_set_environment_for_analysis_compiler__milepost_gcc.sh - normally all environment variables should be already properly set (check variable CCC_UUID - the uuid tool). You have to source this file before using cTuning CC - it tells cTuning CC to use MILEPOST GCC for program analysis (extraction of features and access to fine-grain optimizations through ICI).
Importantly, cTuning CC can now use any compiler that supports ICI and cTuning/MILEPOST technology for code analysis and characterization, that is configured through the following environment variables (using GCC as an example):
CTUNING_ANALYSIS_CC=gcc CTUNING_ANALYSIS_CPP=g++ CTUNING_ANALYSIS_FORTRAN=gfortran
File ./_set_environment_for_plugin_tests.sh sets up environment variables for low-level ICI tests and should also be already properly set. If you plan to use only high-level cTuning CC, you can skip it.
Example of _set_environment_for_analysis_compiler__milepost_gcc.sh:
#!/bin/bash
# Author: Grigori Fursin (http://fursin.net/research) # # (C) 2010, University of Versailles at Saint-Quentin-en-Yvelines, France
BUILD_EXT=install BUILD_DIR=$PWD/$BUILD_EXT PWD_SAVE=PWD
export PATH=$BUILD_DIR/bin:$PATH export LD_LIBRARY_PATH=$BUILD_DIR/lib:$BUILD_DIR/lib64:$LD_LIBRARY_PATH
#set MILEPOST compiler names (in the future it may be some other compiler #that supports ICI and plugins) export CTUNING_ANALYSIS_CC=gcc export CTUNING_ANALYSIS_CPP=g++ export CTUNING_ANALYSIS_FORTRAN=gfortran
#ICI PLUGINS export ICI2_PLUGIN_VER=gcc-plugin-ici2 export ICI_LIB=$BUILD_DIR/lib export ICI_PLUGIN_VER=$ICI2_PLUGIN_VER
export CCC_ICI_USE=ICI_USE export CCC_ICI_PLUGINS=ICI_PLUGIN
export CCC_ICI_PASSES_FN=ici_passes_function export CCC_ICI_PASSES_EXT=.txt
#export CCC_ICI_PASSES_RECORD_PLUGIN=$ICI_LIB/$ICI_PLUGIN_VER-save-executed-passes.so export CCC_ICI_PASSES_RECORD_PLUGIN=$ICI_LIB/$ICI_PLUGIN_VER-extract-program-structure.so export CCC_ICI_FEATURES_ST_FN=ici_features_function export CCC_ICI_FEATURES_ST_EXT=.ft export CCC_ICI_FEATURES_ST_EXTRACT_PLUGIN=$ICI_LIB/$ICI_PLUGIN_VER-extract-program-static-features.so
export ICI_PROG_FEAT_PASS=fre
export ML_ST_FEAT_TOOL=$BUILD_DIR/bin/featlstn.P export XSB_DIR="$BUILD_DIR/3.2" export ICI_PROG_FEAT_EXT_TOOL=$BUILD_DIR/bin/ml-feat-proc
export CCC_ROOT=$PWD/ccc-framework
export CCC_PLUGINS=$CCC_ROOT/src-plat-indep/ export PATH=$CCC_ROOT/src-plat-indep/plugins:$PATH
export CCC_UUID=uuidgen
Configuration for demos
- You can find how to use cTuning Compiler Collection either transparently without Makefile modifications or explicitly using multiple benchmarks in the demo directory (bitcount, bzip2, libvorbis, matmul).
You need to first configure environment variables in the ___common_environment.sh which are user-dependent:
cTuning CC can use 2 separate compilers - one for analysis (should support ICI and cTuning/MILEPOST technology for program and architecture characterization, self-tuning and adaptation such as MILEPOST GCC) and another can be any user compiler (GCC, LLVM, ICC, ROSE, Open64, XL, etc) driven by the analysis compiler.
User compiler is defined using the following environment variables (using GCC as an example):
CTUNING_COMPILER_CC=gcc CTUNING_COMPILER_CPP=g++ CTUNING_COMPILER_FORTRAN=gfortran
CCC_CTS_USER and CCC_CTS_PASS should be set to your username and password when self-registering at http://cTuning.org/wiki/index.php/Special:UserLogin
NOW YOU CAN TEST cTuning CC wrapper and communication with the cTuning database by invoking __test_milepost_gcc.sh. If everything is installed correctly, you should get a response from the cTuning web-service: "Test passed successfully".
In order to continue using cTuning CC, you can check the following variables: Note that they already have default parameters so you do not have to change that unless you want to tune cTuning CC:
CCC_CTS_URL=cTuning.org/wiki/index.php/Special:CDatabase?request=
- points to the cTuning web-service.
CCC_CTS_DB=cod_opt_cases - points to the database with optimization cases
from the community.
ICI_PLUGIN_VERBOSE=1 - if set to 1, additional diagnostic information from ICI plugins. ICI_VERBOSE=1 - if set to 1, additional diagnostic information from ICI. ICI_PROG_FEAT_PASS=fre - sets pass after which to extract static program features.
CCC_COMPILER_FEATURES_ID=129504539516446542 - sets compiler ID which was used
to extract static program features for all programs
at cTuning.org. Do not changed it unless you really
understand what you are doing ;) !..
CCC_OPTS="-O3" - sets combination of flags to be used if cTuning prediction web-service
did not return optimization flags.
CCC_OPT_ARCH_USE=1 - if set to 1, cTuning CC will also use architecture-dependent flags
(such as -march=athlon64) from cTuning.org. If set to 0, architecture
dependent flags will be ignored.
TIME_THRESHOLD=0.3 - when calculating speedups at cTuning.org, only optimization cases
with EXECUTION TIME more than this threshold are considered.
NOTES= - when <>"", only those optimization cases are returned that have this NOTES.
PG_USE=0 - if set to 1, only those optimization cases are returned that have function and other
level profiling. If unset or set to 0, use only those cases that do not have profiling
to avoid speedup skewing due to profiling.
OUTPUT_CORRECT=1 - if set to 1, only those optimization cases are returned that have been
checked for correctness by comparing benchmark outputs for the original
and transformed program (note that it still does not guarantee that
the combination of optimizations is correct, but it helps to reduce
obvious wrong cases).
RUN_TIME=RUN_TIME - sets which execution time to use when calculating speedups
(RUN_TIME - overall program execution time,
while RUN_TIME USER - only user execution time)
SORT=012 - when predicting optimizations, the best combinations of optimizations
are selected from the most similar program. Naturally, that program
can have flags that improve not only execution time, but also code
size and compilation time among other parameters. Hence a user can
suggest an order of sorting speedups by:
0 - execution time
1 - code size,
2 - compilation time
before returning the top optimization. For example, when setting this variable to
012 - cTuning returns the optimization case with the highest execution time
and only then sorts them by code size improvement and compilation time speedup;
102 - cTuning returns the optimization case with the highest code size improvement,
then execution time speedup and then compilation time;
201 - cTuning returns the optimization case with the highest compilation time speedup,
then execution time speedup and only then code size.
CT_OPT_REPORT=1 - when set to 1, cTuning returns all optimization cases sorted according to SORT
environment variable together with the associated optimization ID so that user
could later force different optimization case, particularly when having multi-objective
optimization scenarios.
Here is an example of such output:
****************************************************************************
Checking program features (and aggregating them if generated) ...
Static program features:
ft1=9, ft2=2, ft3=1, ft4=0, ft5=4, ft6=1, ft7=0, ft8=2, ft9=1, ft10=0, ft11=0,
ft12=0, ft13=5, ft14=0, ft15=0, ft16=8, ft17=0, ft18=0, ft24=27, ft25=13.50, ft19=0,
ft39=0, ft20=1, ft21=0, ft33=0, ft21=24, ft35=2, ft22=11, ft23=0, ft34=6, ft36=3,
ft37=0, ft38=0, ft40=0, ft41=8, ft42=0, ft43=0, ft44=0, ft45=0, ft46=1, ft48=3, ft47=9,
ft49=0, ft51=0, ft50=55, ft52=21, ft53=0, ft54=2, ft55=0, ft26=0, ft27=0, ft28=0, ft29=0,
ft30=5, ft31=0, ft32=0
Submitting features to the cTuning web-service to predict good optimizations ...
cTuning Optimization Report (optimal optimization cases):
Distance from most close program (462.libquantum) = 0.639
Selected opt. case = 23011215880571251
Optimal cases on frontier (averaged speedups):
Ex.time: Code size: Comp. time: cTuning opt. case:
1.18 0.80 1.00 15423655473087225
1.21 0.80 0.80 29686176401405
1.25 0.70 0.80 4614589283098526
1.29 0.67 0.80 23011215880571251
1.25 0.70 0.80 15721270875126789
1.26 0.69 0.80 15128754576807000
1.29 0.67 1.00 19230939973657069
1.07 1.02 1.00 3258730975700728
1.21 0.80 1.00 23810155474721838
1.24 0.71 1.00 4699569679776380
1.26 0.68 0.83 15492934568598271
Predicted flags:
-O2 -fdelete-null-pointer-checks -fno-tree-pre -funroll-all-loops
Invoking command:
gcc -O2 -fdelete-null-pointer-checks -fno-tree-pre -funroll-all-loops bitarray.c
bitcnt_1.c bitcnt_2.c bitcnt_3.c bitcnt_4.c bitcnts.c bitfiles.c bitstrng.c
bstr_i.c loop-wrap.c
****************************************************************************
Multi-objective optimizations:
When there are many optimization cases that improve at the same time execution time, code size
and compilation time, the selection of an optimal optimization case depends on depends on end-user
usage scenarios: improving both execution time and code size is often required for embedded applications,
improving both compilation and execution time is important for data centers and real-time systems,
while improving only execution time is common for desktops and supercomputers. Hence, we provided several
other environment variables to select optimization cases on the frontier of the optimization space:
DIM=012 - returns optimization cases only on the frontier of all optimization cases.
For example DIM=01 produces 2D frontier for execution time speedup and code size improvement,
DIM=02 produces 2D frontier for execution time and compilation time speedups,
DIM=12 produces 2D frontier for code size improvement and compilation time speedup,
DIM=012 produces 3D frontier for all constraints.
CUT=0,0,0 - cuts optimization cases frontier on each dimension, i.e. if CUT=0,0,1.2
the frontier optimization cases should have compilation time speedup > 1.2,
if CUT=1,1,1, all optimization cases on frontier should have execution time
speedup > 1, code size improvement > 1 and compilation time > 1.
When using this mode with DIM=012 and CUT=1,1,1, only one optimization case will be returned
(when using CT_OPT_REPORT=1):
1.07 1.02 1.00 3258730975700728
Note, that you have to select such cases manually, because cTuning CC will still use
the top optimization case before building frontier since the last one really depend on
user scenario.
The following info is very important to find optimization cases from similar program for the following architecture (you can most similar architecture to yours at with optimization case at http://cTuning.org/cdatabase)
CCC_PLATFORM_ID=2111574609159278179 (example for AMD Athlon 64 3700+) CCC_ENVIRONMENT_ID=2781195477254972989 (example for Linux Mandriva 2.6.17-10alchemy) CCC_COMPILER_ID=331350613878705696 (example for GCC 4.4.0)
When compiling large applications, feature extraction can take a very long time (and this is part of the future work to speed it up), so a user may want to extract features only of a few functions. In this case, a user should add the file _ctuning_select_functions.txt to the compilation directory where only those functions should be listed that need to be processed (one function per line).
- If you want to test low-level plugins, you can find self-explanatory tests in plugins-ici-2.05/tests directory.
Feature extractor
- Low level pass ml-feat (gcc-4.4.x/gcc/ml-feat.c) invoked through ICI after a given pass (currently fre). It saves low-level info about program into external file that is later processed by high-level feature extractor.
- High level feature extractor (plugins-ici-2.05/src-ml/extract-program-static-features.legacy/ml-feat-proc/featlstn.P) is written in Prolog to calculate features based on low-level information obtained from ml-feat pass).
V2.1 - featlstn.P - 55 features with removed duplicate feature ft21.
featlstn1.P - 56 features (move duplicate feature to ft56).
featlstn2.P - 57-65 features added by Jeremy Singer.
NOTE: Current cTuning.org prediction web-services, etc are hardwired to work with
the original feature list featlstn.P. In the future we should change that to
support any feature list. For example, we plan to add polyhedral program representation
as a feature set and then use cTuning learning and prediction services directly.
V2.0 - bug fixes
V1.0 - featlstn.P - had two duplicate features ft21 (thanks to Jeremy Singer who reported that bug).
Usage
- cTuning web-services test:
ctuning-cc --ct-test *.c
You can also use test script ./__test_ctuning_web_service_for_ctuning_cc.sh
- Using optimization cases directly from the Collective Optimization Database (referenced by unique ID) - it is useful for multi-objective optimization, to share optimization cases within the community or when publishing papers and results on program optimization:
ctuning-cc --ct-opt=11475790782770590 *.c
You can also use demo script ./__compile_using_ctuning_cc_with_fixed_optimization.sh to understand how to configure your own system.
- Predict good optimizations (execution time, code size, compilation time) based on correlation of program features and optimizations using collective optimization knowledge (empirical iterative feedback-directed compilation performed by multiple users and shared in the Collective Optimization Database):
ctuning-cc -Oml *.c You can also use demo script ./__compile_using_ctuning_cc_with_predicted_optimization.sh (or ./__compile_using_ctuning_cc_with_predicted_optimization_tr.sh for transparent invocation of this mode without flags through environment variables) to understand how to configure your own system.
- Extract program structure:
ctuning-cc -O3 --ct-extract-structure *.c You can also use demo script ./__extract_program_structure_using_ctuning_cc.sh (or __extract_program_structure_using_ctuning_cc_tr.sh. for transparent invocation of this mode without flags through environment variables) to understand how to configure your own system.
- Extract program features:
ctuning-cc -O3 --ct-extract-features *.c
You can also use demo script ./__extract_program_features_using_ctuning_cc.sh (or __extract_program_features_using_ctuning_cc_tr.sh. for transparent invocation of this mode without flags through environment variables) to understand how to configure your own system.
- Some of the above methods can be invoked transparently without any Makefile modifications, using CTUNING_* environment variables. Look at the scripts in demo directory.
Demos
Directory demo contains some benchmarks, real programs and libraries written in C, C++ and Fortran that demonstrate the usage of cTuning CC.
- All directories have file ___common_environment.sh - it is needed to configure cTuning CC and MILEPOST GCC including cTuning web-services and a username (you can self-register at cTuning.org to access web-services), default optimization level, how to balance predicted optimizations for improve execution time, code size and compilation time, and the most similar platform/environment/compiler to the user one in order to predict optimizations.
Example of ___common_environment.sh:
#!/bin/bash
# Author: Grigori Fursin (http://fursin.net/research) # # (C) 2010, University of Versailles at Saint-Quentin-en-Yvelines, France # (C) 2007-2010, UNIDAPT Group, INRIA, France (http://unidapt.org)
export ICI_PLUGIN_VERBOSE=1 export ICI_VERBOSE=1 export ICI_PROG_FEAT_PASS=fre
#set cTuning web-service parameters export CCC_CTS_URL=cTuning.org/wiki/index.php/Special:CDatabase?request= #export CCC_CTS_URL=localhost/cTuning/wiki/index.php/Special:CDatabase?request= export CCC_CTS_DB=cod_opt_cases #set cTuning username (self-register at http://cTuning.org/wiki/index.php/Special:UserLogin) export CCC_CTS_USER=gfursin
#set user compiler (currently gcc, but can be used with any other # compiler such as LLVM, ICC, XL, ROSE, Open64, etc) export CTUNING_COMPILER_CC=gcc export CTUNING_COMPILER_CPP=g++ export CTUNING_COMPILER_FORTRAN=gfortran
#misc parameters - don't change unless you understand what you do!
#compiler which was used to extract features for all programs to keep at cTuning.org #do not change it unless you understand what you do ;) ... export CCC_COMPILER_FEATURES_ID=129504539516446542
#use architecture flags from cTuning export CCC_OPT_ARCH_USE=0
#retrieve opt cases only when execution time > TIME_THRESHOLD export TIME_THRESHOLD=0.3
#retrieve opt cases only with specific notes #export NOTES=
#retrieve opt cases only when profile info is !="" #export PG_USE=1
#retrieve opt cases only when execution output is correct (or not if =0) export OUTPUT_CORRECT=1
#check user or total execution time #export RUN_TIME=RUN_TIME_USER export RUN_TIME=RUN_TIME
#Sort optimization case by speedup (0 - ex. time, 1 - code size, 2 - comp time) export SORT=012
#produce additional optimization report including optimization space froniters export CT_OPT_REPORT=1
#Produce optimization space frontier #export DIM=01 (2D frontier) #export DIM=02 (2D frontier) #export DIM=12 (2D frontier) #export DIM=012 (3D frontier) #export DIM=012
#Cut cases when producing frontier (select cases when speedup 0,1 or 2 is more than some threshold) #export CUT=0,0,1.2 #export CUT=1,0.80,1 #export CUT=0,0,1
#find similar cases from the following platform export CCC_PLATFORM_ID=2111574609159278179 export CCC_ENVIRONMENT_ID=2781195477254972989 export CCC_COMPILER_ID=331350613878705696
- Demo benchmarks also have the following scripts:
- __test_ctuning_web_service_for_ctuning_cc.sh - test cTuning web-service for cTuning CC / MILEPOST GCC.
- __compile_using_ctuning_cc.sh - compile program using cTuning CC / MILEPOST GCC.
- __compile_using_ctuning_cc_with_ccc_opts_tr.sh - compile program and transparently substitute original optimizations with the ones presented in environment variable CCC_OPTS (to enable empirical iterative and collective optimization).
- __compile_using_ctuning_cc_with_fixed_optimization.sh - compile program using cTuning CC and transparently use optimizations referenced by the unique ID from the Collective Optimization Database.
- __compile_using_ctuning_cc_with_predicted_optimization.sh - compile program using cTuning CC, extract features and attempt to predict good optimizations in an attempt to improve execution time, code size and compilation time depending on user optimization scenario (note that it doesn't always work well and it is an on-going R&D to improve prediction, the quality of features and association between features and optimizations).
- __compile_using_ctuning_cc_with_predicted_optimization_tr.sh - transparent invocation of program optimization using environment variables.
- __extract_program_structure_using_ctuning_cc.sh - extract program structure using MILEPOST GCC.
- __extract_program_structure_using_ctuning_cc_tr.sh - transparent invocation of program structure extractio using environment variables .
- __extract_program_features_using_ctuning_cc.sh - extract program features using MILEPOST feature extractor.
- __extract_program_features_using_ctuning_cc_tr.sh - transparent invocation of program feature extraction using environment variables.
- __run.sh to run compiled program
- All directories have additional files:
- _ctuning_select_functions.txt - users can specify which functions to process during optimization prediction (since feature extraction is quite slow at the moments and requires more work/local caching/etc). Users can remove this file to process the whole program .
- _ctuning_remove_opt_flags.gcc44p - users can specify which original optimization flags to remove when predicting or substituting optimizations (i.e. to avoid conflicts that original optimization has say flags -O3 -funroll-loops while predicted optimization using machine learning is -O2 -funroll-all-loops - in this case, all original optimization flags should be removed). The format of this file is described in CCC documentation.
- Additionally, directory libvorbis-1.2.3 has scripts:
- __configure_with_ctuning_cc_and_fixed_optimization.sh - to configure library with cTuning CC flags to substitute optimizations with the ones from Collective Optimization Database referenced by unique ID. The library is then recompiled using make command.
- __configure_with_ctuning_cc_and_optimization_prediction.sh - to configure library with cTuning CC flags to predict optimizations in an attempt to improve execution time, code size and compilation time depending on user optimization scenario. The library is then recompiled using make command.
cTuning compiler unique IDs (COMPILER_ID)
- cTuning CC 2.5 GCC 4.4.0 ICI 2.05 MILEPOST 2.1, COMPILER_ID=164654947683234
- cTuning CC 2.5 GCC 4.4.1 ICI 2.05 MILEPOST 2.1, COMPILER_ID=327397845688213
- cTuning CC 2.5 GCC 4.4.2 ICI 2.05 MILEPOST 2.1, COMPILER_ID=5432645853305414
- cTuning CC 2.5 GCC 4.4.3 ICI 2.05 MILEPOST 2.1, COMPILER_ID=93216642957846
- cTuning CC 2.5 GCC 4.4.4 ICI 2.05 MILEPOST 2.1, COMPILER_ID=674893631465783
Acknowledgments (cTuning CC / MILEPOST GCC / Interactive Compilation Interface / cTuning tools)
- MILEPOST project colleagues (University of Edinburgh, INRIA, CAPS Entreprise, IBM)
- Fabio Arnone (STMicroelectronics, France)
- Phil Barnard (ARC, UK)
- Francois Bodin (CAPS Entreprise, France)
- Zbigniew Chamski (InfraSoft IT Solutions, Poland)
- Bjorn Franke (University of Edinburgh, UK)
- Grigori Fursin (INRIA/UVSQ, France)
- Taras Glek (Mozilla, USA)
- Nikhil Kapur
- Yuriy Kashnikov (UVSQ, France)
- Abdul Wahid Memon (UVSQ, France)
- Cupertino Miranda (INRIA, France)
- Mircea Namolaru (IBM, Israel)
- Jose Noudohouenou (UVSQ, France)
- Diego Novillo (Google, USA)
- Sebastian Pop (AMD, USA)
- Joern Rennecke (UK)
- Jeremy Singer (University of Manchester, UK)
- Basile Starynkevitch (CEA, France)
- Ayal Zaks (IBM, Israel)
Other colleagues from IBM, NXP, STMicroelectronics, ARC, CAPS Enterprise, Mozilla, UVSQ ...