|
|
| (25 intermediate revisions not shown.) |
| Line 1: |
Line 1: |
| - | '''Continuous Collective Compilation Framework documentation'''
| |
| - |
| |
| | {{CMenu:CTools|CCC|}} | | {{CMenu:CTools|CCC|}} |
| | | | |
| - | ''This is the first draft of the documentation. It should be updated. Any help is appreciated.'' | + | '''Continuous Collective Compilation Framework documentation''' |
| - | | + | |
| - | === License ===
| + | |
| - | | + | |
| - | This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License version 2 as published by the Free Software Foundation.
| + | |
| - | | + | |
| - | This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the [http://www.gnu.org/copyleft/gpl.html GNU General Public License for more details].
| + | |
| - | | + | |
| - | If you found this software useful, you are welcome (but not obliged) to reference http://ctuning.org website and these publications {{Ref|FT2009}},{{Ref|FMTP2008}} in your derivative works.
| + | |
| - | | + | |
| - | === Framework high-level overview ===
| + | |
| - | | + | |
| - | http://ctuning.org/wiki/images/figure_ccc.gif
| + | |
| - | | + | |
| - | === Requirements ===
| + | |
| - | | + | |
| - | To successfully install CCC Framework, you need:
| + | |
| - | | + | |
| - | * C/C++ compiler (GCC) (''we hope to convert 1 program that uses C++ to C'').
| + | |
| - | * MySQL client and development headers /include/mysql/* ''(not strictly required but functionality will be reduced)''.
| + | |
| - | * PHP ''(not strictly required but functionality will be reduced)''.
| + | |
| - | * PAPI library ''(not strictly required - used in auxiliary tools)''.
| + | |
| - | | + | |
| - | === Directory structure ===
| + | |
| - | | + | |
| - | * '''/'''
| + | |
| - | ** ''README.txt'' - brief README and GPL license info
| + | |
| - | ** ''INSTALL.sh'' - main installation script that calls ccc-configure* scripts one by one
| + | |
| - | ** ''ccc-configure*'' - distributed configuration scripts for different tasks (''See Installation section'')
| + | |
| - | ** ''ccc-build.cfg'' - CCC Framework build number
| + | |
| - | ** ''ccc-build-db.cfg'' - lower and upper version number of the [[CDatabase|Collective Optimization Database]] which this version is intended to work with (to avoid incompatibility when both CCC Framework and COD are evolving).
| + | |
| - | | + | |
| - | * '''/src-plat-dep''' - platform dependent tools and plugins
| + | |
| - | ** ''/include/ccc'' - CCC Framework header files
| + | |
| - | ** ''/lib'' - auxiliary functions
| + | |
| - | ** ''/plugins'' - plugins sources
| + | |
| - | *** ''/compilation'' - iterative feedback-directed compilation plugins
| + | |
| - | *** ''/ml-prediction'' - machine learning prediction plugins
| + | |
| - | ** ''/tools'' - low-level tools
| + | |
| - | *** ''/ccc-time'' - substitution for standard time program to collect different profile info about program including hardware counters support
| + | |
| - | *** ''/ccc-comp'' - basic tool dealing with program compilation
| + | |
| - | *** ''/ccc-run'' - basic tool dealing with program execution and profiling collection
| + | |
| - | *** ''/ccc-db-send-stats-comp'' - send compilation statistics to COD
| + | |
| - | *** ''/ccc-db-send-stats-comp-passes'' - send info about compiler optimizations (at function-level) to COD
| + | |
| - | *** ''/ccc-db-send-stats-prog-feat'' - send info about program static features (machine learning) to COD
| + | |
| - | *** ''/ccc-db-send-stats-run'' - send execution statistics to COD
| + | |
| - | ** ''/tools-aux'' - auxiliary tools if system supports that
| + | |
| - | *** ''/hardware-counters-papi'' - collecting hardware counters statistics (dynamic program features for machine learning or statistical analysis)
| + | |
| - | | + | |
| - | * '''/src-plat-indep''' - platform independent tools and plugins
| + | |
| - | ** ''/include/ccc_script_functions.php'' - library that supports platform independent plugins and deals mostly with COD
| + | |
| - | ** ''/plugins'' - plugins and scripts
| + | |
| - | | + | |
| - | * '''/cfg''' - CCC configuration directories for different architectures/environments/compilers
| + | |
| - | ** ''/default'' - default configurations including optimizations for several compilers (GCC,Open64,PathScale)
| + | |
| - | | + | |
| - | * '''/install''' - installation directory for (platform-dependent) tools and scripts
| + | |
| - | | + | |
| - | * '''/apps''' - applications converted to work with CCC and scripts to automate iterative compilation
| + | |
| - | | + | |
| - | === Installation ===
| + | |
| - | | + | |
| - | Installation is performed using INSTALL.sh script for each platform (architecture/environment) on which iterative compilation experiments will be performed.
| + | |
| - | This script calls individual ccc-configure-* scripts to configure the following:
| + | |
| - | | + | |
| - | * '''ccc-configure-1-platform-name.sh''' - Select local name of the platform (the directory with this name will be created in ''cfg'' and ''install'' directories).
| + | |
| - | | + | |
| - | * '''ccc-configure-2-database.sh''' - Configure [[CDatabase|Collective Optimization Database]] access if it is used for experiments. If it is not used, all compilation and execution statistics is recorded in local files and can later be send to database in case you would like to share your optimization cases.
| + | |
| - | | + | |
| - | * '''ccc-configure-2b-database-test.sh''' - Test COD access using parameters from the previous step.
| + | |
| - | | + | |
| - | * '''ccc-configure-3-platform.sh''' - Provide architecture info for experiments (you can view if similar already [http://ctuning.org/wiki/index.php/Special:CDatabase?request=view_architectures] exists and use its unique ID or add new architecture info [http://ctuning.org/wiki/index.php/Special:CDatabase?request=add_architecture directly]). Each architecture has its unique ID to be able to share optimization cases.
| + | |
| - | | + | |
| - | * '''ccc-configure-4-environment.sh''' - Provide environment info for experiments (you can view if similar already [http://ctuning.org/wiki/index.php/Special:CDatabase?request=view_environments] exists and use its unique ID or add new environment info [http://ctuning.org/wiki/index.php/Special:CDatabase?request=add_environment directly]). Each environment has its unique ID to be able to share optimization cases.
| + | |
| - | | + | |
| - | * '''ccc-configure-5-compiler.sh''' - Provide compiler info for experiments (you can view if similar already [http://ctuning.org/wiki/index.php/Special:CDatabase?request=view_compilers] exists and use its unique ID or add new compiler info [http://ctuning.org/wiki/index.php/Special:CDatabase?request=add_compiler directly]). Each compiler has its unique ID to be able to share optimization cases.
| + | |
| - | | + | |
| - | * '''ccc-configure-6-compile-all-plugins.sh''' - Compile and configure all plugins
| + | |
| - | | + | |
| - | * '''ccc-configure-6-compile-all-tools-aux.sh''' - Compile all auxiliary tools if platform supports them
| + | |
| - | | + | |
| - | * '''ccc-configure-6-compile-all-tools.sh''' - Compile all low-level tools
| + | |
| - | | + | |
| - | * '''ccc-configure-7-update.sh''' - Check for update
| + | |
| - | | + | |
| - | * '''ccc-configure-8-set-environment.sh''' - Set environment based on the information entered in all previous steps. The environment files ''ccc-env.sh'' and ''ccc-env.csh'' for your platform will be created in the directory ''cfg/<platform_name>/''. You can edit them to correct paths to specific compilers such as GCC with ICI, MILEPOST GCC, LLVM, Open64, PathScale, Testarossa, Intel, etc.
| + | |
| - | | + | |
| - | This script has to be invoked for a given platform before performing any experiments.
| + | |
| - | The ccc-configure* scripts can later be invoked individually if needed.
| + | |
| - | | + | |
| - | === Collective Optimization Database ===
| + | |
| - | | + | |
| - | COD has been recently separated into 2 parts: common and experimental.
| + | |
| - | The '''common database''' keeps information about architectures, environments, compilers, programs, compiler flags, architecture configurations, etc - the information that can be common for many users.
| + | |
| - | The '''experimental databases''' keep local information about optimization cases after iterative feedback-directed compilation. They can have user-sensitive information and hence should not always be shared. User can later select interesting optimization cases to share.
| + | |
| - | | + | |
| - | Some more info about COD web-services/API is [http://ctuning.org/wiki/index.php/CDatabase:Documentation:API here].
| + | |
| - | | + | |
| - | ''To be extended''
| + | |
| - | | + | |
| - | === Applications ===
| + | |
| - | | + | |
| - | CCC Framework is intended to automate a large number of iterative compilation experiments. Application has to be slightly modified to work with CCC. For example, [[CTools:CBench|Collective Benchmark]] is already prepared to be used directly with the latest CCC Framework. [[CTools:CBench:Format Here]] you can find more info about benchmark format. Eventually, we plan to add full support to enable application optimizations transparently without any Makefile modifications.
| + | |
| - | | + | |
| - | === Low-level tools ===
| + | |
| - | | + | |
| - | There are 3 main low-level tools that abstract platform from iterative compilation experiments:
| + | |
| - | | + | |
| - | * '''ccc-time'''
| + | |
| - | | + | |
| - | Command line: ccc-time -fe <name_of_executable> -fp <command_line_for_executable> -ft <file to save time>
| + | |
| - | | + | |
| - | This program substitutes native time to profile program and potentially support different architectural features such as hardware counters, etc.
| + | |
| - | Normally, if users work with ported to CCC Framework applications such as CBench, they will manipulate only with ccc-comp and ccc-run or high-level plugins and will not use this tool directly.
| + | |
| - | | + | |
| - | * '''ccc-comp'''
| + | |
| - | | + | |
| - | | + | |
| - | | + | |
| - | === Iterative compilation example ===
| + | |
| - | | + | |
| - | === Plugins ===
| + | |
| - | ==== Iterative compilation plugins ====
| + | |
| - | | + | |
| - | ==== Machine learning plugins ====
| + | |
| - | | + | |
| - | ==== Data Analysis plugins ====
| + | |
| | | | |
| - | <BR>
| + | * [[CTools:CCC:Documentation:CCC_V2.5|CCC V2.5]] |
| - | <BR>
| + | * [[CTools:CCC:Documentation:CCC_V2.0x|CCC V2.0x]] |
| - | <BR>
| + | |
| - | <BR>
| + | |
| - | <BR>
| + | |
| - | <BR>
| + | |
| - | <BR>
| + | |
| - | <BR>
| + | |
| - | <BR>
| + | |
| - | <BR>
| + | |
| - | <BR>
| + | |
| - | <BR>
| + | |
| - | <BR>
| + | |
| - | <BR>
| + | |
| - | <BR>
| + | |
| - | <BR>
| + | |
| - | <BR>
| + | |
| - | <BR>
| + | |
| - | <BR>
| + | |
| - | <BR>
| + | |
| - | <BR>
| + | |
| - | <BR>
| + | |
| - | <BR>
| + | |
| - | <BR>
| + | |
| - | <BR>
| + | |