CTools:ICI:Projects:GSOC09:Scripts

From cTuning.org

Current revision

Scripts for Google Summer of Code 2009 Projects Contributers:

• Yuanjie Huang (ICT, China)
• Liang Peng (ICT, China)
• Yang Chen (ICT, China)

Overview

A set of scripts have been writen to automate the installation and compilation process of GSoC 2009 projects:

• Fine-grain optimization tuning
• Function cloning and program instrumentation

These scripts can be placed in other directories as long as they are in the same directory, here's an overview of their functionalities:

• ici-build-adapt-plugin.sh: build and install plugin
• ici-build-gcc.sh: build and install gcc-ici
• ici-build-debuggable-gcc.sh: build gcc-ici for debug
• ici-check-environment.sh: print environment variables
• ici-compile-with-adapt.sh: compile with adapt plugin
• ici-set-environment.sh: set environment variables
• ici-xml-util-cloning.py: modify XMLs for cloning automaticly
• ici-xml-util-tuning.py: modify XMLs for tuning automaticly

A step by step description will be given below to demostrate how you can use ICI easily with them:

Build and Install GCC-ICI and Adapt Plugin

Set shared parameters

Modify ici-set-environment.sh to specify important paths:

• ICI_PREFIX: where you want to install everything
• ICI_BUILD: where you would like to build libs and GCC
• ICI_SOURCE: where gcc-ici/branches lies
• ICI_BRANCH: which branch to build, this can be set to either "tuning" or "adapt"

Please use absolute path for these three variables

To config the place of XML files, you can set ICI_ADAPT_XMLDIR variable to where you save temporary XML files, If you set an relative path, the files will be generated relative to the current/working directory. For example, if you set ICI_ADAPT_XMLDIR="xmls" and your current/working dir is /somepath/, then XML files will be stored to /somepath/xmls/.

Remeber to extract the src-third-party tarball.

Build and install GCC

Build and install by runing:

$ ./ici-build-gcc.sh 0

This script would remove everything and perform clean build from souce given parameter 0, run this script without any parameters to access usage:

$ ./ici-build-gcc-adapt.sh
Usage: ici-build-gcc.sh <build>
build: 0 - remove all and build everything from source
       1 - remove gcc objects and rebuild gcc
       2 - rebuild gcc

Build and install adapt plugin

Run the following script after build and install GCC: </pre> $ ./ici-build-adapt-plugin.sh </pre>

This script would build and install libmxml, adapt plugin. Remember to run this script after the first step, because the plugin will be removed with gcc and other libs.

Build and a debuggable stage1 GCC

This is an optional step. A stage 1 gcc for debugging can be generated with the following script:

$ ./ici-rebuild-gcc-tuning-debug.sh

Remember to run this script after a clean all build of GCC-ICI.

Compile with ICI and Adapt Plugin

The compilation can be devided into three steps, and a script is provided to automate the process below:

Build 1

Create current XML with the compilation flow and info

$ ./ici-adapt-compile.sh 1

All xml files will be put in a directory specified by $ICI_ADAPT_XMLDIR variable in ici-set-environment script. This script can be used to compile with a make file or call gcc directly. A script which contains the compilation follow with stage1 gcc can be generated for debug too.

Run this script without any parameters to access usage:

$ ./ici-adapt-compile.sh
Usage: ici-adapt-compile.sh [lang] <ctrl> [func] [vrbs]
lang: -c (default)
      -fortran
ctrl: 1 - record XML
      2 - reuse XML
func: 1 - compile with make file
      2 - compile with GCC directly
      3 - compile with stage1 GCC
      4 - generate asm with GCC directly
      5 - generate asm with stage1 GCC
      9 - generate a script to debug stage1 gcc
vrbs: 1 ~ 3 plugin verbose level

Modify compilation flow

To enable function cloning and apply function specific optimizations, extra information should be added to XML files. A Python script is provides as a shorthand.

$ ./ici-xml-util.py [option] file.ini xmldir

• file.ini:controls how to perform function cloning and instrumentation,
• xmldir: dicrectory which contain XML files to be modified to enable cloning
• option -i: enable instrumentation
• option -t <filename>: generate external library template

The INI file format is a de facto standard for configuration files. We defined three kind of sections in script ici-xml-util.py with name [clone%d], [instrument%d], [opt] for function cloning, instrumentation and function specific optimizations respectively.

Here is an example ini file:

[clone1]
include=*                 ; which functions are included to be cloned
exclude=main*             ; which functions are excluded to be cloned
cloneno=4                 ; how many times will these functions be cloned
clonename=_clone          ; what is the postfix of clones' name
option0=-O3               ; optimization for the first clone,  i.e. function $ORIGINAL_FUNCTION_NAME_clone_0
option1=-O3               ; optimization for the second clone
option2=-O3               ; optimization for the third clone
option3=-O3               ; optimization for the fourth clone
selectname=clone1_select  ; name of external function that will be called to determine to run original function or one of the clones. 

[instrument1]
include=*                 ; which functions are included to be instrumented
exclude=*clone*           ; which functions are excluded to be instrumented
before=_instr_start       ; postfix of external function's name that will be instrumented at the beginning of function
after=_instr_end          ; postfix of external function's name that will be instrumented at the end of function

[opt]
include=test_nrv          ; which function is included to be specified optimization
option=-O3                ; optimization for this function

This Python script runs on Python-2.3 and above.

There is another Python script for tuning parameters automatically, which changes the parameters in current supported optimizations. This script is supposed to be used with some iterative compilation driver. This python script This script finds XML files in ICI_ADAPT_XMLDIR, so if you want to run it directly in shell, make sure the ICI_ADAPT_XMLDIR environment variable is valid.

ICI_ADAPT_XMLDIR="adapt-xmls" ici-xml-param-tuning.py

This Python script runs on Python-2.4 and above.

Please feel free to edit these two scripts for your own experiments.

Build 2

Compiles program with heuristics from XML files(function cloning, function specific optimization options, optimization pass sequence and paramters):

$ ./ici-adapt-compile.sh 2

All in One

A script is provided to automate the process above:

$ ./ici-auto-compile-with-adapt.sh

This script is designed to run in the benchmark source directory with an ini file providing instructions to modify XML files named "adapt.ini", and if there is a file named "__ici_run.sh", it will be called at the end of compilation.