CTools:CCC

From cTuning.org

(Difference between revisions)

Revision as of 21:58, 16 March 2010

	Continuous Collective Compilation Framework
	*Enabling collective optimization*

Navigation: cTuning.org > CTools

Continuous Collective Compilation Framework (CCC) is a collaborative modular plugin-enabled R&D infrastructure to automate program and architecture optimizations (i.e. iteratively search for good program and architecture optimizations in a feedback-directed manner) and gather various static and dynamic optimization profile data in a Collective Optimization Database. It supports both global program optimization and fine-grain procedure, loop or instruction level optimizations if compiler supports Interactive Compilation Interface. CCC is used to help end-users optimize their programs, libraries and whole OS automatically (improve execution time/code size, etc), test and tune compiler optimization heuristic. It also enables collaborative R&D and optimization knowledge reuse with statistical and machine learning techniques (FMTP2008, FT2009). Current version of CCC supports cBench/MiBench/EEMBC/Polyhedron/SPEC_CPU95,2000,2006 benchmark suites.

CCC Framework is a community-driven project - you are welcome to join the project, extend it, provide smart search and data analysis plugins, leave feedback and add your optimization data to help the community. You can also communicate with cTuning community through our mailing lists.

Background:

Downloads

Plugin repository

Documentation and APIs

FAQs - Why do we need CCC framework? Is it similar to ACOVEA tool? etc...

Projects to extend CCC Framework (development plan)

Current development:

Related:
- Collective optimization repository to continuously collect profitable optimization cases from the community that improve program execution time, code size, compilation time, etc...
- MILEPOST GCC - machine learning-enabled compiler that automatically correlates program features and optimizations collected using CCC framework to be able to predict profitable optimizations to improve execution time, code size and compilation time for unseen programs.

Example of complex optimization search spaces for susan_c from Collective Benchmark and MILEPOST GCC 4.4.0 after randomly selecting about 80 optimization flags:

We can automatically improve execution time of the program by nearly 2 times over the highest GCC optimization level after using CCC framework (we obtain similar results on LLVM, Open64, Intel and IBM compilers). We can also use CCC framework to perform multi-objective optimizations (selecting optimization cases on the optimization space frontier shown by red circles and blue dots) such as:

optimize both execution time and code size (important for optimizing libraries and embedded/mobile computing systems)
optimize just execution time (important for desktop computers and HPC servers/supercomputers)
optimize both execution time and compilation time (important for cloud computing services and real-time systems)

Furthermore, we are developing MILEPOST GCC to correlate program features and behavior to predict good optimizations for unseen programs based on prior learning.

News

2010.January.28 - Proceedings and slides from GROW'10 and SMART'10 are now available online.

2010.January.25 - We added Speedup Test beta plugin to cTuning collective optimization database to enable precise and rigorous statistical analysis of the performance of benchmarks.

2010.January.4 - Call for participation: GROW'10 and SMART'10 workshops will be held on the 23rd and 24th of January in Pisa, Italy co-located with the HiPEAC conference. Preliminary programs are available: SMART'10 program and GROW'10 program.

2010.January.1 - We wish you all a very happy and prosperous New Year with lots of exciting achievements maybe even related to cTuning technology ;) !

2009.November.11 - Small CCC analysis plugins update available at SVN to speedup queries when dealing with large amount of optimization data in cDatabases (gigabytes of data).

2009.November.7 - Submission deadline for SMART'10 workshop has been extended until the 22nd of November, 2009.

2009.October.2 - We successfully passed the final MILEPOST review and the project is officially over. We would like to thank all the partners from the University of Edinburgh, IBM Haifa, CAPS and ARC for a great collaborative work during last 3 years and cTuning community for a very interesting feedback and extensions! We released all the tools from the project and hope to continue extending them within community-driven cTuning.org. This infrastructure should open up many interesting research opportunities for performance auto-tuning based on statistical and machine learning techniques so we hope to see many more interesting extensions to the MILEPOST/cTuning technology soon ;) !..

2009.September.25 - New CFP for SMART'10 workshop co-located with HiPEAC'10 conference in Pisa, Italy is now available. Prof. Keith Cooper from Rice University kindly agreed to give a keynote talk.

2009.September.22 - Congratulations to Dr.Christophe Dubach who has won a presitigious BCS/CPHC Distinguished Dissertation Award for his thesis "Using Machine-Learning to Efficiently Explore the Architecture/Compiler Co-Design Space" supervised by Prof. Michael O'Boyle. This topic is related to cTuning technology!

2009.September.1 - The documentation of MILEPOST GCC/GCC ICI extensions by Yuanjie and Liang during GSOC'09 program is now available: Function cloning and program instrumentation and Fine grain program tuning. We would like to fully test and sync these developments with mainline GCC within next month or two.

2009.August.05 - The colleagues from the UNIDAPT Group started investigating the use of cTuning/MILEPOST technology and the UNIDAPT framework to predict good optimization and parallelization schemes for hybrid heterogeneous CPU/GPU-like architectures together with CAPS Entreprise based on run-time adaptation and profiling, empirical iterative compilation, statistical analysis, machine learning, program and dataset features and run-time decision trees (FT2009, LCWP2009, Fur2009, JGVP2009, TWFP2009, FMTP2008, LFF2007, FCOP2005). They plan to add new optimization cases to the Collective Optimization Database in Autumn, 2009.

2009.July.27 - The paper "Portable Compiler Optimization Across Embedded Programs and Microarchitectures using Machine Learning" (DJBP2009) has been accepted for the 42nd IEEE/ACM International Symposium on Microarchitecture (MICRO). The research has been led by the colleagues from the University of Edinburgh - congratulations!

2009.July.22 - cTuning-related CFP: SMART'10 and GROW'10 workshops will be co-located with HiPEAC'10 conference in Pisa, Italy at the end of January, 2009. You are warmly invited to submit your novel research results and developments to our workshops! At the websites of these workshops you can find all information about topics, PC, deadlines and submission procedures.

2009.July.15 - New optimization cases to improve execution time, code size and compilation time are now available in Collective Optimization Database for cBench, EEMBC, SPEC and other applications with multiple datasets using MILEPOST GCC 4.4.0 and 2 architectures (Intel and AMD) from GRID5000 (infrastructure for large scale parallel and distributed computing research).

2009.July.08 - The preprint of a paper "Collective Tuning Initiative: automating and accelerating development and optimization of computing systems" describing cTuning infrastructure is now available online (Fur2009).

2009.June.29 - Stable MILEPOST GCC 4.4.0 has been released. Follow further community developments at cTuning GCC ICI page and cTuning development mailing list.
Next, we plan to use MILEPOST/cTuning technology to enable realistic adaptive parallelization, data partitioning and scheduling for heterogeneous multi-core systems using statistical and machine learning techniques.

2009.June.26 - The pdf of the paper that describes Collective Tuning Infrastructure and cTuning concept (presented at the GCC Summit'09) will be available in a few weeks here.

2009.June.10 - Extended version of the "Collective Optimization" paper (FT2009) describing collective tuning concept has been accepted for ACM Transactions on Architecture and Code Optimization (TACO).

2009.June.01 - After nearly 1 year of developments we released/updated all our open-source collaborative R&D tools:
- fully redesigned and documented Interactive Compilation Interface v2.0 for GCC 4.4.0 synchronized with the official plugin GCC branch - transforming compilers into plugin-enabled research toolsets
- MILEPOST GCC 4.4.0 pre-release version at SVN - automating program optimization and compiler optimization heuristic tuning using machine learning
- Continuous Collective Compilation Framework v2.0 - enabling automatic collaborative program optimization based on statistical and machine learning techniques
- Collective Benchmark/MiDataSets v1.0 - enabling realistic program optimization research and benchmarking using multiple open-source programs/datasets.
  
  We also updated Collective Optimization Database with various optimization cases for Intel and AMD processors and comparison of different compilers including GCC, LLVM, Open64, Intel, etc - enabling sharing and reuse of optimization knowledge.
  
  We would like to thank cTuning community for feedback, help and support! You are welcome to join this community effort to automate program optimization and compiler/architecture design.

2009.April.27 - We gave a talk at the University of Illinois at Urbana-Champaign about Collective Tuning Initiative and tools and MILEPOST project ("Collective Optimization, run-time adaptation and machine learning"). Presentation is available here. We would like to thank all the UIUC colleagues for a very interesting and useful feedback.

2009.April.21 - Several projects to enable automatic fine-grain program optimization and run-time adaptation in GCC using iterative compilation and machine learning (based on our cTuning/UNIDAPT/ICI/MILEPOST technology) have been accepted by the Google Summer of Code. You are welcome to join cTuning community and follow or participate in the developments using our dedicated mailing lists.

2009.March.08 - CCC pre-release version has been uploaded to the SVN. We are currently documenting it.

2009.March.02 - Google Summer of Code'2009 will be accepting development applications soon - you are welcome to check CCC extension projects and CCC feedback/suggestion pages if you would like to help us to extend our collaborative tools within this program!

Old news archive (2006-2008)

News archive

You are welcome to join us and participate in CCC developments, discussions, provide feedback or suggestions to extend CCC or add new functionality. The framework currently supports multiple compilers including GCC, Open64, PathScale, Intel ICC, IBM XLC and a large number of server, desktop and embedded architectures.

CCC supporters:

@@ Line 6: / Line 6: @@
 {{CMenu:CTools|}}
-Continuous Collective Compilation Framework (CCC) is a collaborative modular plugin-enabled R&D infrastructure to automate program and architecture optimizations (i.e. iteratively search for ''good'' program and architecture optimizations in a feedback-directed manner) and gather various static and dynamic optimization profile data in a [[CDatabase|Collective Optimization Database]]. It supports both global program optimization and fine-grain procedure, loop or instruction level optimizations if compiler supports [[CTools:ICI|Interactive Compilation Interface]]. It is used to help end-users optimize their programs, libraries and whole OS automatically (improve execution time/code size, etc), test and tune compiler optimization heuristic. It also enables collaborative R&D and optimization knowledge reuse with statistical and machine learning techniques ({{Ref|FMTP2008}}, {{Ref|FT2009}}).
+Continuous Collective Compilation Framework (CCC) is a collaborative modular plugin-enabled R&D infrastructure to automate program and architecture optimizations (i.e. iteratively search for ''good'' program and architecture optimizations in a feedback-directed manner) and gather various static and dynamic optimization profile data in a [[CDatabase|Collective Optimization Database]]. It supports both global program optimization and fine-grain procedure, loop or instruction level optimizations if compiler supports [[CTools:ICI|Interactive Compilation Interface]]. CCC is used to help end-users optimize their programs, libraries and whole OS automatically (improve execution time/code size, etc), test and tune compiler optimization heuristic. It also enables collaborative R&D and optimization knowledge reuse with statistical and machine learning techniques ({{Ref|FMTP2008}}, {{Ref|FT2009}}).
+Current version of CCC supports '''cBench/MiBench/EEMBC/Polyhedron/SPEC_CPU95,2000,2006''' benchmark suites.
 CCC Framework is a community-driven project - you are welcome to [[Join|join]] the project, [[CTools:CCC:Projects|extend]] it, provide smart search and data analysis plugins, leave [[CTools:CCC:Feedback|feedback]] and add your optimization data to help the community. You can also communicate with cTuning community through our [[Community|mailing lists]].