Contents
Disclaimer
This documentation is written by Grigori Fursin and is gradually evolving together with the project (until the stable version is released). Please contact us if you notice mistakes or inconsistencies to collaboratively improve this document!
Introduction
cM framework is fully documented using DoxyGen (basically, all modules' functions and OpenME interface functions). Live documentation is available at c-mind.org website.
User can regenerate latest documentation locally from CMD as cm internal gen_docs or using cM web front end: Usage -> Regenerate cM APIs (DoxyGen) (just making sure that doxygen is installed). This documentation can be easily browsed using cM web front end: Usage -> Regenerate cM APIs (DoxyGen)
At the same time, description of "high level" modules' actions is available by using CMD: cm <module> help or on this wiki for collaborative update and remarks. For example, description of the module core is available here. For your convenience, when browsing modules from cM, you can see links to module API and wiki, as can be seen here.
Finally, we specially provided many modules that implement common academic and industrial experimentation scenarios so that developers could reuse them as basic blocks, understand their functionality by example, and extend or combine them to implement new scenarios.
cM development methodology
Developing any new research infrastructure is a challenging process since often one of the primary objectives of such frameworks is to quickly prototype multiple new and often high-level research ideas while focusing less on implementation. Typically, software is being developed in parallel with research and after several years it often becomes too complex, unmanageable, hardwired and undocumented that frequently leads to complete rewriting or abandoning such tools. To avoid such problems, we are developing a new generic, modular and unified infrastructure that enables collaborative and reproducible R&D, quick prototyping of ideas and sharing of knowledge. Though there is an associated start-up cost to design such a framework, we expect that later it will be easy to specialize it to various end-user scenarios such as application and architecture characterization and optimization.
Since developing such kind of infrastructure from scratch often requires very large teams and many years of work which we can't afford, we decided to use Agile software development methodology that we successfully applied to most of the cTuning tools and in the MILEPOST project. This methodology is very effective for new research projects that has to be completed with a very limited amount of time and manpower while conducting research, developing software and documenting best found variants at the same time to be able to quickly start up long-term R&D. Such methodology is also very common for goal-oriented research and development projects rather than process-oriented straightforward development projects.
When prototyping new ideas/concepts, we use the following methodology:
- discuss high-level idea/concept;
- discuss requirements and schedule;
- agree on several possible implementation variants (unless implementation is straightforward);
- quickly prototype various variants, test them and select the best working variant;
- stabilize and clean up the final working variant, add tests, prepare developer specifications, demos and user guides;
- prepare releases and obtain user feedback.
When extending existing framework or providing new relatively straight-forward features, we use the following methodology:
- discuss high-level feature;
- discuss requirements and schedule;
- prepare/update specification on how to implement it;
- implement and test it;
- test the whole framework;
- update specifications, demos and user-guides if needed;
- prepare release and obtain user feedback.
Note for contributors
First version of cM (base line) is written from scratch by Grigori Fursin without reusing any previous code and is planned to be released under standard or BSD-style license with OpenME interface under LGPL license to connect to other libraries, tools and applications.
cTuning and Collective Mind community can contribute by:
- improving and extending base line cM (low-level functionality and high-level modules)
- adding new tools (compilers and their descriptions, run-time adaptation systems, tools for hardware counters monitoring, etc)
- adding new modules and data (auto-tuning, models, run-time adaptation, visualization, tables, etc)
List of possible improvements (both cM functionality and research projects) is available here.
When improving existing base-line cM, patches can be sent to the mailing list with the remark about the author and the purpose. Patches will be reviewed and added to cM under official cM license.
When adding new modules and data, we suggest to use cM license too, but it is not strictly necessary.
Development conventions
Collective Mind framework enables collaborative R&D through modules (plugins) and distributed repositories. Plugins allow flexibility in development and coding conventions while focusing more on required high-level functionality. However, we still suggest to follow specific development and coding conventions to make Collective Mind and Collective Mind-powered tools easily extensible by other users. If some code/text is not following conventions, you are very welcome to help us to update the code!
- Collective Mind modules are documented using Doxygen tool (http://www.doxygen.org).
C example:
/** * @brief brief description * @param parameters * @param parameters * @todo: ToDo notes * @return return parameters */
Python example:
""" brief description Input: input parameters Output: output parameters """
- Collective Mind generally follows GNU Coding Standards (http://www.gnu.org/prep/standards/), though the first focus is on functionality and research.
- Collective Mind should NOT HAVE TABS in the sources, but SPACES.
- We use "_" and not "-" in the names of the files.
- The end of line should always be \n.
- We suggest to use one space indent in sources.
- We should not hardwire constants that can be user/environment-dependent inside the code. Instead we move them to configuration files or to configuration arrays.
- When visualizing something in format "X:Z", we should have a space after : and not before, i.e. X: Z.
- When visualizing something in format "X=Z", we should not have spaces before and after = or should have both spaces, i.e. X=Z or X = Z.
- We use lower-case for:
- data/module aliases
- names of parameters in json files
- names of commands in modules
- We use upper-case for:
- types of parameters ("URL", "MODULE_UID", etc)
- In C, we suggest to add void in function declarations if there are no parameters:
int function(void);
Testing methodology
Collective Mind has a module test that implements self-testing of high-level functionality. It can be invoked as
cm test all
It is in a very preliminary state and we expect that we will gradually add more tests with the community when needed. We also plan to add test function to each module to perform self-tests using some prepared inputs and outputs.
Unitests are not available yet.
Extending cM functionality (high-level)
Adding new module
Since modules are entries in cM repository abstracted by module with the same name module, adding a new module is easy. For example, user can add module 'mytest' from CMD as following:
cmx add module:mytest
Now, it is immediately possible to test/use this module:
cm mytest help
This command should return preset functions for this module (from core module).
Adding new function
First, user should find path to an entry, using:
cmx find module:mytest
Then user should edit <PATH_TO_MODULE>/.cm/data.json
User should add the following sub-dictionary to "cm_actions" list:
{
"cm_index": FUNCTION_NAME,
"desc": TEXTUAL_DESCRIPTION,
"func": FUNCTION_NAME
}
For example,
{
"cm_index": "myfunc",
"desc": "My super function",
"func": "myfunc"
}
Then user should add to the end of the module's python source code <PATH_TO_MODULE>/module.py the following: def myfunc(i)
print 'HELLO'
print i
return {'cm_return':0}
Now, user can try the following command in CMD:
cm mytest myfunc
Normally, user will see the following:
HELLO
{'cm_run_module_uoa': 'mytest', 'cm_action': 'myfunc', 'cm_module_uoa': 'mytest', 'cm_console': 'txt'}
Note, that all functions should always return 'cm_return':0 if success or 'cm_return':<error_code> if error with 'cm_error' as error text.
After adding parameter myparam=great to CMD as 'cm mytest myfund myparam=great', the output should change as following:
HELLO
{'cm_run_module_uoa': 'mytest', 'cm_action': 'myfunc', 'cm_module_uoa': 'mytest', 'cm_console': 'txt', 'myparam': 'super'}
The same function can be called from any other module as following:
r=cm_kernel.access({'cm_run_module_uoa':'mytest', 'cm_action':'myfunc', 'myparam':'super'})
if r['cm_return']>0: return r # let cM deal with errors
That's all! Now, you can follow cM specification or existing modules and data as example to extend cM!
