Our achievements

• 2023.September » The cTuning foundation is proud to deliver the new version of our Collective Knowledge Technology v3 with the open-source MLCommons CM automation language, CK playground and modular inference library (MIL) that became the 1st and only workflow automation enabling mass submission of more than 12000 performance results in a single MLPerf inference submission round with more than 1900 power results across more than 120 different system configurations from different vendors (different implementations, all reference models and support for DeepSparse Zoo, Hugging Face Hub and BERT pruners from the NeurIPS paper, main frameworks and diverse software/hardware stacks) in both open and closed divisions! See HPCWire article and contact Grigori Fursin for more details!
• 2023.June » We are honored to give a keynote about our community projects at the 1st ACM conference on reproducibility and replicability.
• 2023.May » We joined forces with AVCC and MLCommons to develop the industry's first Automotive Benchmark based on our related collaboration with General Motors.
• 2023.April » We have successfully validated our new MLCommons CM reproducibility and automation language during the 1st MLPerf inference community submission challenge. We are honored that our technology and expertise has helped the community and companies automate, unify and reproduce more than 80% of recent MLPerf inference benchmark submissions (and 98% of power results) with very diverse technology from Neural Magic, Qualcomm, Krai, DELL, HPE, Lenovo, Hugging Face, Nvidia, AMD, Intel and Apple across diverse CPUs, GPUs and DSPs with PyTorch, ONNX, QAIC, TF/TFLite, TVM and TensorRT using popular cloud providers (GCP, AWS, Azure) and individual servers and edge devices provided by our volunteers and contributors. Following this success, we are leading an open MLCommons taskforce on automation and reproducibility to develop Collective Knowledge Playground - a free, open-source and technology-agnostic platform to collaboratively benchmark and optimize AI, ML and other emerging applications via reproducible challenges and tournaments.

  See the Forbes article abour our technology and LinkedIn report.

• 2022.September » We have donated our open-source Collective Knowledge v1 technology to MLCommons to benefit everyone (new GitHub repo). We also established a public MLCommons task force on automation and reproducibility to automate MLPerf benchmarks and make them easier to run out of the box with any software, hardware, data sets and models.
• 2021.Februrary » We are honored to give an ACM Tech Talk about "Reproducing 150 Research Papers and Testing Them in the Real World: Challenges and Solutions".
• 2020.December » We are honored to join MLCommons as a founding member to accelerate machine learning innovation along with leading companies and universities including AMD, Alibaba, Arm, Baidu, Cerebras Systems, Centaur Technology, Cisco, Dell, d-Matrix, Facebook AI, Fujitsu, FuriosaAI, Gigabyte, Google, Grai Matter Labs, Graphcore, Groq, HPE, Horizon Robotics, Inspur, Intel, Kalray, Landing AI, MediaTek, Microsoft, Myrtle.ai, Neuchips Corporation, Nettrix Information Industry, Nvidia, Qualcomm, Red Hat, SambaNova Systems, Samsung, Shanghai Enflame Technology, Syntiant, Tenstorrent, VerifAI, VMind Technologies, Xilinx, Gungdong Oppo Mobile Telecommunications Corp, Harvard University, Indiana University, Stanford University, University of California, Berkeley, University of Toronto, and University of York.
• 2020.November » We have completed the prototyping phase of our Collective Knowledge framework (CK) and successfully validated it in multiple industrial and academic projects as described in this white paper and the FASTPath'20 presentation. We have helped our partners and the community to use CK as an extensible playground to implement reusable components with automation actions for AI, ML, and systems R&D. We used such components to assemble portable workflows from reproduced research papers during our reproducibility initiatives at ML and systems conferences. We then demonstrated that it was possible to use such portable workflows to automate the co-design process of efficient software, hardware and models, simplify MLPerf inference benchmark submissions, and quickly deploy emerging AI, ML, and IoT technology in production in the most efficient way (speed, accuracy, energy, costs) across diverse platforms from data centers to edge devices.
• 2020 » Our unified artifact appendix and reproducibility checklist is now used at many systems and ML conferences including MLSys, ASPLOS, Supercomputing, CGO and PPoPP.
• 2019.January » We started organizing reproducible quantum machine learning hackathons
• 2018.March » We published results from the 1st reproducible ACM ReQuEST tournament at ASPLOS
• 2017.September » Microsoft sponsors non-profit cTuning foundation
• 2017.May » We organized the first reproducible tournament to co-design SW/HW stacks for deep learning at ACM ASPLOS
• 2017.February » We helped ACM prepare policy on Result and Artifact Review and Badging
• 2017.February » Our technology received ACM CGO test of time award
• 2016 » CK framework helped General Motors to collaboratively benchmark different deep learning implementations.
• 2016 » Arm made a press-release about using our Collective Knowledge Technology [ PDF (page 17) ]
• 2016 » We initiated crowd-tuning campaign crowdsourcing GCC/LLVM tuning and combining it with ML-based learning across diverse hardware including mobile devices and cloud services provided by volunteers using the CK framework. You can see crowd-results in our SOTA scoreboard!
• 2016 » Our Collective Knowledge framework was used in the TETRACOM-funded project to crowd-source compiler bug detection together with Imperial College London and dividiti
• 2016 » Ed Plowman (director of performance analysis strategy in ARM) says "contribute to CK and workload automation" [ Slides ]
• 2016 » We have presented our Collective Knowledge approach at DATE'16
• 2015 » cTuning's Collective Knowledge technology has received the HiPEAC technology transfer award
• 2015 » We have received funding from the EU TETRACOM project to enhance the Collective Knowledge Framework and validated it in industry with Arm!
• 2015-cur. » We initiated open and community-driven reviewing of publications and artifacts at the ADAPT workshop
• 2014-cur. » We helped to initiate artifact evaluation at CGO and PPoPP conferences
• 2014 » cTuning technology was referenced by Fujitsu as closely related to their long-term initiative on "big data" driven optimization of Exascale computer systems
• 2014-2016 » cTuning technology has been accepted as a new theme on reproducible research and experimentation in computer engineering for the EU HiPEAC network of excellence [More]
• 2012 » Grigori Fursin received INRIA award for "making an outstanding contribution to research" and a 4-year fellowship for his cTuning technology
• 2012 » We developed a Collective Mind Technology (cTuning v3) and crowdsourced program optimization using commodity mobile phones similar to SETI@home [J10, P28, P10, P50]
• 2006-2009 » The cTuning technology demonstrated the possibility to fully automate the construction of compiler optimization heuristics and speed up benchmarking, optimization and co-design of multi-core reconfigurable systems by several orders of magnitude thus dramatically reducing time to market for the new systems and increasing ROI. This technology is considered by IBM to be the first in the world [M4, P19, P28, press about our project].
• 2009 » Our novel crowd-tuning approach (crowdsourcing SW/HW optimization) together with the cTuning framework, unified interfaces, a public repository of knowledge, and machine learning web services enabled practical ML-based self-tuning compiler (MILEPOST GCC) [project website, new live optimization repository, framework, IBM's press-release]
• 2009 » We integrated the Interactive Compilation Interface to the mainline GCC 4.6+ to crowdsource autotuning and ML-based learning of optimization heuristics (sponsored by Google) [S17, M7, P19, P28, F6]
• 2008 » We integrated our novel run-time adaptation technique for statically compiled programs based on code multi-versioning and fast decision trees to the mainline GCC 4.8+ to automatically maximize program performance and power consumption without the need for complex recompilation and JIT infrastructures across a variety of systems such as mobile devices or data centers with VM (sponsored by Google and Mozilla) [S14, F6, P50, P32, P28, P10, P6]