Chemora

Chemora! (Computational Hierarchy for Engineering Model-Oriented Re-adjustable Applications) is a code generation and optimization framework. In simple terms, it enables efficient and hybrid computations to be generated from high level mathematical equations (including discretization, transcription to a programming language, and optimizations).

Chemora starts from a high-level problem description in terms of partial differential equations using abstract tensor notation, the Chemora framework discretizes, optimizes, and generates complete high performance codes for a wide range of compute architectures. Chemora extends the capabilities of Cactus, facilitating the usage of large-scale CPU/GPU systems in an efficient manner for complex applications, without low-level code tuning. Chemora achieves parallelism through MPI, multi-threading (using OpenMP), and accelerators (using CUDA or OpenCL). Optimizations include high-level code transformations, efficient loop traversal strategies, dynamically selected data and instruction cache usage strategies, and JIT compilation of GPU code tailored to the problem characteristics.

In Chemora, we take an integrative approach to addressing programmability and performance at all levels, from enabling large-scale collaborations, to separating physics, numerical analysis, and computer science in an application, to disentangling kernel implementations from performance optimization annotations. Chemora is based on the Cactus computational toolkit, a well-known framework used in several scientific communities for developing HPC applications. Cactus is a component-based software engineering framework providing key abstractions to simplify significantly parallel programming for a large class of problems, in particular solving systems of partial differential equations (PDEs) on block-structured grids -- i.e. adaptive mesh refinement (AMR) and multi-block systems.

Chemora enables existing Cactus-based applications to continue scaling their scientific codes and make efficient use of new hybrid systems, without requiring costly re-writes of application kernels or adopting new programming paradigms. At the same time, it also provides a high-level path for newly developed applications to efficiently employ cutting-edge hardware architectures, without having to target a specific architecture.

Technically Chemora consist of Cactus and several thorns (modules) handling the heterogeneous computations. In particular:

• Carpet ([1], [2], [3]) - a Cactus driver, which provides adaptive mesh refinement (AMR) and multi-block capabilities;
• CaKernel - ([4], [5], [6]) a set of high level programming abstractions, and the corresponding implementations. Based on the Cactus computational infrastructure, CaKernel provides two major sets of programming abstractions: (1) Grid Abstractions that represent the dynamically distributed adaptive grid hierarchy and help to separate the application development from the distributed computational domain; (2) Kernel Abstractions that enable automatic generation of numerical kernels from a set of highly optimized templates and help to separate the development,scheduling, and execution of numerical kernels;
• Kranc - ([7], [8], [9]) a code generation package. The user of Kranc provides a Kranc script containing a section describing the partial differential equations to solve, and a section describing the numerical algorithm to use. Kranc translates this high-level description into a complete Cactus module, including C++ code implementing the equations using the specified numerical method, as well as code and metadata for integrating this into the Cactus framework;
• Accelerator - tracks which data (and which parts of the data) is read and written by a particular routine, and where this routine executes (host or GPU). Data is copied only when necessary, and then only those portions that are needed. Note that data is not only accessed for computations, but also by inter-process synchronization and I/O.

The Chemora itself also has been described in a separate publication.

Chemora is supported by NSF grants 1265449/1265434/1265451. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.