Title: High-Order Methods for HPC Turbulence Applications - Plenary Talk 

Authors: Paul Fischer (1,2,3), Misun Min (3), Ananias Tomboulides (3,4), Vishal Kumar (3) 

Abstract: We describe scaling developments in the high-order open-source simulation package Nek5000/RS, which is designed to solve turbulent thermal-fuids applications on platforms ranging from laptops to exascale computers. Our approach tackles two key ingredients for this class of problems: 

(i) effcient discretizations and (ii) effcient and scalable implementations. Regarding (i), large-scale sim ulations of turbulence imply multiscale interactions and relatively long integration times wherein small-scale ow features of size l are transported over long distances L >> l. Such conditions place stringent requirements on numerical accuracy lest the small scale features be distorted by numerical dispersion or dissipation. As noted by Kreiss and Oliger in 1972, high-order numerical meth ods are particularly e cient in this regime. We demonstrate that the bene ts of high-order pertain not only to the primitive Navier-Stokes equations (NSE), but also to model equations such as used for large-eddy simulations (LES) and Reynolds-Averaged Navier-Stokes (RANS) formulations. Like the NSE, LES and RANS are based on advection-dominated PDEs where high-order yields e ciency through rapid convergence by requiring a lower number of grid points, n, than their low-order counterparts. Regarding (ii), we discuss design and performance issues for high-order methods that are particularly critical at the strong-scale limit of high-performance computers (i.e., where parallel e ciency begins to deviate from unity). For distributed-memory solutions of PDEs run ning on P processors, this limit is tied to the local problem size, that is, the number of grid points per MPI rank, n/P. As this number decreases, communication effects become important. On GPUs, kernel launch latency is another factor controlling achievable speed-ups. For large processor counts these e ects are further ampli ed by the presence of a coarse-grid solver, which is essential for e cient solution of the pressure Poisson problem that governs incompress ibility. We describe a novel approach to solution of this coarse-grid problem on exascale platforms. We illustrate the e effectiveness of the overall approach with several Nek5000/RS applications throughout the talk, including recent trillion grid point computations on the Frontier supercomputer at Oak Ridge National Laboratory 

1) Mechanical Science and Engineering, 2) Computer Science, University of Illinois,

Urbana Champaign, USA

3) Mathematics and Computer Science Division, Argonne National Laboratory, USA

4) School of Mechanical Engineering, Aristotle University of Thessaloniki, Greece

Title: Exascale Advances in CFD 

Abstract: We discuss recent developments in the spectral-element-based computational fluid dynamics code, NekRS, which is the GPU-based successor to Nek5000. Through numerous simulation examples, we illustrate that NekRS sustains 80% parallel efficiency for local problem sizes, n/P, ranging from 3M points per MPI rank on OLCF's Frontier (2 ranks per AMD MI250X) to 5M points per rank on ALCF's NVIDIA A100-based Polaris. On 72,000 ranks of Frontier, NekRS is sustaining 0.39 TFLOPS per rank or a total of 28 PFLOPS for thermal hydraulics simulations in a full reactor core. In addition to nuclear energy applications, we describe recent developments in SEM-based wall modeled LES for atmospheric boundary layer simulations relevant to wind energy applications. We also present several technical developments that are important to exascale workflows. These include meshing and mesh partitioning for large meshes having in excess of 1B spectral elements; in situ visualization advances that avoid writing multi-TB output files; and GPU-based interpolation utilities that are essential for particle tracking and for support of overset grids. Performance scaling results are presented for each of these developments. NekRS is under development at Argonne National Laboratory and was initially funded as part of the Center for Efficient Exascale Discretizations (CEED), which was supported by the US Department of Energy's Exascale Computing Project.

Title: Fifth-Order Bound-, Positivity-, and Equilibrium- Preserving Affine-Invariant AWENO Scheme for Two-Medium γ-based Model of Stiffened Gas 

Abstract: We describe a quasi-conservative finite difference AWENO scheme with the affine-invariant Z-type nonlinear weights (Ai-AWENO) for the γ based model. The shock-capturing scheme should always but often fail to preserve the constant velocity and pressure. One leading cause is that switch ing the equation of state between different mediums generates numerical oscillations around the medium interface. In the Ai-AWENO scheme, the conservative variables, instead of the primitive variables, are used, and the equilibriums of velocity and pressure are preserved. A hybrid flux-based bound- and positivity-preserving (BP-P) limiter, which is a convex combina tion of the high-order (for resolution) and first-order (for BP-P) numerical f luxes, is also implemented to enforce the physical constraints. The theoreti cal analysis yields the exact CFL conditions of the first-order Lax-Friedrichs numerical flux for the stiffened gas. The numerical diffusion coefficient de pends nonlinearly on the local Mach number. Various one-, two-, and three dimensional benchmark two-medium shock-tube problems illustrate the pro posed scheme’s high-order accuracy and enhanced robustness.

Title: Mixed Virtual Volume  Methods for Elliptic Problems 

Abstract: We develop a class of mixed virtual volume methods for elliptic prob- lems on polygonal/polyhedral grids. Unlike the mixed virtual element methods introduced in Brezzi et. al. (ESAIM: M2AN, 48 (2014)) and Beira ̃o da Veiga et. al. (ESAIM: M2AN, 50 (2016)) which yield saddle point problems, our methods are reduced to symmetric, positive definite problems for the primary variable without using Lagrangian multipliers. We test the mixed equation by the gradient of certain nonconforming space and orthogonal complement of gradient of ploynomial spaces. By integrating by parts, we obtain a nonconforming virtual element method for the primal variable p.

Once the primary variable is computed, all the degrees of freedom for the Darcy velocity are locally computed. Also, the L2-projection onto the polynomial space is easy to compute. Hence our work opens an easy way to compute Darcy velocity on the polygonal/polyhedral grids.

An optimal error analysis is carried out and numerical results are pre- sented which support the theory.

Title: Machine Learning for Molecular Dynamics Simulation: Acceleration and Analysis 

Abstract: Molecular dynamics (MD) simulation is a powerful computational method to calculate physical properties and analyze molecular mechanisms for various molecules.  However, the high computational costs of large-scale and long-time MD simulations limit the method's applicability.  In addition, the analysis of molecular mechanisms is typically complex due to the complexity of molecular behavior.  To address these general problems of MD simulations, we proposed two novel machine learning methods focusing on (1) the acceleration of time evolution in MD simulations and (2) the detection of representative molecular behavior that characterizes systems.  We proposed the machine learning method based on generative adversarial nets (GANs) to accelerate time evolution in MD simulations.  This method obtains the efficient time evolution method from the short-term MD data to generate long-term dynamical statistics.  To detect representative molecular behavior in systems, we applied deep neural networks to calculate statistical distances between different ensembles, which are probability distributions over the possible states of systems.  Using statistical distances between all systems, their low-dimensional embeddings are provided.  The structure of embeddings highlights the meaningful variables to explain the system differences.  Furthermore, the molecular behavior contributing to those differences can also be detected using the trained function of deep neural networks.  The applicability of our two schemes was demonstrated through experiments for several types of MD data. 

Title: Discrete moving frames, and symmetry-preserving variational integrators   

Abstract: In this talk, we will introduce the discrete counterpart of moving frame theory and demonstrate its application to the computation of difference invariants. Discrete moving frames are further applied to construct variational integrators that preserve symmetries and, consequently, conservation laws, while conserving the symplectic structure. We will also explore the mathematical foundations of these techniques, and illustrate their effectiveness through various examples.

Title: Level Set Methods on Polyhedral Meshes

Abstract: : In this talk, a cell-centered finite volume method is used to solve the G-equation on polyhedral meshes in three-dimensional space, that is, a general type of the level-set equation including advective, normal, and mean curvature flow motions. Numerical experiments quantitatively show that the size of time step proportional to an average size of computational cells is enough to obtain the second-order convergence in space and time for smooth solutions of the general level set equation. A qualitative comparison is presented for a nontrivial example to compare numerical results obtained with hexahedral and polyhedral meshes. Furthermore, we propose to use a nonlinear boundary condition, that is, eikonal equation, when advective or normal flow equations in the level-set formulation are numerically solved on polyhedral meshes. Since a common choice of the initial condition in the level set method is a signed distance function, the eikonal equation on the boundary is compatibly correct at the starting point. Enforcing the eikonal equation on the boundary along time can effectively eliminate an inflow boundary condition which is typically necessary in the transport equation. For the chosen examples, the numerical results confirm that the eikonal boundary condition provides comparable accuracy and robustness in the evolution of the surface using the exact Dirichlet boundary condition.

Title: High order method with moving frames for the ephaptic coupling of the general 2D neural fibers with non-aligned Ranvier nodes 

Abstract:  A novel multidimensional mathematical model is proposed for the ephaptic coupling between the general-shaped neural fibers containing the non-aligned Ranvier nodes. Using this partial differential equation model, we aim to investigate the effects of the diverse configuration of the Ranvier nodes, for example, the neural fibers with different locational frequencies along the fiber or misaligned Ranvier nodes. Moreover, this model implies the meaning of the neural fiber's geometry, particularly the curvature, which is critical to the function of the white matter neural fibers. The proposed governing equation, known as a bidomain model, is derived from the Frankenhaeuser-Huxley model, with the domain being composed of the nonoverlapping intracellular space and extracellular space with physiologically verified conductivity tensors. The governing equation is validated against the previous literature on the one-dimensional neural fiber with the aligned Ranvier nodes. A high-order continuous Galerkin scheme is used for computational simulation with moving frames representing the intracellular and extracellular conductivity. Moreover, the proposed model is simulated to confirm the effects of the non-aligned Ranvier nodes with various shapes of the neural fiber bundle, which has only been conjectured by theories.  

Title: XCCELS: Exascale Computing Center for Sparse Linear Systems 

Abstract: In this presentation, we provide a comprehensive overview of the ongoing research activities at the eXascale Computing Center for parsE Linear System (XCCELS). The core objective driving XCCELS is the development of an exascale linear solver numerical library, with a specific focus on enhancing high-performance computing (HPC) capabilities. During the initial phase of development, XCCELS concentrates on creating a prototype numerical library tailored specifically to meet the demands of many-core computing environments. This phase involves rigorous testing, algorithm refinement, and optimization procedures to ensure that the library can effectively exploit the parallel computing systems with many-core architectures. As XCCELS progresses into the second phase of its roadmap, the focus shifts towards fine-tuning and parallelizing the numerical library for exascale HPC environments based on heterogeneous architectures. This phase involves optimizations at both algorithmic and implementation levels, aimed at fully leveraging the massive parallelism inherent in exascale computing systems. Techniques such as task parallelism, data parallelism, and hybrid approaches are explored and optimized to achieve peak performance on exascale platforms. XCCELS aims to provide a versatile numerical library capable of supporting real-time and large-scale analyses across diverse scientific and engineering domains as an output. Furthermore, XCCELS is committed to foster a various user community. Educational initiatives, technical support services, and interactive sessions such as lectures and hands-on tutorials are also action plans of XCCELS's outreach strategy. These initiatives not only equip users with the knowledge and tools needed to effectively utilize HPC resources but also contribute to the overall growth and advancement of national research capabilities in HPC and related scientific disciplines. By adopting this multifaceted approach focusing research excellence, optimization strategies, community engagement, and educational outreach, XCCELS aims to significantly enhance both national and global research capabilities in the realm of high-performance computing and its applications in diverse scientific and engineering fields. 

Title: Autotuning framework for HPC applications on exascale computers

Abstract: In this talk, I would like to present an autotuning framework designed for HPC applications on exascale-based supercomputers. Leveraging Apache TVM (Tensor Virtual Machine), the potential of the framework is verified by automating HPC parallel programming models and loop optimizations. Furthermore, the framework is extended to encompass a high-dimensional HPC operator interface and support optimizations such as sparse format layout. Apache TVM, renowned for its hardware-agnostic computation optimization capabilities primarily in machine learning contexts, will be showcased for its adaptability to typical HPC applications on exascale computers. The versatility of the framework will be highlighted by optimizing not only dense matrix operations but also sparse matrix operations, encompassing optimization processes and initial findings on the KISTI-5 supercomputer. Additionally, the framework for future developments will be included the multi-objective optimization such as power efficiency and computational performance. 

Title: High-Order WENO-based Semi-Implicit Projection Method for Incompressible Turbulent Flows 

Abstract: This study introduces a high-order projection method for simulating incompressible turbulent flows, using weighted essentially non-oscillatory (WENO) schemes. For spatial discretization, WENO schemes are applied to nonlinear convection while standard central differences (CD) handle the viscous term. The method integrates different orders of WENO and central difference schemes (WENO3/CD2, WENO5/CD4, and WENO7/CD6) to achieve third, fifth, and seventh-order spatial accuracy for velocity fields. To validate this approach, extensive testing was conducted using the 2D Taylor Green vortex problem in both space and time, as well as simulations of 3D Taylor-Green vortex problems at Reynolds numbers (Re) of 1, 600, 16,000, and 160,000. The results, which included analyses of kinetic energy, enstrophy, energy spectra, and vortex fields, demonstrate the method's capability to accurately simulate high Reynolds number turbulent flows without needing additional sub-grid scale models. 

Title: High-order Method for Very High-speed Compressible Flows

Abstract: This seminar focuses on the recent advances in high-order methods for simulating high-speed compressible flows. High-speed aerodynamics presents significant challenges, including strong shock waves, extreme aerodynamic heating, vibrational and electronic energy excitations, and chemical and thermal nonequilibrium. Traditional low-order numerical methods often fail to capture these complex phenomena with the necessary accuracy. High-order methods show great potential for providing improved precision and stability for such simulations without a significant increase in computational cost. This talk will cover the theoretical foundations for achieving high-order accuracy, computational implementations on modern computing architectures, and thermochemical models for simulating high-temperature gas. These advancements highlight the potential for high-order methods to significantly progress the field of high-speed aerodynamics.

Title: On the connection between the stabilization-free polygonal element and staggered discontinuous Galerkin methods 

Abstract: We present a connection between the stabilization-free polygonal element and staggered discontinuous Galerkin methods. By introducing a gradient reconstruction operator, the staggered discontinuous Galerkin method can be transcribed into a class of stabilization-free polygonal element methods. Motivated by this connection and some existing stabilization-free polygonal element methods, we present some new variants of the staggered DG methods and their error estimates. We also present a static condensation process for the staggered discontinuous Galerkin method for its efficient implementation, which also can be applied to the stabilization-free polygonal element methods. The theoretical results are verified by numerical experiments. 

Title: A third-order finite difference weighted essentially non-oscillatory scheme with shallow neural network 

Abstract: When solving hyperbolic conservation law, discontinuity is hard to deal with and important to get an accurate approximation. For several decades, heuristic ENO and WENO methods have been improved for treating discontinuity. Recently since the success of neural network, neural network has been introduced in numerical analysis, including WENO. Many works applying neural network for WENO faces with maximum convergence order, more accuracy without spurious oscillation. In this paper, we use a shallow neural network to introduce two WENO weight functions with finite difference scheme. We use the same scale- and translation-invariant preprocessing layer in previous work and the WENO-JS weight as a label in both models. Each model is trained by using each the modified logarithm error loss and mean squared error loss with a linear weight term. The WENO-SNN1 and WENO-SNN2 perform well enough to capture smoothness and discontinuity, and achieves the convergence error more than WENO-Z. Besides, the WENO-SNNs show the less error than WENO-Z without spurious oscillation.

Title: Enhancing Stability in Physics-Informed Neural Networks with the Re-spacing Layer for Stiff Differential Equations and Plasma Applications 

Abstract: We address the challenges of using physics-informed neural networks (PINNs) for solving stiff differential equations. Solutions to stiff differential equations exhibit sharp gradients, typically necessitating a high concentration of samples in certain regions, which leads to data imbalance and slower convergence. We introduce the Re-spacing Layer (RS-layer), a pre-trained encoding layer that redistributes sampling points uniformly. RS-layer effectively regularizes solution gradients and improves training and model accuracy. Our method is validated through tests on various stiff differential equations and plasma problems.

Title: Application of Multi-GPU TDMA Algorithms to Incompressible Navier-Stokes Equations 

Abstract: Multi-GPU TDMA (Tri-Diagonal Matrix Algorithm) Algorithms, which is suitable for simulations on grids exceeding 1 billion cells, is presented. If the direction of solving the tridiagonal system is aligned with direction of partitioned domain, traditional methods use all-to-all communication to redistribute data among processors, which negatively impacts computational speed. Here, we avoid all-to-all communication by using parallelization methods based on the divide and conquer parallel computation model, namely PDD (Parallel Diagonal Dominant) for diagonal dominant systems and PPT (Parallel Partitioned LU) for general systems. The presented Algorithms demonstrate high computational performance and scalability in simulation up to 1.4 billion cells. It can also be extend to higher-order compact scheme using Padé approximation. 

Title: Defects of Liquid Crystals in the Landau-de Gennes theory 

Abstract: Liquid crystals have been paid attention by many scientists due to their rapid advances in liquid crystal material technolgy and practical applications such as imaging and optical probing, etc. In this talk, we  discuss the Landau-de Gennes energy functional which governs a system of a liquid crystal and investigate molecular directions in two-dimensional domains. In particular. we study defects formations by way of spectral collocation method. 

Title: Variational integrator for underwater vehicle dynamics via Euler-Poincaré reduction

Abstract: This presentation explores the dynamics of underwater vehicles using the Euler-Poincaré reduction framework, which incorporates advected parameters representing additional dynamics into a variational problem. We will outline the theoretical foundation of Euler-Poincaré reduction with advected parameters and derive the corresponding Euler-Poincaré equations for underwater vehicle dynamics. From the reduced discrete Lagrangian, the discrete Euler-Poincaré equations from the reduced discrete Lagrangian will be derived, forming a variational integrator that ensures the preservation of the system’s geometric structure. Numerical simulations will illustrate the integrator’s ability to preserve geometric properties over long time periods, highlighting its potential for practical applications in the control and navigation of underwater vehicles.

Title:  Dispersed microbubble-laden turbulent flow based on high-order Euler-Lagrange appraoch 

Abstract: To resolve multiphase flow, specifically dispersed phase flow, tracking the dispersed phase's trajectory is crucial. The Euler-Lagrange approach is adopted to predict the interaction between the dispersed phase (microbubble) and the continuous phase (turbulence). The Lagrangian tracking code, ppiclF (parallel particle-in-cell library written in Fortran), and the spectral element method code, Nek5000, are combined to simulate microbubble-laden turbulent flows.  In this presentation, the developed microbubble models and microbubble dynamics in the turbulent channel flow will be introduced.  Turbulent quantities such as turbulent boundary layer and Reynolds stresses are compared with respect to the size and number of bubbles.  Additionally the drag reduction mechanism by microbubbles is analyzed.