Overview

The goal of this article is to show weak scaling and performance sensitivity to hardware choices on Google Cloud for interFoam, a multiphase solver within Open Source Field Operation and Manipulation(OpenFOAM). To illustrate these points, we work with a 2M cell dam break simulation as the chosen benchmark. Key takeaways from this work include :

1. For interFoam, we find good scaling efficiency with >25K cells / MPI rank on both c2 and c2d instances on Google Cloud. When the number of cells per MPI rank is less than 20K, we observe poor scaling efficiency and eventually an increase in simulation wall-time as this ratio is reduced.
   
   

2. Scaling efficiency losses are attributed to MPI_AllReduce calls, which account for ~25% of simulations wall-time at 10K cells/rank, and cannot necessarily be attributed to just cross-VM communication.
   
   

3. The c2d instance type on Google Cloud provides the lowest runtime and cost in the 20K cells/MPI rank regime.

In addition to these results, we share our methods for creating VM images for running OpenFOAM on Google Cloud and our process for benchmarking and profiling using open-source technologies. 

Further, for users wishing to quickly get started with OpenFOAM on Google Cloud, without having to create and manage their own VM images, we have published updates to the RCC-CFD (formerly CFD-GCP) Marketplace solution on Google Cloud based on this work.  Documentation on using RCC-CFD, including how to leverage the target architecture optimized images can be found at the RCC ReadTheDocs.

Methods

Benchmark Description

For this work, we focus on the interFoam application, included with OpenFOAM, and use a higher resolution version of the Dam Break simulation; the input files are made publicly available on the rcc-apps github repository. The resolution is increased, relative to the Dam Break test case included with OpenFOAM, by modifying the blockMeshDict, directly adding additional cells in each region of the model domain. The resulting mesh has about 2.8 Million grid cells.

The dam break simulation uses the interFoam application, which solves the incompressible Navier-Stokes equations for a two-phase fluid using a Volume-of-Fluid (VOF) method. Relative to single phase Navier-Stokes, the two-phase fluid model also requires an advection model for the volume fraction of one of the fluid phases. The fluid density in this approach is calculated as a weighted average of the phase densities where the averaging weight for each grid cell is given by the volume fraction.

For gradient, divergence, and laplacian calculations linear differencing methods are used, with linear upwinding applied for tracer advective flux divergences. For the volume fraction advection equation, we use the interface compression scheme with van-Leer’s flux limiter. The equation for the pressure is solved using the preconditioned conjugate gradient method with the Diagonal-based Incomplete Cholesky (DIC) preconditioner. To forward step the model, we use first order Forward Euler with a maximum courant number of 1.

The initial conditions consist of a block of a liquid fluid phase occupying the lower left region of the domain. Buoyancy forces accelerate this mass of fluid down and no normal flow leads to a bore that encounters a step in the model domain and develops shear flow instabilities at the fluid interface which promotes mixing.

Building with Packer, Spack, & Google Cloud Build

RCC-Apps is an open-source Git repository that maintains Google Cloud VM image build scripts for commonly used research applications. In general, the RCC-Apps build workflow consists of the following steps:

1. Provision an imaging node

2. Install Spack dependencies and Spack

3. Install the desired compiler

4. Install a Spack environment

5. Deprovision imaging node, create VM image from the image node disk, and delete the image node disk.

To achieve this workflow, RCC-Apps uses Hashicorp’s Packer with the googlecompute builder. Packer manages the image node provisioning, deprovisioning, imaging, and error handling during the setup process. RCC-Apps provides a common script that is used to install Spack’s dependencies, Spack, and the desired compiler. Each application defines a Spack environment that can be used during the build process to install the desired application.

To install OpenFOAM, we use the RCC-Apps infrastructure, which provides steps to install Spack’s dependencies, Spack, and the requested compiler as part of the build system. This allows the user to simply provide an installation script or a Spack environment file that can be used to install their desired application. Using Spack also has the advantage that we can simply specify the target architecture to ensure we are using the appropriate compiler flags to enable specific architecture optimizations. 

This setup allows for a methodical approach to generating VM images using a matrix of compilers, target architecture, and optimization flags. Additionally, by using a Spack environment file alongside Terraform infrastructure as code, we are able to completely version control the benchmarking process.

Target Architectures

Google Cloud provides access to a variety of compute platforms. The machine types are named using a convention that indicates the machine family and series, vCPU-to-memory ratio, and the number of cores. Since our goal in this work is to understand which combination of compiler and architecture provides optimal performance for OpenFOAM on Google Cloud, we need to take advantage of compiler features that can produce OpenFOAM binaries that are targeted for each specific platform.

When building OpenFOAM from source code, you have the ability to request the compiler attempt to leverage various optimizations. Typically, a suite of optimizations are grouped into the -O options. For example, -O0 instructs the compiler to not attempt any special optimizations, while -O3 is usually the most aggressive. Depending on which compiler you are using (e.g. Intel, GCC, etc.) the various optimization levels implement different strategies. In addition to optimization levels, compilers can also be instructed to generate tuned instructions for specific cpu hardware models and generations.

Spack uses a library called archspec to resolve target architectures and defines the appropriate compiler flags for many commonly used compilers to generate tuned instructions. Although in this work, we did not explore the impacts of other compiler flag settings, Spack also provides the ability to set compiler flags and override the decisions made by a package’s build system. Spack effectively reduces the problem of creating targeted builds with various compilers to simply making statements about which compiler we want to use and what architecture we want to target. The table below shows some of the machine types available on Google Cloud alongside the target architecture name in Spack.

With these ideas in mind, during our build process we create the following images : 

• Compiler = GCC 10.2.0, Target Architecture = x86_64

• Compiler = GCC 10.2.0, Target Architecture = cascadelake

• Compiler = GCC 10.2.0, Target Architecture = zen3

• Compiler = Intel OneAPI Compilers 2022.1.0, Target Architecture = x86_64

• Compiler = Intel OneAPI Compilers 2022.1.0, Target Architecture = cascadelake

Profiling with HPC Toolkit

On Google Cloud, we want to determine how well OpenFOAM scales for fixed size problems. Ideally, adding more vCPU to a specific problem and simultaneously increasing the number of MPI ranks will result in speedup that is proportional to the number of vCPU. However, MPI communications are expected to become increasingly expensive as the number of MPI ranks is increased. Additionally, public cloud platforms like Google Cloud leverage “ethernet-like” networking that typically has higher communication latency than on-premises HPC networking (e.g. Infiniband). This latter fact is a source of concern for researchers attempting to tackle problems that require high resolution computational fluid dynamics simulations where clusters of tightly coupled compute nodes work together to simulate the evolution of a fluid.

Results

When running the benchmarks on Google Cloud, we are looking to find the ideal hardware and software configuration for running OpenFOAM. An ideal configuration, in our view, is one that provides a balance between shortest runtime and lowest cost. When time-to-solution is not a factor, obtaining the minimal cost is an obvious goal for anyone running on public cloud systems. For some applications, the lowest time to solution and the lowest cost configurations are not equivalent to the shortest runtime. 

This section covers the behavior of the Dam Break (2.8M) benchmark as we vary the machine type, target architecture, compiler, number of MPI ranks and task affinity. For the machine types, we compare two of the compute optimized instance types available on  Google Cloud, the c2 (Intel Cascadelake) and c2d (AMD Epyc Milan). When running on these instances, we use the largest machine type available for each type and use compact placement policies. Wider VMs allow more MPI communications to occur on VM rather than across VM when compared to instances with fewer vCPU per node. Additionally, compact placement helps minimize communication latency between VMs by ensuring VMs are hosted physically close together in Google Cloud datacenters. On the c2 instances, we use the cascadelake targeted builds with Intel and GCC compilers; on the c2d’s, we use the zen3 targeted builds with the GCC compilers. 

For tightly coupled CFD applications, like OpenFOAM, simulations can often be sped up through domain decomposition and parallel execution with MPI. In domain decomposition, the model’s physical domain is divided as equally as possible and distributed for execution to multiple MPI processes (or “ranks”). At the boundaries of each rank’s domain, information must be shared regularly between neighboring ranks to ensure correctness of the execution. Additionally, the algorithm used to calculate the pressure field in interFoam uses an implicit solver that also requires all-to-all communication (AllReduce) to verify convergence of the preconditioned conjugate gradient solver. 

This additional MPI communication overhead typically grows with the number of MPI ranks and leads to inefficiencies in scaling. Figure 1 shows the runtime for the Dam Break (2.8M) benchmark as a function of the number of MPI ranks and various build and runtime configurations. In all scenarios, we see that the runtime decreases as the number of MPI ranks increases, out to about 120 MPI ranks. Beyond this point, we see that the runtime increases, presumably because the MPI overhead is becoming more prevalent.

Since MPI communication is required when running in parallel, obtaining the best possible runtime for a fixed number of MPI ranks depends (in part) on the placement of the MPI processes; ideally, communications off-VM are minimized. On the other hand, providing memory bound applications, like OpenFOAM, providing each MPI rank more computational real estate can provide more memory bandwidth per rank, potentially improving model runtime. In our study, we looked at binding MPI ranks to hardware threads and to cores. Rank binding prevents the operating system from evicting each MPI rank from the resource it is bound to. Binding a rank to a core provides each MPI rank residency on a physical core and less contention for memory resources. On the other hand, binding a rank to a hardware thread (a vCPU on Google Cloud) allows us to pack more MPI ranks per node and reduce cross-VM communications.