General considerations and rules of thumb

In general one should minimize the size of the system (i.e. the total number of atoms) as this is the main factor in determining the time it will take to simulate 1 ns.

• Keep a minimum water buffer of 1.2 nm to 1.5 nm between protein and box edge to prevent nonphysical interactions with periodic images.

• Use a dodecahedral box where possible (70% of the volume of a cubic box).

• Have at least 3 layers of lipids between a membrane protein and the box edge.

A system with <20,000 atoms is small, <80,000 atoms is medium, >150,000 is pretty big.

Before benchmarking your simulation system, you have to install Gromacs (see bottom) you want to use.

Running jobs on Bridges: https://www.psc.edu/bridges/user-guide/running-jobs

Download the benchmarking slurm file, gromacs.slu (see below) and SFTP to Bridges.

SFTP your simulation system (eg step7_production.tpr) to Bridges. In order to get your simulation system ready for benchmarking, you must run equilibration simulations.

SSH to Bridges.

Create a folder for benchmarking:

mkdir benchmark-date (eg benchmark-072119)

cp gromacs.slu /benchmark-date/

cp step7_production.tpr /benchmark-date/

Run gromacs.slu

cd benchmark-date

sbatch gromacs.slu

qstat

After the completion of the benchmarking run, check the results:

vi md_default.log

vi md_1gpu.log

vi md_2gpu.log

vi PERFORMANCE.txt

In order to close the vi editor, type ":q!".

The content in the Slurm file (gromacs.slu)

#!/bin/bash

#SBATCH -J gmxminibench

## Give your job a name, so you can recognize it in the queue overview

#SBATCH -N 1

## Number of nodes

#SBATCH --ntasks-per-node=32

## Bridges has 16 CPU cores per GPU and 2 GPUs per node

#SBATCH --gres=gpu:p100:2

## Generic Resource Scheduling (GRES) 2 P100 GPU cards

#SBATCH -p GPU

## Intended partition

#SBATCH -t 1:00:00

## Time (H-MM-SS)

TPR=step7_production.tpr

module load cuda/9.2 gcc/7.3.0

source /home/yohei/Library/gromacs/versions/2018.3/cuda-9.2/gcc-7.3.0/threads/bare/bin/GMXRC

gmx mdrun -v -nsteps 40000 -s $TPR -deffnm md_default -pin on

## See explanations for affinity and pin.

gmx mdrun -v -nsteps 40000 -s $TPR -deffnm md_2gpu -pin on -ntmpi 4 -ntomp 8 -pme gpu -pmefft gpu -npme 1

## See explanations for MPI, ntmpi and ntomp. 4 x 8 = 32 CPU cores: one thread-MPI rank per core

gmx mdrun -v -nsteps 40000 -s $TPR -deffnm md_1gpu -pin on -nt 16 -ntmpi 4 -ntomp 4 -pme gpu -pmefft gpu -npme 1 -gputasks 0000 -nb gpu

## See explanations for PME and npme, for PME Fast Fourier Transform, and also for gputasks. All 4 tasks will be performed by GPU 0 (0000).

grep Performance: *.log | tee PERFORMANCE.txt