NWChem

These tests were performed on Taavare (16 procs and 64 GB RAM per node and with Infini band) in the year 2013. The input file can be found at this link. (http://www.nwchem-sw.org/images/Input_c240_pbe0.nw)

Performance with the above input file

64_proc - 972 sec

48_proc - 1202 sec

32_proc - 1761 sec

16_proc - 3366 sec

Performance when the memory of the above input file is changed.

64_proc + memory option removed - 1616 sec

64_proc + memory option and with "noverify" - 1609 sec

64_proc + memory total 20GB - 968 sec

64_proc + memory total 20GB - 974 sec

So, leaving memory option (as mentioned in the NWCHEM manual) is NOT GOOD. Give memory to get best performance depending on your system's memory.

Environment to create NWCHEM

This environment is created with the help of Karteek Kumar Bejagam. This may need to be changed to enhance the performance further. The below environment gave the performance similar to what NWCHEM website has mentioned.

export NWCHEM_TOP="/lustre/home/pati/Softwares/nwchem-6.3_2013-10-17"

export TARGET=LINUX64 # may not be required and it may be NWCHEM_TARGET

export LARGE_FILES=TRUE

export ENABLE_COMPONENT=yes

export TCGRSH=/usr/bin/ssh

export NWCHEM_TARGET=LINUX64

export NWCHEM_MODULES="all"

export LIB_DEFINES="-DDFLT_TOT_MEM=16777216"

export USE_MPI=y

export USE_MPIF=y

export USE_MPIF4=y

export IB_HOME=/usr

export IB_INCLUDE=$IB_HOME/include/infiniband

export IB_LIB=$IB_HOME/lib64

export IB_LIB_NAME="-libumad -libverbs -lpthread -lrt"

export ARMCI_NETWORK=OPENIB

#export ARMCI_NETWORK=MELLANOX

export MKL_INCLUDE=$MKLROOT/include/intel64/ilp64

export BLAS_LIB="-L$MKLROOT/lib/intel64 -lmkl_intel_ilp64 -lmkl_sequential -lmkl_core -lpthread -lm"

export BLASOPT="$BLAS_LIB"

export BLAS_SIZE=8

export SCALAPACK_SIZE=8

export SCALAPACK="-L$MKLROOT/lib/intel64 -lmkl_scalapack_ilp64 -lmkl_intel_ilp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_ilp64 -lpthread -lm"

export SCALAPACK_LIB="$SCALAPACK"

export USE_SCALAPACK=y

export MPI_HOME=/home/intel-2013/impi/4.1.0.024

export MPI_LOC=$MPI_HOME

export MPI_LIB=$MPI_LOC/lib64

export MPI_INCLUDE=$MPI_LOC/include64

export LIBMPI="-L/share/apps/intel/impi/4.1.0.024/intel64/lib -Xlinker --enable-new-dtags -Xlinker -rpath -Xlinker /share/apps/intel/impi/4.1.0.024/intel64/lib -Xlinker -rpath -Xlinker /share/apps/intel/mpi-rt/4.1 -lmpi -lmpigf -lmpigi -lrt -lpthread -lmpi_ilp64"