Installation of LAMMPS on Intel Xeon Cluster

Installation workflow with LAMMPS for Intel Xeon, Intel Xeon Phi, and Intel Xeon Phi Coprocessor clusters


No any introductions, just scroll down and follow it!

System info (Hardware)

This installation workflow is tested on RHEL 7 and TAIWANIA cluster, Intel Xeon Gold 6148 2.40GHz (64 bit system) cluster equipped with Intel Omni-Path high-speed interconnect technology. LAMMPS is built by Intel compiler, Intel MPI and MKL in Intel Parallel Studio XE 2018 update 1 and GCC 6.3.0 compiler.

Prerequisites (Software)

  1. The tarball of LAMMPS source code (download from https://lammps.sandia.gov/)
  2. Intel Parallel Studio XE Cluster Edition. Student/Lecturer can get the latest version and updates of this Intel suite in free of charge from https://software.intel.com/en-us/parallel-studio-xe.
  3. GCC compiler (download from https://gcc.gnu.org/)

Build LAMMPS with USER-INTEL (BASIC)

This is going to install LAMMPS with USER-INSTALL. If you want LAMMPS to include more packages, please skip this one and find an intermediate version below!


1. Download program source code from LAMMPS website or clone LAMMPS repository using Git clone.

2. Uncompress the tarball file and navigate to LAMMPS top directory.

tar -xzvf lammps.tar.gz

3. Create a bash script, for example, install_lmp_intel.sh, like this:

#!/bin/bash

# Install LAMMPS with USER-INTEL package

module purge
module load intel/2018_u1
module load gcc/6.3.0

export LMP_ROOT=$HOME/lammps-stable_16Mar2018_RK
export LMP_LIB=$LMP_ROOT/lib

export INTEL_TOP=/pkg/intel/2018_u1
source $INTEL_TOP/parallel_studio_xe_2018/psxevars.sh
source $INTEL_TOP/compilers_and_libraries/linux/mpi/intel64/bin/mpivars.sh

cd $LMP_ROOT/src/

make yes-asphere yes-class2 yes-kspace yes-manybody yes-misc yes-molecule
make yes-mpiio yes-opt yes-replica yes-rigid
make yes-user-omp yes-user-intel

make intel_cpu_intelmpi -j 16 > $LMP_ROOT/make_lmp_cpu_intel64_std.log
#make knl -j 16

4. Change file permission and execute the script:

chmod +x install_lmp_intel.sh
./install_lmp_intel.sh

5. When installation is done, LAMMPS executable, lmp_intel_cpu_intelmpi, will be created at src directory.

Build LAMMPS (Intermediate)

The following installation workflow is mainly designed myself for TAIWANIA cluster. Please use it carefully !


1. Suppose that a LAMMPS directory is located at /home/rangsiman/lammps-stable_16Mar2018

2. Create a bash script, for example, install_lmp_intel_mpi.sh, like this:

#!/bin/bash

# This LAMMPS installation workflow was tested on Intel Xeon Gold 6148 cluster.
# Check package table in this website for library dependence.
# https://lammps.sandia.gov/doc/Section_packages.html

module purge
module load intel/2018_u1 gcc/6.3.0

export LMP_ROOT=$HOME/lammps-stable_16Mar2018_FULL2
export LMP_LIB=$LMP_ROOT/lib
export LMP_LOG=$LMP_ROOT/make_lmp_intel64_cpu.log

export INTEL_TOP=/pkg/intel/2018_u1
source $INTEL_TOP/parallel_studio_xe_2018/psxevars.sh
source $INTEL_TOP/compilers_and_libraries/linux/mpi/intel64/bin/mpivars.sh

echo "Start on `date`" > $LMP_LOG
echo "Build LAMMPS 'lmp_intel_cpu_intelmpi'" >> $LMP_LOG

echo "######################################################" >> $LMP_LOG
echo "1. Build library for some packages" >> $LMP_LOG

#make clean-all

# make lib-package args="-m mpi" means "cd $LMP_LIB/package && make -f Makefile.mpi"

cd $LMP_LIB/reax
make -f Makefile.gfortran
echo `ls *.a` >> $LMP_LOG

cd $LMP_LIB/meam
make -f Makefile.ifort
echo `ls *.a` >> $LMP_LOG

cd $LMP_LIB/poems
make -f Makefile.mpi
echo `ls *.a` >> $LMP_LOG

cd $LMP_LIB/colvars
make -f Makefile.g++
echo `ls *.a` >> $LMP_LOG

cd $LMP_LIB/awpmd
make -f Makefile.mpi
echo `ls *.a` >> $LMP_LOG

cd $LMP_LIB/linalg
make -f Makefile.gfortran
echo `ls *.a` >> $LMP_LOG

cd $LMP_LIB/qmmm
make -f Makefile.mpi
echo `ls *.a` >> $LMP_LOG

#cd $LMP_LIB/atc
#make -f Makefile.mpi | tee -ai $LMP_LOG
#echo `ls *.a` >> $LMP_LOG

cd $LMP_ROOT/src/
make lib-smd args="-b"

echo "Done on `date`" >> $LMP_LOG

echo "######################################################" >> $LMP_LOG
echo "2. Include all packages." >> $LMP_LOG

cd $LMP_ROOT/src/
make yes-all

echo "Done on `date`" >> $LMP_LOG

echo "######################################################" >> $LMP_LOG
echo "3. Exclude some packages that are not required." >> $LMP_LOG

make no-VORONOI
make no-KIM
make no-GPU
make no-LATTE
make no-MSCG
#make no-KOKKOS
#make no-USER-ATC
make no-USER-QUIP
make no-USER-VTK
make no-USER-H5MD
#make no-USER-QMMM
#make no-USER-NETCDF
#make no-USER-MOLFILE
make no-USER-VTK

# If error while compiling user-intel package, uninstall and install package again.
#make no-USER-INTEL
#make yes-USER-INTEL

echo "Done on `date`" >> $LMP_LOG

echo "######################################################" >> $LMP_LOG
echo "4. Install LAMMPS with Intel makefile." >> $LMP_LOG

make intel_cpu_intelmpi -j 16 >> $LMP_LOG

echo "Done on `date`" >> $LMP_LOG

echo "######################################################" >> $LMP_LOG

3. Change file permission and execute the script:

chmod +x install_lmp_intel_mpi.sh
./install_lmp_intel_mpi.sh

4. When installation is done, a LAMMPS executable, lmp_intel_cpu_intelmpi, will be created at src directory.

Errors and Workarounds

The errors that I found while I was compiling LAMMPS using make and solution.


1. If Makefile.lammps of package is not found in its library directory. You should build the static library of package again.

1.1 For example, Makefile.lammps not found in ../../poems , you should build the static library of POEM package by following.

cd $LMP_LIB/poems
make -f Makefile.mpi

1.2 Then check if Makefile.lammps and libpoems.a are created.

ls Makefile.lammps libpoems.a

1.3 If these files exists, try making LAMMPS again.

cd $LMP_ROOT/src
make intel_cpu_intelmpi -j 16 | tee -ai $LMP_ROOT/make_lmp_cpu_intel64_log


2. If compilation ran into error of "ld: unable to to locate -lompstub.", you can make symbolic link of libiompstubs5.so to libompstub.so for workaround.

cd $INTEP_TOP/lib/intel64
ln -s libiompstubs5.so libompstub.so

Then add FULL PATH of directory where libompstrub.so is to your PATH and LD_LIBRARY_PATH environment variables in .bash resource configuration file as well as add to Makefile.intel_cpu_intelmpi file. For example,

2.1 Add to PATH and LD_LIBRARY_PATH

export PATH=$INTEL_TOP/lib/intel64:$PATH
export LD_LIBRARY_PATH=$INTEL_TOP/lib/intel64:$LD_LIBRARY_PATH

2.2 Edit Makefile.intel_cpu_intelmpi file in MAKE directory

vi $LMP_ROOT/src/MAKE/OPTIONS/Makefile.intel_cpu_intelmpi

and add library in format -L<full_path_directory_of_dynamics_library> -l/<name_of_library> to LIB

LIB =           -ltbbmalloc -L/home/rangsiman/intel_parallel_studio_xe_2018_update1_cluster_edition/lib/intel64/  -lompstub

2.3 Finally, try making LAMMPS executable again.

cd $LMP_ROOT/src
make intel_cpu_intelmpi -j 16 | tee -ai $LMP_ROOT/make_lmp_cpu_intel64_log


3. Error "more undefined references to `for_dealloc_allocatable' follow" occurs at compilation is almost finished.

3.1 Edit Makefile.intel_cpu_intelmpi file in MAKE directory

vi $LMP_ROOT/src/MAKE/OPTIONS/Makefile.intel_cpu_intelmpi

and add following library to LIB

LIB =           -ltbbmalloc -lifcore

3.2 Then try making again

cd $LMP_ROOT/src
make intel_cpu_intelmpi -j 16 | tee -ai $LMP_ROOT/make_lmp_cpu_intel64_log


4. Error while compiling shows "mpi.h not found"

Make sure that you have loaded or added MPI library of Intel MPI and GCC compiler to your current PATH. Also you can use following command to check which package provides this file.

locate mpi.h

This should help your compilation to finish smoothly.

How-to run LAMMPS on Intel Xeon cluster.

1. Create bash shell script called run_lmp_intel.sh

#!/bin/bash

# Normal usage: ./run_lmp_intel.sh inut.in N
# N is number of MPI ranks

if [ ! -e $1 ]; then echo "Error: $1 not found!" ; exit 1 ; fi
if [ $2=="" ]; then export NPAL="1"; else export NPAL="$2" ; fi

export LMP_INP="$1"
export INP_NAME="`basename $LMP_INP .in`"
export LMP_OUT="$INP_NAME".log

export MPI="mpiexec"                                    #MPI run environment
export LMP_ROOT="$HOME/lammps-stable_16Mar2018"         #LAMMPS top directory
export LMP_EXE="lmp_intel_cpu_intelmpi"                 #name of executable
export LMP_BIN="$LMP_ROOT/src/$LMP_EXE"                 #Binary file
export LMP_CORES="$NPAL"                                #number of cores (mpi ranks)
export LMP_THREAD_LIST="1"                              #OMP Threads per process

export KMP_BLOCKTIME="0"
export I_MPI_PIN_DOMAIN=core                            #pin each MPI process to a core
export I_MPI_FABRICS=shm                                #communication fabric = shared mem

#Following line add -suffix intel for USER-INTEL package.
#export LMP_ARGS="-scrren none -pk intel 0 -sf intel"   #arg list
export LMP_ARGS="screen none -pk intel 0"               #arg list

export OMP_NUM_THREADS=$LMP_THREAD_LIST
echo "Running $LMP_OUT"

$MPI -np $LMP_CORES $LMP_BIN -in $LMP_INP -log $LMP_OUT $LMP_ARGS

2. Execute script with LAMMPS input as 1st argument and number of MPI processes as 2nd argument. For example,

./run_lmp_intel.sh  atoms.in  48

This command is going to run LAMMPS with 48 MPI processes and spawn OMP thread = 1 on each process.

Miscellaneous

You can check package table in following website for library dependence https://lammps.sandia.gov/doc/Section_packages.html


Rangsiman Ketkaew