Installation of LAMMPS on Intel Xeon Cluster

Installation workflow with LAMMPS for Intel Xeon, Intel Xeon Phi, and Intel Xeon Phi Coprocessor clusters

No any introductions, just scroll down and follow it!

Installation workflow with LAMMPS for Intel Xeon, Intel Xeon Phi, and Intel Xeon Phi Coprocessor clusters

Contents

System info (Hardware)

Prerequisites (Software)

Build LAMMPS with USER-INTEL (BASIC)

Build LAMMPS (Intermediate)

Errors and Workarounds

How-to run LAMMPS on Intel Xeon cluster.

Miscellaneous

System info (Hardware)

This installation workflow is tested on RHEL 7 and TAIWANIA cluster, Intel Xeon Gold 6148 2.40GHz (64 bit system) cluster equipped with Intel Omni-Path high-speed interconnect technology. LAMMPS is built by Intel compiler, Intel MPI and MKL in Intel Parallel Studio XE 2018 update 1 and GCC 6.3.0 compiler.

Prerequisites (Software)

The tarball of LAMMPS source code (download from https://lammps.sandia.gov/)
Intel Parallel Studio XE Cluster Edition. Student/Lecturer can get the latest version and updates of this Intel suite in free of charge from https://software.intel.com/en-us/parallel-studio-xe.
GCC compiler (download from https://gcc.gnu.org/)

Build LAMMPS with USER-INTEL (BASIC)

This is going to install LAMMPS with USER-INSTALL. If you want LAMMPS to include more packages, please skip this one and find an intermediate version below!

1. Download program source code from LAMMPS website or clone LAMMPS repository using Git clone.

2. Uncompress the tarball file and navigate to LAMMPS top directory.

tar -xzvf lammps.tar.gz

3. Create a bash script, for example, install_lmp_intel.sh, like this:

#!/bin/bash

# Install LAMMPS with USER-INTEL package

module purge

module load intel/2018_u1

module load gcc/6.3.0

export LMP_ROOT=$HOME/lammps-stable_16Mar2018_RK

export LMP_LIB=$LMP_ROOT/lib

export INTEL_TOP=/pkg/intel/2018_u1

source $INTEL_TOP/parallel_studio_xe_2018/psxevars.sh

source $INTEL_TOP/compilers_and_libraries/linux/mpi/intel64/bin/mpivars.sh

cd $LMP_ROOT/src/

make yes-asphere yes-class2 yes-kspace yes-manybody yes-misc yes-molecule

make yes-mpiio yes-opt yes-replica yes-rigid

make yes-user-omp yes-user-intel

make intel_cpu_intelmpi -j 16 > $LMP_ROOT/make_lmp_cpu_intel64_std.log

#make knl -j 16

4. Change file permission and execute the script:

chmod +x install_lmp_intel.sh

./install_lmp_intel.sh

5. When installation is done, LAMMPS executable, lmp_intel_cpu_intelmpi, will be created at src directory.

Build LAMMPS (Intermediate)

The following installation workflow is mainly designed myself for TAIWANIA cluster. Please use it carefully !

1. Suppose that a LAMMPS directory is located at /home/rangsiman/lammps-stable_16Mar2018

2. Create a bash script, for example, install_lmp_intel_mpi.sh, like this:

#!/bin/bash

# This LAMMPS installation workflow was tested on Intel Xeon Gold 6148 cluster.

# Check package table in this website for library dependence.

# https://lammps.sandia.gov/doc/Section_packages.html

module purge

module load intel/2018_u1 gcc/6.3.0

export LMP_ROOT=$HOME/lammps-stable_16Mar2018_FULL2

export LMP_LIB=$LMP_ROOT/lib

export LMP_LOG=$LMP_ROOT/make_lmp_intel64_cpu.log

export INTEL_TOP=/pkg/intel/2018_u1

source $INTEL_TOP/parallel_studio_xe_2018/psxevars.sh

source $INTEL_TOP/compilers_and_libraries/linux/mpi/intel64/bin/mpivars.sh

echo "Start on `date`" > $LMP_LOG

echo "Build LAMMPS 'lmp_intel_cpu_intelmpi'" >> $LMP_LOG

echo "######################################################" >> $LMP_LOG

echo "1. Build library for some packages" >> $LMP_LOG

#make clean-all

# make lib-package args="-m mpi" means "cd $LMP_LIB/package && make -f Makefile.mpi"

cd $LMP_LIB/reax

make -f Makefile.gfortran

echo `ls *.a` >> $LMP_LOG

cd $LMP_LIB/meam

make -f Makefile.ifort

echo `ls *.a` >> $LMP_LOG

cd $LMP_LIB/poems

make -f Makefile.mpi

echo `ls *.a` >> $LMP_LOG

cd $LMP_LIB/colvars

make -f Makefile.g++

echo `ls *.a` >> $LMP_LOG

cd $LMP_LIB/awpmd

make -f Makefile.mpi

echo `ls *.a` >> $LMP_LOG

cd $LMP_LIB/linalg

make -f Makefile.gfortran

echo `ls *.a` >> $LMP_LOG

cd $LMP_LIB/qmmm

make -f Makefile.mpi

echo `ls *.a` >> $LMP_LOG

#cd $LMP_LIB/atc

#make -f Makefile.mpi | tee -ai $LMP_LOG

#echo `ls *.a` >> $LMP_LOG

cd $LMP_ROOT/src/

make lib-smd args="-b"

echo "Done on `date`" >> $LMP_LOG

echo "######################################################" >> $LMP_LOG

echo "2. Include all packages." >> $LMP_LOG

cd $LMP_ROOT/src/

make yes-all

echo "Done on `date`" >> $LMP_LOG

echo "######################################################" >> $LMP_LOG

echo "3. Exclude some packages that are not required." >> $LMP_LOG

make no-VORONOI

make no-KIM

make no-GPU

make no-LATTE

make no-MSCG

#make no-KOKKOS

#make no-USER-ATC

make no-USER-QUIP

make no-USER-VTK

make no-USER-H5MD

#make no-USER-QMMM

#make no-USER-NETCDF

#make no-USER-MOLFILE

make no-USER-VTK

# If error while compiling user-intel package, uninstall and install package again.

#make no-USER-INTEL

#make yes-USER-INTEL

echo "Done on `date`" >> $LMP_LOG

echo "######################################################" >> $LMP_LOG

echo "4. Install LAMMPS with Intel makefile." >> $LMP_LOG

make intel_cpu_intelmpi -j 16 >> $LMP_LOG

echo "Done on `date`" >> $LMP_LOG

echo "######################################################" >> $LMP_LOG

3. Change file permission and execute the script:

chmod +x install_lmp_intel_mpi.sh

./install_lmp_intel_mpi.sh

4. When installation is done, a LAMMPS executable, lmp_intel_cpu_intelmpi, will be created at src directory.

Errors and Workarounds

The errors that I found while I was compiling LAMMPS using make and solution.

1. If Makefile.lammps of package is not found in its library directory. You should build the static library of package again.

1.1 For example, Makefile.lammps not found in ../../poems , you should build the static library of POEM package by following.

cd $LMP_LIB/poems

make -f Makefile.mpi

1.2 Then check if Makefile.lammps and libpoems.a are created.

ls Makefile.lammps libpoems.a

1.3 If these files exists, try making LAMMPS again.

cd $LMP_ROOT/src

make intel_cpu_intelmpi -j 16 | tee -ai $LMP_ROOT/make_lmp_cpu_intel64_log

2. If compilation ran into error of "ld: unable to to locate -lompstub.", you can make symbolic link of libiompstubs5.so to libompstub.so for workaround.

cd $INTEP_TOP/lib/intel64

ln -s libiompstubs5.so libompstub.so

Then add FULL PATH of directory where libompstrub.so is to your PATH and LD_LIBRARY_PATH environment variables in .bash resource configuration file as well as add to Makefile.intel_cpu_intelmpi file. For example,

2.1 Add to PATH and LD_LIBRARY_PATH

export PATH=$INTEL_TOP/lib/intel64:$PATH

export LD_LIBRARY_PATH=$INTEL_TOP/lib/intel64:$LD_LIBRARY_PATH

2.2 Edit Makefile.intel_cpu_intelmpi file in MAKE directory

vi $LMP_ROOT/src/MAKE/OPTIONS/Makefile.intel_cpu_intelmpi

and add library in format -L<full_path_directory_of_dynamics_library> -l/<name_of_library> to LIB

LIB =           -ltbbmalloc -L/home/rangsiman/intel_parallel_studio_xe_2018_update1_cluster_edition/lib/intel64/  -lompstub

2.3 Finally, try making LAMMPS executable again.

cd $LMP_ROOT/src

make intel_cpu_intelmpi -j 16 | tee -ai $LMP_ROOT/make_lmp_cpu_intel64_log

3. Error "more undefined references to `for_dealloc_allocatable' follow" occurs at compilation is almost finished.

3.1 Edit Makefile.intel_cpu_intelmpi file in MAKE directory

vi $LMP_ROOT/src/MAKE/OPTIONS/Makefile.intel_cpu_intelmpi

and add following library to LIB

LIB =           -ltbbmalloc -lifcore

3.2 Then try making again

cd $LMP_ROOT/src

make intel_cpu_intelmpi -j 16 | tee -ai $LMP_ROOT/make_lmp_cpu_intel64_log

4. Error while compiling shows "mpi.h not found"

Make sure that you have loaded or added MPI library of Intel MPI and GCC compiler to your current PATH. Also you can use following command to check which package provides this file.

locate mpi.h

This should help your compilation to finish smoothly.

How-to run LAMMPS on Intel Xeon cluster.

1. Create bash shell script called run_lmp_intel.sh

#!/bin/bash

# Normal usage: ./run_lmp_intel.sh inut.in N

# N is number of MPI ranks

if [ ! -e $1 ]; then echo "Error: $1 not found!" ; exit 1 ; fi

if [ $2=="" ]; then export NPAL="1"; else export NPAL="$2" ; fi

export LMP_INP="$1"

export INP_NAME="`basename $LMP_INP .in`"

export LMP_OUT="$INP_NAME".log

export MPI="mpiexec"                                    #MPI run environment

export LMP_ROOT="$HOME/lammps-stable_16Mar2018"         #LAMMPS top directory

export LMP_EXE="lmp_intel_cpu_intelmpi"                 #name of executable

export LMP_BIN="$LMP_ROOT/src/$LMP_EXE"                 #Binary file

export LMP_CORES="$NPAL"                                #number of cores (mpi ranks)

export LMP_THREAD_LIST="1"                              #OMP Threads per process

export KMP_BLOCKTIME="0"

export I_MPI_PIN_DOMAIN=core                            #pin each MPI process to a core

export I_MPI_FABRICS=shm                                #communication fabric = shared mem

#Following line add -suffix intel for USER-INTEL package.

#export LMP_ARGS="-scrren none -pk intel 0 -sf intel"   #arg list

export LMP_ARGS="screen none -pk intel 0"               #arg list

export OMP_NUM_THREADS=$LMP_THREAD_LIST

echo "Running $LMP_OUT"

$MPI -np $LMP_CORES $LMP_BIN -in $LMP_INP -log $LMP_OUT $LMP_ARGS

2. Execute script with LAMMPS input as 1st argument and number of MPI processes as 2nd argument. For example,

./run_lmp_intel.sh  atoms.in  48

This command is going to run LAMMPS with 48 MPI processes and spawn OMP thread = 1 on each process.

Miscellaneous

You can check package table in following website for library dependence https://lammps.sandia.gov/doc/Section_packages.html

Rangsiman Ketkaew

Google Sites

Report abuse