Installation of LAMMPS on Intel Xeon Cluster
Installation workflow with LAMMPS for Intel Xeon, Intel Xeon Phi, and Intel Xeon Phi Coprocessor clusters
No any introductions, just scroll down and follow it!
Contents
System info (Hardware)
This installation workflow is tested on RHEL 7 and TAIWANIA cluster, Intel Xeon Gold 6148 2.40GHz (64 bit system) cluster equipped with Intel Omni-Path high-speed interconnect technology. LAMMPS is built by Intel compiler, Intel MPI and MKL in Intel Parallel Studio XE 2018 update 1 and GCC 6.3.0 compiler.
Prerequisites (Software)
- The tarball of LAMMPS source code (download from https://lammps.sandia.gov/)
- Intel Parallel Studio XE Cluster Edition. Student/Lecturer can get the latest version and updates of this Intel suite in free of charge from https://software.intel.com/en-us/parallel-studio-xe.
- GCC compiler (download from https://gcc.gnu.org/)
Build LAMMPS with USER-INTEL (BASIC)
This is going to install LAMMPS with USER-INSTALL. If you want LAMMPS to include more packages, please skip this one and find an intermediate version below!
1. Download program source code from LAMMPS website or clone LAMMPS repository using Git clone.
2. Uncompress the tarball file and navigate to LAMMPS top directory.
tar -xzvf lammps.tar.gz
3. Create a bash script, for example, install_lmp_intel.sh, like this:
#!/bin/bash
# Install LAMMPS with USER-INTEL package
module purge
module load intel/2018_u1
module load gcc/6.3.0
export LMP_ROOT=$HOME/lammps-stable_16Mar2018_RK
export LMP_LIB=$LMP_ROOT/lib
export INTEL_TOP=/pkg/intel/2018_u1
source $INTEL_TOP/parallel_studio_xe_2018/psxevars.sh
source $INTEL_TOP/compilers_and_libraries/linux/mpi/intel64/bin/mpivars.sh
cd $LMP_ROOT/src/
make yes-asphere yes-class2 yes-kspace yes-manybody yes-misc yes-molecule
make yes-mpiio yes-opt yes-replica yes-rigid
make yes-user-omp yes-user-intel
make intel_cpu_intelmpi -j 16 > $LMP_ROOT/make_lmp_cpu_intel64_std.log
#make knl -j 16
4. Change file permission and execute the script:
chmod +x install_lmp_intel.sh
./install_lmp_intel.sh
5. When installation is done, LAMMPS executable, lmp_intel_cpu_intelmpi, will be created at src directory.
Build LAMMPS (Intermediate)
The following installation workflow is mainly designed myself for TAIWANIA cluster. Please use it carefully !
1. Suppose that a LAMMPS directory is located at /home/rangsiman/lammps-stable_16Mar2018
2. Create a bash script, for example, install_lmp_intel_mpi.sh, like this:
#!/bin/bash
# This LAMMPS installation workflow was tested on Intel Xeon Gold 6148 cluster.
# Check package table in this website for library dependence.
# https://lammps.sandia.gov/doc/Section_packages.html
module purge
module load intel/2018_u1 gcc/6.3.0
export LMP_ROOT=$HOME/lammps-stable_16Mar2018_FULL2
export LMP_LIB=$LMP_ROOT/lib
export LMP_LOG=$LMP_ROOT/make_lmp_intel64_cpu.log
export INTEL_TOP=/pkg/intel/2018_u1
source $INTEL_TOP/parallel_studio_xe_2018/psxevars.sh
source $INTEL_TOP/compilers_and_libraries/linux/mpi/intel64/bin/mpivars.sh
echo "Start on `date`" > $LMP_LOG
echo "Build LAMMPS 'lmp_intel_cpu_intelmpi'" >> $LMP_LOG
echo "######################################################" >> $LMP_LOG
echo "1. Build library for some packages" >> $LMP_LOG
#make clean-all
# make lib-package args="-m mpi" means "cd $LMP_LIB/package && make -f Makefile.mpi"
cd $LMP_LIB/reax
make -f Makefile.gfortran
echo `ls *.a` >> $LMP_LOG
cd $LMP_LIB/meam
make -f Makefile.ifort
echo `ls *.a` >> $LMP_LOG
cd $LMP_LIB/poems
make -f Makefile.mpi
echo `ls *.a` >> $LMP_LOG
cd $LMP_LIB/colvars
make -f Makefile.g++
echo `ls *.a` >> $LMP_LOG
cd $LMP_LIB/awpmd
make -f Makefile.mpi
echo `ls *.a` >> $LMP_LOG
cd $LMP_LIB/linalg
make -f Makefile.gfortran
echo `ls *.a` >> $LMP_LOG
cd $LMP_LIB/qmmm
make -f Makefile.mpi
echo `ls *.a` >> $LMP_LOG
#cd $LMP_LIB/atc
#make -f Makefile.mpi | tee -ai $LMP_LOG
#echo `ls *.a` >> $LMP_LOG
cd $LMP_ROOT/src/
make lib-smd args="-b"
echo "Done on `date`" >> $LMP_LOG
echo "######################################################" >> $LMP_LOG
echo "2. Include all packages." >> $LMP_LOG
cd $LMP_ROOT/src/
make yes-all
echo "Done on `date`" >> $LMP_LOG
echo "######################################################" >> $LMP_LOG
echo "3. Exclude some packages that are not required." >> $LMP_LOG
make no-VORONOI
make no-KIM
make no-GPU
make no-LATTE
make no-MSCG
#make no-KOKKOS
#make no-USER-ATC
make no-USER-QUIP
make no-USER-VTK
make no-USER-H5MD
#make no-USER-QMMM
#make no-USER-NETCDF
#make no-USER-MOLFILE
make no-USER-VTK
# If error while compiling user-intel package, uninstall and install package again.
#make no-USER-INTEL
#make yes-USER-INTEL
echo "Done on `date`" >> $LMP_LOG
echo "######################################################" >> $LMP_LOG
echo "4. Install LAMMPS with Intel makefile." >> $LMP_LOG
make intel_cpu_intelmpi -j 16 >> $LMP_LOG
echo "Done on `date`" >> $LMP_LOG
echo "######################################################" >> $LMP_LOG
3. Change file permission and execute the script:
chmod +x install_lmp_intel_mpi.sh
./install_lmp_intel_mpi.sh
4. When installation is done, a LAMMPS executable, lmp_intel_cpu_intelmpi, will be created at src directory.
Errors and Workarounds
The errors that I found while I was compiling LAMMPS using make and solution.
1. If Makefile.lammps of package is not found in its library directory. You should build the static library of package again.
1.1 For example, Makefile.lammps not found in ../../poems
, you should build the static library of POEM package by following.
cd $LMP_LIB/poems
make -f Makefile.mpi
1.2 Then check if Makefile.lammps
and libpoems.a
are created.
ls Makefile.lammps libpoems.a
1.3 If these files exists, try making LAMMPS again.
cd $LMP_ROOT/src
make intel_cpu_intelmpi -j 16 | tee -ai $LMP_ROOT/make_lmp_cpu_intel64_log
2. If compilation ran into error of "ld: unable to to locate -lompstub.", you can make symbolic link of libiompstubs5.so to libompstub.so for workaround.
cd $INTEP_TOP/lib/intel64
ln -s libiompstubs5.so libompstub.so
Then add FULL PATH of directory where libompstrub.so is to your PATH and LD_LIBRARY_PATH environment variables in .bash resource configuration file as well as add to Makefile.intel_cpu_intelmpi file. For example,
2.1 Add to PATH and LD_LIBRARY_PATH
export PATH=$INTEL_TOP/lib/intel64:$PATH
export LD_LIBRARY_PATH=$INTEL_TOP/lib/intel64:$LD_LIBRARY_PATH
2.2 Edit Makefile.intel_cpu_intelmpi file in MAKE directory
vi $LMP_ROOT/src/MAKE/OPTIONS/Makefile.intel_cpu_intelmpi
and add library in format -L<full_path_directory_of_dynamics_library> -l/<name_of_library> to LIB
LIB = -ltbbmalloc -L/home/rangsiman/intel_parallel_studio_xe_2018_update1_cluster_edition/lib/intel64/ -lompstub
2.3 Finally, try making LAMMPS executable again.
cd $LMP_ROOT/src
make intel_cpu_intelmpi -j 16 | tee -ai $LMP_ROOT/make_lmp_cpu_intel64_log
3. Error "more undefined references to `for_dealloc_allocatable' follow" occurs at compilation is almost finished.
3.1 Edit Makefile.intel_cpu_intelmpi file in MAKE directory
vi $LMP_ROOT/src/MAKE/OPTIONS/Makefile.intel_cpu_intelmpi
and add following library to LIB
LIB = -ltbbmalloc -lifcore
3.2 Then try making again
cd $LMP_ROOT/src
make intel_cpu_intelmpi -j 16 | tee -ai $LMP_ROOT/make_lmp_cpu_intel64_log
4. Error while compiling shows "mpi.h not found"
Make sure that you have loaded or added MPI library of Intel MPI and GCC compiler to your current PATH. Also you can use following command to check which package provides this file.
locate mpi.h
This should help your compilation to finish smoothly.
How-to run LAMMPS on Intel Xeon cluster.
1. Create bash shell script called run_lmp_intel.sh
#!/bin/bash
# Normal usage: ./run_lmp_intel.sh inut.in N
# N is number of MPI ranks
if [ ! -e $1 ]; then echo "Error: $1 not found!" ; exit 1 ; fi
if [ $2=="" ]; then export NPAL="1"; else export NPAL="$2" ; fi
export LMP_INP="$1"
export INP_NAME="`basename $LMP_INP .in`"
export LMP_OUT="$INP_NAME".log
export MPI="mpiexec" #MPI run environment
export LMP_ROOT="$HOME/lammps-stable_16Mar2018" #LAMMPS top directory
export LMP_EXE="lmp_intel_cpu_intelmpi" #name of executable
export LMP_BIN="$LMP_ROOT/src/$LMP_EXE" #Binary file
export LMP_CORES="$NPAL" #number of cores (mpi ranks)
export LMP_THREAD_LIST="1" #OMP Threads per process
export KMP_BLOCKTIME="0"
export I_MPI_PIN_DOMAIN=core #pin each MPI process to a core
export I_MPI_FABRICS=shm #communication fabric = shared mem
#Following line add -suffix intel for USER-INTEL package.
#export LMP_ARGS="-scrren none -pk intel 0 -sf intel" #arg list
export LMP_ARGS="screen none -pk intel 0" #arg list
export OMP_NUM_THREADS=$LMP_THREAD_LIST
echo "Running $LMP_OUT"
$MPI -np $LMP_CORES $LMP_BIN -in $LMP_INP -log $LMP_OUT $LMP_ARGS
2. Execute script with LAMMPS input as 1st argument and number of MPI processes as 2nd argument. For example,
./run_lmp_intel.sh atoms.in 48
This command is going to run LAMMPS with 48 MPI processes and spawn OMP thread = 1 on each process.
Miscellaneous
You can check package table in following website for library dependence https://lammps.sandia.gov/doc/Section_packages.html
Rangsiman Ketkaew